Object storage on your cluster with Garage

Object storage keeps uploads and other files outside the application. Replicas share them through the S3 API, a common interface for uploading and downloading files. Tenkai runs Garage on your cluster, with a hosted bucket for each use, its own keys and a size limit. Your servers hold the files. You need no separate storage provider to serve them.

The files live on one node, the data node. To recover from its loss, configure a copy to a storage bucket at a provider outside the cluster. Tenkai can schedule copies as often as every 15 minutes.

One storage service per cluster

An owner or admin enables cluster storage and chooses its data node. Tenkai runs one Garage service there, with all hosted buckets in one local volume. It reserves 128 MB of memory.

Tenkai gives the storage an HTTPS address, such as https://storage-k7m4p9.apps.tenkai.cloud, and issues its certificate. The edge node routes requests to Garage. Garage publishes no port on the host.

Each hosted bucket is a resource in a project's environment. Its keys allow access to that bucket only, not to other buckets in the shared storage.

How an application reaches its bucket

Bind the bucket to an application, then deploy it. The binding supplies the endpoint, region, bucket name and keys in variables such as S3_ENDPOINT and S3_SECRET_ACCESS_KEY. Configure the application's S3 client to use them.

• The application and browsers use the same public endpoint. The application can give the browser a signed URL that grants access to a file. The browser then downloads it from storage without passing the file through the application.
• Sign upload URLs without a checksum, as Garage checks it against the uploaded file. With the AWS SDK for JavaScript, set requestChecksumCalculation to WHEN_REQUIRED.
• Set the client to path-style addressing. The bucket goes in the path, as in https://storage-k7m4p9.apps.tenkai.cloud/uploads-k7m4p9/photo.jpg.
• Each bucket has a limit of 1 to 1000 GiB, shown as GB in Tenkai. Garage refuses uploads that exceed it, though uploads at the same moment can go a little above it. This does not reserve disk space: other buckets and resources share the disk and can fill it.
• Use multipart uploads for large files. Each request through the edge node has 60 seconds to send its body, so each part must fit within that time. Check your client's upload settings.

Applications in the same environment can share a bucket. They share its files and keys.

A hosted bucket or a kept directory

A web application can instead keep a directory, such as /app/data, on one node. Compare both in the last scene above.

A kept directory needs no S3 client. The application writes to its filesystem, but runs one replica on the node that holds its files. Each deployment interrupts it. When that node is down, the application is down.

A hosted bucket separates files from application replicas. A web application with no kept directories can run up to 10 replicas on eligible nodes and use blue-green deployments. If the storage's data node goes down, replicas on other nodes can keep running, but requests that need the bucket fail.

Choose a kept directory for software that needs local files, such as WordPress or an SQLite database. Choose a hosted bucket when your application supports S3 and needs shared files across replicas.

Keep a copy outside the cluster

Tenkai runs Garage on one node, without replication to another node or a way to move its volume. Losing that node's disk loses every hosted bucket's files.

Choose a storage bucket of your team to keep a copy outside the cluster. Set the schedule to daily, hourly or every 15 minutes. A container on the data node runs rclone to update the copy:

• new and changed files go to the copy;
• deleted files leave the copy too;
• files go straight to the provider, not through Tenkai or the edge node.

Tenkai does not encrypt the copied files with the backup key. You can read them at the provider without Tenkai. The copy mirrors your bucket, so a mistaken deletion reaches it at the next copy. To keep older versions, enable versioning at the provider.

A 15-minute schedule does not guarantee that you can lose at most 15 minutes of changes. Failed, missed or slow copies leave files out of date. Check the transfer history and test a restore before relying on the copy.

If the data node is permanently lost, remove it from the cluster by force. Enable storage on another node, choose Create again on each lost bucket, then restore its copy. Redeploy bound applications to give them the new endpoint and keys. Scheduled copies pause after recreation so the empty bucket does not erase the copy. A successful restore resumes them. See When the data node is lost.

Hosted buckets hold application files, never Tenkai backups. Backups go to storage buckets outside the cluster they protect.

When to choose another option

• Without a copy, use a hosted bucket only for files you can lose or recreate, such as thumbnails or a cache. If you cannot lose changes between copies, choose storage with stronger durability guarantees.
• For more data than one node can hold, consider a storage provider. Compare storage, request and transfer costs with your servers' cost.
• For distant visitors, consider a CDN or storage closer to them. Hosted buckets serve files from your cluster's location.
• For recovery from mistaken overwrites or deletions, use versioning at the provider, either for your copy or for a bucket your application uses directly.

Go further

• Run object storage on your cluster
• Keep a copy of the files
• Keep the files of a web application
• Back up your databases to a storage bucket
• How traffic reaches your application