Disaster recovery: what Tenkai lets you recover from
See what keeps running when a replica, a node, a whole cluster or Tenkai itself is lost, and how you bring a database back from a backup in your storage bucket.
Tenkai restarts application replicas while enough managers and capacity remain. You recover lost data from your own backups. A surviving edge node can rebuild a lost manager quorum. Losing the whole cluster means rebuilding on new servers.
Try losing a node or Tenkai. In this example, surviving nodes have spare capacity. Losing the data node also means losing its disk.
Scene 1 of 9: Everything runs
Three managers with spare capacity. node-1 is the edge node. An application with no kept files runs on node-1 and node-2. Its database runs on node-3. Successful daily backups are in an external storage bucket.
What the cluster repairs alone
Swarm restarts web application and worker replicas on nodes that accept workloads and have room. It needs a majority of managers, so three can lose one. Public traffic also needs the edge node. See load balancing and replicas.
A web application that keeps files stays on its data node. A scheduled job does not retry a failed run, and skips times that pass while its scheduler is down.
What you bring back from a backup
A database keeps its data on one data node. It stops while that node is down. If the disk is gone for good, only a backup brings the data back.
In the Backups tab, choose New database on a successful backup. Choose a project whose cluster has room on a ready node. Bind the new database to your application in place of the lost one, and deploy.
Check the restored data before deleting the old database. Deletion makes Tenkai forget its backups. They stay in your bucket, readable without Tenkai with your backup key.
For a hosted bucket, replace the lost cluster storage, choose Create again, then restore its copy from your storage bucket. Deploy its bound applications again for the new address and keys. The copy mirrors your bucket: files deleted before the last copy are gone from it too.
How old is the data you get back
A daily backup recovers one copy, leaving out writes made after it. This timeline shows a database using daily backups alone. It uses 03:00 as an example, not a fixed schedule.
Scene 1 of 4: A daily backup
With backups enabled, Tenkai attempts one about once a day. Here, each copy represents the data at 03:00. Real backups have no fixed start time and take time to finish.
Tenkai checks hourly for a backup due 23 hours and 45 minutes after the latest attempt started. A failed backup counts too. There is no immediate retry or alert, so your latest successful backup can be older than a day.
Check the timeline. Fix the cause of a failure, then choose Back up now. If the bucket refused its keys, replace them first.
What stops until you act
The edge node
Public web traffic enters through the edge node. If it fails, visitors cannot reach the replicas. Tenkai can move the edge node only while it still answers. If it is gone for good, rebuild on a new cluster.
The quorum
Without a majority of managers, running containers continue, but Swarm cannot replace replicas or start scheduled job runs. Wait for enough managers to return. If they are permanently lost and the edge node survives, choose Recover the cluster. Tenkai rebuilds the quorum around the edge node. Lost disks still need backups.
An application that keeps files
Tenkai restores its backup only on its data node. If that node is gone, recover its files and included database backups without Tenkai.
When the whole cluster is lost
Losing every server and disk loses the containers, volumes and cluster registry images. Your configuration in Tenkai and copies in an external storage bucket remain.
To rebuild applications and databases:
- Add servers to a new cluster, and create a project on it.
- Create each database from a successful backup in the new project.
- Recreate your applications, variables and bindings, then deploy from your source or image registry. Set up their domains on the new cluster.
- Check the restored data and applications before deleting the old projects and cluster. See the servers of a cluster are gone.
Keep the old projects until you have recovered and checked the data. Deleting them makes Tenkai forget their backups.
When Tenkai is down
Your cluster runs without Tenkai. With its nodes healthy, it keeps serving, replaces replicas and runs scheduled jobs. The edge node serves the certificates it holds, but Tenkai cannot renew them while down.
You cannot start a deployment or restore while Tenkai is down. Once Tenkai returns, it queues due backups. See When Tenkai is unavailable.
Keep the team's backup key file outside Tenkai, and download it again after every rotation. With it, your backups stay readable even if you lose access to Tenkai.
Limits
- Tenkai keeps no standby cluster.
- Choosing a storage bucket enables backups. You have nothing to restore until one succeeds.
When another option suits better
- You cannot lose a day's writes. Compare your latest successful backup and a measured restore time with your target. Choose a database service that meets it.
- You cannot wait for a rebuild. Choose a platform with automatic regional failover. Tenkai leaves you in control of your servers and backups, but you rebuild a lost cluster yourself.