How a Tenkai cluster works

A Tenkai cluster is one or more servers that run your resources with Docker Swarm. Manager nodes keep the cluster's state and decide where replicas run. Worker nodes run workloads without taking part in those decisions.

Changes need agreement from more than half the managers. This majority is the quorum. With 3 managers, the cluster can lose 1 without losing quorum. That does not protect its public traffic or local data.

Manager nodes and worker nodes

Both roles can run your web applications and workers. Choose Stop accepting workloads to move applications off a manager. Databases, applications with kept files and cluster storage stay on that node.

Add a worker node for more CPU or memory. It can run at another provider or at home, through the cluster network. Distance adds network delay, but workers take no part in the quorum.

The Edge node is always a manager. It receives public traffic and keeps the cluster registry. It also runs Git builds when the cluster has them enabled.

The quorum

The managers agree by majority:

• 1 manager. The quorum is 1. The cluster cannot lose its manager.
• 3 managers. The quorum is 2. The cluster can lose 1.
• 5 managers. The quorum is 3. The cluster can lose 2.

2 managers need both votes, so they tolerate no more failures than 1. Losing either blocks changes, not running containers. Tenkai promotes or demotes two nodes at a time and supports at most 5 managers.

Slow links between managers delay decisions. Keep them in one region, in separate availability zones if your provider has them.

When a node fails

With quorum, Swarm replaces a failed node's stateless replicas without Tenkai. The remaining nodes must accept workloads, have room and have or be able to pull the image. A node that restarts with its Swarm data intact rejoins automatically. Healthy replicas do not move back.

Some resources cannot move:

• A managed service or a web application that keeps files stops until its data node returns. A lost disk needs a usable backup.
• The cluster storage also stays on its data node. Hosted buckets stop answering while that node is down.

If the edge node fails, visitors cannot reach your applications, even if replicas run elsewhere. Nodes also cannot pull missing images from its cluster registry.

When the quorum is lost

Without more than half the managers, Swarm cannot change services or replace lost replicas. Containers on surviving nodes keep running, but their applications still need their database and the edge node to answer requests.

Changes resume once enough managers communicate again. If the other managers are permanently lost and the edge node survives, an owner or admin can choose Recover the cluster. Tenkai rebuilds the quorum around that edge node. The lost managers become worker nodes in Tenkai's records.

Recovery does not restore lost disks. Check databases and applications that keep files afterwards. If the edge node is lost too, wait for it to return or rebuild on a new cluster. See Recover a cluster.

Drain a node

A drain moves applications off a node before maintenance. Tenkai relocates each web application and worker, one at a time, using the same release. A web application switches traffic once its replacement passes its health check. A worker has no route to switch, and its old and new replicas briefly run together.

After these moves, Swarm runs no containers on the node until you resume it. Any scheduled job run still there stops. The drain guide explains the steps and failure handling.

A drained manager still votes. A reboot takes it out of the quorum until it returns. Reboot one manager at a time, and check that the others are healthy first.

Move and reboot uses the same application moves before restarting a node. Moved applications stay on their new nodes after it returns. Databases, kept files and scheduled job runs do not move. Move the edge node before rebooting it to keep public traffic flowing.

Some nodes cannot drain:

• The edge node. Move it to another manager first.
• A node that holds a managed service's data, a web application's kept files or the cluster storage. On a manager, use Stop accepting workloads to move only applications without local data.
• The last node accepting workloads. Another node must accept them and have room.

What a cluster does not protect against

• Losing the edge node. More managers do not provide automatic traffic failover. A planned edge node move needs the old node to remain reachable.
• Losing local data. Extra managers do not copy your volumes. Keep backups outside the cluster.
• Losing too many managers together. Separate availability zones reduce that risk.
• Running out of capacity. Leave enough room for replacements after a node fails.

When another option suits better

• One server may suit a small project that can accept downtime. An outage alone does not erase data. If the disk is lost, restore a backup. More managers alone do not prevent data loss or a traffic outage.
• If your servers span regions, keep managers in one region and use worker nodes elsewhere. Account for the delay and traffic costs between them.

Go further

• Clusters and node roles
• Make a cluster highly available
• Drain a node before you work on it
• Install updates and reboot a server
• Load balancing and replicas