Tenkai
Sign inStart for free
Skip to guide
Help center

Clusters and the roles of their nodes

Understand the edge node, manager nodes, worker nodes and data nodes, how many failures a cluster survives, and how to keep replicas on separate nodes.

A cluster is where your resources run. Its servers, the nodes, work as one: you choose no server for a web application, and the cluster places its replicas on the nodes that have room.

A server added on its own starts a new cluster. A server added to a cluster becomes one of its nodes.

The roles of the nodes

  • The Edge node receives the public traffic of every web application of the cluster: your DNS points to it. It also keeps the cluster registry and runs the builds. A cluster has one, and it is always a manager node.
  • Manager nodes run the cluster and decide where each replica runs. They also run your workloads, unless you choose Stop accepting workloads.
  • Worker nodes only run workloads.

The Data node of a stateful resource, a managed service or a web application that keeps files, is the node that holds its volume. The resource always runs there. Tenkai prefers a worker node when the cluster has one. You choose the data node of the cluster storage.

How many failures a cluster survives

A change to the cluster needs a majority of the managers, the quorum. Without a quorum, running containers continue, but nothing can change or move until the managers come back.

  • 1 manager: the cluster cannot lose a manager.
  • 3 managers: it can lose 1.
  • 5 managers: it can lose 2.

Two managers are worse than one: losing either loses the quorum. Tenkai promotes and demotes managers in pairs to keep the number odd. See Make a cluster highly available.

Managers are sensitive to latency. Keep them in one region, spread over its availability zones if your provider has them. Worker nodes can run anywhere, even at home.

When a node fails

The managers restart the replicas of its web applications on other nodes, if the cluster keeps its quorum and the other nodes have enough free CPU and memory. Without a quorum, they stay stopped until the managers come back or you recover the cluster by hand.

A managed service, or a web application that keeps files, cannot move: it stops until its data node comes back. A PostgreSQL with high availability is the exception. If the node or its disk is lost, only your backups remain. See the storage and recovery limits.

If the edge node fails, the public traffic of the cluster stops. With one manager, that is also the only manager, so nothing moves either. Treat the edge node as the most important server of the cluster.

Keep replicas on separate nodes

The cluster may place several replicas of a web application or a worker on one node, as when the other nodes are full. A node that fails then stops all of them at once.

To make sure two replicas never run on the same node, open the application, choose Change configuration… and turn on Keep replicas on separate nodes. It applies at the next deployment, or with Apply configuration.

It only sets where the replicas run, not how many: three nodes and two replicas still run two replicas, on two different nodes. With one replica, it changes nothing until you add a second.

The trade-off: a deployment then needs as many nodes that accept workloads, each with room for one replica, as the application runs replicas. With fewer, it stops before it starts and says how many nodes have room: add a node or run fewer replicas. While a node is drained or down, its replica waits for a node without one.

Leave it off when you run several replicas on one server on purpose, as for more throughput.

One CPU architecture per cluster

All the nodes of a cluster are x86 or all are ARM, so any replica can run on any node. With both kinds of servers, make two clusters.