The etcd of a cluster

What etcd does on a cluster

The two instances of a PostgreSQL with high availability agree on which one is the primary in etcd, a small database with one member on each manager node. etcd accepts a write only when more than half of its members agree, so an instance on a node cut from the others can neither keep nor take the role.

You never create or configure etcd. Tenkai starts it with the first PostgreSQL with high availability of the cluster and removes it with the last one. It answers on the cluster network only.

Each member reserves 64 MB of memory on its manager node, and no CPU.

How many failures it survives

With three manager nodes, etcd keeps working when one is lost; with five, when two are. A cluster that must survive the loss of two manager nodes needs five. See How many failures a cluster survives.

When etcd loses its majority, a PostgreSQL with high availability keeps taking writes while its primary reaches its standby. A primary that loses its standby too steps down within seconds: its hostname closes each connection, reads included, until enough manager nodes are back. See What it does not protect against.

Read its state on the cluster page

The etcd section of the cluster page shows its state and each member. When it says Failed, read the message, fix what it names, then an owner or admin chooses Retry. A PostgreSQL whose provisioning waited for etcd failed with it: retry the database once etcd runs.

When no member answers

Tenkai could not confirm that etcd has its majority means no member answered. Check that more than half of the manager nodes run and that Tenkai reaches them: see cluster is not ready. Read a member's logs with docker service logs on its tenkai-etcd-<node id> service on a manager node.

Do not stop or remove the tenkai-etcd services by hand: Tenkai does not start a missing member again, and etcd runs with one member less.

If most manager nodes are gone for good, recover the cluster: etcd starts again with the member of the edge node alone.

What etcd changes on your manager nodes

Only one member stops at a time, so etcd keeps its majority:

• Tenkai refuses to reboot a node that holds a member while another such node reboots or while the members change. Request it again afterwards.
• A manager node that holds a member cannot be drained. Choose Stop accepting workloads instead.
• The managers stay at three or five while the cluster holds a PostgreSQL with high availability: delete these databases before reducing them to one.
• A manager that becomes a worker node keeps its member until Tenkai takes it out. Until then, it can neither be drained nor leave its cluster.