Redis Internals: Cluster Bus Gossip Protocol

Redis Cluster keeps a second network plane beside client RESP: the cluster bus. Nodes use it to discover peers, share slot and epoch views, detect failures, vote on failovers, and propagate selected cluster events. This post covers architecture (planes and gossip mechanism), then implementation (protocol wire format and code paths) in git/redis (cluster_legacy.h, cluster_legacy.c).

Related: Network / command path, Pub/Sub, Build from source.

1. Architecture

Redis Cluster’s control architecture is a node-to-node mesh on a dedicated cluster bus port, separate from client RESP. Each node holds a local membership and slot view and exchanges binary clusterMsg packets—periodic gossip for liveness and topology, plus discrete messages for failure, failover, and selected events.

1.1 Two planes

Redis Cluster separates data traffic from control traffic. Clients stay on RESP; only Redis nodes speak the binary bus protocol.

 clients --RESP--> [ Redis node: data + slots ]
                        |
                        | bus TCP (clusterMsg)
                        v
                   peer ... peer

The bus does not carry ordinary key commands. Gossip scales membership and health without shipping a full node list on every packet.

1.2 Gossip mechanism

Gossip is how Redis Cluster spreads membership and health without shipping a full node list on every packet. It rides only on PING / PONG / MEET: the fixed header always describes the sender (epochs, myslots, flags, …); the body carries a sample of rumors about other nodes (clusterMsgDataGossip rows). Discrete types (FAIL, UPDATE, Pub/Sub, failover votes, …) are separate events on the same links—they are not the gossip sampler.

Wire layouts for those types are in §2.3.

1.2.1 Exchange

PING, PONG, and MEET share the same packet kind (data.ping). Semantics differ:

sequenceDiagram
  participant A as Node A
  participant B as Node B
  A->>B: MEET or PING
  Note right of B: apply header, merge gossip sample
  B->>A: PONG
  Note left of A: apply header, merge gossip sample

Default frequency. clusterCron runs every 100 ms. Ping timing is driven by cluster-node-timeout (default 15000 ms / 15 s in redis.conf / config.c):

So under defaults, a healthy peer is typically re-pinged on the order of every ~7.5 s when its last pong ages out, plus occasional random pings (~1/s cluster-wide toward the stalest sample). Manual-failover wait may ping the chosen replica continuously.

Each successful exchange updates the sender from the peer’s header and merges the peer’s gossip rows into the local nodes dictionary.

1.2.2 Sampling

The body never lists the whole cluster. clusterSendPing builds a bounded random sample, then forces failure opinions into that sample:

Why ~1/10 (and at least three). Source comment on clusterSendPing: within about two cluster-node-timeout windows, nodes exchange several pings/pongs. With ~N/10 gossip slots per packet, the chance that a given master appears often enough in others’ samples is high enough that a node already in PFAIL can collect majority failure reports before reports expire—without paying full-membership bandwidth every round.

What each sampled row carries. A rumor about one third party: id, coarse ping_sent / pong_received (seconds), IP and ports, and that peer’s flags as the sender sees them (MASTER / SLAVE / PFAIL / FAIL / NOADDR, …). Exact field sizes are under §2.3.1.

1.2.3 Merge

clusterProcessGossipSection validates node ids, then for each row:

1. Corrupt ids — reject the entire gossip section.
2. Known node, not myself — if the packet sender is a master and the row has PFAIL/FAIL, record a failure report (and maybe promote to FAIL); if the row clears failure flags, drop that master’s report. Optionally refresh local pong_received. If the subject is down locally but gossip says it is up at a new address, update IP/ports and drop the old link.
3. Unknown node — if the sender is a trusted cluster member, the row is not NOADDR, and the id is not blacklisted, insert a new clusterNode.
4. Gossip about myself — ignored for state updates.

Over many rounds, random samples plus PFAIL bias yield cluster-wide convergence on membership and failure views with (O(N)) rumor bytes per packet rather than (O(N^2)) full dumps.

1.2.4 Failure detection (weak quorum)

Failure is a local opinion that becomes a cluster flag only with majority support among masters that serve slots. Gossip sampling is what spreads those opinions:

1. If A cannot reach X within cluster-node-timeout, A sets local PFAIL on X.
2. Masters advertise PFAIL/FAIL in gossip rows; peers record failure reports from masters. PFAIL bias in §1.2.2 Sampling accelerates this.
3. If A already has PFAIL on X and reports from a majority of masters (quorum ((\mathit{cluster_size}/2)+1)), A sets FAIL, clears PFAIL, and broadcasts a discrete FAIL message (data.fail) so others can adopt the flag without waiting for another gossip round.
4. Reachability again can clear FAIL under defined conditions (especially for replicas).

This agreement is intentionally weak and time-based: partitions may delay visibility, and without majority no replica promotion is authorized.

1.2.5 Replica promotion (to master)

Once the failed master is FAIL, a replica of that master may try to take over on the same bus:

1. Replica R broadcasts FAILOVER_AUTH_REQUEST (header carries epochs and claimed myslots; body empty). For manual failover, MFSTART runs first and FORCEACK may be set on the request.
2. Slotted masters that accept the vote reply with unicast FAILOVER_AUTH_ACK.
3. When R collects enough acks (failover_auth_count reaches quorum), it promotes: SLAVE → MASTER, claims the former master’s slots, bumps configEpoch.
4. Peers learn the new owner through subsequent gossip headers and optional UPDATE packets so client MOVED targets point at R.

Wire layouts: §2.3.4–§2.3.6, §2.3.7.

1.2.6 Design properties (gossip)

2. Implementation

2.1 Links and packet ingress

/* cluster_legacy.h */
typedef struct clusterLink {
    connection *conn;
    list *send_msg_queue;
    char *rcvbuf;
    clusterNode *node;   /* NULL until sender is known */
    int inbound;
    /* ... */
} clusterLink;

clusterCron (and related paths) open outbound links and schedule pings. Inbound data accumulates in rcvbuf until a full clusterMsg is present; clusterProcessPacket branches on type.

2.2 Protocol design

2.2.1 Framing

Every bus packet is one binary clusterMsg: fixed header, then a type-specific data body. Signature RCmb, protocol version 1; field offsets are ABI-stable across upgrades.

1. Header always describes the sender — id, IP, ports, flags, full slot bitmap (myslots), currentEpoch / configEpoch, replication offset, cluster state. Receivers refresh the sender from the header even when they care mainly about the body.
2. Body is typed — gossip arrays for the ping family; failed node name for FAIL; channel+payload for publish; slot config for UPDATE; module blob for MODULE; often empty for vote / MFSTART (meaning is in type + header epochs/flags).
3. Trust boundary — unknown senders are ignored for sensitive types. MEET is the intentional exception that creates knowledge of the sender.

2.2.2 Header fields

The fixed header ends at offset 2256 (data begins there). Multi-byte integers are sent in network byte order. Offsets below are from static_assert in cluster_legacy.h.

How receivers use the header. After validating sig, ver, and totlen, the node looks up or creates the clusterNode for sender, then copies reachability-related fields (ports, myip, flags, epochs, myslots, slaveof, offset, state) according to the processing rules for that type. Gossip rows in data describe other nodes; the header is always the sender’s self-description.

Flags worth recognizing on the wire.

In the header, flags describe the sender (normally MASTER or SLAVE, plus capability bits). PFAIL / FAIL matter most in gossip rows (see §1.2), where they are the sender’s opinion about other nodes and feed failure reports.

2.3 Data by message type

hdr->type selects the union clusterMsgData arm. The fixed header (§2.2.2) is always present; only data (and how count / mflags / epochs are interpreted) changes.

2.3.1 PING / PONG / MEET (0–2) — data.ping

Role. Continuous gossip: liveness round-trip plus membership rumors. MEET additionally inserts the sender if unknown.

Body layout.

data.ping
+------------------+-----+------------------+------------------------+
| gossip[0]        | ... | gossip[count-1]  | ping extensions (opt.) |
+------------------+-----+------------------+------------------------+
  sizeof(clusterMsgDataGossip) each           hdr->extensions records

• hdr->count = number of gossip entries (see §1.2.2 Sampling).
• Optional extensions follow &gossip[count] when mflags[0] has CLUSTERMSG_FLAG0_EXT_DATA (getInitialPingExt).

Gossip entry (clusterMsgDataGossip) — rumor about a third node, filled by clusterSetGossipEntry:

Extensions (after the array, 8-byte aligned):

Example (PING / PONG). A sends PING to B with count = 4 (three random peers + one PFAIL). B merges gossip, replies PONG with its own header and sample.

Example (MEET). Operator runs CLUSTER MEET 10.0.0.5 7000 on node A. A opens a bus TCP connection to 10.0.0.5:17000 (client port + 10000) and sends a packet with type = MEET, the same data.ping layout as a ping (header describes A, body is a gossip sample). New node N does not yet know A: because the type is MEET, it inserts A into nodes, then replies with PONG (same body shape). Later membership grows through ordinary PING/PONG gossip samples, not further MEET packets.

2.3.2 FAIL (3) — data.fail

Role. After local PFAIL plus majority master reports, broadcast so peers adopt FAIL quickly.

Body (clusterMsgDataFail):

No gossip array; count unused.

Example. Master A promotes X to FAIL and broadcasts FAIL with about = X. Peer D adopts the flag from known sender A.

2.3.3 PUBLISH (4) / PUBLISHSHARD (10) — data.publish

Role. Cross-node Pub/Sub. PUBLISH is cluster-wide; PUBLISHSHARD goes only to other nodes in the sender’s shard.

Body (clusterMsgDataPublish; lengths in network order):

Example. PUBLISH alerts "disk-full" on A → bus PUBLISH to all nodes. SPUBLISH → PUBLISHSHARD only within the shard.

2.3.4 FAILOVER_AUTH_REQUEST (5) — empty data

Role. Replica asks masters for votes to promote.

Body. None beyond the common header. Meaning is in:

Broadcast; only slotted masters vote (clusterSendFailoverAuthIfNeeded).

Example. Replica R of failed master X broadcasts FAILOVER_AUTH_REQUEST with X’s former slots in myslots.

2.3.5 FAILOVER_AUTH_ACK (6) — empty data

Role. One master’s yes vote, unicast to the candidate.

Body. Empty. Validity is the sender being a slotted master and senderCurrentEpoch >= failover_auth_epoch. Candidate increments failover_auth_count.

Example. Master A sends FAILOVER_AUTH_ACK to R; at quorum, R promotes.

2.3.6 UPDATE (7) — data.update

Role. Push another node’s slot ownership when a peer’s view is stale.

Body (clusterMsgDataUpdate):

Example. After failover, an UPDATE about R with a higher epoch rewrites B’s slot map so MOVED targets R.

2.3.7 MFSTART (8) — empty data

Role. Replica asks its master to start manual failover (pause writes, set mf_end / mf_slave).

Body. Empty. Sent from a replica to its master to start manual failover (mf_end / mf_slave on the master side).

Example. R sends MFSTART to master M; M pauses clients and pings R with the offset for mf_can_start.

2.3.8 MODULE (9) — data.module

Role. Opaque module cluster API traffic.

Body (clusterMsgModule):

Example. Module on A unicasts or broadcasts a typed blob; B dispatches to registered receivers.

2.4 Slots and epochs on the wire

Slot ownership and config freshness ride in every gossip header (myslots, configEpoch, currentEpoch). When a node learns a higher config epoch for a peer’s slots than it currently believes, an UPDATE packet’s body (§2.3.6) can push that peer’s name, epoch, and full slot bitmap so redirects stay correct after failover or resharding.

2.5 Design properties (wire)

Gossip sampling and failure quorum are covered in §1.2.

2.6 Wire format: clusterMsg

Binary framing as designed in §2.2. Concrete header fields:

typedef struct {
    char sig[4];              /* "RCmb" */
    uint32_t totlen;
    uint16_t ver;
    uint16_t port;            /* primary client port */
    uint16_t type;            /* CLUSTERMSG_TYPE_* */
    uint16_t count;           /* gossip entries for ping family */
    uint64_t currentEpoch;
    uint64_t configEpoch;
    uint64_t offset;
    char sender[40];
    unsigned char myslots[CLUSTER_SLOTS/8];
    char slaveof[40];
    char myip[NET_IP_STR_LEN];
    uint16_t extensions;
    /* pport, cport, flags, state, mflags, ... */
    union clusterMsgData data;
} clusterMsg;

Gossip entry

typedef struct {
    char nodename[CLUSTER_NAMELEN];
    uint32_t ping_sent;
    uint32_t pong_received;
    char ip[NET_IP_STR_LEN];
    uint16_t port;
    uint16_t cport;
    uint16_t flags;
    uint16_t pport;
    uint16_t notused1;
} clusterMsgDataGossip;

Optional extensions (CLUSTERMSG_FLAG0_EXT_DATA): hostname, human nodename, forgotten-node TTL, shard id, internal secret—8-byte aligned after the gossip array.

2.7 Sending: clusterSendPing

/* cluster_legacy.c — behavior */
wanted = max(3, floor(dictSize(nodes) / 10));
/* random eligible nodes: not myself, not receiver, not HANDSHAKE/NOADDR/... */
/* then append every PFAIL node */
clusterSetGossipEntry(hdr, gossipcount++, node);

2.8 Receiving: clusterProcessGossipSection

Called for ping-family packets after length validation.

markNodeAsFailingIfNeeded(node):
  require local PFAIL
  failures = reports from masters (+ self if master)
  if failures >= (cluster_size/2)+1
      set FAIL; clusterSendFail(name)

2.9 Key files