Failure Scenarios

Failure Scenarios

This document describes how T4 behaves under various failure conditions, what data is at risk, and how to recover. Each scenario maps to automated tests.

---

Leader crash

Before WAL flush

Scenario: the leader process dies (kill -9 or OOM) after accepting a write request but before fsyncing it to the local WAL.

Outcome:

• The write was never fsynced → it never existed durably. The client connection is broken; the client receives an error or a timeout.
• No data is lost because nothing was durably stored.

Recovery:

• A follower wins election in ~7–10 s (it waits 7 s after it last heard the leader, and until its reconnect attempts are exhausted; ~0.3 s with a graceful shutdown broadcast).
• The new leader loads the latest S3 checkpoint and replays any WAL segments ahead of it.
• Clients retry against the new leader.

Tested by: TestLeaderCrashBeforeWALFlush

---

After WAL flush, before ACK to client

Scenario: the leader fsyncs the entry to its local WAL and broadcasts to followers, but crashes before returning to the client.

Outcome:

• The entry is on the leader’s WAL (and on follower WALs if ACKs were received before the crash).
• The client gets a broken connection.

Recovery:

• The new leader replays the leader’s WAL from S3 and applies all entries up to the highest committed revision.
• The write is visible after recovery. The client may retry safely — a Create (conditional) will fail with ErrExists (already committed), while a Put will be idempotent.

Tested by: TestLeaderCrashBeforeWALFlush (covers post-flush path)

---

Follower crash

Scenario: a follower process dies while the leader and at least one other follower are healthy.

Outcome:

• Leader detects the disconnect immediately, or within 2 s through missed heartbeats if the connection is not closed.
• With default FollowerWaitMode=quorum, the leader needs ACKs from ⌊n/2⌋ + 1 nodes. In a 3-node cluster losing one follower still leaves the leader + 1 follower = quorum. Writes continue.
• The crashed follower’s writes that were in-flight are lost locally; but those writes were not quorum-ACKed, so they were not acknowledged to the client.

Recovery:

• Restart the follower. On startup it loads the latest S3 checkpoint and replays any missing WAL segments from S3 before reconnecting to the leader.
• If the follower restarted with a fresh disk (container replacement), the S3 path restores the full state.

Tested by: TestFollowerKilledDuringCommit

---

Full cluster restart

Scenario: all nodes are stopped or fail simultaneously.

Outcome:

• No writes are accepted while all nodes are down.
• All nodes restart independently. Each loads the latest S3 checkpoint and replays WAL segments.
• Leader election proceeds via S3 conditional PUT. The first node to successfully write the lock becomes the new leader.

Recovery:

# Restart each node normally — they self-coordinate via S3.
t4 run --data-dir /var/lib/t4 --s3-bucket my-bucket --s3-prefix t4/ \
           --peer-listen 0.0.0.0:3380 ...

Any write that completed quorum ACK before the cluster went down exists on all nodes’ WALs and survives.

Tested by: TestE2EThreeNode

---

S3 unavailability

Cluster mode

Scenario: S3 becomes unreachable while the cluster is running.

Outcome:

• WAL segment uploads and checkpoints queue in the background.
• The leader cannot renew its lock. While its followers still hear it, none of them may take over, and the leader keeps serving on its last renewal: until 2 s before that renewal’s validity ends, at most about a minute. After that it stops acknowledging writes and serving linearizable reads, and returns ErrNoLeader (etcd: Unavailable), because it can no longer rule out that another node has taken over.
• Writes that need the election fence moved (after a follower leaves or falls behind) fail with the same error while S3 is down.
• Followers cannot write a new lock either, so no election succeeds and there is no split brain. Serializable reads keep working on every node.
• 2 s after its lease ended, if it still cannot renew, the leader steps down: it stops its peer server and t4 run exits with status 1, for its supervisor to restart it as a follower.

Recovery:

• An outage shorter than the lease: the next renewal restores the lease and writes resume. Queued uploads resume automatically.
• A longer outage: once S3 is back, a follower takes over (about 7 s after the old leader stepped down) and the restarted old leader follows it.
• No manual intervention is needed, provided a supervisor restarts t4 run when it exits.

Tested by: TestObjectStoreUnavailableWritesSucceed, TestObjectStoreUnavailableRecovery

Only the leader loses S3

Scenario: the leader cannot reach S3, while its followers can, for example a network fault on the leader’s path to S3.

Outcome: as in a full outage, the leader keeps serving until its lease ends, then stops serving and, 2 s later, steps down. Its followers no longer hear it, and about 7 s later one of them takes over through S3 and serves. An outage of the leader’s path to S3 that outlasts its lease costs a failover, not an outage of the cluster.

Tested by: TestLeaderStepsDownWithoutObjectStorage

---

Single-node mode

Scenario: S3 becomes unreachable in single-node mode (where WALSyncUpload=true, the default).

Outcome:

• Writes succeed until the current open WAL segment fills and a rotation is attempted. On rotation, the seal-and-upload step fails. The node fences itself to prevent unacknowledged data from accumulating silently.
• All writes acknowledged before the outage are durable — their segments are already on S3.

Recovery:

1. Restore S3 access.
2. Restart the node. It replays all local WAL segments (including any partial segment) before resuming.

A restart during the outage is also safe: with local data present, the node starts from it when S3 does not answer within 15 s, without replaying from S3. Only a node with an empty data directory needs S3 to start.

Tested by: TestObjectStoreUnavailableWritesSucceed, TestSingleNodeRestartsDuringObjectStoreOutage

---

Network partition

Follower partitioned from leader

Scenario: a follower cannot reach the leader or S3, but the leader can still reach S3 and the other followers.

1. Follower exhausts --follower-max-retries reconnect attempts.
2. Follower reads the S3 lock. The leader noticed the silence through missed heartbeats and renews the lock every 2 s, so the follower finds it fresh and does not promote.
3. Leader continues writing with the remaining nodes.
4. When the partition heals, the follower reconnects and resyncs from the leader’s ring buffer.

Result: no data loss, no split-brain, no service interruption.

Tested by: TestNoTakeoverFromLiveLeader, TestLeaderHeartbeatNotBlockedByPendingWrites

---

Leader partitioned from followers and S3

1. The leader’s lock renewals fail and it stops hearing its followers. Its lease falls back to 4 s after its last renewal, and it stops acknowledging writes and serving linearizable reads.
2. 7 s after they last heard the leader, and once the lock has not been renewed for 6 s, followers race to write a new lock with If-Match: <etag>. Only one wins.
3. The new leader begins serving. By then the old leader has already stopped serving, and 2 s after its lease ended it steps down.
4. The old leader’s writes that were not acknowledged are dropped. A client retrying them gets an error, never a false success.

The lease relies on clocks: any two nodes’ clocks must differ by less than 2 s. NTP keeps them far closer.

Failover time: ~7 s with default settings. ~0.3 s with graceful shutdown broadcast.

Tested by: TestNetworkPartitionNoSplitBrain, TestFailoverTime, TestDeposedLeaderStopsServing, TestLeaderWithoutObjectStoreKeepsServing, TestTakeOverChecksTheRecordItReplaces

---

WAL corruption

CRC mismatch mid-segment

Scenario: a WAL segment file is partially corrupted (bit flip, torn write, truncated upload).

Outcome:

• On replay, ReadEntry detects the CRC mismatch and returns an error.
• All entries before the corrupted one are applied. The partially-written entry and all subsequent entries in that segment are discarded.
• A warning is logged: wal: CRC mismatch at rev=N.

Recovery:

• If only the trailing entry of the last segment is corrupted (typical torn-write on crash), the node recovers all entries up to the corruption point. The last write that failed the CRC was not durably acknowledged, so no data is lost.
• If corruption is deeper, entries after the corruption point in that segment are lost. Restore from a checkpoint (see Backup and Restore) and replay WAL segments from S3.

Tested by: TestWALCorruptionMidSegment, TestWALReplayAfterPartialUpload

---

Checkpoint corruption

Scenario: a checkpoint manifest or archive on S3 is corrupted or incomplete.

Outcome:

• ReadManifest or ReadCheckpointIndex fails with a JSON parse or integrity error.
• The node logs the error and falls back to the previous checkpoint.
• If no valid checkpoint exists, the node fails to start and prints a clear error.

Recovery:

• Use t4 restore list to identify the last good checkpoint.
• Use t4 restore checkpoint to download it to a fresh --data-dir.
• Start the node normally pointing at the new data directory.

Tested by: TestCheckpointCorruptionManifest, TestCheckpointCorruptionArchive

---

Chaos: random node kills

Scenario: random cluster nodes (leaders and followers) are killed and restarted from fresh data directories across multiple rounds of concurrent writes.

Outcome verified by TestChaos:

• Every write acknowledged before a kill is visible on all surviving nodes after recovery.
• The cluster re-elects a leader after each kill without split-brain.
• Nodes restarted from a fresh disk recover fully via the S3 checkpoint + WAL replay path.

To run extended chaos testing:

T4_CHAOS_ROUNDS=100 go test -run TestChaos -timeout 600s

---

Failover timing

Event	Typical time
Graceful leader shutdown (broadcast)	~12 ms
Graceful leader shutdown, follower missed the broadcast	next takeover attempt (the released lock carries no liveness to wait out)
Crash failover (default settings)	~7–10 s (7 s after followers last heard the leader, and their reconnect attempts exhausted)
Follower reconnect after network heal	< 1 s

Tested by TestFailoverTime.

---

See also

• Consistency model — durability definitions, CAP properties
• Operations guide — S3 unavailability, network partitions, recovery procedure
• Backup and restore — recovering from checkpoint after severe data corruption