Testing

This page lists the test suites that back T4’s correctness claims: what each one checks, how it injects faults, and when it runs in CI. The guarantees themselves are described in the Consistency Model; the behaviour under specific failures, with the tests that cover each one, is in Failure Scenarios.

“Every PR” below means every pull request and every push to main and release-* branches; for Jepsen, pushes run only when they change jepsen/. The workflows are in .github/workflows.

Suite	Runs
Go unit tests and compatibility fixtures	Every PR (Linux, macOS and Windows)
Go integration tests, including the chaos test	Every PR
Race detector over the whole main module	Every PR
S3 end-to-end tests	Every PR
Upgrade from earlier releases	Every PR
Differential tests against etcd	Every PR
etcd robustness tests, single node	Every PR
etcd robustness tests, three nodes on S3	Nightly and on every release tag
Kubernetes apiserver storage and watch cache tests	Every PR
Kubernetes conformance on k0s, quick mode (one test)	Every PR
Kubernetes conformance on k0s, full suite	Nightly and on every release tag
Jepsen smoke matrix (3 jobs)	Every PR
Jepsen full matrix (18 jobs)	Nightly and on every release tag
30-minute stress test	Nightly and on every release tag
Go benchmarks	Pushes to `main`

Go unit and integration tests

What they prove: the storage engine, WAL, replication, leader election, checkpointing, leases, auth and the etcd adapter behave as specified, including under crashes, object store outages and partitions between nodes.

Unit tests cover the root package, cmd/t4, the etcd adapter (etcd, etcd/auth), internal/... and pkg/object. They run with -short on Linux, macOS and Windows.
Integration tests in tests/integration are black-box scenarios through the public t4 API: multi-node clusters, scale down and up, recovery from checkpoints and WAL, leader failover, watches and leases across a leader change, and corruption of the WAL and checkpoints. Faults are injected in-process: nodes are closed or killed mid-commit, the object store is wrapped to fail on demand, and peer traffic goes through a proxy that can be blocked to partition a follower from the leader.
TestChaos runs rounds of concurrent writes on a three-node cluster, kills a random node each round (the leader about a third of the time), restarts it from an empty data directory so it must recover from the object store, and checks that every acknowledged write is still readable. CI runs 5 rounds; T4_CHAOS_ROUNDS runs more.
Race detector: a separate job runs go test -race ./... over the whole main module without -short, so the integration tests, TestChaos and a 3-minute run of the stress test below also run under the race detector.

When: every PR.

Compatibility fixtures

What they prove: databases written by earlier releases still open and keep working.

tests/compat holds data directories and object store contents generated by tagged releases (starting with v0.19.1) and checked into testdata/. The current code must open a previous release’s local data directory, restore and replay its object store contents, and restore its branch checkpoints. A separate fixture holds lease and auth state written by v1.1.11, the last release before the meta keyspace, and checks it keeps working in its legacy layout.

When: every PR, with the unit tests.

Upgrade tests

What they prove: a real binary of an earlier release and the current binary agree on the data, end to end.

tests/upgrade downloads an earlier release’s t4 binary (checksum-verified), has it write a varied dataset through the etcd API, stops it, and starts the current binary on the same data. Every key, revision, lease and auth entry must read back unchanged, history included. With S3 enabled, the old release also writes to S3 and the new one restores from there into an empty data directory.

CI picks the releases to upgrade from with hack/upgrade-matrix.sh: the latest patch of the previous minor release, the first patch of the latest minor release, and the latest release. Each runs against RustFS as the S3 store.

When: every PR.

See also Upgrade and Downgrade and the v1 Compatibility contract.

S3 end-to-end tests

What they prove: T4 works against a real S3-compatible server, not only the in-memory object store most Go tests use.

tests/e2e runs against RustFS in Docker: the object store client (put, get, list, delete, versioned reads), the CLI, restore from an S3 checkpoint (plain and encrypted), an encrypted cluster through leader failover and a late-joining node (every object written to the bucket must be encrypted), and a TLS-enabled S3 endpoint.

When: every PR.

Differential tests against etcd

What they prove: T4 gives the same answers as etcd, revision for revision.

tests/etcddiff runs the same randomized workload of puts, deletes, transactions, lease grants, keepalives and revokes against a real embedded etcd and against T4’s etcd adapter, and requires identical results: the header revision after every operation, the responses, per-key metadata (create and mod revision, version, lease), and the full watch history. A second test covers lease expiry: an expiring lease deletes its keys in exactly one revision, and an expiring lease without keys consumes none.

Revision-exact agreement is what lets a replicator map T4 revision R to etcd revision R. Each run uses a random seed and logs it; ETCDDIFF_SEED=<seed> reproduces a failure.

There is no fault injection: both servers run as a single node.

When: every PR.

etcd robustness tests

What they prove: T4 passes etcd’s own correctness checks for linearizability and watch guarantees while nodes crash, lose each other, or lose S3, and while history is compacted.

tests/robustness drives real t4 run processes with the traffic generators and validators from etcd’s robustness suite (go.etcd.io/etcd/tests/v3/robustness). Validation requires that:

every result a client observed is linearizable;
watches deliver one ordered, gap-free history on every node, resumable from any revision;
reads at a past revision return the state at that revision;
every node ends with the same HashKV.

Two traffic scenarios run: EtcdPutDeleteLease (puts, deletes and leases) and Kubernetes (traffic shaped like the Kubernetes apiserver’s).

Fault injection: the harness, not etcd’s gofail failpoints, injects one fault per run once traffic is up to speed:

SIGKILL of a random node, the leader, a follower, or every node, followed by a restart; or a graceful restart of the leader.
In a three-node cluster, network faults through a proxy in front of each node’s peer and S3 traffic: the followers cut off from the leader, the leader’s peer traffic delayed by 50–200 ms, or the leader or a follower cut off from S3. A cut connection drops what is sent rather than closing, as on a lossy network.

On a single node only the random-node kill applies. Like a supervisor, the harness restarts a node whose t4 process exits on its own, as a leader does when it steps down after losing S3.

Every scenario also compacts the history every 200 ms, including to revisions already compacted and to revisions that do not exist yet, which must be refused as etcd refuses them.

Differences from upstream:

etcd validates against the requests it persisted, read from its raft WAL. T4 has no raft WAL, so the history is rebuilt from a watch over the whole keyspace.
A compaction takes no revision, so that history has no place for one. A successful compaction is left out of it and placed by linearization alone; the compaction loop never compacts past what the history watch has seen.

When: every PR runs the single-node scenarios. The nightly workflow, and every release tag, also runs every scenario and fault on a three-node cluster against RustFS.

Jepsen

What it proves: a five-node cluster sharing one S3 store stays linearizable (single key and multi-key transactions), strict-serializable, and gives every watcher the same totally ordered history, while the network is partitioned, nodes are killed, and nodes lose access to S3.

The Jepsen suite runs nodes n1–n5 in Docker against RustFS. Workloads, defined in core.clj:

register — single-key read, write and CAS (50% / 30% / 20%), checked by Knossos against a CAS-register model.
multi-register — three keys written and CASed together in one Txn, modelled as a single CAS register, so a partially applied transaction would show up as a linearizability violation.
append — Elle list-append, ported from Jepsen’s etcd suite: multi-key transactions guarded on ModRevision, checked for strict serializability.
wr — Elle read-write registers in one Txn per transaction, checked for strict serializability.
watch — writers put successive values to one key while watchers, served by different nodes, run a series of short watches, each resuming after the last revision seen. All watchers must observe the same sequence, with no event lost, duplicated or reordered.
set — adds to a set stored in one key via CAS; every acknowledged add must be visible to every read that starts after it completes.

Nemeses:

partition-halves — splits the five nodes 2/3 at random, then heals.
kill — kills and restarts the T4 process on a random node.
partition-s3 — drops one random node’s traffic to S3 with iptables, then heals.

Each job runs for 60 seconds of faults and traffic, then heals, waits for recovery and runs final reads.

When:

Every PR runs a smoke matrix: partition-halves × register, multi-register, append (3 jobs).
Nightly runs the full matrix: all three nemeses × all six workloads (18 jobs).
Every release tag runs the full matrix on the release commit before any image or binary is built.

Kubernetes apiserver storage tests

What they prove: the Kubernetes apiserver’s own storage tests pass on T4, both for the etcd3 storage backend and for the watch cache in front of it.

tests/apiserver builds the upstream tests from k8s.io/apiserver (v0.35.4 by default) and runs them against a real t4 run process. Only the embedded etcd test server is swapped for T4; the tests themselves are unchanged. Two packages run:

pkg/storage/etcd3 — the etcd3 storage backend the apiserver uses.
pkg/storage/cacher — the watch cache that sits in front of it.

Three TestLists cases in the cacher suite are skipped. They expect the watch cache to delegate an exact list at a revision to storage, which it does only until a watch progress notification has told it about that revision. etcd sends those every 10 minutes by default; T4 sends them every second, so the cache already holds the revision and serves the list itself.

There is no fault injection: T4 runs as a single node.

When: every PR.

Kubernetes conformance

What it proves: a Kubernetes cluster that keeps its state in T4 passes the Kubernetes conformance tests.

tests/conformance builds a T4 image from the tree and starts a two-node k0s cluster in Docker (a controller that also runs workloads, and a worker), with T4 as its external etcd. Sonobuoy then runs the conformance suite (certified-conformance mode) on it. This covers what the apiserver storage tests do not: the whole control plane on T4, with its controllers, watches, leader election leases and the apiserver’s own compactions.

There is no fault injection: T4 runs as a single node. Run it locally with make test-conformance; T4_CONFORMANCE_MODE=quick runs a single test.

When: every PR runs quick mode, which boots the cluster and runs a single test in a few minutes. The full suite runs nightly and on every release tag, and takes about two hours.

Stress test

What it proves: a cluster stays consistent through a long run of membership changes, leader changes and compactions under continuous writes.

TestLongRunningConsistency in tests/integration writes continuously through any live node while it compacts periodically and adds or removes a random node every ~12 seconds, keeping the cluster between 1 and 5 nodes. New nodes bootstrap from the object store, and removing the leader forces an election. Every ~15 seconds, and once at the end, every live node must hold every committed write.

When: nightly and on every release tag, for 30 minutes. Every PR also runs it for its default 3 minutes in the race detector job.

Static analysis

go vet and golangci-lint run on every PR. golangci-lint reports only issues new in the change.

Benchmarks

The Go benchmarks for the root package and the etcd adapter run on every push to main, and their results are kept as workflow artifacts. They are not compared against a baseline and do not fail the build. The head-to-head comparison with etcd is a separate Docker harness under bench/; its results and methodology are on the Benchmarks page.

Releases

Pushing a release tag starts the release workflow, which:

waits for the CI workflow to have passed on the tagged commit, and refuses to release otherwise;
runs everything the nightly workflow runs on that commit: the full Jepsen matrix, the 30-minute stress test, the three-node robustness tests and Kubernetes conformance;
builds and publishes images and binaries only after both pass.

Known gaps

What the suites above do not cover today:

No lease workload in Jepsen. Leases are exercised by the robustness tests’ EtcdPutDeleteLease traffic, under process kills and while the followers are cut off from the leader or a node from S3, by the differential tests (no faults), and by the integration tests for keepalive, expiry and revoke across a leader change. No test drives lease keepalive through a network partition: the robustness traffic grants and revokes leases but never keeps one alive.
Network faults in the robustness tests run only nightly and on release tags. They need a three-node cluster on S3; pull requests run the single-node scenarios.
The differential, Kubernetes apiserver and conformance tests run a single node without faults. They check API semantics, not behaviour under failure.
Jepsen runs are short. Each job runs 60 seconds of traffic with three clients, on one host in Docker.