Upgrade and Downgrade

What operators need to know to move a running T4 deployment from one v1.x release to another. For the underlying compatibility rules see the v1 Compatibility Contract.

What v1.x guarantees

A newer v1.x binary can open a data directory and an object-store prefix written by any older v1.x binary.
Newer v1.x can also add fields, flags, default values, exported types, and metrics — but never remove or change the meaning of existing ones without a major version bump.
Default values listed in the defaults table do not change across v1.x unless explicitly documented as tuning guidance.

What fails closed

These are the failure modes the binary refuses rather than silently corrupting state. They protect against an unsupervised downgrade onto data the older binary cannot interpret. None of them is recoverable without action; the binary exits with a clear error and the cluster’s other nodes (if any) keep serving.

Trigger	Behaviour
Checkpoint `format_version` newer than this binary understands	Startup refuses to restore. Operator must roll the binary forward.
WAL frame magic / version newer than this binary understands	Startup refuses to replay. Same remedy.
Follower on an earlier release connecting to the leader of a database created with the meta keyspace	The leader refuses the stream with `wal_format_unsupported`. Upgrade the follower.
WAL entry with an op code this binary does not know (local replay, S3 replay, or the leader’s peer stream)	Replay or apply is refused. A follower stops replicating and logs an error instead of retrying or attempting a leader takeover; it does not apply the entry. Same remedy.
Object store does not enforce the conditional PUTs leader election needs, and `--peer-listen` is set	Startup refuses with `ErrConditionalWritesIgnored`. Use a store that enforces them — on GCS that means `--gcs-bucket` rather than the S3-compatible endpoint — or run a single node. A single node starts with a warning.
`manifest/latest` references a checkpoint missing referenced SSTs	Startup error. Recoverable from an earlier checkpoint via `t4 restore`.
Branch registry entry points at a checkpoint older than the safe `format_version`	GC refuses to reclaim. Inspect with `t4 inspect`.

Rolling upgrade (cluster)

For multi-node clusters, upgrade one node at a time. The leader stays available throughout; the follower being upgraded re-syncs from S3 on restart.

Pick the oldest node that is currently a follower (avoid the leader on the first pass).
Stop the t4 process. Followers resync automatically on restart, so the data directory does not need to be cleared.
Install the new binary and restart with the same flags as before.
Wait for the follower to catch up — t4 status shows current_revision >= leader_current_revision - tolerable_lag and the leader’s metrics show the follower reconnected.
Repeat for the remaining followers.
Restart the leader last. A follower wins the takeover election (~6 s; see Consistency). Promote it gracefully by signalling the leader with SIGTERM; the new leader is the highest-CommittedRev follower.

The cluster keeps accepting writes throughout, modulo the ~6 s window when the leader is being restarted.

Single-node upgrade

Stop the t4 process.
Install the new binary.
Start with the same flags. T4 replays any unsealed WAL entries from the local data directory; if the local directory is gone (ephemeral storage), it restores the latest checkpoint from S3.

Databases created with the meta keyspace

Databases created by this release or later keep T4’s own bookkeeping (etcd leases, auth users, roles and tokens) in the meta keyspace, which is replicated through the WAL but does not consume revisions. As in etcd, granting a lease, keeping it alive, revoking a lease with no attached keys, and changing auth leave the revision unchanged.

The meta keyspace is enabled when the database is created, as its first WAL entry, and never changes afterwards:

Existing databases keep their format. When this release opens a database created by an earlier one, lease and auth state stays in revisioned keys as before, and the database can still be downgraded.
New databases cannot be opened by earlier releases. Their WAL segments containing meta entries use format 3 and their checkpoints use format 2, which earlier releases refuse.
A leader of a new database refuses followers that run an earlier release. They are turned away with wal_format_unsupported before any entry is streamed to them. Create new databases only once every node runs this release or later.

Moving off the GCS S3-compatible endpoint

A deployment configured with --s3-endpoint https://storage.googleapis.com is on a store that accepts the conditional writes leader election depends on and ignores them. Since the release that added the startup check, a cluster on that configuration refuses to start, so it has to move; a single node starts with a warning and should still move.

Switch to the GCS API, which enforces the preconditions:

# before
--s3-endpoint https://storage.googleapis.com --s3-bucket my-bucket --s3-prefix t4/ \
  --s3-access-key-id ... --s3-secret-access-key ...

# after
--gcs-bucket my-bucket --gcs-prefix t4/

The bucket and the object names do not change, so the existing data is read as it stands — no copy or re-upload. What changes is how T4 authenticates: Application Default Credentials instead of HMAC keys, or --gcs-credentials-file for a service account key. The HMAC keys can then be deleted. --gcs-bucket and --s3-bucket are mutually exclusive, so remove the --s3-* flags rather than leaving them alongside.

Downgrade

A downgrade to an older v1.x release is best-effort, not guaranteed. The newer binary may have written checkpoint or WAL formats the older binary does not understand.

Check these before downgrading:

Is the latest checkpoint in a format the older binary understands? Neither t4 restore list nor t4 inspect meta reports a format version, so read it from the manifest:
Terminal window
```
# Absent or 1 → the older binary can restore it. 2 → it will refuse.
aws s3 cp s3://your-bucket/your-prefix/manifest/latest - | jq '.format_version // 1'
```
Check the local WAL segments the same way — the third byte of each header is the format version:
Terminal window
```
for f in /var/lib/t4/wal/*.wal; do
  printf '%s format %d\n' "$(basename "$f")" \
    "$(dd if="$f" bs=1 skip=2 count=1 2>/dev/null | od -An -tu1 | tr -d ' ')"
done
```
If either is above what the older binary knows, it refuses to start (see fail-closed table above).
Are any branch registry entries (branches/<id> in S3) newer than the older binary’s safe format_version? List them:
Terminal window
```
aws s3 ls s3://your-bucket/your-prefix/branches/
```
Active branches pin their ancestor checkpoint format, so downgrading past that version is unsafe even if the latest checkpoint is older.
Has any newly-added t4.Config field been used at runtime? See git diff of config.go between the two versions. Embedders relying on a newer field will get a compile error against the older library; the standalone binary silently ignores unknown CLI flags from a newer env-var set, which is a recipe for silent misconfiguration.

If all three answers are “no”, the downgrade is safe. If any answer is “yes”, roll forward to a release without the regression instead — moving forward with a fix is always safer than moving backward.

What to do when fail-closed fires

The binary exits with an error referencing the schema version it rejected. Steps:

Note the version it refused and the corresponding compatibility contract section.
Do not delete the offending object — it is needed by the newer binary you must roll back to.
Roll the binary forward to a version that supports that schema. The data directory and S3 prefix are unchanged by the failed start.