How XKV Was Tested With 1000 Kills, Torn Writes and a Million Revisions

A store that keeps the reason behind every value is only useful if it keeps the value too. So before 0.1.0 went out, each promise XKV makes got a test aimed straight at it, and every number below sits in the repository’s test results, written by one script on a 2-CPU Linux machine.

The promises are short. A transaction is all or nothing. An acknowledged write survives a crash. Committed data is never dropped quietly. Compaction never breaks a chain of causes. Here is how each was tried.

Killing the Writer

A helper process opens a store and writes as fast as it can, each transaction naming earlier revisions as causes. After every acknowledged write it prints the revision number. The test kills it with SIGKILL at a random moment, reopens the file, and checks three things: every acknowledged revision is there, no transaction is half visible, and every cause in the file still points at a real revision.

That ran 1000 times on the same file. 53,701 transactions were acknowledged, and none was lost. A kill can’t split a single write call, so the file never even had a torn tail to trim; that case gets its own test below.

Cutting the Last Write

A power cut can leave half a write on disk. To copy that, the test takes a file, cuts its last transaction at each of its 397 byte positions, and opens the result. Then it does the same with a damaged byte in place of a cut. Every open trimmed back to the last whole transaction, answered Get and Why as before, and the next write carried on the numbering.

The bytes that get cut off aren’t thrown away. XKV copies them to a file beside the store, named .trimmed- and the time, before it trims.

Damage in the Middle

A cut at the end is a crash. A changed byte in the middle of the file is something else, and XKV treats it that way: it finds a whole, valid transaction after the break, so it refuses to open with ErrCorrupt and changes nothing. xkv check says where the damage starts. xkv repair trims there on purpose, keeping the cut bytes the same way, for the day you decide the rest is lost.

The fuzzers fed both the file reader and the record decoder random bytes for two minutes each. Neither crashed, and both refuse anything that isn’t in the exact form the writer makes.

Compaction Keeps the Chains

Compaction drops old revisions, so it is where a chain of causes could break. The test builds 20 random workloads, compacts each one, and asks the same questions before and after: Get for every key, the full Why tree for every kept revision, and the kept history. It asks again after reopening and again after a full rescan. Every answer matched.

13,659 of the 15,299 revisions kept across those runs stayed only because a kept revision names them as a cause. Those are the ones a plain “keep the last N” rule would have lost.

A second test killed the process 40 times in the middle of a compaction. Whether the kill landed before the rename or after it, the store opened and answered as it had before.

A Million Revisions

The long test writes one million revisions across ten thousand keys, each with a reason and a cause. On the test machine:

Speed

A Put that waits for fsync took 157 µs; without the fsync 1.8 µs. Get took 0.76 µs and a Why through a 16-revision chain 25 µs. The fsync is most of the cost of a write, and it is the part that makes an acknowledged write safe.

The tests also run on every push, on Linux and macOS, under the race detector on Linux, on Windows, in 32-bit mode and in WebAssembly under Node. The download page has the binaries those checks produced.