XKV Can Now Swap Changes Between Stores and Keep the Reasons Behind Them
XKV 0.2.0 is out, and it adds exchange. One store writes its recent changes to a file, another store reads them in, and every change arrives with its actor, its reason, its ref and its causes. Ask the second store why a value is what it is, and you get the same chain the first store would give you.
That matters wherever the same settings live in more than one place. A service and its staging copy. Two sites that each take local writes. A laptop that was offline for a day. Copying values between them is easy. Copying the story behind them is what usually gets lost, and with it any chance of answering “who changed this, and why” on the receiving side.
How It Works
$ xkv export site-a.xkv --since 120 --out a-changes.xkvx
$ xkv import site-b.xkv a-changes.xkvx
Every store now has an id, and every revision keeps the id of the store it was first written to and its number there, however many times it moves. That one rule does most of the work. Importing the same file twice changes nothing. A store recognises its own changes when they come back. Causes land on the right revisions on the other side, even when the numbers differ. And since an import adds only what other people wrote, it needs no reason of its own: each change keeps the one it was written with.
An export also carries the older revisions its changes point at, so a chain arrives whole. Those older ones come in as history: Why can follow them, but they never become a key’s value.
When Both Sides Changed the Same Key
This is where sync tools usually pick a winner and throw the loser away. XKV picks a winner too, but keeps the loser. Each change carries what its writer had seen of the key, so an import can tell a change made on top of the current value from one made without knowing it. Only the second kind is a conflict, and clocks have no say in that.
In a conflict, one change becomes the value and the other stays in the key’s history, where History shows it and marks it. By default the later change wins, and every store picks the same one, so stores exchanging in both directions end up with the same values, whether there are two of them or a ring of four. A clock that runs slow can’t undo a later edit either: XKV never records a write as earlier than a change to that key it already holds. You can also tell an import to keep your side or take theirs. If both sides made the same change, there’s nothing to settle.
Checked Before Anything Is Written
An exchange file ends with a line that gives its own length and a SHA-256 of everything before it. The import reads the whole file and checks it first, so a file cut short in transit or changed by hand is refused whole. Each transaction in the file goes in as one transaction, so an import that dies part-way can just be run again.
A copied store file keeps its id, and two stores with the same id would mix up each other’s revisions, so an import between them is refused. xkv new-id gives the copy its own id, and the history the two share stays shared.
How It Was Tested
Exchange got its own tests, aimed at what it promises, and the numbers are in the repository’s test results.
Twenty pairs of stores took random writes on both sides, swapped files in both directions at random and compacted now and then. That made 2,045 revisions, 749 conflicts and 174 changes both sides had made alike. Every pair ended with the same values. Then 100 edits made after settling each reached the other store as an ordinary change, with no conflict.
Pairs are the easy case, so 90 more runs linked three or four stores in a ring, a star or a full mesh, with clocks up to 74 minutes apart. They settled 7,185 conflicts between them, and every run ended with the same values on every store. A separate random network of three to five stores added deletes, reopening, replayed old files and imports cut part-way, and a model of what each change had seen checked all 5,739 reported conflicts. None was spurious, none was missed.
A process importing a file was killed 40 times at random points; 20 of the kills landed part-way. Every time the file checked out, and running the import again ended exactly like a clean import. An export was cut at every byte, and 3,000 copies had a few bytes changed: each was refused whole, except 3 whose change still left a valid file. A fuzzer resealed rewritten files with a correct checksum 3.6 million times, and nothing panicked. At scale, 100,000 revisions exported in 536 ms and imported in 1.8 seconds.
Independent reviews went through the design and the code before the release. They found a record that could grow past what a reader accepts, a way for a far-future date to break “the later change wins”, and a compaction case that kept the wrong revision, among others. Each is fixed and has its own test.
Upgrading From 0.1
Exchange needs a new file format, version 2, which adds the store id and what each change had seen. Stores made by 0.1 still open and work as before. xkv upgrade rewrites one as version 2, keeping every revision and its number, and an export or import does it on its own. One thing to know: XKV 0.1 can’t open a version 2 file, so upgrade the program before the files.
The reference lists the new calls and commands, and the exchange file is described line by line in the repository. Binaries for Linux, macOS and Windows are on the download page.