How the HyperCrux Beta Was Tested, With a Power Cut at Every Write and 261 Planted Bugs

HyperCrux 0.1 left the hardest part of a database to SQLite, which has decades of testing behind it. The Beta, the new engine in Go that an AI coding agent has built since 7 October 2026, does that part itself. It keeps one append-only file and a copy of the data in each process’s memory, with its own rules for a crash and for compaction. So it has to show, with tests, that a crash or a failed write at any moment loses nothing that was committed. The progress report of 14 October gave the Beta’s speed and a short account of the checks. This post goes through them in depth.

Every task in the plan closed with an automatic test of its own, which then ran on every push, on x86 and ARM Linux. Eight of the 48 tasks built tests and nothing else: 0.x’s tests as a suite for either engine, a harness that compares the two engines, a corpus of SQL answers, two layers that fake a failing disk, a driver that crashes the engine at every point, a harness of many processes, and the benchmarks. Every figure below comes from the task records, which are on GitHub with the code.

The tests are tested too. The Beta’s code holds 261 bugs put there on purpose, and each has to make its tests fail.

A Power Cut at Every Write

The Beta reaches its file only through a small Go interface, so a test can put a fake disk under it. The fake, the fault layer, holds the disk in memory. When a test cuts the power, each sector written since the last sync is kept, lost or torn, as a real drive can leave it. A new file or a rename that its folder hasn’t synced yet can vanish too, as on ext4. Any call can be made to fail at its nth use, and a seed decides every choice, so a failure replays exactly.

The fault layer’s own tests show why the rules matter. A file renamed over another, as a compaction swaps in its new file, came back as the old file in 109 seeds of 200 after a cut when its folder wasn’t synced. With the folder synced, the rename lasted in all 200.

On that disk, the crash-point driver runs a workload of commits and cuts the power at each write in turn, from many seeds. Then it fails each call in turn, and copies the file in the middle of each commit. After every point the database has to open at the last commit that succeeded, or at the one under way, with exactly that commit’s state, and the check of the four handles has to find nothing. Through the public Go package, the smallest workload, eight commits on a new database, makes 86 calls. A full run cuts the power 1,008 times, from 48 seeds, and fails a call 688 times, and as many again on a disk that stops working after the failure. Four more workloads add compactions: one made by the same handle, one by another handle, and two where the engine compacts by itself after a commit. In the long run of 14 October, every crash point ran from 3,840 seeds, which took 1 hour 42 minutes on x86 and 2 hours 21 minutes on ARM.

What the Crash Tests Caught

They caught real bugs, well before any release. When a compaction’s sync of its folder failed after the rename, its handle stopped writing, as it should. But once that handle had closed, another could commit into the new file while the rename might still not be on the drive, and a power cut could put the old file back without the commit. In the test that showed it, 19 seeds of 40 lost a commit that had been reported done. The fix: every writer syncs the folder once before its first commit into a file whose name it didn’t make itself. Since then no seed has lost one, and a planted bug that takes the sync away is caught by the tests.

The same suites found that a commit could report a failure after it had succeeded, when only letting go of the lock failed. A caller would then have made the commit twice. Now the commit stands, the call returns no error, and the lock goes at the handle’s next commit, or when it closes.

The driver had to show it can fail, too. Its first test plants a known mistake in a toy log: each marker written before its batch is synced. A cut in the marker’s write catches it in 2 to 6 per cent of seeds, and a failed sync in 76 to 83 per cent. That’s why the suites run many seeds at every point.

200 Killed Writers and Many Processes at Once

A simulated power cut tests one process. Kills test many. 0.x’s kill test runs on the Beta through a thin adapter: it kills a writer with SIGKILL at a random moment, 200 times, and checks after each kill that the file holds exactly the last committed transaction, with every record, link and vector. On the Beta it made 17,512 transactions in 62 seconds. The package’s own version goes on until 200 writers have died in the middle of a commit. In one recorded run, 447 writers were killed, 200 of them mid-commit, and 74 of those commits were in the file at the end, whole, since their writers died after the batch was written. The other 126 left no trace.

The many-process harness runs writers and readers on one database, all of them copies of the test binary, and kills them at random as they go. At the end, every reader must have seen every commit once and in order, and the file must hold every commit a writer saw succeed. In a toy log, a planted bug that writes each commit’s marker before its batch was caught in each of 15 runs, 5 to 48 milliseconds in. The harness also moves backups into place at random moments, and checks each stretch of the log between two restores against its own file. In one full run through the public package, 470 commits began, with 27 compactions and 7 restores among them, and every reader saw every commit once and in order. In the long run of 13 October, five runs of 42 minutes each passed on both architectures.

Checked Against 0.x, Answer by Answer

The Beta keeps 0.x’s Go API, so 0.x can judge it. 0.x’s own suite, the 25 tests that don’t depend on SQLite, runs on both engines. The Beta runs it twice, once as it is and once with the file opened afresh before every read, so nothing passes from memory that didn’t reach the file. 0.x’s tests of the command run on both binaries as well: all 11 sections on 0.x, and on the Beta the 9 that don’t use SQLite’s own features.

The differential harness goes further. It plays random sequences of operations on both engines, compares every answer and then the whole database, and shrinks any sequence they disagree on to its shortest form, which it keeps as a test. To show it can fail, it also runs copies of 0.x with one thing each done wrong. Each was caught within 11 sequences, and shrank to between one and five steps. Its first long run against the Beta found one difference, which is there on purpose: a write that fails in the Beta leaves no trace, where 0.x keeps the new fields it named. On 14 October the harness ran 800,000 sequences on each architecture, in 2 hours 51 minutes on x86 and 2 hours 12 minutes on ARM, and the two engines agreed.

SQL has a judge of its own: a corpus of 3,911 cases with the answers 0.x gave, from statements inside and outside the Beta’s subset to 3,000 generated expressions and hundreds of dates. The Beta has to give 0.x’s answer to each, or refuse a statement the corpus marks as outside the subset.

Fuzzing and the Long Runs

Go’s fuzzer feeds code random input, keeps what reaches new paths, and stops at the first crash or wrong answer. Six targets run for 30 seconds each on every push: four in the Beta’s own code, the decoders of the file’s parts and the SQL parser, and two in the reader of the export format, which 0.x shares. The race detector runs over the whole suite on every push as well.

A long workflow, started by hand, gives each target 70 minutes on x86 and on ARM, beside hours of the crash suites, the many-process harness and the differential harness. In its first run, on 13 October, the batch decoder took 250 million inputs on x86 and found nothing. The SQL parser found a real bug, after 22 minutes on x86 and 54 on ARM: a condition naming a column whose name is a space from outside ASCII, such as a non-breaking space, followed by a comment, didn’t print back as itself. It was fixed the next day with a test and a planted bug, and in the second run the parser took 203 million inputs on x86 and 219 million on ARM without a fault.

Bugs Planted on Purpose

A test that never fails proves little. So each planted bug sits in a branch that an ordinary build leaves out, and a script switches each one on alone and needs its tests to fail. The bugs sit where data could be lost or wrong: in commits, recovery, compaction, the copy in memory, vectors, links and SQL. Each task that changed the code added its own, which is how the count grew to 261. The push workflow tries them all in five shards side by side. In the long run of 14 October, the 235 planted by then were all caught, on x86 and on ARM, in 21 minutes each.

The harnesses were tested by hand as well. While the crash driver was built, 17 bugs went into the driver itself, one at a time, and its tests caught all 17. The many-process harness caught 22 of the 23 put into it. The one it missed can’t be seen from outside: a final check that read the file before the killed writers had ended, which waits for their lock anyway.

Two Rounds of Independent Review

Then two more AI coding agents, neither of which wrote any of the Beta, reviewed it in two rounds of their own, recorded in full on GitHub. The progress report lists four of their fixes. Here’s the rest of what they found.

The first round reviewed the file’s safety. It wrote a test of its own that damages random files, with zeros, a changed byte, a cut or random bytes to the end, and opens each one: 2,000 files a run, and 60,000 once by hand, all handled as the format says. The one finding that needed code was the decoder the progress report mentions, which a damaged batch could make take about 89 times its size in memory. Four more it answered without code:

• A compaction can still undo a backup moved into place, if the move lands in the tens of microseconds between its last check and its rename. The docs now say so.
• A drive that leaves a torn sector unreadable stops the database opening until the end of the file is cut back by hand. That fails safe, since nothing committed is lost.
• ext4 mounted with data=writeback can bring back a commit that was cut out of the file after it failed, so the docs rule that setting out.
• One task’s record claimed a kill can’t leave half a batch. It can, and it’s harmless, since half a batch fails its checksum and the next writer cuts it.

The second round reviewed the copy in memory, SQL, the Go package and the command. Beside the fixes in the progress report, a filter for a vector search that ends inside a comment now gives the error 0.x gives, and the command builds on macOS and Windows again, where it says it runs on Linux only. One error message no longer names the package twice. Each fix came with a test, and all but one with a planted bug. It also found that the code an export writes to mustn’t call the database, or the two can wait for each other for ever, and the docs now say so. Each round ended with questions for the project’s owner, and two of the answers became small tasks of their own: compacting by dead data, and holding text and bytes to 0.x’s limit of 1,000,000,000 bytes.

What the Tests Can’t Show

The fault layer is kinder than some disks, as the unreadable sector shows. The tests run on Linux, on the file systems of CI’s runners and of the machine the work is done on, and nothing tests a network file system, which the Beta rules out. Passing every test shows that the cases tried work. Trust with real data takes years of use, as it did for SQLite.

Moving from 0.x to the Beta covers what changes for a program and its data. The task board shows where the work stands, and each task’s record there gives its tests and their figures.