What Happens When Your Database Crashes Halfway Through a Write?

"Crash-safe" is one of those phrases that shows up in every database's marketing copy and almost never gets demonstrated. It's easy to say. It's much harder to actually prove, because proving it means doing something most engineers go out of their way to avoid: killing the process on purpose, at the worst possible moment, and seeing what's left afterward.

I've been building VellumDB, a small LSM-tree key-value store in Rust, specifically to understand what PostgreSQL, RocksDB, and LevelDB are actually doing under the hood rather than just calling one. Once the write-ahead log, memtable, SSTables, and compaction were all in place, I stopped writing new features for a while and started trying to break it instead. This post walks through what I found, including one bug that a planned test would never have caught.

The one guarantee everything else depends on

Before any of this makes sense, there's one fact worth being precise about: when your code writes to a file, the OS usually doesn't touch the disk at all. It copies your bytes into the page cache, a chunk of RAM the kernel uses to buffer disk I/O, and returns immediately. If the machine loses power right then, that write is gone, even though your program was told it succeeded.

fsync is the system call that closes that gap. It blocks until the kernel has actually pushed the buffered data down to the disk. Every single write in VellumDB calls fsync before returning control to the caller:

pub fn append(&mut self, seq: u64, ops: &[WalOp]) -> io::Result<()> {
    // ...build the payload...
    write_frame(&mut self.file, &payload)?;
    self.file.sync_data()
}

That one line is the entire reason the experiments below don't just end in data loss. It's also expensive: fsync is orders of magnitude slower than a RAM write, and in my own benchmarks it dominates write latency (a few milliseconds per write, versus microseconds for a read). That's a real, deliberate trade this design makes, and it's worth knowing you're making it before you make it.

Experiment 1: kill the process mid-write

The simplest possible test: write two keys, then kill the process externally before it ever gets to shut down cleanly.

printf 'put foo bar\nput baz qux\n' | timeout 2 cargo run --quiet -- data-dir

No exit, no flush, just a hard kill. Reopening the same directory afterward:

vellum> bar
vellum> qux

Both writes survived. That's the fsync from above doing its job: by the time the REPL printed ok, the write was already durable, independent of whatever happened to the process a moment later.

Experiment 2: corrupt a record by hand

Durability against a clean kill is one thing. What about a record that's actually corrupted, a bit flipped somewhere on disk, not by a crash but by bad luck?

I wrote two records normally, then flipped a single byte inside the second one without touching its length field:

let mut bytes = fs::read(&path).unwrap();
bytes[bytes.len() - 2] ^= 0xFF;
fs::write(&path, &bytes).unwrap();

Replaying the log afterward recovers exactly the first record and stops. It doesn't error out, and critically, it doesn't silently load the corrupted second record as if it were valid data either.

This works because every record is wrapped in a small envelope: a length, a CRC32 checksum, then the payload. Corruption changes the payload's bytes without changing its stored checksum, so a mismatch is basically guaranteed. The database can tell the difference between "nothing left to read" and "something here I shouldn't trust," and treats them differently.

Experiment 3: the crash that actually matters for a database like this

Here's where it gets interesting. An LSM engine doesn't write data in one atomic step, it periodically flushes an in-memory buffer out to an immutable file on disk, then updates a small piece of metadata (called a manifest) that says "this file is now part of the database." That's two separate writes. What if the process dies between them?

Rather than trying to time a real process kill to land in that exact window, I reproduced the artifact such a crash would leave behind directly: a fully-formed, valid data file written straight to the directory, without ever going through the normal flush path, so nothing ever recorded that it should be considered part of the database.

Reopening the directory:

• the orphan file is not counted as active
• its data is nowhere to be found
• everything written earlier through the normal log still comes back correctly

The manifest is installed by writing to a temporary file and then renaming it over the real one, and a rename is atomic on both Windows and POSIX filesystems: it either fully lands or it doesn't happen at all. A file the manifest doesn't name is invisible, no matter how complete or well-formed it looks sitting on disk. That's what "the old version or the new version survives, never a mix of both" actually means in practice, it isn't a slogan, it's one specific file being ignored.

Experiment 4: multi-key writes, atomically

What if you want to write several keys as a single logical operation, where a reader should never see half of them applied? I wrote a small before-key normally, then a two-key batch, then truncated the file so it cut a few bytes into the batch:

get(before)  -> "1"    // untouched, survives
get(batch-a) -> None   // neither half of the batch landed
get(batch-b) -> None

This one didn't need new machinery. A batch is written as a single checksummed envelope covering every operation in it, not one envelope per key, so a torn write during it loses the entire envelope. There's no code path where batch-a can land and batch-b can't, because they were never two separate durability events in the first place.

Experiment 5: the bug I didn't go looking for

Everything above was a test I designed on purpose. This last one wasn't, and it's the one I actually learned something from.

While working on the corruption-detection logic from experiment 2, I tried opening a database directory that had been created before that change, back when the on-disk format didn't have the length-and-checksum envelope yet. The result:

failed to open vellum-data: failed to fill whole buffer

The whole database refused to open. That's strictly worse than the crash it was supposed to protect against.

Tracing it down: the new code was reading old-format bytes, and by pure coincidence, the first few of them (a zero byte for an old field, followed by more zeros) happened to satisfy the new envelope's length-and-checksum check as an empty, technically-valid record. The outer check passed. But the code that parsed what was supposedly inside that envelope hadn't been written defensively, it assumed that if the envelope was valid, the data inside it must be too, and used a bare ? that turned any problem there into an unhandled crash instead of routing it through the same lenient recovery path as everything else.

The fix was small: treat a structural failure while parsing an already-checksum-verified payload exactly the same as a torn write, drop it and stop, rather than let it propagate as a hard error. But finding it wasn't small. It took actually trying to open a real, older database, not just running the test suite, because the test suite only exercises the failure modes I'd thought to write down in advance. This one only showed up because I used the software the way an actual user eventually would.

Why this is worth doing at all

Anyone can write "handles crashes gracefully" in a README. The difference between that sentence and something you can actually trust is whether someone tried to break it and can show you exactly what happened when they did: which byte got flipped, which file got orphaned, which assertion passed or failed. "I built a write-ahead log" is a claim. "I killed the process mid-write, corrupted a record by hand, faked the exact artifact an interrupted flush leaves behind, and found a real bug by opening an old database" is evidence.

The full writeup, with the actual commands and file offsets, lives in the VellumDB repository if you want to see the rest of it.