Storage layout

The database directory

A database is a directory named by the database name. Inside it sits a single directory named by 32 hex characters, which are the 16 bytes of the root node ID. Every node ID derives from that root, so the hex name fixes the identity of the data. Rename the outer directory and the database carries on working.

chat/
  6f2a91c4e85b0d37a1fe94cc20b8735d/
    alice/
      laptop-3f9a/
      desktop-81c2/
    bob/
      macbook-0d77/

Each user has a directory. Each machine belonging to that user has a directory inside it. A machine writes its own directory and reads all the others. That single rule keeps a shared folder such as Dropbox free of write conflicts, because two machines never touch the same file.

A machine name comes from the machine's own configuration, held outside the shared folder, and carries a random suffix. Two machines that share a hostname still get separate directories.

A machine is the unit that owns files. A user is the unit that owns data: records under users/<user> are valid only from that user's machine directories. See Data model for the shared and per-user areas.

Days and log files

Inside a machine directory, files sit under a UTC date.

alice/laptop-3f9a/
  index                                  root index
  2026/index                             year index
  2026/09/index                          month index
  2026/09/15/index                       day index, written when the day is sealed
  2026/09/15/00000000.log
  2026/09/15/00000000.00000000.small.dat
  2026/09/15/00000000.00000001.small.dat
  2026/09/15/00000000.00000000.large.dat
  2026/09/15/00000001.log
  2026/09/16/00000000.log                today: logs and data files only
  2026/09/16/00000000.00000000.small.dat

Log files are numbered from 00000000, eight digits, with no gaps within a day. A log file holds 64-byte records end to end and nothing else: no header, no framing, no padding. Record 812 of a log starts at byte 812 × 64, so a position within a machine's history is a day, a file number and a record index. A log file holds at most 16384 records, which makes a full file 1 MB.

A machine writes to the day max(today, last day written). A clock that moves backwards leaves positions increasing.

Days with no writes have no directory. The index tree lists the days that exist, so a reader can tell a quiet day from a day that has yet to arrive.

Values and data files

A record carries the length of its value and a 16-byte hash field. Short values live in the record itself. Longer values live in data files beside the log.

Data files are named after their log file: 00000000.00000003.small.dat is the fourth small data file of log 00000000.log. Small and large files are numbered separately, so a run of large values leaves the current small file open and the next small value continues it.

No record stores a data file number or an offset. A reader works both out by walking the log in order.

small = 0, offset = 0, large = 0

for each record in the log, in order:
    skip tombstones, references, and values of 16 bytes or fewer

    if length >= 1 MiB:
        value is <log>.<large>.large.dat, the whole file
        large = large + 1
    else:
        if offset >= 16 MiB:
            small = small + 1
            offset = 0
        value is <log>.<small>.small.dat at [offset, offset + length)
        offset = offset + length

The rollover check runs before the append, so a small value always lands whole in one file and a file may finish a little over 16 MiB. A value is never split across two files.

A log holding four long values of 300 bytes, 2 MiB, 900 bytes and 40 bytes fills its data files like this:

From the log alone, a reader knows the final size of every data file. A shorter file holds values still in transit, and the reader knows exactly which records are waiting on them. Each value is checked against the hash in its record when it is read.

Today

Today's files take appends and nothing else. A machine writes no index for today and never compacts today's files. Copies made by compaction land in today's log, so today's log grows from two sources: new writes, and records carried forward from older files.

Other machines read today's logs as live data. Decisions that depend on something being absent wait for a sealed day, which is the point at which the machine has published a complete account of what that day contains.

Write order

1. Append the value to its data file.
2. Append the 64-byte record to the log.

The value goes first, so a complete record always has its value on disk behind it. A reader that holds a record never waits for a value that the machine never wrote.

Startup recovery

A machine may have stopped in the middle of a write. At startup it repairs its own current day:

1. Truncate the current log file to a multiple of 64 bytes, which drops a torn record.
2. Walk the log and compute the expected size of every data file belonging to it.
3. Truncate any data file past its expected size, which drops a value whose record never reached the log.

Sealed days need no repair. Nothing writes them after they are sealed.

Sealing a day

A machine seals a day when it first writes on a later day, and at startup when the last day it wrote is earlier than today. Sealing writes the day index, then updates the month, year and root index files. Indexes covers what those files hold.

A machine that stops in the middle of a day leaves that day unsealed until it runs again. Its records stay readable. Entries born that day stay unverified, so tombstones for them stay in place.

One writer per directory

A machine keeps its own last written position in its local configuration, outside the shared folder. At startup it compares that position with what the directory holds.

A restored directory breaks the rule that a machine's files only move forward. Records that the rest of the database has already accounted for would come back. The position check catches that at startup, before any write.

A conflicted copy left by the sync service inside a machine directory means the same thing: two machines wrote it. The server reports it and stops.

Related