Skip to content

Comparing with the reference

harness/compare.mjs holds the server to the reference PDS (@atproto/pds), and you can run each of its runs by hand. The PDS: accounts and their records says what’s compared, and each area’s page has its own “How it is checked”.

You’ll need a clone of the repository, with the harness’s packages installed and the server built (Contributing).

spec/PDS.md says what the server does and how compare.mjs checks it. The server and the reference PDS get the same random requests, and their answers have to agree, apart from what differs between any two servers. The reference’s own library has to verify the server’s whole repository, and its MST root must equal the reference’s.

Some requests go out at once: puts of different keys, and batches racing on the same commit (exactly one of those may land).

When the server forwards a request to the AppView, a chat service or any service an atproto-proxy header names, both sides send it to stub services (harness/lib/services.mjs). The stubs record what each side sent. That way the harness compares the forwarded requests, their headers and their service-auth tokens, as well as the answers.

An account on a third PDS that shares the PLC directory publishes OAuth’s permission sets (OAuth: permission sets), and both servers point at it. So each one resolves the harness’s sets over the network, with their proofs, the same way it would any app’s. The harness compares PAR’s answers, the page’s data, the token’s expanded scope at sign-in and refresh, and what calls a set’s session may make with the reference.

Every run ends by checking the server’s operations (Operations). The harness checks its /xrpc/_health against the reference’s, its metrics against its log and what the harness sent, and the admin’s _logs and _stats against what it wrote and counted. It checks every span against the request it belongs to, at a stub OTLP receiver that the run takes away for a while. It checks a shutdown, and starts at another log level, with a PLC directory that fails, after a kill, and with variables that don’t parse. Last, the whole log has to hold a line for every request and none of the passwords, tokens, codes, emails or keys the run handled.

An agent working independently of the first built a second server from spec/. It passed the same harness, with the same repository encoding and the same behaviour where the specs leave it open.

Run it from harness/, with the server’s command after --:

cd harness && npm ci
node compare.mjs --steps 400 --seed 1 -- ../target/release/pds serve

A seed makes the same requests every run. Each flag below starts a different kind of run.

node compare.mjs --steps 1000 --seed 21 --crash-every 15 --firehose -- ../target/release/pds serve

--crash-every N makes the harness kill the server with SIGKILL every N steps, often during a write. It starts the server again on the same data and checks that every answered write survived, and that the interrupted write landed whole or not at all. pds-server’s own tests check what survives a power cut, by cutting its log short at every byte of its last write.

--firehose also subscribes the harness to both servers’ subscribeRepos. It checks each of the server’s events the way a consumer would (signature, the chain of since and prevData, the ops undone on the tree in the event giving the previous root, and the reference library’s verifyProofs). It compares the server’s commits with the reference’s by their ops, and replays from random cursors. When crashes are on too, the stream has to carry on after each restart with no gap or repeat.

node compare.mjs --steps 400 --seed 31 --commands -- ../target/release/pds serve
node conformance.mjs -- ../target/release/pds serve

--commands sends some of its requests through the MCP tool, the CLI command or pds xrpc instead of XRPC, in any of the three MCP revisions. They’re held to the reference like the rest. pds xrpc gets each argument in the forms the request allows: ==, =, :=, an @path file, and - with a Content-Type: header. Before that, the run checks every tool’s title, schema and hints, and every command’s flags against the lexicons. It also checks each legacy revision’s handshake, pds xrpc on the server’s own routes and on a forwarded method, and its usage errors (Commands: every method as an MCP tool and a CLI command). conformance.mjs runs MCP’s conformance suite against the endpoint, for 2026-07-28 and 2025-11-25.

node compare.mjs --steps 400 --seed 1 --invites -- ../target/release/pds serve
node compare.mjs --steps 400 --seed 1 --limits -- ../target/release/pds serve

--invites runs both the reference and the server with invite codes required.

--limits turns the rate limits on and compares each with the reference’s, under both password sessions and OAuth tokens. It also starts both with an upload limit of their own, and compares them at its edge. Then it holds only the server to the rest of Limits: file modes, the session and app-password caps, firehose messages and subscribers, calls to the PLC directory, timeouts, the free-space floor, the blob quota and its time for unreferenced uploads, and starting again with more accounts than open files allowed.

node compare.mjs --lifecycle -- ../target/release/pds serve

--lifecycle (Account lifecycle) deactivates accounts, has the admin take them down, and deletes them by emailed token. It checks what each state does to every method and to the firehose. When you add --crash-every N, the harness kills the server every N changes of state.

node compare.mjs --moves -- ../target/release/pds serve

--moves (Moving an account between servers) moves accounts to the server from a third PDS running the reference, and from the server back to it. Their handles are under the server’s domains, or external ones served by DNS and HTTPS. The harness compares every step with the reference, holds both firehoses to what each move puts on them, and checks the methods of a move by account state. It moves accounts but not blobs. --migrate, below, moves blobs.

node compare.mjs --steps 600 --seed 51 --crash-every 40 --store -- ../target/release/pds serve
node compare.mjs --scale -- ../target/release/pds serve
node compare.mjs --upgrade -- ../target/release/pds serve

--store holds the server to The store: what is kept, the backfill window, and the bounds. It gives both sides a five-second backfill window, then rewrites a few accounts’ records, blobs and space records many times over. It kills the server while it reorganizes its data, and compares every read and the firehose’s OutdatedCursor replays with the reference. Then it holds the data directory to three times the live state and a constant for each account, and each start to twice the median of the empty starts timed around it, plus 250 ms. A start over that is tried again up to twice before it fails.

--scale holds only the server to STORE.md’s scale rules, with a 16 MiB cache. The harness samples its resident memory while it holds 2000 accounts, touches each, writes a 64 MiB repository, streams it back and replays the firehose. It gives one account 4000 blobs and a space repo of 40000 records, and reads them back after a start, the whole space repo among them. Then it holds 100000 accounts, finds one in 50 of them by DID, handle, email and session, and refreshes 10000 sessions several times over. Through all of that, memory has to stay within its base, the cache and 4 KiB an account. The server has to start as fast with all of that as on an empty data directory. After a kill, a shutdown and a start, the firehose’s first message from a cursor has to come within a second. Making the accounts takes most of the run’s time, some minutes.

--upgrade starts only the server on the data directories the last release wrote, in harness/upgrade/ (one shut down and one killed). Every read has to match what that release answered there, and the server’s next writes have to carry on from that release’s (The store: what is kept, the backfill window, and the bounds).

--migrate moves the server’s blobs from its data directory into a bucket and back with pds migrate-blobs. The command is killed at random moments and run again, one move is turned back halfway, and one is stopped by a missing object. After each, the harness reads every blob against the reference’s (Blobs). It also holds the data directory to one process at a time.

--s3 keeps both sides’ blobs in buckets of an S3 store that the harness runs in process, and the store checks every request’s signature (Blobs). During the random steps and crashes, or --lifecycle, the store fails uploads, reads and the server’s deletes for both sides alike, and the harness compares the answers. The server’s bucket has to match what its accounts hold within a minute, and its data directory must hold no blob’s bytes.

node compare.mjs --pages --seed 1 -- ../target/release/pds serve

--pages drives the OAuth authorization pages in Chromium on both sides (OAuth: what the authorization pages show). It takes each flow the same way on both: from a PAR to the client’s redirect URI, signing in, remembered accounts, the email code, two tabs of one request, denials and backs. The harness compares the API calls each page made and where it landed, and exchanges its codes. Only the server’s pages are held to the rest of OAUTH-PAGES.md: their views and words, the consent list for each scope, focus, busy buttons, headers and HTML, assets, console, the origins asked, the width on a phone, and axe-core’s WCAG rules.

The spaces run (Spaces) checks the server against the reference’s pinned spaces alpha. It has its own packages:

cd harness/spaces && npm ci
node compare.mjs --seed 1 -- ../../target/release/pds serve

Besides the alpha’s PDS and a second one for the spaces’ peers, it starts a third as the run’s lexicon authority. That one publishes the space type declarations that OAuth space: scopes name, and permission sets that grant space permissions. The server is started with PDS_LEXICON_AUTHORITY_DID set to that account’s DID, and the alpha resolves from it too (OAuth: access tokens on every method, and what scopes allow, “How it is checked”; OAuth: permission sets, “How it is checked”).

node perf.mjs -- ../target/release/pds serve

perf.mjs runs one workload against both servers, each in its own process. For each part it takes the server’s best p99 and throughput as ratios over the reference’s. The server’s p99 ratio must be within 1.5 times the baseline’s in harness/perf-baseline.json, and never over 1.5. Its throughput ratio must be at least 0.8 times the baseline’s, and never under 0.8.

The baseline is the weakest of at least five runs on the runner, from the nightly or the perf workflow. The owner can ask for perf on any branch (knotless run rerun knotless://chadtmiller.com/monolith --ref refs/heads/<branch> --workflow perf). Each run prints every part’s ratios, then all of them on one line the way the baseline holds them. A pull that makes the server faster raises the baseline from those, and one that lowers it says why.

Then subscribers to the firehose come and go, 20 cohorts of 20. After the last cohort, the server must hold no more open file descriptors or threads than after the first, and no more than 8 MiB more memory (Operations).

harness/bench.mjs benchmarks the server beside other PDSes on this machine, one at a time. Every one runs in Docker the same way, with the server as its image builds it. It measures:

  • perf’s workload with applyWrites, listRecords and more
  • the routes a client calls most
  • reads and writes by the number of callers, and a mix of them at once
  • subscribers coming and going
  • each server’s start time and memory
  • what its client can send at all, so a rate near that is marked

A run refuses a busy machine unless you pass --busy, and it records the load. Each implementation is an adapter in it, pinned in bench/impls.json. The PDS benchmark says how to run and extend it, and Benchmark: the reference PDS, Monolith, vlpds, zds and tranquil holds the last results, with the machine and every caveat.

cd harness && node bench.mjs --setup && node bench.mjs --rounds 3 --json ../bench/results.json

just bench runs the same.

bench/store/ benchmarks how the server keeps its data, separately from the rest of the server. It runs the server’s logs and tables (crates/pds-store) beside SQLite, RocksDB and fjall under the same commits, each synced before it’s answered. Its parts are:

  • writes by 4 to 256 writers, and with no sync at all
  • a log replayed, and a firehose replayed from each store
  • reads of a record, a listing and an account not yet open, on one thread and sixteen, with what the process then holds
  • sizes on disk, compressed and not
  • the server’s start and memory by the number of accounts in its service log, beside STORE.md’s bounds

Rows next to the server’s own each change one thing about how it writes or reads (frames as bytes, a log allocated ahead, a table’s file held open, smaller blocks, another check), to show what each is worth. bench/store/README.md says what each part measures and what’s fair to each store, and bench/store/RESULTS.md holds the last results, by machine. The store-bench workflow runs it on the runner when the owner asks, on main or a branch. A pull that changes it or crates/pds-store runs it small (store-bench-small, BENCH_SMALL), only to show that it builds and runs.

The first build compiles RocksDB from source, which takes some minutes. Its bindings need libclang (apt-get install libclang-dev, and Xcode’s tools have it on macOS). On macOS, bench/store/.cargo/config.toml builds it to sync with F_FULLFSYNC, like the logs do. Without that flag, its writes there aren’t as durable as the others’.

just store-bench # arguments: counts of accounts for its last part

bench/soak/soak.mjs runs the server, or several PDSes side by side, under a steady mixed load for hours. As it goes, it checks that nothing answered is lost. It reads every account’s records back whole now and then, and holds every answered commit to coming through on the firehose. It’s for what only hours show and the benchmark’s short rounds don’t: memory over time, how the data directory grows and what it settles to, and what a crash in the middle of a load costs. It needs only Node. bench/soak/README.md says how to run it against a server on this machine or ones on a network, and covers its settled check of the data directory. bench/soak/RESULTS.md holds the runs, beside the reference.

OUT=/tmp/soak TARGET_URL=http://127.0.0.1:2594 TARGET_DOMAIN=test node bench/soak/soak.mjs

bench/soak/analyze.mjs makes every table of RESULTS.md from a run’s files: the soak’s own, a sampler’s where one ran beside it, and the server’s log cut to its requests and starts. Each run in RESULTS.md is its description, its command and that command’s output, so running the command again checks it. You add a run the same way, with its files in a directory of bench/soak/data/.

node bench/soak/analyze.mjs bench/soak/data/at-rest

harness/compat.mjs checks how closely any PDS behaves like the reference, area by area. It runs the core of compare.mjs’s checks (OAuth and its scopes, sessions, accounts, records, validation, the repository, sync, the firehose, identity, blobs) against each implementation with an adapter in harness/lib/adapters/, and keeps going past a difference. Compatibility with the reference PDS says which checks it runs and how to add an implementation. Compatibility results holds the last run, against Monolith, zds, vlpds, pds.js and tranquil.

just compat

That’s cd harness && node bench.mjs --setup && node compat.mjs --steps 200 --json ../compat/results.json. Arguments pass through, such as just compat --impls zds --areas oauth.