Skip to content

Compatibility with the reference PDS

harness/compat.mjs checks how closely a PDS behaves like the reference (@atproto/pds, the version harness/package-lock.json pins), area by area. It runs the core of harness/compare.mjs’s checks against any implementation that has an adapter in harness/lib/adapters/. It sends the same random requests to the server and to a reference running beside it, and compares their answers the way compare.mjs does. Unlike compare.mjs, it keeps going past a difference. Each area records which of its checks agree and which differ, plus the first difference in full. Compatibility results is a run against Monolith, zds, vlpds, pds.js and tranquil.

just compat # from the repository's root: everything below, in one

Or run it by hand:

cd harness
node bench.mjs --setup # once: Monolith's binary, and the other implementations' images (bench/README.md)
node compat.mjs --steps 200 --json ../compat/results.json > ../compat/tables.md
node compat.mjs --report ../compat/results.json # the tables again

--impls monolith,zds and --areas oauth,records run only some implementations or areas. --seed draws other requests, and --jobs sets how many runs go at once (3). just compat passes these flags on (just compat --impls zds --areas oauth), and a flag you give there wins over the recipe’s. The tables go to standard output, and RESULTS.md holds them along with what they show.

Monolith’s own gates are separate. just gates runs compare.mjs with every check, and it stops at the first difference.

compare.mjs’s checks fall into three kinds. Core checks compare the server with the reference, or hold both to the same rule, and they run against every implementation. Not yet checks compare with the reference too, but in areas the first cut leaves out (email, invites, rate limits, the admin lifecycle, proxying). Monolith’s own checks hold Monolith to its own pages where the reference can’t say, or to the reference’s implementation rather than the protocol. They run only against Monolith, in its gates.

check (compare.mjs step)modulekindarea
checkOAuthMetadataoauthcore (but see below)oauth
oauthLogin, oauthParRefused, oauthDpopRefused, oauthTokenRefused, oauthRevokeRefused, oauthRefresh, oauthRevoke, oauthGetSession, oauthClientLibraryoauthcoreoauth
oauthScopedCall, oauthScopedRefused, oauthScopedEdgesoauth-scopescoreoauth scopes
oauthSetPar, oauthSetLogin, oauthSetCall (the sets are published on a third PDS, which is the lexicon authority each server is given)permission-setscore, where an implementation can be given a lexicon authorityoauth permission sets
oauthClientDocument, oauthClientCache, oauthClientLogin, oauthAssertionRefused, oauthConfidentialSessionoauth-clientscoreoauth clients
oauthAuthorizeRefused (the page, its API and its redirect endpoint, each with one thing wrong)oauthMonolith’s own (the reference’s page API)
oauthDeviceCookies (device cookies, renewed at sign-in, with 30 s of grace)oauthMonolith’s own (the reference’s page API)
oauthSilentSignIn (a remembered device is sent back at once)oauth-clientsMonolith’s own (the reference’s page API)
oauthDepartures, oauthClientDepartures (spec/OAUTH.md’s and spec/OAUTH-CLIENTS.md’s departures, held to the spec)oauth, oauth-clientsMonolith’s own
oauthAccountAuth, oauthAccountSessions, oauthAccountDevices, and with --commands, oauthAccountCommands (spec/OAUTH-ACCOUNT.md’s routes, which the reference doesn’t serve)oauth-accountMonolith’s own
passkeyAuth, passkeyAdding, passkeySigningIn, passkeySignInRefused, passkeyRemoving, passkeyStates, and with --commands, passkeyCommands (spec/PASSKEYS.md’s passkeys, which the reference doesn’t have)passkeysMonolith’s own
createSession, getSession, describeServer, refreshSession, deleteSession, wrongTokens, appPasswords, sessionRaceaccounts, session, sessionscoresessions
createAccount, createAccountRefused, accountRaceaccountscoreaccounts
createRecord, putRecord, deleteRecord, applyWrites, getRecord, listRecords, describeRepo, createWithTid, concurrent, oddRequests, crossWritesrecords, odd, accountscorerecords
validatedWrite, duplicates, validatedBatch, validatedElsewherevalidationcorevalidation
checkRepos (getRepo verified by the reference’s library, the MST root, since, getLatestCommit)repositorycorerepository
listRepos, getRepoStatus, getBlocks, getGoneBlocks, syncGetRecord, noCrawlsYet, checkCrawlsfederationcoresync
--firehose: startFirehose, checkFirehose, checkReplayfirehosecorefirehose
updateHandle, updateHandleRefused, updateHandleNoToken, resolveHandle, wellKnownHandle, recommendedCredentials, handleRace, describeRepoidentity, recordscoreidentity
uploadBlob, blobRecord, getBlob, listBlobsblobscoreblobs
requestEmailConfirmation, confirmEmail, requestEmailUpdate, updateEmail, emailFactor, passwordReset, appPasswordEmailemailnot yet (email)
inviteCodes, --inviteslimitsnot yet (invites)
bigBatch, bodyCaps, appPasswordScopes, --limitslimits, oauth-limitsnot yet (limits, though their exact numbers are Monolith’s own)
loginTiming, oauthSignInTiminglimitsMonolith’s own (timing floors)
--lifecyclelifecyclenot yet (the admin lifecycle)
forwarded, refused, serviceAuth, chatCase, preferences, readAfterWrite, foreignRecord, feedAndPushproxynot yet (proxying)
--commands: checkCommands, withoutAccount, calls through MCP tools and CLI commandscommands, viaMonolith’s own (spec/COMMANDS.md)
outage, checkOperations (logs, traces, metrics, health, lifecycle restarts)operations, loggingMonolith’s own (spec/OPERATIONS.md)
--store, --scalestore, scaleMonolith’s own (spec/STORE.md)
--crash-every: crash (a server killed and started again on its data)crashescore, but not in this first cut. It needs the server killed and restarted, which the adapters can do (server.kill, server.start), and it’s left for later

Some parts of core checks are Monolith’s own, and they hold only Monolith (cand.adapter.own in the code):

partwherewhy
describeServer’s spacesAlpha: truechecks/session.mjsspec/SPACES.md, not the reference
the atproto_kem key required in the genesis operation and in getRecommendedDidCredentialschecks/accounts.mjs, checks/identity.mjsspec/SPACES.md (another implementation may have one, see below)
a blob at exactly the 5 MiB limit and one byte past it, plus describeServer’s blobUploadLimitchecks/blobs.mjsthe exact number of a limit
PAR bodies of exactly 100 KiB and one byte past itchecks/oauth.mjsthe exact number of a limit
OAuth access tokens’ expires_in of 899 to 900checks/oauth.mjsspec/OAUTH.md’s departure (below)
the authorization server metadata’s request_parameter_supported: falsechecks/oauth.mjsspec/OAUTH.md, held to the spec
a GET of the page API’s sign-in, and a POST to an endpoint it doesn’t havechecks/oauth.mjsthe reference’s page API

Each area runs its own steps on a fresh pair of servers, with two or three accounts made first. It draws the steps by weight, the way compare.mjs draws its own. The repository, sync and firehose areas need writes to have something to check. So createRecord, putRecord, deleteRecord and applyWrites are drawn among their steps as supporting steps (the firehose also draws createAccount and updateHandle), and making the first accounts is a supporting step in every area. A supporting step’s difference is reported separately and isn’t counted in the area. The repository’s checkRepos and the firehose’s checks run every --check-every steps (10) and at the end. Sync’s crawl checks run with the first accounts, every 10 steps and at the end.

A server that’s built to host one account uses that account in every area (Compatibility: any PDS against the reference, “One account”). Its adapter’s claim makes the account, and the reference makes its own beside it with createAccount, both with the password the server was started with. The checks that make an account (the accounts area, the firehose’s createAccount) or need a second one (crossWrites, handleRace, and updateHandle’s refusal of a handle another account has) don’t run. Every other check runs on the one account.

A check is one step. It keeps going past a difference (in a compatibility run, differ in lib/report.mjs keeps the difference instead of ending the run), and it counts as failed if anything in it differed. A check that throws (on an answer of a shape it can’t read) or takes more than 3 minutes counts as one that couldn’t go on. For each check, an area records how many ran and failed, each thing that differed and how often, and its first difference. That’s the request and both answers, printed the way compare.mjs prints them.

harness/lib/adapters/ has one adapter per implementation, each a module named in its index.mjs. common.mjs holds what they share (the server as a process or a container, a rotation key, a browser’s way through a form). compare.mjs --impl <name> picks one (Monolith by default). An adapter says how to make the server for the run’s setup, and what that server is:

  • start(o) makes the server for the setup o (port, data directory, PLC directory, the reference’s recovery key, crawlers, AppView, the web of OAuth client documents, rate limits and invites on or off). It answers with the process the harness starts, kills and starts again, the URL the harness calls, the URL the server’s DID documents name, and its OAuth issuer if that’s a different one. It also gives the rotation keys it lists in each DID and its own key as a signer, its admin’s Authorization, and takeMail(), which gives the emails it sent. A server that hosts one account also answers with its password and claim, which makes that account.
  • unsupported lists each setting it can’t be given, with the reason. The checks that depend on one are skipped, and the report says why.
  • pageApi, or else signIn (below).
  • own says whether Monolith’s own rules hold it. Only Monolith’s does.

The checks don’t read any of this. When a check needs something from the server beyond what the protocol says, it asks harness/lib/server.mjs. That offers own() (whether Monolith’s own rules hold it), can(setting) (whether it can be given a setting unsupported names), pageApi() and issuer(side). For Monolith, each one answers so the check does exactly what it does without asking. Anything only a compatibility run needs, such as making a one-account server’s account, lives in harness/lib/areas.mjs (the compatibility run itself) instead of checks/.

Monolith’s adapter runs it the way the harness always has. It runs the command after -- with the PDS_* environment that compare.mjs’s header lists, reads its mail from PDS_DEV_MAIL_DIR, and kills and restarts it as a process.

zds, vlpds, pds.js and tranquil run in Docker, from the images bench.mjs --setup builds and pulls (pinned in bench/impls.json). They’re configured the way the benchmark configures them (harness/lib/impls.mjs, shared with bench.mjs), plus whatever a compatibility run needs. pds.js, which the benchmark doesn’t run, is configured the way its container runs by default. The harness reaches each one on a loopback port, and each one reaches the harness’s PLC directory, AppView and crawlers through host.docker.internal. The server is the container. The harness starts it, kills it with a signal, and follows its log with docker logs.

pds.js hosts one account. Its adapter starts it with a fresh account password (PDS_PASSWORD, which it requires) and claims the account the way its onboarding page does. That’s a same-origin POST /account/onboarding/create with that password, the handle and the email, then createSession by the new DID. pds.js names its endpoint https://<PDS_HOSTNAME> in the DID documents it makes, and its OAuth issuer is the URL it’s called at. So the adapter gives https://localhost as its public URL and the loopback URL as its issuer, and each is taken out of answers the way a server’s URL is.

tranquil runs on a fresh PostgreSQL of its own, on a Docker network with it, as the benchmark runs it. It names itself https://<PDS_HOSTNAME> everywhere: its OAuth issuer, the endpoint in its DID documents, and the htu its DPoP proofs must name. So the adapter gives it the hostname localhost and https://localhost as its public URL, which is its issuer too, while the harness reaches it on a loopback port: the checks sign each proof’s htu with the issuer and send the request to the port. Its one operator key, PLC_ROTATION_KEY, is a public did:key, listed first in each DID it makes, so it takes the reference’s recovery key; its other rotation key is each account’s own signing key, which it holds and signs with, so the harness has none of its own.

settingMonolithzdsvlpdspds.jstranquil
PLC directoryPDS_PLC_URLZDS_PLC_DIRECTORYVLPDS_PLC_URL, VLPDS_PLC_MODE=directoryPLC_URLPLC_DIRECTORY_URL
rotation key, the harness’sPDS_PLC_ROTATION_KEY_HEXZDS_PLC_ROTATION_KEYVLPDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEXno: only the account’s signing key, which the server holdsno: each account’s own signing key, which the server holds
the reference’s recovery keyPDS_RECOVERY_DID_KEYZDS_RECOVERY_DID_KEYVLPDS_PLC_RECOVERY_DID_KEYno: the samePLC_ROTATION_KEY, its one operator key
server DID did:web:localhostPDS_HOSTNAMEZDS_SERVER_DIDVLPDS_SERVICE_DIDPDS_HOSTNAME until it has its account, then that account’s DIDPDS_HOSTNAME
hostname localhost (requestCrawl’s)PDS_HOSTNAMEno: it comes from ZDS_PUBLIC_URL, which must be the URL the harness calls (DPoP’s htu)no: the same, and it doesn’t ask a relay on another host to crawl when its own host is localno: it tells one relay (RELAY_URL) at most every 20 minutesno: with the hostname localhost it asks no crawler to crawl
handle domains .test, .examplePDS_HANDLE_DOMAINSZDS_HANDLE_DOMAINS.test only: VLPDS_HANDLE_DOMAIN=test (another domain needs two labels)no: its hostname is its only onePDS_USER_HANDLE_DOMAINS=test,example, which it names with no leading dot
accounts, more than oneyesyesyesno: one, made by claimyes
admin password admin-passPDS_ADMIN_PASSWORDZDS_ADMIN_TOKENno: its admin token must be 32 bytes or moreno: it has no admin methodsno: an admin is an account marked so in its database
invites off (or on)PDS_INVITE_REQUIREDZDS_INVITE_REQUIREDVLPDS_INVITE_REQUIREDoff, since it has noneINVITE_CODE_REQUIRED
rate limits off (or on)PDS_RATE_LIMITS_ENABLEDno general limits to turn on, and full getRepo exports are raised to 8 at onceVLPDS_NO_RATE_LIMITSno limits to turn onDISABLE_RATE_LIMITING
the harness’s AppViewPDS_BSKY_APP_VIEW_*ZDS_PROXY_SERVICE_URL, _DIDVLPDS_APPVIEWAPPVIEW_URL, APPVIEW_DIDno setting: it proxies by the atproto-proxy header, resolving the DID that names
the harness’s crawlersPDS_CRAWLERSZDS_CRAWLERSnot given (see hostname)not given (see hostname)not given (see hostname)
OAuth client documents from the harness’s webPDS_OAUTH_CLIENT_FETCHno: it fetches the client_id’s own https URLno: the same (http only with --dev-mode)no: it fetches the client_id’s own URLno: it fetches the client_id’s own URL
no SSRF guard, the way the reference runsPDS_DISABLE_SSRF_PROTECTIONZDS_ALLOW_PRIVATE_FETCHits default for operator-set URLsits defaultALLOW_PRIVATE_FETCH
mailPDS_DEV_MAIL_DIRZDS_MAIL_PROVIDER=outbox, read with dev.zat.admin.listMailno: it’s readable only with --dev-modeno: its container has no transportno: it sends only to an SMTP relay and keeps none to read back
reached at the address it namesyesyesyesyesno: it names itself https://localhost, reached on a loopback port
consent to a narrower scopethe page APIno: there’s no consent stepno: its form only takes scopes awayno: its form approves or denies the scope asked forno: its consent takes the scopes approved, but the hook carries none (below)

Each “no” skips these checks:

  • the hostname: sync’s crawl checks
  • the handle domains: describeServer
  • client documents: the whole oauth clients area
  • a narrower scope: the narrowing in oauthLogin (the scope asked for is granted)
  • more than one account: those above
  • the harness’s rotation key: its own PLC operation before an updateHandle
  • the recovery key: recommendedCredentials
  • the address it names: oauthClientLibrary, as the atproto client library refuses an https address on a loopback host

No core check needs mail, the admin password or rate limits.

The harness signs in on the reference’s authorization page through its page API (/@atproto/oauth-provider/~api/sign-in, consent, reject, with its CSRF cookie and device cookies). Monolith serves that API too, so with Monolith every step of a sign-in is compared. No other server serves it, and the protocol leaves the sign-in step to each server. So an adapter without pageApi has a hook:

signIn(account, authorizeUrl, consent)
// account: {identifier, password}; authorizeUrl: the page a PAR's request_uri names;
// consent: {approve}. Answers the answer that sends the browser back to the client,
// with its code: a redirect, or a form_post page.

zds’s hook posts its password form (POST /oauth/authorize: request_uri, username, password, or deny), and zds grants at once with a 303 to the client. vlpds’s hook loads the page (its device cookie and csrf field), posts /oauth/authorize/sign-in (identifier, password), then posts /oauth/authorize/consent (did, action=allow or deny). That redirects to the client or answers with a form_post page. pds.js’s hook posts its consent form (POST /oauth/authorize: request_uri, client_id, password, action=approve or deny). The form doesn’t ask for an identifier, since the server has one account, and it redirects to the client with a 302. tranquil’s page is a script calling its JSON endpoints, and its hook calls them as the script does: it posts request_uri, username and password to /oauth/authorize, which answers the next page as redirect_uri. That’s the consent page, whose scopes /oauth/authorize/consent answers and then takes back approved, or at once tranquil’s /oauth/authorize/redirect, which redirects to the client with a 303. To refuse, it posts request_uri to /oauth/authorize/deny, which answers the client’s URL with error=access_denied as JSON, and the script navigates there; the hook answers that as a 303 there. Its consent can approve fewer scopes than were asked, but the hook is given no scope to approve, so it approves the scope asked. None of them needs a browser.

When an adapter has a hook, the reference still signs in through its page API, but its page’s answers are its own and aren’t compared. The server signs in through its hook. What’s compared is the protocol around the sign-in: PAR, the redirect to the client (its target, code, state and iss, or error), the token exchange with PKCE, DPoP and its nonces, refresh, revocation, getSession, every scoped XRPC call with the session’s tokens, and the client documents’ rules. oauthClientLibrary signs in through the hook too, and compares only the pages’ redirect status.

Compatibility: any PDS against the reference lists what they’re allowed, section by section, with the reason for each:

  • the same things are taken out of every answer as for Monolith
  • the KEM key is optional (held to Monolith’s form if it’s published)
  • an OAuth access token may last 15 or 60 minutes
  • only three of describeServer’s fields are compared
  • the exact numbers of limits aren’t checked
  • sign-in happens on the server’s own page, and it’s compared from the redirect on
  • Monolith’s own departures aren’t held against them
  • a server that hosts one account uses that one
  • a server with password rules of its own gets each area’s first accounts with a password its adapter makes to fit them
  • a server named by an https address it isn’t reached at has the requests for that address sent where it is reached
  • the settings an adapter can’t give have their checks skipped and reported

The harness allows nothing else, and Monolith is held to all of spec/PDS.md.

spec/PDS.md and the area pages list “what the reference does that the specs leave open” (the order a write’s checks refuse in, the 500s, and the rest). Those aren’t allowed to differ. Another implementation that does otherwise differs from the reference, and RESULTS.md says where a difference is one of those.

The vlpds documentation the report cites as deliberate

Section titled “The vlpds documentation the report cites as deliberate”
  • listRepos’ order (DESIGN.md:5905): “listRepos order is (slot, DID), because listings are served per shard, not by creation order.”
  • getBlob of an upload that no record names (DESIGN.md:5906-5907): “Fresh uploads are readable. Blobs are written under their final key at upload, so getBlob serves an upload before any record references it, until the blob GC collects it.” Also docs/blobs.md:65-66: “Unlike the reference PDS, vlpds serves a blob through getBlob as soon as it’s uploaded, before any record names it.”

harness/checks/ is what Monolith’s gates run against the reference. Adding an implementation mustn’t change it for Monolith, not by one request and not by one draw from the seeded source. So an implementation brings these, in order of preference:

  1. An adapter, plus the settings it can’t be given and why (their checks are skipped).
  2. A capability in spec/COMPAT.md, where it differs from what the harness assumes in a way the protocol allows (one account, its own TLS). The capability is answered in lib/server.mjs or handled in lib/areas.mjs. It works the same for every implementation that needs it, and leaves Monolith unchanged.
  3. Nothing else. A quirk of its own that no protocol rule allows is a difference or a skipped check. A check never gets bent around it.

No check names an implementation. A change of the second kind has to show that it leaves Monolith’s runs as they were. compare.mjs --tally <file> writes how many requests of each kind a run made, and that’s the same for two runs of one seed. So you compare a few seeds’ tallies from before and after the change, and they come out the same.

Here are the steps:

  1. Pin it in bench/impls.json and give it a benchmark adapter (bench/README.md), unless it can’t serve the benchmark’s workload (like pds.js, which hosts one account). Put its Docker configuration in harness/lib/impls.mjs.
  2. Give it an adapter, a module of its own in harness/lib/adapters/ named in its index.mjs. The adapter has start, port, info, unsupported with reasons, signIn unless it serves the reference’s page API, and readyMs if it starts slowly. For a server that hosts one account, it also lists accounts in unsupported and gets password and claim from start. For a server that refuses passwords by a rule of its own, it has passwordFor(p) (spec/COMPAT.md, “Password rules”). For a server that names itself by an https origin it isn’t reached at, start answers that origin as publicUrl and fronted: true (spec/COMPAT.md, “Its own address”).
  3. Add its name to IMPLS in harness/compat.mjs, and add the differences its documentation says are on purpose to DELIBERATE there.
  4. Run node compat.mjs --impls <name> --steps 60 to try it, then do a full run.