Operations
What the server tells whoever runs it, and how fast it must be: its log, its health, its metrics and its traces, what its admin reads of them over HTTP, and the performance it is held to beside the reference. What the server serves, and how it is checked, is in The PDS: accounts and their records.
Of this, the reference shows only /xrpc/_health, and the harness holds the
server to it. The reference’s log is its own (pino’s), and it serves no
metrics or traces, so the rest is this server’s design, following the
ordinary conventions: a log of JSON lines, Prometheus’ text format for
metrics, and OpenTelemetry for traces (OTLP, W3C Trace Context, its HTTP
semantic conventions and its standard OTEL_* variables). The harness holds
the server to this page.
Environment
Section titled “Environment”Besides those the other pages list:
| Variable | Description | Default |
|---|---|---|
PDS_LOG_LEVEL | the least level of log line written: debug, info, warn or error; or off, for none | info |
PDS_LOG_DIR | a directory the log is kept in too, for the admin to read back (“Kept”, below); made if missing; unset, the log is kept nowhere but standard output | unset |
PDS_LOG_KEEP_BYTES | the most bytes the files in PDS_LOG_DIR take together; at least 65536 | 67108864 (64 MiB) |
PDS_METRICS_PORT | the port /metrics is served on, on 127.0.0.1; unset, metrics are not served | unset |
OTEL_EXPORTER_OTLP_ENDPOINT | the base URL of the OTLP receiver traces go to: they are posted to it with v1/traces appended, after a / if it does not end in one | unset |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | the URL traces are posted to, as it is; before OTEL_EXPORTER_OTLP_ENDPOINT | unset |
OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL | http/protobuf, the only protocol the server speaks | http/protobuf |
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS | headers sent with each export, key=value pairs joined by ,, each value percent-decoded; the second’s pairs win | none |
OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSION | gzip or none | none |
OTEL_SERVICE_NAME | the service.name of the server’s spans | unknown_service:pds |
OTEL_SDK_DISABLED | true turns traces off, whatever else is set | false |
OTEL_BSP_SCHEDULE_DELAY | milliseconds between exports | 5000 |
OTEL_BSP_MAX_QUEUE_SIZE | spans held for export at most; past it, new spans are dropped | 2048 |
OTEL_BSP_MAX_EXPORT_BATCH_SIZE | spans in one export at most | 512 |
OTEL_BSP_EXPORT_TIMEOUT | milliseconds an export may take, retries and all, before its spans are dropped | 30000 |
A value that does not parse, a level not listed, a protocol other than
http/protobuf (as grpc or http/json), a PDS_METRICS_PORT equal to
PDS_PORT, a PDS_LOG_KEEP_BYTES below 65536, or a PDS_LOG_DIR that
cannot be made or written to, makes pds serve exit 1 before it writes
anything to standard output, with why on standard error, as Limits
says of its variables. An empty variable is as if unset.
The log
Section titled “The log”The server writes its log to standard output: one JSON object per
line, UTF-8, each ended by \n; and with PDS_LOG_DIR set, keeps each line
in that directory too (“Kept”, below). Standard error is for one thing
only, why pds serve exits 1 (Commands: every method as an MCP tool and a CLI command, Durability,
Limits): a server that runs writes nothing there.
Note: standard output is the stream a container runtime or a supervisor collects; standard error is kept for the human reading why the server would not start.
Each line is written whole, by one write, so lines of requests answered at
once never mix. A line is not held in the process: once written, it
survives the server being killed with SIGKILL.
Every line
Section titled “Every line”| Field | Description |
|---|---|
time | when the line was written, RFC 3339 in UTC with milliseconds: 2026-11-09T17:23:00.123Z |
level | debug, info, warn or error |
msg | what happened, a fixed text: one of the events below, or a short text of the server’s own that holds no value from a request |
and the event’s own fields. A field with no value is left out, never
null. Lines below PDS_LOG_LEVEL are not written; the levels go debug,
info, warn, error, and off writes none.
Events
Section titled “Events”msg | level | Fields | When |
|---|---|---|---|
start | info | version, port, sync, and metrics_port when it is set | the first line of each start, once the environment is read and before the data is |
recovered | warn | dropped_bytes | a start on a data directory the last server did not shut down: killed, crashed, or the machine lost power |
request | info, or error for a status of 500 or more | method, status, ms, trace_id, and did when known; of a getBlob, repo, cid and bytes | each request answered |
plc failure | warn | error, status when the directory answered, and did when the call concerns one | each call to the PLC directory that fails: no answer, a timeout, or a status other than 2xx |
blobstore failure | warn | op: put, get or delete; error, status when the store answered, and did | each request to the bucket that fails (Blobs): no answer, a timeout, or a status other than 2xx |
trace export failed | warn | error | an export whose spans are dropped (Export, below), at most one line a minute while exports keep failing |
store failure | error | error | what the server does to its data in the background and could not (The store: what is kept, the backfill window, and the bounds, Durability): a compaction; a checkpoint of the journal; or a sync of the journal, after which it answers no write until it is started again |
shutdown | info | signal: SIGTERM or SIGINT | the last line of a server shut down |
start:versionis the server’s version, as/xrpc/_healthanswers it;portandmetrics_portare numbers;syncistogetheroreach, how waiting writes are made durable (Durability,PDS_SYNC).recovered:dropped_bytesis a number, the bytes of writes cut short that recovery dropped from the end of the logs the server keeps its state in (Durability, The store: what is kept, the backfill window, and the bounds): a torn tail, and the logs of generations after one cut short or not sealed; 0 when it dropped none. A start on an empty directory, or on one the last server shut down, writes norecovered. The server may choose how it knows, as by a mark it makes on shutdown and takes away as it starts.request: one line for each request the server answers, written once its answer is whole and before the last byte of it is sent, so that a client that has the whole answer knows the line is written. A response streamed, such asgetRepo’s, has its line once its body is made. An upgrade to a WebSocket (subscribeRepos) is answered by its101: the line is written then, not when the subscription ends. A request whose client leaves before the answer is sent whole still has its line.method: the request’s route, as the metrics label it (below);status: the HTTP status, a number;ms: a number, the milliseconds from reading the request’s head to writing the line, with any fraction;trace_id: the request’s trace, 32 lowercase hex digits (Traces, below), whether or not traces are exported;did: the DID of the account whose credentials the request carried and the server verified, as a bearer token it gave out; left out otherwise;repoandcid: ofcom.atproto.sync.getBlobalone, the DID of the account and the CID of the blob asked for, when the account holds that blob and the server answers with it,200, or failed to read it,500(Blobs, “getBlob”, “The objects”); left out of agetBlobanswered otherwise, and of every other line.didstays the caller’s, so agetBlobwith no token hasrepoand nodid;bytes: withrepoandcid, when the answer is200: a number, the bytes of the blob the server had handed to the connection by the time it wrote the line, which it writes as it hands on the last of them. That is the blob’s size when it was sent whole, and fewer when the client left first or the store’s read failed after its first byte;0forHEAD. It counts what the server handed on, not what reached the client: a client that left may have had fewer.
plc failure:errorsays why, astimeoutor the status;statusis a number.shutdown: on SIGTERM or SIGINT the server stops taking connections, lets the requests it has finish, for up to 10 seconds, closes the firehose’s subscriptions, syncs what it has written, exports the spans it holds, for up to 5 seconds, writes this line, and exits 0.
Besides these the server may write lines of its own, as when a crawler does
not answer or a write fails, at warn or error, with fields of their own,
and at debug whatever helps.
What is never logged
Section titled “What is never logged”No line holds, in any field, whole or in part:
- a password, an account’s or an app password; the admin password;
- a token: an access or refresh JWT, a delegation token or space credential
(Spaces: credentials), an OAuth code or token, a DPoP proof, or any
Authorizationheader; - an invite code, or a code sent by email, as an OAuth sign-in’s
emailOtp; - a cookie’s value, as a device’s or its session’s id (OAuth: the authorization server);
- a key: the PLC rotation key, an account’s signing or KEM key, the key the server signs its JWTs with;
- the values of
OTEL_EXPORTER_OTLP_HEADERS; - an account’s email;
- a request’s body, query string or headers, or an answer’s body; but for
getBlob’srepoandcid(Events, above): the DID of an account here and the CID of a blob it holds, as public as the blob.
method is a route the server serves, never the path asked, so nothing
else of a URL is logged.
With PDS_LOG_DIR set, each line written to standard output is written to
a file in PDS_LOG_DIR too, the same bytes, by one write of its own,
just after its write to standard output and before the next line is
written to either. So a line, once on standard output, is kept as well,
and survives the server being killed with SIGKILL as it does, but for the
one line a server killed between the two writes was writing: it is on
standard output alone. Lines below PDS_LOG_LEVEL are written to
neither. A write to the directory that fails is not tried again, and the
server serves on: standard output is the log, and the files a copy of it
to read back.
The files in PDS_LOG_DIR take at most PDS_LOG_KEEP_BYTES bytes
together, at every moment. Past that the oldest lines are let go first, as
many at once as the server likes, but never one of the newest lines whose
bytes, with their \n, add up to no more than half of PDS_LOG_KEEP_BYTES:
those are always kept. A line whose bytes, with its \n, are more than
half of it is not kept at all.
A server started on a PDS_LOG_DIR that holds lines carries on after
them, so the kept lines run across starts, oldest first. How the files are
named and laid out is the server’s own; only their total size, and the
lines _logs reads back, are seen. One server writes to a PDS_LOG_DIR at
a time.
Note: PDS_LOG_DIR is apart from PDS_DATA_DIR: the lines are no state
of the server’s, no backup needs them, and the data directory’s size is
held to its bounds without them (The store: what is kept, the backfill window, and the bounds, “Bounds”). A container’s
standard output is lost with the container, as when a deploy makes it
again; a volume of their own keeps the lines across it.
Health
Section titled “Health”GET /xrpc/_health answers as the reference does: 200 with
Content-Type: application/json; charset=utf-8 and
{"version": "0.0.0"}version being the server’s version, that of the pds package. Its query,
if any, is ignored. HEAD answers the same status and headers with no body;
POST is 400 InvalidRequest. It is answered while the server serves, and
is in no rate limit: past global (Limits) it is still 200, with no
RateLimit-* headers. Its route is _health.
The root
Section titled “The root”GET / says what the server is, as the reference’s does, and how to
connect an AI app to it (Commands: every method as an MCP tool and a CLI command, OAuth for MCP clients). It answers 200
with Content-Type: text/plain; charset=utf-8 and this text, where
<origin> is the origin of PDS_PUBLIC_URL, every line ending in \n,
the first line empty, and none with spaces at its end:
__ __ /\ \__ /\ \__ __ \ \ ,_\ _____ _ __ ___\ \ ,_\ ___ /'__'\ \ \ \/ /\ '__'\/\''__\/ __'\ \ \/ / __'\ /\ \L\.\_\ \ \_\ \ \L\ \ \ \//\ \L\ \ \ \_/\ \L\ \ \ \__/.\_\\ \__\\ \ ,__/\ \_\\ \____/\ \__\ \____/ \/__/\/_/ \/__/ \ \ \/ \/_/ \/___/ \/__/\/___/ \ \_\ \/_/
This is an AT Protocol Personal Data Server (aka, an atproto PDS)
Most API routes are under /xrpc/
Connect an AI app
1. Add this address to your AI app as a connector:
<origin>/mcp
In Claude: Settings > Connectors > Add custom connector, and choose "Sign in when needed".
2. When it asks, sign in here and approve it.
3. Ask it things like:
What have I posted this week? Post "hello from Claude" to Bluesky Who have I followed lately? Set my Bluesky bio to "Software engineer in Seattle"
Code: https://knotless.dev/chadtmiller.com/monolith Protocol: https://atproto.comThe banner is the reference’s; the rest is this server’s own, and only the
status and Content-Type are compared. Its query, if any, is ignored.
HEAD answers the same status and headers with no body; any other method
is 404, as the reference’s. It is in no rate limit and carries no
RateLimit-* headers, as the reference’s does not. Its route is /.
Metrics
Section titled “Metrics”GET /metrics on 127.0.0.1:PDS_METRICS_PORT answers the server’s metrics,
200 with Content-Type: text/plain; version=0.0.4; charset=utf-8, in
Prometheus’ text exposition format, version 0.0.4. Any other request on
that port is 404. With PDS_METRICS_PORT unset nothing listens, and
/metrics on the server’s own port is 404 as any route it does not serve.
It is a port of its own, with no auth. Being apart, a scrape is no request of the server’s: it has no log line or span, is in no metric and in no rate limit.
Note: the server’s own port is the one put before the internet; a separate one, on the loopback, is never in front of it, so nothing there needs guarding, and a Prometheus scrape, which carries no credentials, needs none.
Each family has a # HELP line and a # TYPE line, then its samples;
every family below is there from the start, its series once they have a
value. Counters and histograms count from the start of the process, so they
start again from 0 when it does, as Prometheus expects. A histogram has a
_bucket sample for each bound below, cumulative, then le="+Inf", then
_sum and _count. Durations are in seconds.
| Metric | Type | Labels | Description |
|---|---|---|---|
pds_requests_total | counter | method, status | requests answered: one for each request line |
pds_request_duration_seconds | histogram | method | each request’s ms, in seconds |
pds_rate_limited_total | counter | method | requests refused by a rate limit (429) |
pds_firehose_seq | gauge | the seq of the firehose’s last event; 0 before any | |
pds_firehose_subscribers | gauge | subscribeRepos connections open | |
pds_accounts | gauge | the accounts the server holds, those listRepos lists | |
pds_log_bytes | gauge | the bytes of the logs the server keeps its state in, in PDS_DATA_DIR, all of them: every generation’s not yet folded away, and the checkpoints (The store: what is kept, the backfill window, and the bounds) | |
pds_fsync_duration_seconds | histogram | each sync of written data to the disk, every one the server makes: a log’s as a commit is answered and as it is opened, a sealed log’s, a directory’s, a table’s, a blob’s file, the firehose’s index, its keys and marks, and the whole data directory’s after a start that followed a kill | |
pds_outbound_requests_total | counter | target, outcome | each request the server makes |
pds_outbound_request_duration_seconds | histogram | target | each request the server makes, until its answer or failure |
pds_log_bytes grows by what each write puts in the logs, and falls only
as the server lets go of what they held: a log’s generations folded into
its checkpoint (an account’s, its table: The store: what is kept, the backfill window, and the bounds, “Bounds”), which a write
may start and which the server may run in the background at any time;
and the journal’s records, once what they hold is durable where it is
kept (Durability). It lets go of nothing but as it makes, deletes,
replaces or cuts short a file in PDS_DATA_DIR, and what it lets go of
leaves the gauge as that file changes, neither before nor after: a read
of the gauge begun once the change can be seen in the directory counts
none of it, and one ended before the change counts all of it, however
long the server takes over what it does next (a journal’s segment
deleted leaves the gauge as it goes, not once the directory is synced or
the other segments are deleted). So across a write during which no file
there was made, deleted, replaced or cut short, each only grown or left
as it was, the gauge grows, by however little; across one during which
one was, it may be less after than before.
The buckets:
- requests and outbound requests:
0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10(Prometheus’ own default); - syncs:
0.0001,0.00025,0.0005,0.001,0.0025,0.005,0.01,0.025,0.05,0.1,0.25,1.
The labels:
-
methodis the request’s route, the same as its log line’s:- a method the server serves, by its NSID, for
/xrpc/<nsid>, whatever its HTTP method; _health,_logsand_stats, for/xrpc/_healthand the admin’s reads (below),_oauthSessions,_oauthRevokeSession,_oauthDevicesand_oauthRevokeDevice, for the account’s OAuth routes (OAuth: the account's own sessions and devices), and_passkeyList,_passkeyAddand_passkeyRemove, for its passkey routes (Passkeys);- a route outside
/xrpc/the server serves, as its path:/(“The root”),/mcp(Commands: every method as an MCP tool and a CLI command),/.well-known/atproto-did(Identity), and OAuth’s (OAuth: the authorization server):/.well-known/oauth-protected-resource,/.well-known/oauth-authorization-server,/oauth/jwks,/oauth/par,/oauth/token,/oauth/revoke,/oauth/authorize,/oauth/authorize/redirect, and the authorization page’s/@atproto/oauth-provider/~api/sign-in,~api/consent,~api/reject,~api/sign-out,~api/passkey-optionsand~api/passkey-sign-in, the passkey page/oauth/passkeyand its~api/passkey-register(Passkeys), and/oauth/assets/{name}for every path under/oauth/assets/, whatever its name (OAuth: what the authorization pages show); the MCP issuer’s (OAuth for MCP clients):/.well-known/oauth-protected-resource/mcp,/.well-known/oauth-authorization-server/oauth/mcp,/oauth/mcp/authorize,/oauth/mcp/tokenand/oauth/mcp/revoke; a page that adds such a route adds it to this list; _unknownfor anything else: an NSID the server does not serve, a path under/xrpc/that is no NSID, any other path.
Note: so no label holds what a client chose, and the series are bounded by the routes served.
- a method the server serves, by its NSID, for
-
statusis the HTTP status, in decimal:"200". -
targetis where the request goes:plcfor the PLC directory,crawlerfor a crawler (Federation: what a relay reads),servicefor the AppView and the servicesatproto-proxynames (Proxying, service auth and preferences),did-webfor adid:webdocument,clientfor a client’s metadata document or key set (OAuth: web, native and confidential clients, Spaces: credentials),spacefor a space’s notification to another service (Spaces: write notifications, the writer set and syncers),lexiconfor a permission set’s record (OAuth: permission sets),handlefor a handle’s/.well-known/atproto-did(Identity), andblobstorefor the bucket blobs are kept in (Blobs);outcomeisokfor a 2xx answer anderrorfor anything else, no answer included.
The admin’s reads
Section titled “The admin’s reads”The admin, whoever has PDS_ADMIN_PASSWORD, reads what the server logged
and counted over the server’s own port, without reaching its machine:
GET /xrpc/_logs and GET /xrpc/_stats. They are the server’s own, as
/xrpc/_health is, of no lexicon and no NSID: so they are listed under no
page of The PDS: accounts and their records, and their MCP tools and CLI commands are given in
Commands: every method as an MCP tool and a CLI command, “The admin’s reads”.
Each takes the admin’s Authorization: Basic as Limits’s “The admin”
says, and checks it first: anything else is 401 AuthenticationRequired.
Then a method other than GET is 400 InvalidRequest; then a parameter
that does not parse, or is out of its range, is 400 InvalidRequest. A
parameter neither takes is ignored. Each answers 200 with Content-Type: application/json; charset=utf-8, costs a point of global (Limits),
and has its route, _logs or _stats, in its request line, its metrics
and its span, as any request.
The lines kept in PDS_LOG_DIR (“Kept”, above), oldest first:
{"lines": [{"time": "2026-11-09T17:23:00.123Z", "level": "info", "msg": "request", "method": "com.atproto.repo.createRecord", "status": 200, "ms": 3.2, "trace_id": "…", "did": "did:plc:…"}]}lines holds the last limit of the lines kept that match every
parameter given, each the JSON object the line is, as written. They are
the lines kept when the read began: its own request line is written once
its answer is made, so it is never among them, and is among those of the
next read. A line in the files that is not a JSON object, as one cut short
by a lost power, is skipped.
| Parameter | Matches a line |
|---|---|
trace | whose trace_id is this |
msg | whose msg is this |
method | whose method is this: a route, as com.atproto.repo.createRecord or /oauth/token |
level | at this level or above: debug, info, warn or error; any other value is 400 |
minMs | with an ms of this many milliseconds or more, an integer of 0 or more; a line with no ms does not match |
limit | how many, the last of those matched: an integer from 1 to 500, 100 if not given |
With PDS_LOG_DIR unset, _logs is 404 LogNotKept, once the auth and the
parameters are checked.
_stats
Section titled “_stats”What the server counted since it started, as its metrics count it (above),
whether or not PDS_METRICS_PORT is set:
{ "version": "0.1.0", "started": "2026-11-09T17:20:01.004Z", "accounts": 3, "logBytes": 1048576, "firehose": {"seq": 120, "subscribers": 1}, "routes": [ {"method": "com.atproto.repo.createRecord", "count": 40, "statuses": {"200": 38, "400": 2}, "rateLimited": 0, "meanMs": 3.1} ], "outbound": [{"target": "plc", "ok": 3, "error": 0, "meanMs": 41.7}], "fsync": {"count": 212, "meanMs": 0.4}}versionis the server’s, as/xrpc/_healthanswers it;startedis thetimeof this start’sstartline.accounts,logBytes, andfirehose’sseqandsubscribersarepds_accounts,pds_log_bytes,pds_firehose_seqandpds_firehose_subscribers.routeshas one entry for each route with a request answered, the mostcountfirst, then bymethod:countis itspds_request_duration_seconds_count,statusesitspds_requests_totalby status, each status as a string,rateLimiteditspds_rate_limited_total, 0 if none, andmeanMsitspds_request_duration_seconds_sumover itscount, in milliseconds.outboundhas one entry for eachtargetwith a request made, bytarget:okanderrorare itspds_outbound_requests_totalof eachoutcome, 0 if none, andmeanMsitspds_outbound_request_duration_secondssum over its count, in milliseconds.fsyncispds_fsync_duration_seconds’ count, and its sum over its count in milliseconds, or 0 with no sync.
Every number is as the metrics are when the answer is made: the _stats
request is counted once answered, so not in its own answer. meanMs may
have any fraction.
Traces
Section titled “Traces”With OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
set, and OTEL_SDK_DISABLED not true, the server exports a span for each
request it answers, and spans within it, by OTLP over HTTP with protobuf
bodies: a POST of an ExportTraceServiceRequest with Content-Type: application/x-protobuf, gzipped with Content-Encoding: gzip if the
compression is gzip. With neither set, no span is exported, but each
request still has its trace id.
Note: protobuf because it is OTLP/HTTP’s default: an operator who sets only the endpoint, as the OpenTelemetry specification lets them, gets what every collector takes.
The request’s trace
Section titled “The request’s trace”A request’s traceparent header (W3C Trace Context) names its trace: a
request with a valid one is in that trace, its span the child of the span
it names. A header that is not valid is ignored, as if absent: valid is
00-<trace id, 32 lowercase hex>-<parent id, 16 lowercase hex>-<flags, 2 lowercase hex>, neither id all zeros; a version other than 00 and not
ff is read for those fields alike, and what follows them ignored. A
request with no valid one starts a trace of its own, with a new random
trace id. tracestate is not kept.
Spans are sampled as OpenTelemetry’s parentbased_always_on: a request
whose traceparent has its sampled flag (01) clear exports no span, nor
any span within it; every other request’s spans are exported.
Each span has the resource attributes service.name, as
OTEL_SERVICE_NAME, and service.version, the server’s version, and the
instrumentation scope pds.
-
The request: one span, of kind
SERVER, from reading the request’s head to writing its log line, named<HTTP method> <http.route>, or the HTTP method alone for a route of_unknown, asGET /xrpc/com.atproto.repo.getRecord. Its attributes, of OpenTelemetry’s semantic conventions for an HTTP server:Attribute Description http.request.methodthe HTTP method, as GET; one the server does not know,_OTHERurl.paththe request’s path as it came, percent-encoding and all, without its query url.schemehttphttp.route/xrpc/<nsid>for a method served,/xrpc/_health,/xrpc/_logs,/xrpc/_stats,/xrpc/_oauthSessionsand the account’s other OAuth and passkey routes, or the route’s path, as/mcp; none for_unknownhttp.response.status_codethe status, an integer error.typethe status, as a string, when it is 500 or more Its status is
ERRORwhen the status is 500 or more, and unset otherwise. -
A sync: a span of kind
INTERNALnamedfsync, the child of the request’s, for each time it waits on its writes reaching the disk. Every request whose answer says a commit was made (commitin its body) has at least one. -
A request the server makes, to any target above: a span of kind
CLIENTnamed with its HTTP method, asPOST, withhttp.request.method,server.address,server.port,url.full(without its query, fragment or credentials), andhttp.response.status_codewhen it was answered,error.type(the status as a string, ortimeout) when it failed, and statusERRORthen. One made for a request is the child of the request’s span; one made on its own, as a crawl or settling a handle change after a start, is the root of a trace of its own. Each such request carries atraceparentnaming this span, with the trace’s sampled flag.
No attribute holds a query, a header, a body or anything the log may not.
Export
Section titled “Export”Spans are exported in the background, in batches, as the OTEL_BSP_*
variables say. No request waits on an export: a receiver that refuses
connections, or takes them and never answers, changes no answer and no
request’s time. A full queue drops new spans; an export that fails may be
tried again until OTEL_BSP_EXPORT_TIMEOUT has passed since its first try,
and its spans are then dropped. Once the receiver answers again, the next
export goes to it, within OTEL_BSP_EXPORT_TIMEOUT.
Performance
Section titled “Performance”harness/perf.mjs runs one workload against the reference and against the
server, each in a process of its own (the reference in a Node process of
its own, with its PLC directory, which the server uses too), with the rate
limits off. The workload is in parts, run in order:
| Part | What | Requests | At once |
|---|---|---|---|
createAccount | new accounts, each with a password | 40 | 4 |
createRecord | records of about 100 bytes, spread over the accounts | 2000 | 16 |
getRecord | records written, chosen at random | 4000 | 16 |
getRepo | an account’s whole repository, of 50 records | 200 | 8 |
firehose | 20 subscribers to subscribeRepos from now, each from an address of its own, then writes one at a time; each time is from sending the write to a subscriber receiving its #commit | 200 writes, 4000 deliveries | 1 write |
uploadBlob | 64 KiB of random bytes, application/octet-stream | 200 | 8 |
Each part is run in three rounds of a third of its requests, the sides
taking turns: the reference’s first round, the server’s, the reference’s
second, and so on. A round’s throughput is its requests, or for firehose
its deliveries, a second over the round’s whole time. Each side is judged
by its best round: the lowest 50th and 99th percentile of a round’s times,
and the highest throughput.
Note: other work on the machine only ever makes a round slower, so the best of three is the nearest to what the side can do, and the turns spread a disturbance over both sides.
A part’s ratios are the server’s best over the reference’s, in the same run: its p99 ratio, of the best 99th percentiles, and its throughput ratio, of the best throughputs.
Note: the two sides take turns on the same machine, so a slower or busier machine slows both; the ratios keep what is the server’s own and drop most of what is the machine’s, which times in milliseconds would not.
harness/perf-baseline.json is the baseline: each part’s ratios as
runs on the runner measured them, the nightly’s or the perf workflow’s
(the same run, asked for alone), each part’s the weakest of at least five
runs (the highest p99 ratio and the lowest throughput ratio), with the runs
named in it. The run prints each part’s ratios beside the baseline’s, and
all of them on one line as the baseline holds them. A pull that makes the
server faster raises the baseline, from what those runs print; one that
lowers it says why. A part the baseline does not name is held to the
reference alone, as if its ratios were 1.
Note: the weakest of several runs, rather than one, so that one lucky run does not make the baseline more than the server does on most nights; on a laptop with other work on it, three of nine runs fell short of the weakest of the others by a little, so five is the least.
The server must, for every part:
-
have a p99 ratio no more than 1.5 times the baseline’s, or than 1 where the baseline’s is above 1: never a best 99th percentile more than 1.5 times the reference’s, and no more than 1.5 times the baseline’s where the server is faster.
Note: on a shared machine, the 99th percentiles of one side’s three rounds of a part spread by 1.05 to 1.4 times where the part is bound by the server’s own work (
uploadBlob,firehose,createAccount), and further where other work lands on a round; the best of three takes out most of that, the weakest of several runs in the baseline more, and 1.5 the rest, so a server no slower than it was passes while one twice as slow fails. -
and a throughput ratio at least 0.8 times the baseline’s, or than 1 where the baseline’s is below 1.
Note: where a part is bound by the same work on both sides (a password hash, the directory, a write’s flush, the firehose’s fan-out of writes made one at a time), the two sides’ best throughputs differ from run to run by a few percent either way; a floor of 1.0 turns that noise into failures of a server that is level, while 0.8 still fails one that is slower. Where the server is many times faster (
getRecord,getRepo), its ratio moves further from run to run, which the weakest of several runs takes in.
Then, against the server alone, subscribers come and go: cohorts of 20
subscribers to subscribeRepos from now, each from an address of its own,
the same 20 addresses each time (Limits: 16 at most from one). In a
cohort, all 20 connect, one putRecord is made, each subscriber receives
its #commit within 10 s, and all 20 close and see their connections
closed within 10 s. One cohort warms the server, and 20 more follow. The
server process’s open file descriptors, threads and memory are read 15 s
after the first cohort, and after the last: on Linux from /proc/<pid>/fd
and /proc/<pid>/status (its memory resident and swapped out, VmRSS and
VmSwap), on macOS from lsof, ps and footprint (its footprint,
resident and compressed). After the last, they are read again every
quarter second, for up to 20 s, taking the least of each reading, until
all three are within these bounds. The server must, after the last
cohort, hold:
-
no more open file descriptors than after the first;
-
no more threads than after the first;
Note: the server’s idle threads end 10 s after their last work, and their memory goes back with them: read at once after the first cohort, the threads the parts before used are still there, and as many threads held for good would be hidden. A cohort starts some 20 again, and the up to 20 s after the last lets them end. A thread or two the server starts and ends on its own may come some seconds later; what is held for good is in every reading, and such a thread in only some, so the least of each leaves it out. Read so, the descriptors and threads after the last cohort come back to those after the first, or fewer, so neither has any allowance.
-
and no more memory than after the first and 8 MiB.
Note: the allocator keeps some of what was freed for a while. On macOS, once the idle threads had ended, the server’s resident memory after the last cohort was 0 to 6.1 MiB above the first’s in three runs (up to 13 MiB before they had), and its footprint 0.2 and 2.3 MiB above in two. 400 subscribers each holding 256 KiB for good would be 100 MiB more. Resident memory alone is not enough on macOS, which compresses pages no one touches, as memory held for good is not: it missed such a leak, which the footprint showed as 107 MiB more.
The reference is not held to this.
The run exits 1 if any of these fails, naming the part, or the descriptors, threads or memory.
How it is checked
Section titled “How it is checked”harness/checks/operations.mjs and harness/checks/logging.mjs, at the end
of every compare.mjs run, of any kind. The server is started with
PDS_METRICS_PORT set, PDS_LOG_DIR a directory of its own apart from
the data directory, PDS_LOG_KEEP_BYTES of 262144, so that the kept
lines are let go of in a run, its traces going to a stub receiver in the
harness (harness/lib/otlp.mjs) with a header in
OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, and
OTEL_BSP_SCHEDULE_DELAY of 200 ms and OTEL_BSP_EXPORT_TIMEOUT of 3 s.
Every request the harness sends the
server carries a traceparent of its own, one in ten unsampled, and is
noted with its answer and how long it took; and every password, token,
code, cookie, email and key the run handles is kept.
In a run of random steps, the receiver stops answering a third of the way through, is gone at half way, once an export could have timed out, and is back at two thirds; the steps go on. Then, at the end:
- Health:
GET,HEADandPOST /xrpc/_health, andGETwith a query, against the reference: the status,Content-Type, and the body’s form;/metricson the server’s own port,404on both; with--limits,_healthpastglobalon both. - The root:
GET,HEADandPOST /, andGETwith a query, against the reference: the status andContent-Type; the server’s text held to “The root” above, read from this page, with its own origin; and noRateLimit-*headers on it. - Routes: paths under
/xrpc/of no method served, one that is no NSID, and another path, each of route_unknown. - Trace context:
_healthwith notraceparent, ones not valid (no parse, a trace id of 0, versionff, upper case), an unsampled one, and one of version01with more after it: each line’strace_idis new, or the one sent; each span the root of a new trace, or the child of the one sent; none for the unsampled. - Metrics: an account and records of the harness’s own, then: one
more subscriber while it subscribes and
pds_firehose_seqthe last event’s;pds_accountsaslistReposlists;pds_log_bytesno more than the data directory holds, and grown across a record written while no file in the data directory was made, deleted, replaced or cut short, the directory read before the metrics and after them, around the write (a record written again, up to 20 in all, while one was); every family’s type, labels, label values and buckets;pds_requests_totalequal to therequestlines since the server started, by route and status, and no less than the harness’s requests; each route’s_countequal to its requests;pds_rate_limited_totalequal to the 429s; the syncs grown by the records; a PLC request for every account made;404for another path. - The admin’s reads:
_logsand_statswith noAuthorization, a bearer token, the useradminwith another password, and another user: each401 AuthenticationRequired;POST,400 InvalidRequest;_logswith alevelnot listed, aminMsof-1, alimitof0, of501and ofx: each400 InvalidRequest. Then_logswith no parameter, and with alimit, its lines the last that many written to standard output by the starts that kept them, before the read; by thetraceof each of the last requests the harness sent, that request’srequestline alone; bymsg,method,levelandminMs, each and together, the last lines of standard output that match, and none that does not._statsbetween two scrapes of/metrics, each number between the two scrapes’,startedthestartline’stime,routesandoutboundin their order, and the second scrape’s_statscount one more than the first’s. At every check, the files inPDS_LOG_DIRtake no more thanPDS_LOG_KEEP_BYTES, and_logsreads back the newest lines of half that, across a kill and a start. Started withPDS_LOG_DIRunset,_logsis404 LogNotKeptand_statsanswers. - Spans: once the server is shut down with SIGTERM, which it must
exit 0 on within 15 s: for every request the harness sent and had
answered, one
SERVERspan in its trace, the child of itstraceparent, with the name, attributes, status, resource and times the spec gives, or none if unsampled; anfsyncspan under each that made a commit; aCLIENTspan under eachcreateAccount, of its call for the DID (thePOSTof its genesis operation to the directory, or, for an account moving in with itsdid, theGETof the DID’s document, from the directory or, for adid:web, from its host, Moving an account between servers); a root span for every request line of a client that sent notraceparent; noSERVERspan without a line; every export a protobufPOST /v1/traceswith the header. Spans of requests near the outage or just before a kill may be lost, and those of an export the server says it dropped (Export, above): a span missing, with atrace export failedline from the request’s start untilOTEL_BSP_SCHEDULE_DELAY, twiceOTEL_BSP_EXPORT_TIMEOUTand 1.5 s after its answer; or with the last such line before its start within a minute and the timeout, and no export taken since, while exports kept failing unlogged. While the receiver hung, the requests’ median time is under a second, and atrace export failedline is written. - Starts: at
warn, with a PLC directory that answers 500 andOTEL_SERVICE_NAMEunset:createAccountis refused, the directory gets atraceparentin the request’s trace, aplc failureline names the status and the DID, itsCLIENTspan is an error ofunknown_service:pds, and an outbound error is counted. Killed, then started with no OTLP endpoint: no span. Where the machine has an IPv4 address on an interface other than the loopback, the server started withPDS_BIND_ADDRESSunset does not answer_healthon it, and started with0.0.0.0answers on it and on127.0.0.1; where the machine has IPv6’s loopback, started with::1it answers on[::1]and not on127.0.0.1(The PDS: accounts and their records). ThenPDS_LOG_LEVEL=loud, the protocolsgrpcandhttp/json, aPDS_METRICS_PORTthat is no number and one equal toPDS_PORT, aPDS_LOG_KEEP_BYTESof65535and ofmany, aPDS_LOG_DIRunder a file, aPDS_BIND_ADDRESSoflocalhostand of[::1], and one of203.0.113.1, an address no interface of the machine has: each exits 1 with why on standard error. - The log, whole: every line one JSON object with the fields its event
has, at no level below the start’s
PDS_LOG_LEVEL; each start’s lines starting withstart, withrecoveredafter a start the last server did not shut down and not otherwise, and ending withshutdownafter a SIGTERM; onerequestline for each request the harness had answered, but one whose answer was cut short once its server was signalled to stop, found by its trace id, with its route and status, anmsno longer than the harness waited, and thedidof its token for a method that acts as one; of eachgetBlobanswered200or500,repoandcidthedidandcidit asked for, and of one answered200,bytesthe bytes the harness read of it, or, of one whose answer ended short (Blobs, “How it is checked”), no fewer than it read and fewer than itsContent-Length; norepo,cidorbyteson any otherrequestline; nothing on standard error but from a start that exited 1; and none of the run’s secrets, nor the text of a record the harness wrote, anywhere in standard output, standard error or the files inPDS_LOG_DIR.
harness/perf.mjs is a gate of its own, run alone after every other.