Operating
Once the server runs, you’ll want to know what it tells you, how it stops and starts again, and what it needs from the machine. The rules are in Operations, The store: what is kept, the backfill window, and the bounds and Limits.
The log
Section titled “The log”The server writes its log to standard output as one JSON object per line.
There’s a line for every request it answers, and lines for events such as
a start, a recovery or a shutdown. It never logs passwords, tokens, codes,
emails or keys. Standard error is only for why pds serve exits 1, for
example when a variable doesn’t parse.
PDS_LOG_LEVEL sets the lowest level written: debug, info (the
default), warn or error, or off for none
(Operations).
Set PDS_LOG_DIR to keep each line in that directory too. Then you can
read it back without reaching the machine, and it outlives a rebuilt
container. Its files take at most PDS_LOG_KEEP_BYTES (64 MiB by
default), and the oldest lines go first. Keep it apart from
PDS_DATA_DIR, on its own volume, since no backup needs it
(Operations).
Reading the log and stats
Section titled “Reading the log and stats”When PDS_ADMIN_PASSWORD is set, the admin can read what the server
logged and counted over its own port. Use GET /xrpc/_logs and
GET /xrpc/_stats with Authorization: Basic and the user admin, the
CLI’s pds admin logs and pds admin stats, or the MCP tools
admin_logs and admin_stats
(Operations):
export PDS_URL=https://pds.example.com PDS_ADMIN_PASSWORD=…pds admin logs # the last 100 lines keptpds admin logs --level warn --limit 20 # the last warnings and errorspds admin logs --trace 4bf92f3577b34da6a3ce929d0e0e4736 # one requestpds admin logs --method com.atproto.repo.createRecord --min-ms 500pds admin stats # requests by route, since the start_logs answers with the last limit lines (100 by default, at most 500)
kept in PDS_LOG_DIR that match every filter given, oldest first. Without
PDS_LOG_DIR it answers 404 LogNotKept. _stats answers with what the
metrics have counted since the server started, as JSON. That’s requests by
route and status with their mean times, the requests it made, its syncs,
its accounts and its firehose. It doesn’t need PDS_METRICS_PORT.
Health
Section titled “Health”GET /xrpc/_health answers 200 with {"version": "..."} while the
server serves, and it isn’t rate limited. Point a load balancer or a
container’s health check at it
(Operations).
Metrics
Section titled “Metrics”Set PDS_METRICS_PORT to serve Prometheus metrics at /metrics on
127.0.0.1 and that port. It has no auth, and it’s never on the server’s
own port. When it’s unset, nothing listens. A scrape doesn’t count as a
request to the server, so it gets no log line, no span and no rate limit
(Operations).
Traces
Section titled “Traces”Set OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT)
and the server exports a span for each request, and spans within it, by
OTLP over HTTP with protobuf. The other OTEL_* variables set headers,
compression, the service name and batching. A receiver that’s down doesn’t
slow any request (Operations).
Stopping
Section titled “Stopping”On SIGTERM or SIGINT the server stops taking connections and gives the
requests under way up to 10 seconds to finish. Then it closes the
firehose’s subscriptions, syncs what it has written, exports its spans for
up to 5 seconds, logs shutdown, and exits 0
(Operations).
If the server is killed or the machine loses power, no answered write is
lost. The next start recovers and logs recovered
(Durability). After a power cut, a write
that was never answered may turn up whole with nothing kept of its event.
That event is never sent. Instead, the next time its account is used, the
firehose has the events an activation makes (its #sync among them),
once (The store: what is kept, the backfill window, and the bounds).
Syncing writes
Section titled “Syncing writes”A write is on disk before the server answers it, and writes waiting at the
same time share a sync. PDS_SYNC picks how:
together(the default): every write goes where its account keeps it, and also to one shared journal inPDS_DATA_DIR/journal/. The write waits for the journal’s sync, so one sync answers every write waiting when it began, whatever accounts they’re for. In the background, the server makes what the journal holds durable where it’s kept, two seconds after the last write or once the journal holds 16 MiB, and then lets it go. A start after a kill or a power cut restores what’s missing from the journal before it serves, and a shutdown leaves none of it. Its files take a few bytes when the server is quiet, and up to twice what was written in the last seconds when it’s busy.each: each account’s writes are made durable by their own sync, which only the writes waiting on that account share.
Any other value makes the server exit 1 as it starts, with pds: PDS_SYNC: and the reason. The log’s start line shows which one it is,
as sync. Either is safe, and a server started with one carries on from a
data directory the other left. together answers more writes a second
when many accounts are written at once, such as on a disk whose sync takes
a millisecond or more
(Durability).
Upgrades
Section titled “Upgrades”Stop the server, put the new binary or image in place, and start it on the same data directory. The server starts on a data directory the last release wrote, whether it was shut down or killed, and carries on from it. Its spaces carry on too when 0.2.0 or a later release wrote it; the spaces 0.1.0 kept are let go on the first start. It may refuse a data directory a later version wrote and exit 1, but it never serves part of one (The store: what is kept, the backfill window, and the bounds). If a release from before the firehose’s index wrote the data directory, the first start reads every account’s files once for the earlier events, and takes that much longer to serve. So back up the data directory before an upgrade. Once a later version has written to it, the earlier one may not start on it.
The data directory stays within three times what its accounts hold, plus a constant for each account, since the server compacts it in the background (The store: what is kept, the backfill window, and the bounds).
When free space on the filesystem of PDS_DATA_DIR drops below
PDS_MIN_FREE_BYTES (256 MiB by default), the server refuses writes with
507 InsufficientStorage but keeps serving reads
(Limits). Watch the free space, and grow
the volume before it gets there.
Memory
Section titled “Memory”The server’s resident memory stays within its base, plus
PDS_CACHE_BYTES (256 MiB by default), plus 4 KiB for each account, no
matter what the accounts hold. Set PDS_CACHE_BYTES to what the machine
can spare (The store: what is kept, the backfill window, and the bounds).
Open files
Section titled “Open files”When it starts, the server raises its file descriptor limit to the hard limit it was given. It keeps at most a quarter of them open as accounts’ logs and a quarter as their tables, each capped at 4096. If an account’s files aren’t among them, the server opens them again the next time it’s read or written. It serves every account under a hard limit of 256, and a limit of 16384 or more lets it hold the most it will (Limits).
Backups
Section titled “Backups”Deploying explains how to back up the server, and what a start on a backup does.