Benchmark: the reference PDS, Monolith, vlpds, zds and tranquil
harness/bench.mjs, run with just bench (node bench.mjs --setup && node bench.mjs --rounds 3 --json ../bench/results.json), from a clean state (see
The PDS benchmark). There are three rounds per implementation, one
implementation at a time, and every one runs in Docker the same way. Each
round is a fresh container on a fresh volume, registering with a fresh
in-memory PLC directory (@did-plc/server) run by the bench. Every round
warms the server the same way first: each part and route at a tenth of its
size, on accounts of its own. It then runs the parts, the routes, the curves
by the number of callers, the mix and the subscribers coming and going. A
storage round of each follows. README.md, “What it measures”, says what each
of those is. The raw numbers of every round, and every server’s recorded
configuration, are in bench/results.json.
Machine and versions
Section titled “Machine and versions”- Apple M1 Pro, 8 cores, 32 GiB, macOS 26.5.2 (arm64), APFS on the internal SSD. Docker Desktop 28.0.1, its Linux VM given 8 CPUs and 7.7 GiB. Every server, and the client’s ceiling responder, runs in that VM.
- The client: Node v24.22.1, 4 worker threads (
harness/lib/loadpool.mjs). - The load on the machine was not light (see “Caveats”). The 1-minute load
average was 3.7 when the run started; at each round’s start and end it
was:
- Monolith 6 to 13;
- the reference 9 to 10;
- vlpds 6 to 29;
- zds 13 to 16;
- tranquil 8 to 22.
- Pins, as
bench/impls.jsonhas them andresults.jsonrecords:- Monolith: this repository’s
Dockerfile(Rust 1.98 on Debian trixie, as deployed), built from42abe3c. That is the last commit to change what the image is built from, and the commit the run was at. Imagesha256:044e78d7…, linux/arm64. - reference:
@atproto/pds0.5.55 from npm, started through@atproto/dev-env0.6.26’sTestPdsasperf.mjsdoes, pinned byharness/package-lock.json. It runs in the imagebench/docker/reference.Dockerfilebuilds (Node 24.22.1 on Debian trixie),sha256:fd421a48…, linux/arm64. - vlpds: release v1.3.0 of
jazware/vlpds, commit91fa452993865b2f91ff5ed5b87fc28356796d0c, its release imageghcr.io/jazware/vlpds@sha256:e1aca7934017bb757ef3717bbd964ae758cf704dc623703262c512fd462c0477(linux/arm64 here). - RustFS 1.0.4, vlpds’s S3 store:
rustfs/rustfs@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2. It passedvlpds-bucket-probe(from the vlpds image): all six correctness checks,SAFE for vlpds. - zds: release v0.7.8 of
tangled.org/zat.dev/zds, commitcef2c63d7d6d7dd87b90100566307e9aa8a4d170, built by--setupwith its ownDockerfile(Zig 0.17.0,-Doptimize=ReleaseSafe) intomonolith-bench-zds:cef2c63, imagesha256:124944cb…, linux/arm64. Its published image (atcr.io/zat.dev/zds:latest) is amd64 only and predates v0.7.8, so it was not used. - tranquil: release v0.8.0 of
tangled.org/tranquil.farm/tranquil-pds, commitce17e8a4948aa216c6c1056122966a11b8a693c8, built by--setupwith its ownDockerfile(Rust 1.99.1 on Debian trixie, the binary UPX-compressed, on distroless) intomonolith-bench-tranquil:ce17e8a, imagesha256:a6a27312…, linux/arm64. Its published image (atcr.io/tranquil.farm/tranquil-pds) needs a login to pull, so it was not used. - PostgreSQL 18.7, tranquil’s database:
postgres@sha256:1018733457011bf88d81db98cf43aac6d5150afc18787dbe204771fedf80ff95(postgres:18-alpine, as tranquil’s compose files run it).
- Monolith: this repository’s
How Monolith was configured
Section titled “How Monolith was configured”As its image runs it (docs/guide/deploying.md): PDS_PORT=2583,
PDS_BIND_ADDRESS=::, PDS_DATA_DIR=/data, with /data a fresh volume each
round. The bench adds only what a local benchmark needs:
PDS_HOSTNAME=localhostandPDS_PUBLIC_URL=http://127.0.0.1:<port>;- handle domains
.test,.example; - the bench’s PLC directory (
http://host.docker.internal:<port>), with a fresh secp256k1 rotation key each round; - rate limits off and invites not required;
- every private range trusted as a proxy, so that subscribers’ addresses in
X-Forwarded-Forcount.
It is also given an AppView, PDS_BSKY_APP_VIEW_URL=http://127.0.0.1:9 with
did:web:api.bsky.app, as every server on the network has one. Without it,
getPreferences is not served (spec/PROXY.md). Nothing in the workload is
forwarded there. Its DID cache is as it comes (PDS_DID_CACHE_SECS, an hour).
In the storage round only, PDS_REPO_BACKFILL_LIMIT_MS=10000.
How the reference was configured
Section titled “How the reference was configured”As perf.mjs runs it: dev-env’s TestPds with rateLimitsEnabled: false,
inviteRequired: false and hostname: localhost, its dataDirectory and
blobstoreDiskLocation under /data, a fresh volume each round
(harness/bench-reference.mjs). It listens inside its container on the port
it is reached at, and registers DIDs with the bench’s PLC directory through
host.docker.internal. In the storage round only, repoBackfillLimitMs
10000.
How vlpds was configured
Section titled “How vlpds was configured”As its docs run a single node: the release image’s tiny profile
(deploy/ansible/roles/vlpds/defaults/main.yml at v1.3.0, the profile
docs/operations/deploy.md describes for one node): a 2560 MiB container
memory limit (--memory 2560m --memory-swap 2560m, which sizes its caches),
--stop-timeout 90, VLPDS_NODE_ID=single, VLPDS_SHARDS=1,
VLPDS_LEASE_TTL_MS=60000, VLPDS_SLATEDB_MANIFEST_POLL=60s, a 4096 MiB SST
disk cache on a Docker volume, and tiny’s firehose, backfill and export sizes
(--firehose-ring-mb 64, live ring 32, merge queue 32, max lag 64, backfill
cache 32 / read-ahead 16, 4 backfills, 4 exports, MST node cache 64, 1
firehose thread). Threads are its defaults (I/O = the 8 cores the VM shows,
workers = 4). VLPDS_PLC_MODE=directory with a fresh secp256k1 rotation key,
and fresh random JWT secret, admin and internal tokens and KEK each round
(not --dev-mode). The bench’s own changes, as for the others: rate limits
off (VLPDS_NO_RATE_LIMITS), invites not required (its default), the bench’s
PLC directory (http://host.docker.internal:<port>), a loopback URL
(VLPDS_PUBLIC_URL=http://127.0.0.1:<port>), and nothing sent off the
machine (--crawlers= for no relays, --asn-lookup off). Each round runs on
a Docker network of its own with a fresh RustFS container and volume, and the
bucket made through the S3 API; vlpds’s container and RustFS’s are made before
the start clock and removed, with their volumes and network, after the round.
In the storage round only, VLPDS_LOG_RETENTION=10s and
VLPDS_LOG_RETENTION_INTERVAL=5s.
How zds was configured
Section titled “How zds was configured”As its deployment runs it (docs/deployment.md, fly.toml), on its
Dockerfile’s ENV (ZDS_HOST=0.0.0.0, ZDS_PORT=2583,
ZDS_DB=/data/zds.sqlite3) with ZDS_BLOBSTORE_PATH=/data/blobs, /data a
fresh Docker volume each round; a fresh secp256k1 ZDS_PLC_ROTATION_KEY,
ZDS_JWT_SECRET and ZDS_ADMIN_TOKEN each round; no memory limit (its
deployment’s 1 GiB is the Fly VM’s, not a zds setting). zds has no dev mode.
Its defaults kept: handle domains .test, invites not required, permissioned
data off, mail provider comail (createAccount sends no mail, and with no
Comail key nothing is sent). The bench’s own changes: its PLC directory
(ZDS_PLC_DIRECTORY=http://host.docker.internal:<port>), a loopback URL
(ZDS_PUBLIC_URL=http://127.0.0.1:<port>, from which it derives
did:web:127.0.0.1%3A<port>), no relays asked to crawl (ZDS_CRAWLERS=, in
place of bsky.network and vsky.network), and
ZDS_MAX_CONCURRENT_REPO_EXPORTS=8. zds has no general rate limits to turn
off; its cap on full getRepo exports at once (4 by default, a 429 past it)
is the one limit the workload hits, and it is raised to the 8 the workload
runs at once. Its container is made before the start clock and removed, with
its volume, after the round; zds is its container’s PID 1 with no SIGTERM
handler, so the bench kills it rather than wait out docker stop.
How tranquil was configured
Section titled “How tranquil was configured”As its container docs run it (docs/2_INSTALL_CONTAINERS.md,
docker-compose.prod.yaml): on PostgreSQL, its default repository backend,
with blobs as files under /var/lib/tranquil-pds/blobs (a fresh Docker
volume each round) and its in-process cache (ripple, one node); the image’s
SERVER_HOST=[::], SERVER_PORT=3000; PDS_HOSTNAME=tranquil.test, its
handles under that; fresh random JWT_SECRET, DPOP_SECRET and
MASTER_KEY each round. It signs PLC operations with each account’s own
key, so it has no rotation key to give. The bench’s own changes: rate limits
off (DISABLE_RATE_LIMITING), invites not required
(INVITE_CODE_REQUIRED=false), its PLC directory
(PLC_DIRECTORY_URL=http://host.docker.internal:<port>), private addresses
fetchable so that it reaches that directory (ALLOW_PRIVATE_FETCH), no
relays asked to crawl (CRAWLERS=,, a list of none), the gate that holds an
account’s logins and record writes until it verifies an email off
(DISABLE_ACCOUNT_VERIFICATION_GATE), and MAX_CONCURRENT_REPO_EXPORTS=8,
raised from 4 as zds’s is. With no sender address its email is off, so
createAccount sends no mail. Its passwords need an upper-case letter, so the
bench’s accounts’ have one. PostgreSQL runs at its defaults
(max_connections 100, shared_buffers 128 MB, fsync and
synchronous_commit on), as user tranquil_pds and database pds with a
fresh password. Each round runs on a Docker network of its own with a fresh
PostgreSQL container and volume, ready for connections before the start
clock; both containers are removed, with their volumes and network, after
the round. tranquil has no setting that makes a backfill window of 10 s, so
its storage round runs on its defaults.
Results
Section titled “Results”Machine and client
Section titled “Machine and client”- Apple M1 Pro, 8 cores, 32 GiB, macOS 26.5.2 (arm64); Node v24.22.1; Docker 28.0.1, its Linux VM 8 CPUs, 8218316800 bytes.
- Load average (1 min): 3.7 at the start, 13.3 at the end; 6.2 to 28.5 at the rounds’ starts and ends.
- The client: 4 worker threads, each with its own connections, kept alive. Its ceiling, a responder in Docker answering every request with 2 bytes and doing nothing else, requests per second (the best of three):
| at once | 1 | 4 | 8 | 16 | 32 | 64 | 256 |
|---|---|---|---|---|---|---|---|
| per s | 1876 | 5351 | 8093 | 13634 | 18122 | 23098 | 28688 |
Summary
Section titled “Summary”Each part’s requests per second (replay: events sent per second), the mix’s, start time and memory, each the best round; for the firehose’s live parts, the p50 of their lag, the best round. Detail follows.
| reference | monolith | vlpds | zds | tranquil | |
|---|---|---|---|---|---|
| createAccount per s | 33 | 63 | 39 | 6 | 9 |
| createRecord per s | 129 | 3507 | 582 | 445 | 316 |
| applyWrites per s | 71 | 2205 | 303 | 323 | 136 |
| getRecord per s | 1155 | 8258 ‡ | 4343 | 2662 | 2454 |
| listRecords per s | 548 | 4230 | 2531 | 1953 | 1608 |
| getRepo per s | 365 | 2954 | 1191 | 690 | 108 |
| uploadBlob per s | 154 | 1175 | 131 | 405 | 315 |
| createRecordSerial per s | 95 | 576 | 68 | 223 | 55 |
| firehose lag p50 ms | 1.0 | 0.5 | 2.4 | 14.0 | 3.7 |
| firehoseLoaded lag p50 ms | 11.5 | 4.3 | 4.6 | not served | 21.0 |
| replay per s | 11825 | 41549 | 20274 | 53781 | 19302 |
| mix per s | 375 | 4090 | 1331 | 614 ◊ | 736 |
| start ms | 3146 | 180 | 1365 | 399 | 1105 |
| idle RSS MiB (median) | 376.1 | 19.3 | 177.1 | 16.5 | 37.6 |
| peak RSS MiB | 828.4 | 152.4 | 501.2 | 54.5 | 138.2 |
‡: at least 50% of the client’s ceiling at the same concurrency (above): the client may be part of what limits it. ◊: the server closed each connection after its answer, the client opened one a request, and it ran out of local ports and waited for them (each closed connection holds its port for a while): the rate is held to how fast they came back, and the times are from when each request was sent.
createAccount
Section titled “createAccount”40 requests, 4 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | max ms (best) | max ms (median) |
|---|---|---|---|---|---|---|
| reference | 33 | 32 | 107.4 | 118.7 | 162.7 | 171.3 |
| monolith | 63 | 62 | 57.6 | 59.7 | 81.5 | 82.1 |
| vlpds | 39 | 34 | 101.8 | 114.4 | 147.8 | 175.3 |
| zds | 6 | 6 | 560.7 | 708.7 | 1444.9 | 1618.2 |
| tranquil | 9 | 8 | 431.6 | 459.5 | 487.3 | 628.8 |
createRecord
Section titled “createRecord”1000 requests, 16 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 129 | 127 | 121.9 | 123.6 | 160.5 | 174.2 |
| monolith | 3507 | 3365 | 3.8 | 3.9 | 20.5 | 20.9 |
| vlpds | 582 | 580 | 21.5 | 24.6 | 50.0 | 55.3 |
| zds | 445 | 440 | 29.6 | 29.9 | 99.0 | 102.0 |
| tranquil | 316 | 303 | 46.2 | 48.2 | 113.7 | 130.6 |
applyWrites
Section titled “applyWrites”200 requests, 8 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p95 ms (best) | p95 ms (median) |
|---|---|---|---|---|---|---|
| reference | 71 | 70 | 109.3 | 111.2 | 133.4 | 133.4 |
| monolith | 2205 | 1987 | 3.4 | 3.7 | 5.0 | 6.2 |
| vlpds | 303 | 283 | 21.8 | 25.3 | 42.4 | 44.5 |
| zds | 323 | 310 | 21.9 | 22.0 | 47.6 | 50.8 |
| tranquil | 136 | 127 | 55.3 | 57.3 | 81.6 | 90.6 |
getRecord
Section titled “getRecord”2000 requests, 16 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 1155 | 1140 | 12.7 | 12.7 | 31.9 | 32.0 |
| monolith | 8258 ‡ | 7860 | 1.6 | 1.7 | 5.9 | 6.8 |
| vlpds | 4343 | 4126 | 2.7 | 2.9 | 19.0 | 20.1 |
| zds | 2662 | 2252 | 4.6 | 5.1 | 21.9 | 27.5 |
| tranquil | 2454 | 2431 | 5.3 | 5.5 | 27.9 | 28.7 |
listRecords
Section titled “listRecords”1000 requests, 16 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 548 | 537 | 26.1 | 26.2 | 54.1 | 70.5 |
| monolith | 4230 | 4110 | 3.0 | 3.1 | 16.1 | 17.5 |
| vlpds | 2531 | 2391 | 4.8 | 4.9 | 23.1 | 30.1 |
| zds | 1953 | 1855 | 6.9 | 7.1 | 24.1 | 25.5 |
| tranquil | 1608 | 1406 | 8.7 | 9.8 | 33.2 | 36.3 |
getRepo
Section titled “getRepo”1000 requests, 8 at once.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 365 | 361 | 19.4 | 19.9 | 42.5 | 48.3 |
| monolith | 2954 | 2644 | 2.4 | 2.4 | 5.9 | 12.5 |
| vlpds | 1191 | 954 | 4.9 | 6.4 | 28.6 | 28.6 |
| zds | 690 | 637 | 10.0 | 10.3 | 31.6 | 32.5 |
| tranquil | 108 | 104 | 69.4 | 70.0 | 128.8 | 154.4 |
uploadBlob
Section titled “uploadBlob”1000 requests, 8 at once, of 64 KiB each; then, not timed, a record of each account referring to its first blob, for getBlob.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 154 | 152 | 48.1 | 48.9 | 90.4 | 93.7 |
| monolith | 1175 | 1136 | 5.3 | 5.4 | 21.0 | 21.5 |
| vlpds | 131 | 131 | 58.2 | 59.7 | 105.4 | 108.2 |
| zds | 405 | 378 | 16.0 | 17.4 | 53.2 | 57.8 |
| tranquil | 315 | 299 | 22.6 | 22.7 | 54.6 | 58.3 |
createRecordSerial
Section titled “createRecordSerial”1000 requests, one after another, as one person posting: no other write to share a sync with.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|---|---|
| reference | 95 | 93 | 9.2 | 9.4 | 28.6 | 29.6 |
| monolith | 576 | 572 | 1.4 | 1.5 | 6.0 | 6.4 |
| vlpds | 68 | 67 | 11.5 | 12.2 | 41.7 | 44.2 |
| zds | 223 | 214 | 3.6 | 3.6 | 20.4 | 22.1 |
| tranquil | 55 | 55 | 15.8 | 15.9 | 46.8 | 50.5 |
firehose
Section titled “firehose”100 putRecords one after another, 20 subscribers; each time is the firehose’s lag, from a write’s answer to a subscriber having its commit, 0 when the commit came first. No rate: the writes’ would be the write path’s, which the write parts measure.
| impl | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|
| reference | 1.0 | 1.1 | 6.3 | 15.0 |
| monolith | 0.5 | 0.5 | 2.9 | 3.0 |
| vlpds | 2.4 | 2.6 | 17.9 | 21.2 |
| zds | 14.0 | 15.3 | 62.3 | 66.5 |
| tranquil | 3.7 | 3.8 | 23.6 | 24.9 |
firehoseLoaded
Section titled “firehoseLoaded”400 putRecords, 16 at once, each to a different account, 20 subscribers; each time is the firehose’s lag, from a write’s answer to a subscriber having its commit, 0 when the commit came first. No rate: the writes’ would be the write path’s, which the write parts measure.
| impl | p50 ms (best) | p50 ms (median) | p99 ms (best) | p99 ms (median) |
|---|---|---|---|---|
| reference | 11.5 | 13.0 | 47.1 | 53.1 |
| monolith | 4.3 | 10.2 | 22.9 | 40.0 |
| vlpds | 4.6 | 4.8 | 23.1 | 25.5 |
| zds | not served: its live firehose skips commits when writes come at once: of a warm-up of 40 putRecords, 4 at once to 4 accounts, each of 20 subscribers was sent 34 to 37 (its log: subscribeRepos end start_cursor=466 end_cursor=506 frames=35), so no lag can be timed for the rest | |||
| tranquil | 21.0 | 22.6 | 62.3 | 75.0 |
replay
Section titled “replay”4 subscribers from cursor 0, as relays catching up: each is sent every event the server made in the round, warm-up and all, then a write made once all are connected. Throughput is the events sent to them all up to that write’s commit, per second; each time is a subscriber’s, from asking to having that commit.
| impl | per s (best) | per s (median) | p50 ms (best) | p50 ms (median) | max ms (best) | max ms (median) |
|---|---|---|---|---|---|---|
| reference | 11825 | 11639 | 1079.7 | 1096.9 | 1079.7 | 1097.0 |
| monolith | 41549 | 40703 | 276.7 | 308.1 | 307.3 | 313.7 |
| vlpds | 20274 | 17750 | 606.0 | 693.2 | 611.8 | 701.7 |
| zds | 53781 | 45960 | 197.9 | 237.5 | 204.7 | 239.5 |
| tranquil | 19302 | 15850 | 663.0 | 811.1 | 670.6 | 816.7 |
Routes
Section titled “Routes”The routes a client calls most, besides the parts above, each measured alone: requests per second, then the p50 and the tail of their times, each the best round. A route a server answered with no 2xx in the warm-up is not served, with its answer.
| route | n, at once | reference per s | p50 ms | tail ms | monolith per s | p50 ms | tail ms | vlpds per s | p50 ms | tail ms | zds per s | p50 ms | tail ms | tranquil per s | p50 ms | tail ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| describeServer | 1000, 16 | 4285 | 3.1 | 20.0 (p99) | 7939 ‡ | 1.6 | 5.6 (p99) | 4944 | 2.4 | 17.8 (p99) | 3991 | 3.4 | 18.1 (p99) | 4718 | 2.6 | 19.5 (p99) |
| oauthMetadata | 1000, 16 | 3474 | 4.2 | 9.1 (p99) | 9120 ‡ | 1.5 | 4.2 (p99) | 2835 | 4.2 | 21.4 (p99) | 3630 | 3.5 | 11.3 (p99) | 5621 | 2.3 | 6.4 (p99) |
| createSession | 200, 8 | 22 | 355.5 | 362.7 (p95) | 22 | 354.7 | 364.0 (p95) | 72 | 109.3 | 146.8 (p95) | 29 | 234.4 | 426.5 (p95) | 15 | 525.9 | 697.9 (p95) |
| getSession | 1000, 16 | 2214 | 6.7 | 23.5 (p99) | 8260 ‡ | 1.7 | 5.2 (p99) | 4243 | 3.0 | 19.8 (p99) | 1848 | 6.4 | 30.4 (p99) | 2128 | 6.6 | 25.2 (p99) |
| getPreferences | 1000, 16 | 1394 | 10.6 | 27.8 (p99) | 8349 ‡ | 1.6 | 6.3 (p99) | 5067 | 2.6 | 18.4 (p99) | 2741 | 4.5 | 20.2 (p99) | 1992 | 6.8 | 28.1 (p99) |
| getServiceAuth | 1000, 16 | 700 | 20.7 | 44.2 (p99) | 9081 ‡ | 1.6 | 3.5 (p99) | 3754 | 3.0 | 21.6 (p99) | 2530 | 5.4 | 22.4 (p99) | 2232 | 6.0 | 25.4 (p99) |
| resolveHandle | 1000, 16 | 3171 | 4.8 | 18.8 (p99) | 8309 ‡ | 1.5 | 13.8 (p99) | 2944 | 4.1 | 32.4 (p99) | 3323 | 4.0 | 12.3 (p99) | 5548 | 2.1 | 19.6 (p99) |
| describeRepo | 1000, 16 | 1317 | 11.1 | 29.6 (p99) | 8176 ‡ | 1.6 | 6.1 (p99) | 2516 | 5.0 | 28.1 (p99) | 2956 | 4.3 | 20.6 (p99) | 2587 | 5.1 | 36.1 (p99) |
| getLatestCommit | 1000, 16 | 1367 | 10.5 | 26.4 (p99) | 9889 ‡ | 1.5 | 3.0 (p99) | 3940 | 2.9 | 21.4 (p99) | 3663 | 3.5 | 21.3 (p99) | 3257 | 4.1 | 27.2 (p99) |
| syncGetRecord | 1000, 16 | 513 | 27.9 | 63.0 (p99) | 8563 ‡ | 1.7 | 3.7 (p99) | 3005 | 4.1 | 20.4 (p99) | 947 | 14.3 | 43.8 (p99) | 1456 | 9.3 | 32.0 (p99) |
| listRepos | 1000, 16 | 1118 | 12.6 | 29.1 (p99) | 5798 | 2.2 | 10.2 (p99) | 640 | 21.7 | 69.8 (p99) | 1416 | 9.3 | 34.7 (p99) | 328 | 44.3 | 86.1 (p99) |
| getBlob | 1000, 16 | 578 | 24.8 | 52.9 (p99) | 1733 | 8.2 | 25.7 (p99) | 795 | 16.6 | 55.4 (p99) | 1544 | 9.0 | 25.8 (p99) | 1136 | 12.4 | 34.9 (p99) |
| refreshSession | 1000, 16 | 1087 | 11.0 | 44.6 (p99) | 4639 | 2.8 | 16.8 (p99) | 620 | 20.8 | 64.1 (p99) | 368 ◊ | 34.6 | 134.8 (p99) | 671 | 21.3 | 48.7 (p99) |
‡: at least 50% of the client’s ceiling at the same concurrency (above): the client may be part of what limits it. ◊: the server closed each connection after its answer, the client opened one a request, and it ran out of local ports and waited for them (each closed connection holds its port for a while): the rate is held to how fast they came back, and the times are from when each request was sent.
By the number of callers
Section titled “By the number of callers”Each at 1, 4, 16, 64, 256 callers at once: requests answered with a 2xx per second, and the p50 and tail of their times, the best round; and how many were answered otherwise, in the median round, counted rather than stopped at.
getRecord
Section titled “getRecord”| callers | reference | monolith | vlpds | zds | tranquil |
|---|---|---|---|---|---|
| 1 | 473/s, p50 1.9, p99 6.1 ms | 1309 ‡/s, p50 0.7, p99 1.9 ms | 557/s, p50 1.2, p99 17.1 ms | 518/s, p50 1.6, p99 7.9 ms | 365/s, p50 2.2, p99 17.0 ms |
| 4 | 1120/s, p50 3.1, p99 16.7 ms | 3855 ‡/s, p50 0.9, p99 2.6 ms | 1493/s, p50 1.9, p99 19.2 ms | 1452 ◊/s, p50 2.3, p99 11.4 ms | 1113/s, p50 2.9, p99 20.3 ms |
| 16 | 1250/s, p50 11.6, p99 29.1 ms | 8742 ‡/s, p50 1.5, p99 4.6 ms | 3993/s, p50 3.0, p99 20.6 ms | 238 ◊/s, p50 4.2, p99 18.5 ms | 2592/s, p50 5.2, p99 26.8 ms |
| 64 | 1289/s, p50 42.6, p99 202.0 ms | 15886 ‡/s, p50 3.7, p99 11.5 ms | 7865/s, p50 6.2, p99 28.5 ms | 317 ◊/s, p50 11.7, p99 42.1 ms | 3785/s, p50 14.0, p99 62.1 ms |
| 256 | 1370/s, p50 76.6, p99 4127.2 ms | 18286 ‡/s, p50 8.8, p99 34.0 ms | 12314/s, p50 19.4, p99 40.6 ms | 923 ◊/s, p50 40.6, p99 220.5 ms | 4260/s, p50 51.4, p99 122.2 ms |
createRecord
Section titled “createRecord”| callers | reference | monolith | vlpds | zds | tranquil |
|---|---|---|---|---|---|
| 1 | 94/s, p50 9.6, p95 17.2 ms | 604/s, p50 1.4, p95 2.6 ms | 71/s, p50 11.3, p95 30.3 ms | 214/s, p50 3.6, p95 9.7 ms | 48/s, p50 18.4, p95 38.9 ms |
| 4 | 112/s, p50 33.3, p95 51.7 ms | 1543/s, p50 2.1, p95 3.8 ms | 164/s, p50 20.5, p95 45.0 ms | 354/s, p50 8.1, p95 28.6 ms | 130/s, p50 26.5, p95 51.4 ms |
| 16 | 115/s, p50 135.5, p95 161.1 ms | 3331/s, p50 4.0, p95 10.5 ms | 596/s, p50 22.4, p95 46.1 ms | 341/s, p50 34.5, p95 97.5 ms | 246/s, p50 60.4, p95 87.7 ms |
| 64 | 119/s, p50 533.1, p95 579.6 ms | 4025/s, p50 11.6, p95 39.5 ms | 1655/s, p50 31.7, p95 56.1 ms | 334/s, p50 111.7, p95 476.9 ms | 292/s, p50 185.6, p95 341.4 ms |
| 256 | 111/s, p50 2185.5, p99 2600.0 ms | 4133/s, p50 48.0, p99 103.7 ms | 3281/s, p50 65.3, p99 135.9 ms | 336/s, p50 523.1, p99 2001.6 ms | 291/s, p50 778.0, p99 1315.1 ms |
The mix
Section titled “The mix”10000 requests, 32 at once, of each 100 so many of each route, with 2 subscribers to the firehose: requests per second, then each route’s p95 in the mix beside its p95 measured alone (the parts’ and the routes’ tables), the best round, in ms.
| route | of 100 | reference p95 in mix | alone | monolith p95 in mix | alone | vlpds p95 in mix | alone | zds p95 in mix | alone | tranquil p95 in mix | alone |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all, per s | 375 | 4090 | 1331 | 614 ◊ | 736 | ||||||
| getSession | 15 | 54.1 | 12.4 | 12.8 | 3.4 | 16.9 | 7.1 | 97.2 | 18.6 | 22.8 | 13.9 |
| getPreferences | 10 | 67.7 | 17.0 | 13.2 | 3.5 | 13.5 | 5.8 | 81.3 | 11.2 | 26.7 | 13.8 |
| getRecord | 25 | 70.0 | 20.0 | 13.6 | 3.3 | 15.9 | 8.1 | 81.0 | 11.8 | 20.8 | 10.4 |
| listRecords | 15 | 108.3 | 47.3 | 14.2 | 7.2 | 22.3 | 12.8 | 83.3 | 15.4 | 23.9 | 17.1 |
| describeRepo | 5 | 68.7 | 16.9 | 13.5 | 3.5 | 19.4 | 12.5 | 65.5 | 11.4 | 12.1 | 10.6 |
| getBlob | 10 | 249.8 | 45.1 | 19.8 | 17.3 | 22.0 | 42.0 | 77.7 | 19.2 | 17.3 | 28.8 |
| getLatestCommit | 5 | 63.0 | 18.4 | 12.2 | 2.4 | 10.9 | 9.2 | 75.1 | 9.7 | 12.3 | 7.8 |
| createRecord | 10 | 149.0 | 149.5 | 23.5 | 8.3 | 104.2 | 42.9 | 121.1 | 75.6 | 365.5 | 83.8 |
| applyWrites | 3 | 194.4 | 133.4 | 24.1 | 5.0 | 102.1 | 42.4 | 118.8 | 47.6 | 331.4 | 81.6 |
| uploadBlob | 2 | 242.1 | 71.4 | 18.1 | 14.3 | 146.1 | 89.2 | 92.4 | 41.1 | 54.7 | 45.3 |
Subscribers coming and going
Section titled “Subscribers coming and going”11 cohorts of 20 subscribers, each from an address of its own: a cohort connects, has one write’s commit and goes. The server’s threads and RSS 2 s after the first cohort went and 2 s after the last, and how many of the subscribers it refused, the median round.
| impl | threads after the first | after the last | RSS MiB after the first | after the last | refused, of all |
|---|---|---|---|---|---|
| reference | 11 | 11 | 843.1 | 843.2 | 0 of 220 |
| monolith | 234 | 234 | 121.0 | 118.1 | 0 of 220 |
| vlpds | 30 | 30 | 505.0 | 504.5 | 0 of 220 |
| zds | 44 | 44 | 48.6 | 48.8 | 0 of 220 |
| tranquil | 13 | 13 | 93.4 | 94.1 | 0 of 220 |
Start time and memory
Section titled “Start time and memory”Start: from docker start of the server’s container, made already, to its first answered describeServer, on an empty volume. Idle RSS: 2 s after start, before any request. Peak RSS: the most sampled (every ~100 ms) through the warm-up and the whole round. Each the server’s own process in its container, from docker top.
| impl | start ms (best) | start ms (median) | idle RSS MiB (median) | peak RSS MiB (best) | peak RSS MiB (median) |
|---|---|---|---|---|---|
| reference | 3146 | 3193 | 376.1 | 828.4 | 843.6 |
| monolith | 180 | 257 | 19.3 | 152.4 | 188.8 |
| vlpds | 1365 | 1394 | 177.1 | 501.2 | 515.2 |
| zds | 399 | 434 | 16.5 | 54.5 | 54.6 |
| tranquil | 1105 | 1164 | 37.6 | 138.2 | 147.0 |
Not in the table above: what a server needs running beside it, its RSS sampled apart (the sum over its container’s processes, from docker top, as fast as that answers; for PostgreSQL, whose processes share its buffers, the sum of their PSS, which counts each shared page once).
| impl | beside it | idle MiB (median) | peak MiB (best) | peak MiB (median) |
|---|---|---|---|---|
| vlpds | RustFS | 172.0 | 435.4 | 487.4 |
| tranquil | PostgreSQL | 57.1 | 522.7 | 534.5 |
Storage
Section titled “Storage”From each implementation’s storage round, apart from the rounds above: a fresh server started with a backfill window W of 10 s where it has a setting for one, the same warm-up of the parts and the parts, whose speed is not kept (a short window changes what replay sends). What it keeps on disk: the sum of the sizes of the files on its container’s volume (its log goes to docker logs, not there). Under load: once the parts end, after replay. At rest: after the parts, no request for 2 W + 1 s (21 s), then one createRecord to each account the round made, the warm-up’s too, then the size sampled every 500 ms for 10 s, the least kept (spec/STORE.md, “Bounds”). Live state: of each of those accounts, its getRepo, the blobs listBlobs lists and the uploads no record references.
| impl | backfill window | data under load MiB (median) | data at rest MiB (median) | live state MiB (median) | at rest ÷ live (median) |
|---|---|---|---|---|---|
| reference | W = 10 s | 94.8 | 95.4 | 69.8 | 1.37 |
| monolith | W = 10 s | 102.6 | 70.8 | 69.8 | 1.01 |
| vlpds | W = 10 s | 1.6 | 1.7 | 69.8 | 0.02 |
| zds | its default: no setting | 94.6 | 95.3 | 69.8 | 1.37 |
| tranquil | its default: no setting | 62.5 | 62.5 | 69.9 | 0.89 |
zds: zds has no setting for it: none of the ZDS_* variables its binary reads is a backfill window or an event retention, so at rest is measured with what it keeps by default.
tranquil: its backfill window is FIREHOSE_BACKFILL_HOURS, in whole hours (72 by default), and it prunes the events past EVENT_RETENTION_MAX_AGE_SECS (7 days by default, and never less than that window) once a minute at most: nothing sets a window of W with a pass in the quiet, so at rest is measured with what it keeps by default.
Not in the table above: what a server keeps in what runs beside it (vlpds: its bucket, the files on RustFS’s volume; tranquil: its database, the files on PostgreSQL’s volume), measured the same way at the same moments; the last column counts the server’s own and this together.
| impl | beside it | under load MiB (median) | at rest MiB (median) | both at rest ÷ live (median) |
|---|---|---|---|---|
| vlpds | RustFS | 76.6 | 78.7 | 1.15 |
| tranquil | PostgreSQL | 120.6 | 121.0 | 2.63 |
Not run
Section titled “Not run”pds.js is pinned, but only the compatibility check runs it: it hosts one
account, and the workload makes 40. Cocoon, millipds and rsky-pds are not
pinned, and tranquil ran only on PostgreSQL, not on tranquil-store, its
embedded store. bench.mjs takes any of them as an adapter (README.md,
“Adding an implementation”).
Every server served every part, route and section, with one exception: zds did not serve the firehose under load, as its table says.
Caveats
Section titled “Caveats”- The machine was not idle for the whole run. The run waited until the 1-minute load average was at most half the cores (3.7 at its start). Other work on the machine then rose: rounds started and ended at load averages of 6 to 29, highest while vlpds and tranquil ran. Every server’s numbers are lower and noisier than an idle machine’s would be. Best-of-three is reported for that reason, and the medians beside each best show the spread.
- Every server ran in Docker Desktop’s Linux VM, behind its port forwarding. All five pay the same forwarding cost and see the same 8 CPUs, the same volume driver and the same path to the PLC directory. The forwarding has a cost of its own, which the client’s ceiling measures: about 1,900 requests a second one at a time, 13,600 at 16 at once, and 28,700 at 256 (the first table). A rate marked ‡ is at least half of that, and the client and the forwarding may be part of what holds it: Monolith’s getRecord at 16 at once, its fastest routes, and its getRecord curve at every number of callers.
- Durability: every server’s syncs land on the VM’s virtual disk.
Whether Docker Desktop passes them through to the Mac’s drive as a full
flush was not established. On Linux, Rust’s
sync_all/sync_data(Monolith), SQLite’sfsync(the reference, zds), PostgreSQL’s WAL flush (tranquil) and RustFS’sfsync(vlpds) are the same kind of call, so the five pay the same kind of sync. The write numbers are therefore far higher than a native run with macOS’sF_FULLFSYNC, for Monolith most of all (createRecord 3,507 a second here, against 369 natively withF_FULLFSYNC). They show each server’s write path under the same sync, not the cost of a sync through a drive’s cache. Blobs: the reference writes them withfs.writeFileand zds withfwrite, and neither syncs them; Monolith syncs a blob’s file when a commit first references it, not when it is uploaded (spec/DURABILITY.md, “Blobs”); tranquil syncs each to a temp file before renaming it into place; vlpds’s are PUT to RustFS, which syncs them. - zds closes every connection after its answer (
connection: close), so the client opens a connection a request. At a few hundred a second the client ran out of local ports, as each closed connection holds its port for a while. A request then waited for one and was sent again (README.md, “A connection a request”). That is marked ◊: zds’s getRecord curve at 4, 16, 64 and 256 callers, its refreshSession and its mix. Its rate there is held to how fast the client’s ports came back (238 a second at 16 callers, against 2,662 in the getRecord part at 16 at once), not to zds. A server’s many clients each have ports of their own. - zds’s mix is its single connection. zds keeps everything in one SQLite database on one connection. In the mix, every read’s p95 rose to 65–97 ms beside the writes, from 10–19 ms alone, where Monolith’s, vlpds’s and tranquil’s reads stayed within 11–27 ms.
- The reference’s writes do not climb with callers. Its createRecord stays at about 115 a second from 4 callers to 256, with p50 growing from 33 ms to 2.2 s. Its getRecord stays at about 1,300 a second from 16 callers on, with p99 4.1 s at 256.
- vlpds’s writes are batched across requests. Every write on the node goes into one log, group-committed into segments that are PUT to the bucket. Its createRecord climbs with callers (71 a second at 1, 3,281 at 256) as more writes share a segment, while one write at a time (createRecordSerial, 68 a second) pays a whole PUT.
- tranquil’s writes level off at about 300 a second. Its createRecord is 48 a second at 1 caller and 291 at 256, with p50 growing to 778 ms; in the mix its createRecord and applyWrites p95 rose to 331–366 ms, from 82–84 ms alone, while its reads stayed within 12–27 ms.
- zds’s firehose skips commits under concurrent writes. Each of the 20
subscribers of the warm-up of firehoseLoaded (40 putRecords, 4 at once,
each to a different account) was sent 34 to 37 of the 40 commits, by its
own log (
subscribeRepos end ... frames=), though the sequence numbers ran on by 40. Written one at a time, every commit reached every subscriber. So the part is not served for it: no lag can be timed for a commit not sent. - zds’s and tranquil’s limits raised. Their default cap of 4 full
getRepoexports at once answers the workload’s 8 at once with a refusal. It was raised to 8 (ZDS_MAX_CONCURRENT_REPO_EXPORTS,MAX_CONCURRENT_REPO_EXPORTS), as rate limits are off for the others. zds’s other caps are not reached: 32 subscribeRepos connections (no part holds more than 20) and 4 cursor backfills at once (replay’s 4). - tranquil’s gates off. Its default holds an account’s logins and record writes until it verifies an email or other channel; with no mail leaving the machine, none could, and the other servers have no such gate. It is off for the run, as invite codes are for all of them.
- replay’s events are each server’s own. Every server makes its own events for the same round, since an account’s creation is several events and not the same several everywhere. So replay compares events sent per second, not the time to send one fixed stream.
- Storage. vlpds keeps almost nothing on its own volume (its SST cache, 1.7 MiB): its data is in the bucket, 79 MiB at rest, counted apart. tranquil keeps only its blobs on its own volume (62.5 MiB); its database, 121 MiB at rest, is counted apart. zds and tranquil have no backfill window to set to 10 s, so their at-rest sizes include every event of the round. The live state is about the same for all five, 69.8 MiB, nearly all of it the uploads no record references, which each server holds for an hour.
- Memory. Every figure is the server’s own process in its container,
from
docker top:pds, Node’sMainThread,vlpds,zds,tranquil-pds. vlpds runs under a 2,560 MiB container limit, which sizes its caches; the others have none. RustFS’s memory is on top of vlpds’s and PostgreSQL’s on top of tranquil’s, each in a table of its own; PostgreSQL’s is the sum of its processes’ PSS, as they share its buffers. The PLC directory runs in the bench’s process and is counted for none. The reference’s figures include the Node runtime and dev-env’s module graph. tranquil’s binary is UPX-compressed, as its Dockerfile builds it, and unpacks into its memory at start. - createAccount includes the PLC operation to the bench’s directory,
through
host.docker.internal, for all five. createSession is about 22 a second for Monolith and the reference alike: it is their password hash. tranquil’s is 15 a second.