Relay
Independent systems study · AI-assisted build · 2026
A Go and PostgreSQL job queue with worker leases, retries, and a recorded crash-recovery experiment. Built to study what happens between accepting a job and finishing it.
An AI-assisted systems study. The results are from local tests. Delivery is at least once, so external writes still need duplicate protection.
Source publication pendingtoken = B · lease is validOne current owner
A fresh lease token separates the current worker from a late one.
RECORDED EXPERIMENT / SEPT 2026
Follow one job.
The job and its first event are committed together in PostgreSQL.
A replay of recorded events, not a live service. Times are relative to enqueue and rounded to milliseconds.
A job after a crash
Recorded local experiment · September 2026
Each job: 10 ms wait + 1,000 hash iterations
More execution slots
3,000 jobs per configuration. One local trial each, including submission and queue drain. These are not production capacity figures.
Implementation notes
Which worker is allowed to finish the job?
Each claim locks a ready row, installs a fresh lease token, and commits before the handler starts. Other workers skip locked rows. A heartbeat renews the lease using database time.
Completion checks both the token and its expiry. A worker that finishes late cannot overwrite the result after another worker has taken the job.
Kill a worker and follow the job through recovery.
The experiment kills a process during a two-second job. Its lease expires, a replacement claims the job, and attempt two succeeds. The old acknowledgement is rejected. In the recorded run, completion took about three seconds after the kill.
Delivery is at least once. Lease tokens protect queue state; a handler that writes to an external service still needs its own way to prevent duplicate effects.
Measure queueing time as well as throughput.
A single-machine burst test processed 3,000 jobs at each configuration: one execution slot, four slots, and twelve slots across three processes. Throughput was about 62, 255, and 862 jobs per second. Each job waited 10 ms and ran 1,000 hash iterations.
These are individual local trials, including submission and queue drain. They are not production capacity figures. The one-slot run had a completion p95 of about 44 seconds. PostgreSQL remains the shared availability dependency.