# Relay

> A Go and PostgreSQL job queue with worker leases, retries, and a recorded crash-recovery experiment. Built to study what happens between accepting a job and finishing it.

- Author: Benjamin Zhuang
- Category: Distributed job processing
- Context: Independent systems study · AI-assisted build
- Years: 2026
- Stack: Go, PostgreSQL, HTTP, Docker, GitHub Actions
- Source: https://github.com/Benjaminnnnnn/relay (publication pending)
- Web page: https://benjaminnnnnn.github.io/portfolio/relay/

**A worker can disappear. Its job shouldn't.**

## Key figures

- 9,000: jobs in the local scaling experiment
- 3: independent worker processes tested
- SIGKILL: worker recovery verified

## Which worker is allowed to finish the job?

Each claim locks a ready row, installs a fresh lease token, and commits before the handler starts. Other workers skip locked rows. A heartbeat renews the lease using database time.

Completion checks both the token and its expiry. A worker that finishes late cannot overwrite the result after another worker has taken the job.

## Kill a worker and follow the job through recovery.

The experiment kills a process during a two-second job. Its lease expires, a replacement claims the job, and attempt two succeeds. The old acknowledgement is rejected. In the recorded run, completion took about three seconds after the kill.

Delivery is at least once. Lease tokens protect queue state; a handler that writes to an external service still needs its own way to prevent duplicate effects.

## Measure queueing time as well as throughput.

A single-machine burst test processed 3,000 jobs at each configuration: one execution slot, four slots, and twelve slots across three processes. Throughput was about 62, 255, and 862 jobs per second. Each job waited 10 ms and ran 1,000 hash iterations.

These are individual local trials, including submission and queue drain. They are not production capacity figures. The one-slot run had a completion p95 of about 44 seconds. PostgreSQL remains the shared availability dependency.

## Evidence

- [Recorded experiment and raw results](https://github.com/Benjaminnnnnn/relay/blob/main/experiments/2026-09-13-local.json)
- [Concurrency and lease tests](https://github.com/Benjaminnnnnn/relay/blob/main/internal/queue/queue_test.go)
- [Failure semantics and limits](https://github.com/Benjaminnnnnn/relay/blob/main/docs/failure-semantics.md)