Skip to main content

Writes Are Jobs, Reads Are Cached

User-facing requests never block on outbound writes. Every write to a third-party system is queued and executed in the background; every read is served from a local cache. The HTTP handler does the minimum amount of work needed to acknowledge the user and return.

This is the discipline that keeps the UI snappy regardless of upstream latency or availability.

The read path

User hits a page → handler reads from the local Postgres cache → renders HTML/JSON and returns.

If the cache is stale or missing, the handler synchronously fetches from upstream, writes through to the cache, and returns. But the common case — warm cache — never touches the upstream.

Two-tier layering where it earns its keep:

  • L1 in-process (Moka, Redis, etc.) — hot small lookups: OAuth tokens, type catalogs, config data that changes hourly at most.
  • L2 persistent (Postgres) — authoritative entity storage: tickets, quotes, invoices, users.

UI reads target L2. L1 is a microsecond optimization for repeated lookups inside a single request or across requests in the same process.

The write path

User submits a form → handler updates the local cache optimistically with a pending_write = true flag → dispatches a background job → returns the HTTP response.

The background job:

  1. Performs the upstream write (Halo POST, Stripe API call, etc.)
  2. On success, clears pending_write and upserts the upstream response back into the cache
  3. On failure, retries with backoff; if it exhausts retries, escalates via the job queue's escalation pipeline

The user sees the result immediately. The upstream write is guaranteed to happen (eventually) but does not block the request.

The shape of the job queue

A Postgres-backed job_queue table with two job types:

  • Fire-once jobs — "perform this write against the upstream." Dispatched, run once (with retries), done.
  • Poll state-machine jobs — "watch this entity until it reaches a terminal state." Examples: watch Stripe PI until succeeded, watch Halo quote until approvalstate=2. Each tick calls a handler that returns Retry / Completed / Cancelled. Has an abandon_after deadline; if hit, escalates.

Both types live in the same table. The worker binary processes them off a single loop.

Why we do this

  • User-facing latency is decoupled from upstream latency. A Halo API call taking 10s doesn't block a page load.
  • Upstream failures don't fail user actions. A 500 from Halo becomes a retry in the job queue, not an error page for the user.
  • Retries are free. Every write is already a job; retrying a failed job is the same code path as running it the first time.
  • Observability is built in. The job queue table is queryable — "show me all pending writes for client X" is a SQL query.
  • Race conditions are contained. Two users clicking "approve" at the same time becomes two jobs that run sequentially, not two competing HTTP handlers.

What makes this hard (and why new engineers get it wrong)

The temptation is always: "just make the API call inline, it's simpler." It is simpler — right up until the upstream goes flaky. Then you rewrite the whole flow under production pressure. Do it the right way from the first commit.

Signs you're doing it wrong:

  • HTTP handlers contain outbound HTTP calls (other than to internal services)
  • "Save" buttons take longer than ~100ms
  • Error messages like "Something went wrong, please try again" reach users from upstream timeouts
  • Tests require mocking the upstream because the handler hits it directly

When this applies

Every integration with a system that can fail, rate-limit, or slow down.

When it does not apply

  • Reads that must be real-time with zero staleness tolerance (live balance checks against a financial ledger). In those cases, the upstream is the cache — but this is rare.
  • Writes that have no cacheable state and no retry value (one-shot fire-and-forget analytics).
Back to top