The outbox pattern, proved by killing the process six times

The dual write can't be removed, only relocated. One payment capture — real schema, real transaction, real relay query, real consumer — killed at six points, with the state and the self-heal at each.

Here is a flow with three steps and no obvious difficulty. Capture a payment. Post the ledger entries. Tell the rest of the system it happened, by emitting payment.captured, so the payouts service and the receipt mailer and the merchant dashboard find out.

Two of those three live in my database. The third lives in a message broker. There is no transaction that spans both.

That’s the dual write, and the standard answer to it is the transactional outbox: write the event into a table in the same transaction as the state change, then have a separate process read that table and publish. Chris Richardson’s write-up is the canonical statement of it, and it ends on a sentence that sounds like the end of the discussion — “Messages are guaranteed to be sent if and only if the database transaction commits” (microservices.io (opens in new tab)).

I believe that sentence. I’ve also come to think it’s the least useful part of the pattern, because it’s an argument, and arguments about crash-safety are cheap. What’s load-bearing is the matrix: for every point where the process can die, what state is the system in, and does it get itself out unattended? Until I’ve filled in every cell, I don’t know whether my outbox is correct. I only know that the outbox is correct, which is not the same claim and won’t help me at 3am.

So here’s one flow written out in full — schema, transaction, relay query, consumer — and then killed at six points. The conclusion I end up at, which I don’t think gets said bluntly enough: the outbox does not remove the dual write. It moves it from a place where a crash loses data to a place where a crash only duplicates it. That’s the entire trade. The matrix is how you see it.

The flow, written out

Scope first, because it decides the first cell. Capturing money at a PSP is itself an external call, and the boundary between “the PSP captured” and “my database knows” is a different dual write with a different fix — an idempotency key on the way out, and reconciliation as the backstop. The outbox protects the boundary between my database and my broker. Nothing else. I’ll come back to what that costs.

The schema:

create table payments (
  id              uuid primary key,
  idempotency_key text unique not null,
  status          text not null,          -- authorized | captured | failed
  amount_minor    bigint not null,
  currency        char(3) not null,
  psp_capture_id  text,
  captured_at     timestamptz
);

create table ledger_entries (
  id           bigserial primary key,
  txn_id       uuid   not null,           -- groups one balanced set
  account      text   not null,
  direction    text   not null check (direction in ('debit','credit')),
  amount_minor bigint not null check (amount_minor > 0),
  currency     char(3) not null,
  created_at   timestamptz not null default now()
);
create unique index on ledger_entries (txn_id, account, direction);

create table outbox (
  id             bigserial primary key,
  event_id       uuid not null unique,    -- the dedupe key, travels to the consumer
  aggregate_type text not null,           -- 'payment'
  aggregate_id   uuid not null,           -- the payment id; also the partition key
  event_type     text not null,           -- 'payment.captured'
  payload        jsonb not null,
  created_at     timestamptz not null default now(),
  sent_at        timestamptz              -- null = pending
);
create index outbox_pending on outbox (id) where sent_at is null;

Three notes. event_id is minted by the writer, inside the business transaction, and it’s the dedupe key that travels all the way through — the consumer’s defense against duplicates has to be a value the producer committed, not something derived on arrival. The partial index means the relay never scans the millions of rows that already went out. And amounts are integer minor units, with the two entries forming a balanced double-entry pair.

The transaction, after the PSP has confirmed the capture:

begin;

update payments
   set status = 'captured', psp_capture_id = $psp_id, captured_at = now()
 where id = $payment_id
   and status = 'authorized';
-- 0 rows: already captured, or never authorized. rollback, return current state.

insert into ledger_entries (txn_id, account, direction, amount_minor, currency)
values ($txn_id, 'psp_receivable',   'debit',  $amount, $ccy),
       ($txn_id, 'merchant_payable', 'credit', $amount, $ccy);

insert into outbox (event_id, aggregate_type, aggregate_id, event_type, payload)
values ($event_id, 'payment', $payment_id, 'payment.captured',
        jsonb_build_object('payment_id', $payment_id, 'txn_id', $txn_id,
                           'amount_minor', $amount, 'currency', $ccy));

commit;

Three writes, one commit. That’s the whole trick: the event isn’t a message you send, it’s a row you write, and it’s in the same atomic unit as the thing it describes.

The relay:

begin;

with claimed as (
  select id from outbox
   where sent_at is null
   order by id
   limit 100
   for update skip locked
)
select o.id, o.event_id, o.aggregate_id, o.event_type, o.payload
  from outbox o join claimed c on c.id = o.id
 order by o.id;

-- publish each to the broker, in id order, keyed by aggregate_id

update outbox set sent_at = now() where id = any($claimed_ids);

commit;

Two things in there I’d argue about in review.

Why a pending flag and not where id > :cursor. A cursor over a bigserial looks tidier and it is quietly, permanently wrong. Postgres assigns the id when the insert runs, not when the transaction commits, and it doesn’t reclaim values from aborted transactions — the docs are explicit that sequence objects “cannot be used to obtain ‘gapless’ sequences” (Postgres sequence functions (opens in new tab)). Meanwhile a Read Committed select “sees only data committed before the query began” (Postgres transaction isolation (opens in new tab)). Put those together: transaction A takes id 100 and is still open, transaction B takes 101 and commits, the relay polls, sees 101, publishes it, advances the cursor to 101. A commits a millisecond later. Event 100 is now invisible forever, and nothing anywhere will tell you. A pending flag has no such hole — a row is marked sent or it isn’t.

Why skip locked. So several relay instances can run without fighting over the same rows. The Postgres docs describe precisely this use: it “provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table” (SELECT docs (opens in new tab)). A queue-like table is exactly what this is.

The cost of that shape, which usually goes unmentioned: the transaction stays open across a network call to the broker. That’s deliberate — the row locks are what stop a second relay from republishing — but a long-lived transaction holding an old backend_xmin is the exact thing Postgres tells you to hunt down when vacuum stops keeping up (routine vacuuming (opens in new tab)). If the broker gets slow, your relay becomes a long-transaction generator. The alternative is a two-phase claim — update ... set claimed_by, claimed_at, commit, publish, then mark sent — which trades the open transaction for a claim-expiry timer and a wider duplicate window. I’d start with skip locked, keep the batch small, put a hard timeout on the publish, and only move to claims if vacuum complains.

The consumer:

begin;

insert into processed_events (event_id, consumer, processed_at)
values ($event_id, 'payouts', now())
on conflict (event_id, consumer) do nothing;
-- 0 rows: already handled. commit, ack, do nothing else.

insert into payout_items (payment_id, amount_minor, currency, source_event)
values ($payment_id, $amount, $ccy, $event_id);

commit;
-- ack the broker only after the commit returns

The single most important property of that consumer is that the dedupe row and the effect share one transaction, and the ack comes after the commit. Get that wrong and you’ll see why in cell five.

Six crash points

Kill point	Database	Broker	Heals unattended?	Cost of healing
1. Before commit	still authorized, no entries, no outbox row	nothing	Not by the outbox	Caller retry, else a reconciliation break at T+1
2. After commit, before poll	captured, balanced, row pending	nothing	Yes	One poll interval of lag
3. Mid-publish	pending (relay txn rolled back)	may or may not have it	Yes	Possible duplicate
4. After publish, before mark-sent	pending	has it, once	Yes	Certain duplicate
5. Mid-consume	nothing — or dedupe row only	unacked	Only if dedupe + effect share a txn	Silent permanent loss in the bad variant
6. After consume, before ack	effect + dedupe row committed	unacked	Yes	One redelivery, suppressed

Kill 1 — before commit. The PSP has taken the money. My transaction is open and now it’s gone; Postgres rolls it back when the connection drops. The payment still says authorized, there are no ledger entries, there is no outbox row. Money moved in the world and my books don’t know.

Does it heal? Not through the outbox. The outbox never got a row, and a pattern whose guarantee begins “if the transaction commits” says exactly nothing about a transaction that didn’t. It heals two other ways. If the caller retries with the same idempotency key, the PSP returns the same capture rather than taking the money twice, my handler re-runs, where status = 'authorized' is still true, and it commits cleanly. If the caller never retries — user closed the tab, upstream job died too — nothing heals until reconciliation finds a settlement line with no matching ledger entry, at T+1, resolved by a repair rule or by a person.

Cell one is the honest one. The pattern everybody reaches for to make money movement reliable does nothing for the step where the money actually moved.

Kill 2 — after commit, before the relay polls. Payment captured, ledger balanced, outbox row sitting there with sent_at null. The broker knows nothing; as far as the rest of the company is concerned this capture didn’t happen.

Heals: yes, unconditionally, and this is the entire reason the pattern exists. The event is durable and it’s in the same commit as the state. It doesn’t matter which process died or how long it stays dead. Any relay instance that starts later runs the same query and finds the row. Recovery is one poll interval after the relay is back. No human, no replay script, no “can someone re-emit yesterday’s events.”

Compare the naive commit; publish();. Crash in that gap and the event is gone — and the part that makes it genuinely dangerous is that nothing knows it’s gone. No row, no counter, no lag metric. You find out when the merchant asks where their payout is.

The gap between committed and published doesn’t disappear here. It’s still there, in cell two. What changed is that it’s now a row in a table and bounded by a poll interval, instead of invisible and permanent.

Kill 3 — mid-publish. The relay sent the record and died before hearing back. Two sub-cases, and the relay cannot tell them apart: the broker never got it, or the broker got it, wrote it durably, and the ack died on the way home. That indistinguishability isn’t an implementation gap — it’s the same thing that makes exactly-once delivery a lie.

State: the relay’s transaction rolls back, sent_at stays null, the locks release. On the broker, the event either exists or it doesn’t.

Heals: yes. The row is pending, so the next poll republishes it. In the first sub-case that’s the repair; in the second it’s a duplicate. You cannot have one without the other, because getting there would require the relay to know something it can’t know.

Broker-side idempotence narrows this and doesn’t close it. Kafka’s producer with enable.idempotence “will ensure that exactly one copy of each message is written in the stream” (producer configs (opens in new tab)) — but the producer that comes back after a restart is a new session, and the docs are clear that spanning sessions is what transactional.id is for: without one, “the producer is limited to idempotent delivery.” In-session retries get deduped. The cross-restart duplicate goes through. Which is fine — that’s what the consumer’s dedupe table is for.

Kill 4 — after publish, before mark-sent. The broker acked. Durable, replicated, done. The relay dies before update outbox set sent_at = now() commits, so that rolls back with everything else.

State: the event exists on the broker exactly once, and the database says it was never sent.

Heals: yes, on the next poll — by publishing it a second time. Not “might duplicate,” like cell three. Will duplicate, deterministically, every time.

This is the cell I’d put on the whiteboard, because it’s the pattern looking at itself in a mirror. Publishing to the broker and recording “published” in the database are two writes to two systems with no shared transaction. That is a dual write. The outbox pattern’s answer to the dual write contains a dual write. The canonical write-up doesn’t hide this, either — it says the relay “might publish a message more than once” (microservices.io (opens in new tab)).

And that’s fine — but only because of which dual write it is. In the original problem the two writes were (state, event), and losing the second means a payment happened and nobody downstream will ever know. In the relocated problem the two writes are (event, sent-marker), and losing the second means the event goes out twice. One failure mode is silent data loss. The other is a duplicate, which is a thing a consumer can be built to absorb. The outbox doesn’t defeat the impossibility. It picks a better place to be defeated by it.

Kill 5 — mid-consume. This is the one cell whose answer can be “no,” and it depends entirely on how the consumer was written.

Written the way I showed it — dedupe insert and effect in one transaction, ack after commit — the kill rolls everything back. No dedupe row, no payout item, no ack. The broker redelivers and the consumer runs the whole thing from scratch. Heals.

Written the other way — insert the dedupe row and commit, then do the effect — the kill leaves the dedupe row committed and the payout missing. The broker redelivers. The consumer looks up event_id, sees “already processed,” skips the work, acks. The event is now permanently consumed and permanently unapplied. No retry fixes it, because the system believes it’s done. No alarm fires, because every metric says the pipeline is healthy.

The advice everyone repeats gets you most of the way and stops one step short: a consumer “must be idempotent, perhaps by tracking the IDs of the messages that it has already processed” (microservices.io (opens in new tab)). True, and both variants above do exactly that. The part that decides this cell is where the tracking write commits relative to the work, and that doesn’t fit in the sentence.

So it’s the worst outcome in the matrix by a distance, and it is not the outbox’s fault. The outbox’s guarantee ends at the broker. The consumer has its own atomicity problem — “apply the effect” and “record that I applied it” are two writes — and it has the same fix: one transaction. If the effect doesn’t live in your database, you’re back at the start and the effect itself has to be idempotent by construction.

Kill 6 — after consume, before ack. The transaction committed: effect applied, dedupe row written. The process dies before acking.

State: everything is correct. The broker just doesn’t know it.

Heals: yes. The broker redelivers, the on conflict do nothing returns zero rows, the consumer skips the effect and acks. One duplicate delivered, one duplicate suppressed, nothing wrong.

This is the mirror of cell four — the same impossibility (you can’t atomically do work and record it in someone else’s system) showing up at the other end of the pipe. Cell four is why duplicates exist. Cell six is why you need somewhere to catch them. They’re one problem, counted twice. Killing it after the ack is a non-event: nothing pending, nothing redelivered, nothing to heal.

What the matrix proves, and what it doesn’t

Reading down the column:

• Five of six heal unattended. The sixth heals only if the consumer put its dedupe write inside the effect’s transaction.
• Five of the six recoveries work by re-doing something — a caller retry, a republish, a full reprocess, a suppressed redelivery. The exception is cell two, where the relay simply publishes the event for the first time, late. That’s the only clean resume in the table, and it happens to be the cell the whole pattern exists for.
• Three cells produce duplicates by construction — 3, 4 and 6 — and all three are the same lost-ack problem in different costumes. Duplicates aren’t a risk you might hit here; in those cells they’re the mechanism of recovery.
• The slowest cell is the one the outbox doesn’t cover. Cells 2 through 4 recover in a poll interval and cell 6 in a redelivery. Cell 1 recovers at T+1, in reconciliation, possibly by hand.

Now the limits, because a crash matrix is a proof about crashes and nothing else.

It assumes the commit is durable. Every “yes” in that column reduces to “the row is in the database.” Run with synchronous_commit = off, or fail over to a replica that hadn’t received the WAL, and the row isn’t there — cell two turns into silent loss and the whole thing collapses. The outbox’s guarantee is exactly as strong as your durability settings and not one bit stronger. Worth checking what yours actually are before you trust the matrix.

It only models crash-stop. A process that dies is the easy failure. A relay that’s alive but can’t reach the broker appears nowhere above: the system stays correct and stops being timely, rows pile up, and only a metric will tell you. So alarm on the age of the oldest pending outbox row, not on whether the relay process is up. Up-and-stuck is the failure that looks like health.

It doesn’t cover poison. One event the broker rejects — oversized, bad schema — comes back on every poll forever, and if you publish in strict id order it head-of-line blocks everything behind it. You need an attempt counter and somewhere to put the failures. That isn’t a crash, so the matrix is silent on it.

It says nothing about ordering. With skip locked and more than one relay, two events for the same payment can be published by different instances and land out of order — the polling-publisher pattern is honest about this, listing “tricky to publish events in order” as its headline drawback (microservices.io (opens in new tab)). If you need per-key order, run a single relay or shard the outbox by a hash of aggregate_id. And downstream, ordering is a guarantee you’ll never get anyway, so the consumer needs its own defense regardless.

It doesn’t cover being wrong. If the handler writes the wrong amount into payload, the outbox durably, reliably, at-least-once delivers the wrong amount. Crash-safety and correctness are different properties and the matrix speaks to one of them.

And the table grows. Every capture writes a row. Pruning is a job you have to build, and a high-churn insert-and-delete table wants an eye kept on autovacuum. Keeping sent rows for a while is worth it — they’re a debuggable log of what you actually emitted — but “a while” has to be a number somebody chose on purpose.

The cell the outbox can’t reach

Cell one stays open, and I don’t think you fix it by finding a stronger pattern. Pat Helland put the underlying limit better than anyone in 2007: real systems have multiple disjoint scopes of transactional serializability, and “you cannot perform atomic transactions across these disjoint scopes of transactional serializability. That’s what makes them disjoint!” (Life beyond Distributed Transactions (opens in new tab), CIDR 2007). The question is never how to remove the gap. It’s where to put it, and what shape you want it to have when it fails.

The thing I’ve come around to is that the outbox table’s shape is right for cell one even though the outbox pattern isn’t. Change what the row means. An event says this happened — backward-looking news, and the reader’s only obligation is not to apply it twice. An intent says this must happen, exactly once, and I am on the hook until it reaches a terminal state — forward-looking, owned, and the obligation is to drive it to a conclusion.

Concretely, in the transaction that changes business state:

begin;
  insert into payments (id, status)
       values ($payment_id, 'authorizing');

  insert into intents (id, kind, payload, idempotency_key, state)
       values ($intent_id, 'psp.capture', $payload, $idem_key, 'pending');
commit;

Same durability trick, different obligations. Three things follow that a plain event row doesn’t give you.

The idempotency key is minted here, not at the attempt. It’s a column, generated inside the transaction, durable before any network call exists. That’s the difference between a key per operation and a key per attempt — the distinction that makes retries safe rather than an elaborate double-charge machine.

The worker executes; it doesn’t publish. It leases the intent, calls the PSP carrying the stored key, records what came back, and only then writes the ledger entries. The duplicate you could never eliminate — the one the relay’s own dual write guarantees — now lands on the provider’s idempotency layer instead of a consumer’s dedupe table. For an irreversible effect that’s the only place it can land, because only the far side can decide not to move the money twice.

unknown is a legal state, not an error. A timeout must not mark the intent failed; “failed” is a claim you can’t support. It goes to unknown, and the worker retries with the same key or queries the provider for the outcome. This is the state most implementations skip and the one that costs real money. It also expires: idempotency keys have retention windows, so an intent still unknown after that window is no longer resolvable by retry, and it becomes a reconciliation item. Which is fine, as long as something is genuinely watching. And when an authorization succeeded but shouldn’t be completed, there’s no rollback either — the undo is a forward message to the scheme, so it gets its own intent row, its own key, its own retries. Compensation as a row in a table, not an exception handler bolted to the side. Check whether the schemes you’re on put a clock on that reversal; some do, and a deadline you don’t alarm on is a deadline you’ll miss.

The cost is real and it’s not small: leases so two workers don’t execute the same intent, a genuine state model with terminal states, an operator-visible queue of unknown intents that a human can act on, alerting for anything stuck. You also make the provider call asynchronous relative to the user’s request, which means the UI has to render “in flight,” and product people hate “in flight.” So don’t reach for this when the effect is internal and replayable — publishing a domain event, reindexing a document, sending a notification that’s merely annoying twice. Plain outbox event publishing is correct there and the intent machinery is theatre. Reach for it when the effect is irreversible and money-shaped, and the thing you’re protecting against is the third outcome: money moved and nothing in your system knows it.

The trade, stated plainly

What the outbox buys: no event is ever lost and no recovery needs a human — given a durable commit and a consumer written correctly.

What it costs, roughly in the order it’ll bother you:

1. Duplicates become certain, not possible. Cell four guarantees them. Every consumer needs a dedupe key and a dedupe table, and that write has to share a transaction with the effect. If a consumer can’t be made idempotent, the outbox hasn’t helped it — you’ve handed the problem to someone who can’t solve it.
2. Latency, sized by the poll interval. Polling every second means a second of lag and a query per second per relay, forever. Polling every 50ms means twenty queries a second, forever. Log tailing removes the poll and adds a CDC pipeline to your on-call rotation. Pick which operational cost you’d rather carry; there is no version with neither.
3. A hot table that needs a partial index, a pruning job, and vacuum attention.
4. A new silent failure mode. The relay is a component that can be down while everything looks fine. This is the cost I’d watch hardest, because it’s the one the pattern introduces rather than inherits.

And when not to bother. If the consumer is in the same database, don’t publish an event at all — write the row and skip every one of these costs. If the event is a nice-to-have, a cache invalidation or an analytics ping where losing one costs nothing, then commit; publish(); is fine and you shouldn’t pay four costs to avoid a harmless loss. The outbox is for the events where “we captured a payment and nobody downstream found out” is an incident with a customer’s name on it.

The argument for the outbox takes one paragraph and it’s convincing. That’s the problem with it. Whether the version in your codebase is actually correct comes down to details the argument never mentions — flag or cursor, whether mark-sent shares a transaction with the publish, whether the consumer’s dedupe row shares a transaction with the effect, whether the ack lands before or after the commit. None of those show up until you go cell by cell and say out loud what happens if it stops right here.

If I had to guess where a real implementation goes quietly wrong, it’s two places: the cursor query in the relay, which loses events invisibly, and the two-transaction consumer, which drops them invisibly. Both look correct. Both would survive my code review if I were reading fast. Both are cells nobody filled in.

Archie