Webhook ordering is a guarantee you'll never get. Pick your defense.

You can't get ordered delivery over a webhook channel, so stop asking for it. Three well-understood defenses — version numbers, refetch, and a state machine — and which one to reach for first.

Here is a bug I have watched land in three different codebases, wearing a different costume each time. A webhook arrives that says a payment succeeded. The handler flips the order to paid, fulfills it, sends the receipt. A moment later, a second webhook arrives — processing — for the same payment, generated before the success but delivered after it. The naive handler dutifully flips the order back to pending. Now you have a shipped product marked unpaid, and a support ticket that will take an afternoon to unwind.

The instinct is to go find the ordering knob. There isn’t one. And once you accept there isn’t one, the problem gets much smaller, because it turns from “how do I get ordered delivery” (impossible) into “which of three well-understood defenses do I reach for” (a design choice with a right answer for most cases). That’s the whole essay: you cannot get global ordering over a webhook channel, so stop asking for it, and instead pick — deliberately — how you’ll tolerate its absence.

Why the guarantee isn’t on the menu

Webhooks are an at-least-once push channel, and both of those words are the problem.

At-least-once means the sender retries until you acknowledge, and any acknowledgement can be lost in flight, so the same event can arrive more than once. I’ve argued the deeper version of this before — exactly-once delivery is a lie you tell yourself — and it’s load-bearing here: if events can be redelivered, then even a sender who tried to send in order can’t keep them in order, because a retry of event #3 can land after event #4’s first, clean delivery.

Push means the sender fires events at your endpoint independently. There is no single ordered log you’re reading from with a cursor. Two events can be generated close together, dispatched from different workers, retried on different schedules, and hit your endpoint in any interleaving. This isn’t a gap a better provider will close. Stripe says it in plain text: it “doesn’t guarantee the delivery of events in the order that they’re generated,” and it gives the exact worst case — creating a subscription can emit customer.subscription.created, invoice.created, invoice.paid, and charge.created, any of which you might see first (Stripe webhook docs (opens in new tab)). Its retry policy — up to three days of exponential backoff in live mode — is precisely what scrambles the order, and precisely why the same event shows up twice (Stripe webhook docs (opens in new tab)).

So global ordering is off the menu. Good. Now here are the three things you can actually reach for.

Defense 1: version numbers with last-writer-wins

Every event about an entity carries a monotonically increasing marker — a version, a sequence number, or a reliably-generated timestamp from the source. You store the highest one you’ve seen per entity, and you drop any event whose marker is less than or equal to it. The processing event that shows up after succeeded has a lower version, so you throw it away. In practice it’s one guarded write: WHERE incoming_version > stored_version, and the stale flip-back simply loses the race. It never happens.

This is cheap and it’s local. One extra column, one comparison, no network call on the hot path. It also doubles as your dedupe: a redelivered event has an equal-or-lower marker, so it’s discarded for free.

The cost is that it only works if the marker is real and total. Provider timestamps often aren’t safe for this — two events in the same millisecond, clock skew between the workers that emitted them, or a created field that reflects object creation rather than event emission. If the provider gives you an explicit per-object version or sequence number, use it and this defense is excellent. If you’re reaching for a wall-clock timestamp as a stand-in, you’re building on sand, and you should know that before you ship it. The other limitation: last-writer-wins tells you which event is newest, not whether the newest event is valid. It will happily apply a legal-looking-but-wrong transition. It orders; it doesn’t judge.

Defense 2: refetch current state on receipt

Treat the webhook as a doorbell, not a delivery. When an event arrives, you ignore its payload almost entirely and instead call the provider’s API to ask: what is the current, authoritative state of this object right now? You act on the answer, not on the event.

This is beautiful because it sidesteps ordering completely. It does not matter which order the doorbells ring in — every ring makes you go read the latest truth, and the latest truth is the latest truth regardless of the path that told you to look. Duplicates become harmless too: two rings, two identical reads, same result. Stripe recommends exactly this — “you can also use the API to retrieve any missing objects” — and it’s the pattern I trust most for correctness (Stripe webhook docs (opens in new tab)).

The cost is real and it’s threefold. You add a synchronous API call to every event, which means latency, rate-limit pressure, and a dependency on the provider being up at the exact moment their own webhook fired. There’s also a subtle race: the “current state” you fetch may be newer than the event that triggered the fetch — usually fine, occasionally surprising if you assumed the fetch corresponds to that specific event. And it only works when there’s an authoritative source to refetch from. For a provider’s webhook, there is. For your own internal events carrying data that lives nowhere else, there’s nothing to go read, and this defense doesn’t apply.

Defense 3: a per-entity state machine that rejects illegal transitions

Model the entity’s lifecycle explicitly — for a payment, something like pending → authorized → captured → settled, with refunded and failed as terminal branches — and encode which transitions are legal. Each event proposes a transition. The machine accepts it only if it’s legal from the current state. succeeded → processing isn’t a legal move, so the late processing event is rejected on principle, not because you compared version numbers.

What I like about this one is that it’s the only defense that encodes meaning. Version numbers know which event is newer; state machines know which event makes sense. That makes it the strongest guard against the genuinely dangerous class of bug — not “we applied things out of order” but “we applied something that should never have been applied,” like capturing a refunded payment. It’s also self-documenting: the set of legal transitions is the spec, sitting in code where you can test it.

The cost is that it’s the most work, and it’s not always sufficient on its own. You have to actually enumerate the lifecycle, keep it in sync with the provider’s real model (which changes), and decide what to do with the events you reject — silently drop, dead-letter, alert? And it doesn’t linearize concurrent legal transitions; if two individually legal events race, you still need a version tiebreaker or a lock underneath. The state machine is a correctness guard, not an ordering mechanism. It rejects the illegal; it doesn’t sequence the legal.

Which one to reach for first

Reach for the state machine plus a version tiebreaker as your default, and add refetch for the handful of events where being wrong is catastrophic.

Here’s the reasoning, made explicit. The three defenses aren’t really competitors; they defend different things. Version numbers defend against staleness — applying an older event over a newer one. The state machine defends against invalidity — applying an event that should never apply from where you are. Refetch defends against both by refusing to trust the payload at all, at the price of a network round-trip. Money systems have to survive both staleness and invalidity, so the honest baseline is a state machine (catches the dangerous invalid transitions) with version-based last-writer-wins underneath it (breaks ties between legal transitions and gives you dedupe for free). That combination is entirely local — no hot-path API call — and it turns “out of order” from a corruption into a rejected event.

Refetch is the specialist tool, not the baseline. I pull it in for the events where a wrong answer means real money moved wrong — a capture, a payout, a refund — because for those, one extra API call to confirm the authoritative state is a trade I’ll take every time, and its immunity to ordering is worth the latency. I would not put a synchronous refetch on every webhook; you’ll drown in rate limits and inherit the provider’s downtime as your own.

And underneath all three sits the backstop I always come back to: state derived from an append-only log rather than a mutable status column (double-entry, still the right data model) forgives more of these mistakes, because a late or duplicate event becomes an entry you can reason about and reconcile, not an overwrite that silently destroys the previous truth.

The mental shift is the whole point. Stop treating out-of-order delivery as a failure the provider owes you a fix for. It’s the normal weather. Pick your umbrella on purpose — usually the state machine, sometimes the refetch — and the bug I opened with simply never happens, because your handler was never naive enough to trust the order in the first place.

Archie