Designing webhooks that survive retries
Providers retry. Networks drop. Your handler will run twice. Build for it on day one, not after the first double charge.
The problem
Every payment provider promises at-least-once delivery. That is the honest promise: they will send the event, and if they are not sure you got it, they will send it again. The second send looks identical to the first. If your handler charges, credits or emails on every call, it will do so twice.
Store first, act second
Write the raw event to a table keyed on the provider's event id before you do anything else. A duplicate hits the unique constraint and stops there. Processing then reads from your own table, in your own order, at your own pace. The webhook endpoint becomes boring, which is the point.
-- One row per provider event. A retry is a no-op.
insert into webhook_events (provider, event_id, body)
values ($1, $2, $3)
on conflict (provider, event_id) do nothing;Make every effect idempotent
The handler is only half of it. Anything it calls — a ledger write, a payout, a notification — needs a key too. Pass the event id down and let each service refuse the second attempt. Retries stop being incidents and start being Tuesday.
What to test
Send the same event twice in a row. Send it twice at the same time. Send it after the first attempt failed halfway. If all three end in exactly one effect, you are done.