Build · Advanced
Missed payment webhooks: the payment succeeded, the shop never heard
A customer pays, the provider confirms, and the notification to your shop is lost, late or delivered twice. This is how to design order processing so that missed payment webhooks cost nothing: no unpaid-looking paid orders, no duplicate orders, no second charge.
Photo: public domain (CC0)
In this guide
Where a payment notification can go wrong
A card or wallet payment involves three parties that do not share a database: the customer’s browser, the payment provider and your shop. The provider learns the result first. Your shop learns it in two ways: the customer’s browser is redirected back to a return page, and the provider sends a server-to-server notification, usually called a webhook.
Neither channel is reliable alone. The customer can close the tab before the redirect. The webhook can hit your server during a deployment, a timeout or a database outage. Because providers retry notifications that were not acknowledged, the same event can also arrive twice, or arrive hours later and after a newer event for the same payment.
The practical consequence: webhook delivery is “at least once, in no guaranteed order”. A shop that assumes “exactly once, in order” will sooner or later show a paid order as unpaid or ship twice. If you are planning a payment integration, design for that from the first day.
| Failure case | Symptom in the shop | Safeguard |
|---|---|---|
| Notification never arrives | Order stays “awaiting payment” although money was taken | Reconciliation job that asks the provider |
| Notification arrives twice | Two confirmation emails, stock reduced twice | Unique event ID and a conditional status change |
| Notification arrives late | Order was cancelled as unpaid, then payment shows up | Check provider state before cancelling, handle “paid after cancel” |
| Events arrive out of order | A refund is processed before the payment is known | State machine that rejects impossible transitions and re-fetches state |
| Handler crashes after acknowledging | Event is gone, nothing was processed | Store the event durably before answering |
| Forged notification | Order marked paid without any payment | Signature verification on the raw request body |
| Shop retries its own request after a timeout | Customer is charged or refunded twice | Idempotency key on every outgoing request |
Durable processing: store first, acknowledge fast, work from a queue
The endpoint that receives webhooks should do very little. Its whole job is four steps:
- Read the raw request body and verify the signature.
- Insert the event into an events table, with the provider’s event ID under a unique constraint.
- Answer with a 2xx status as soon as the insert has committed.
- Leave everything else to a worker that reads unprocessed events.
If you acknowledge before the event is stored and the process dies, the provider believes you have it and never sends it again. If you do the real work (mark the order paid, write the invoice, call the warehouse) before acknowledging, a slow dependency makes the request time out, the provider retries, and you process the same payment in parallel.
If the database is down, answer with a 5xx status so the provider retries later. Never answer 2xx for an event you could not store.
The worker takes events from the table or from a queue fed by it. Queues also deliver at least once, so the worker must be safe to run twice on the same event. After a fixed number of failed attempts, move the event to a dead-letter state and alert a person.
Idempotency: making “mark order paid” safe to repeat
Idempotent means that running an operation twice has the same effect as running it once. Three techniques together achieve it.
Deduplicate by event ID
The unique constraint on the provider’s event ID stops the same delivery from being stored twice. It does not help when two different events describe the same payment.
Change state conditionally
Do not write “set status to paid”. Write the transition: set status to paid where the order ID matches and the status is still “awaiting payment”. The database reports how many rows changed. Only when exactly one row changed do the side effects run: reduce stock, create the invoice, queue the confirmation email. A second run changes zero rows and does nothing.
Record the side effects in the same transaction as the status change, as rows in an outbox table that a sender works through. Otherwise a crash between “paid” and “send email” leaves an order that is paid and never confirmed.
Use idempotency keys on outgoing requests
The same problem exists in the other direction. When your shop asks the provider to capture or refund a payment and the response times out, you do not know whether it happened. Most payment APIs accept an idempotency key: a unique value you generate once per intended operation and send again unchanged on every retry, so the provider performs it once. Store the key with the order before the first attempt, not in memory.
Before trusting any notification, also compare its amount and currency with the order. A valid event for the wrong amount is not a paid order.
Reconciliation: asking instead of waiting
Providers stop retrying after some time, and an endpoint that was misconfigured for a day will have missed everything. Reconciliation is the safety net: a scheduled job that asks the provider for the truth.
- Open orders: for every order that has been “awaiting payment” for longer than a few minutes, query the provider by the stored payment reference and feed the answer through the same handler the webhook uses.
- Reverse direction: once a day, list the provider’s successful payments for the period and check that each one matches an order in a paid state with the same amount.
- Before cancelling: an order is cancelled as unpaid only after the provider has confirmed that no payment succeeded or is pending.
If reconciliation and the webhook worker both call the same conditional transition, it does not matter which of them arrives first.
The figures below are an illustrative example, not measurements: a shop with 10,000 successful payments in a month, of which 150 (1.5%) did not arrive as one clean notification on the first attempt. 96 + 42 + 12 = 150.
Illustrative example: 150 of 10,000 payments with an imperfect notification
With deduplication and reconciliation, all 150 end in the same state as the other 9,850.
A recovery design without duplicate orders or charges
Duplicate orders usually have one cause: the order is created in two places, once on the return page and once by the webhook. The cure is to create it once, before the payment.
A payment that arrives for an order already cancelled needs a business rule, not only code: reopen the order if the stock is still there, otherwise refund automatically and tell the customer. Decide it in advance. For an existing online shop, the quickest audit is to replay a stored webhook twice in a test environment and see what happens. The same patterns apply to any API integration that receives events from another system.
Questions we hear often
Want this sorted without doing it yourself?
Tell us what you are working on. A person replies within one working day.