Article chapter 04 of 08
Treat each integration as a state machine
From Releasing a multi-portal platform without losing state between components
An integration call is hardly ever one clean step. The application prepares a request, sends it, gets a response or doesn't, stores some local state, and might later accept a webhook or run a reconciliation job. Any of those boundaries can fail after the external system has already acted.
Give the local integration record explicit states that suit the operation. The names vary, but you usually want to tell apart work that's waiting, in progress, accepted, rejected, uncertain and reconciled. "Uncertain" is the one people leave out. It's what you've got when a request times out and the caller can't tell whether the remote side finished it, and retrying blindly from there can run the action twice.
Use an idempotency key or a stable operation identifier wherever the external contract supports one, and store it before the request leaves the application. A retry should carry on with the same operation instead of creating a new one. If the provider has no idempotency support, the recovery path might need a lookup or a human decision in place of an automatic retry.
A webhook is an external event, and it still has to get past verification and your local transition rules. Verify where it came from using whatever mechanism the provider defines, keep the external event identifier, and make repeat deliveries safe. Process each event against the current local state, so an old event that turns up late can't overwrite a newer confirmed state.
Then add reconciliation for records that sit in uncertain or in progress for longer than they should. Reconciliation should compare identifiers and the relevant facts, record what it found, and either finish the transition or move the item into an exception queue someone can see. A scheduled job that quietly scans and skips errors looks reliable from the outside while leaving support with no case to look at.