Most diagrams describe the first attempt.

A request leaves the client, crosses a boundary, changes something, and returns a result. The arrows are clean because the answer is known.

The more interesting design begins when no answer returns.

A timeout is uncertainty, not proof

When a client times out, several realities remain possible. The server may never have received the request. It may still be working. It may have finished successfully while the response was lost on the way back.

These cases look identical from the caller’s side, but they demand different actions. Retrying can recover from a transient fault. It can also repeat a payment, create a second resource, or send the same message twice.

The first useful principle is therefore simple: treat a timeout as an unknown outcome, not as a confirmed failure.

Give the intention an identity

An idempotency key lets repeated requests express one intention. The first request and the retry carry the same identifier, allowing the receiving service to recognize that they belong to a single operation.

This is more precise than comparing request bodies. Two identical payloads may represent two legitimate actions. Two requests with the same identifier should represent the same action, even if they arrive seconds apart.

The service must store the identifier and the result with the same care as the change itself. Otherwise it can remember an operation that never completed, or complete an operation it cannot later recognize.

Retry with a boundary

Safe repetition does not mean endless repetition.

Retries need a budget, a delay, and a reason. Validation errors should return immediately. Transient server errors may justify another attempt. Backoff and jitter prevent many clients from returning at the same instant and turning a small interruption into sustained overload.

The caller should also know when to stop trying automatically and expose the uncertainty to a human or a reconciliation process.

Reconciliation closes the loop

Some workflows outlive a single request. In those cases, reliability comes from comparing desired state with observed state and moving the system toward agreement.

Reconciliation is slower than a direct response, but it handles late arrivals and partial progress honestly. It replaces the fiction of perfect delivery with a repeatable way to recover.

Make the second attempt visible

A retry that succeeds can hide the instability that made it necessary. Useful monitoring distinguishes ordinary first attempts from repeated ones, tracks exhausted retry budgets, and measures the effect visible to the user.

The goal is not to page someone for every failed component. It is to preserve enough evidence to understand whether the system is recovering or merely postponing a larger failure.

Further reading