An order-creation step times out. The automation dashboard turns red. An operator clicks “run again.”
That sounds like recovery. It could also create a second order.
A timeout tells you the caller did not receive a usable response. It does not, by itself, establish whether the receiving system completed the action. Before repeating a consequential write, preserve its identity and determine whether the destination can recognise the same request. When that protection cannot be established, reconcile the outcome or escalate instead of blindly creating another record. Amazon’s discussion of safe retries explains this uncertainty and the role of caller-supplied request identifiers. Amazon Builders’ Library
The purchasing decision is practical: can your automation supplier demonstrate safe recovery after the destination succeeds but the response is lost? A normal successful run cannot answer that question.
The missing state between success and failure
Imagine a hypothetical service business that sends accepted orders from its CRM to a delivery-management system.
For order OR-1042, the destination creates delivery job DJ-781. The response does not reach the CRM. The CRM still shows the order as awaiting dispatch.
Two facts now coexist:
- The destination has a delivery job.
- The sending workflow has no confirmation of that job.
Calling this simply “failed” encourages another creation attempt. Marking it “complete” hides the missing confirmation. A more useful state is outcome unknown: no new creation is permitted until the workflow follows its defined recovery path.
This is a worked design example, not a netlinkE implementation report or customer incident. Its identifiers and outcomes are invented solely to explain the decision.
If your process does not already distinguish business states and owners, start with mapping the workflow before automation. The narrower task here is to specify what happens at an uncertain system boundary.
Give one action an identity that survives a rerun
An *idempotent* operation can be repeated without adding another unintended effect. For an order-creation API, the important question is whether repeated attempts for the same intended creation resolve to the same outcome rather than another order.
The workflow needs to distinguish three identities:
| Identity | Example | Why it matters |
|---|---|---|
| Business record | OR-1042 | Identifies the accepted order |
| Intended action | Create the initial delivery job for OR-1042, approved version 1 | Distinguishes this action from a later amendment or genuinely separate job |
| Technical attempt | Attempt 1, attempt 2 | Explains retries without turning each attempt into new business work |
Store an opaque request key against the intended action before sending it. Reuse that key for another attempt at the unchanged action. Do not generate a fresh key merely because a worker restarted or someone pressed a retry button.
Also retain the destination account, operation, approved input version and a protected comparison value for the payload. An identifier meaningful in one account or operation should not accidentally deduplicate work in another. Store only the evidence needed, under appropriate access controls; avoid embedding customer details in request keys.
This is a proposed application design, not a guarantee supplied by a queue or a CRM.
Read the destination’s contract, not the retry checkbox
Provider behaviour is specific. For example, Stripe documents that a saved idempotent result is returned for subsequent uses of the same key, including an initial server-error result. It checks for changed parameters. Keys can be removed after at least 24 hours; using a key after pruning can initiate a new request. Results are saved only once endpoint execution begins. Stripe: Idempotent requests
The lesson is not to copy Stripe’s retention period into every integration. Ask the supplier to document the actual endpoint’s guarantees:
- Does the create operation support idempotency, a unique external reference, or neither?
- What account and operation scope does the protection cover?
- What happens if the same key arrives concurrently or with different inputs?
- How long does the protection last, and what happens after it expires?
- Can the system retrieve the outcome using the original reference?
A setting called “retry three times” answers none of these questions.
A recovery decision table for the buyer
The following table is an original proposed acceptance model for this article. It is not a proprietary certification or a claim that a specific product already implements it.
| What the workflow can establish | Permitted next step | What must not happen |
|---|---|---|
| The exact action succeeded; destination identity is known | Record and link the established result; check current approval and the outcome of each unfinished later action before proceeding | Create the same job again or treat confirmed creation as authority for a new side effect |
| The outcome is unknown, but the unchanged request has a verified, still-valid idempotency contract and continuing authority | Make a bounded retry with the original key, following the endpoint’s retry rules | Substitute a fresh key |
| The outcome is unknown and duplicate protection is absent, expired or uncertain | Look up the original action; escalate if evidence remains insufficient | Treat silence or a weak “not found” search as proof of failure |
| Authoritative evidence establishes that no effect occurred and none remains pending | Retry only as allowed by the contract and current business approval | Assume every error response establishes this condition |
| Inputs or approval have changed | Hold for reconciliation and a new decision about amendment, cancellation or replacement | Reuse the old key with a changed payload, or create a replacement while the original remains unresolved |
Apply the changed-input or changed-approval rule before any new write, including when the original creation is already confirmed. Recording what happened does not authorise what happens next. Each later side effect needs its own current approval and recovery decision.
“Not found” needs interpretation. A search index might lag behind a newly created record; a pending request might still finish. Ask what the destination’s lookup can actually prove before treating absence as permission to create.
A second authorised delivery for the same customer is not automatically a duplicate. The identity represents an intended action, not merely matching customer names or identical form fields.
Walk through the lost-response case
Use the hypothetical order to inspect the supplier’s design:
- Record intent. Persist the approved initial-delivery action for OR-1042 and its request key.
- Send. Submit the create request to the destination.
- Lose the response. The destination has created DJ-781, but the sender records “outcome unknown.”
- Resolve. Under a verified valid contract, retry the unchanged request with the same key; alternatively, retrieve the authoritative result by its original reference.
- Confirm locally. Link DJ-781 to OR-1042. Mark creation confirmed only when the evidence supports it.
- Recheck before continuing. Confirm that each remaining notification or CRM action still has current approval for its intended inputs. Hold changed or revoked work for a named owner. For an approved action, establish whether it already succeeded, remains uncertain or has not been attempted, then apply that action’s own duplicate controls.
The last step matters. Preventing a second delivery job does not automatically prevent a second customer message. Each consequential side effect needs its own completion record and recovery rule. If a notification was sent but its acknowledgement was lost, resolve that uncertain outcome before sending it again.
Here is the same design as a compact state diagram:
The following table gives the complete text equivalent of the diagram. A dash denotes an unlabelled transition.
| Current state | Observed condition | Next state |
|---|---|---|
| Approved action and durable identity | — | Send attempt |
| Send attempt | Confirmed | Record result |
| Send attempt | Uncertain | Resolve outcome |
| Send attempt | Proven no effect | Check endpoint rules and current approval |
| Resolve outcome | Matched result | Record result |
| Resolve outcome | Unresolved | Hold for named owner |
| Record result | — | Check each remaining action |
| Check each remaining action | Changed or revoked approval | Hold for named owner |
| Check each remaining action | Approved but outcome uncertain | Resolve outcome |
| Check each remaining action | Approved and safe to attempt | Check endpoint rules and current approval |
A timeout must not jump directly to “proven no effect.” A confirmed or matched result allows the workflow to record that fact; it does not bypass the approval and recovery checks for later actions. Completed later actions are skipped, and unresolved items stay with a named owner. The diagram's endpoint checks apply to the particular action being considered, using that action's identity.
For this example, the operations owner is responsible for unresolved delivery creation. The handoff contains the order and action identities, attempt times, last response, lookup evidence and allowed next actions. Agree a response deadline suited to the business consequence. Escalation does not grant permission to create a second job.
Two workers and a manual rerun are part of the problem
A design that checks “have we processed this?” and then writes can still fail if two workers pass the check before either records completion.
Require evidence that the destination’s duplicate protection handles concurrent requests. On the sending side, claim work through an atomic state change or equivalent concurrency control. But do not mistake a local lock for protection over a remote system: a worker can fail after the remote effect and before the local result is saved. Amazon’s idempotency design discussion explicitly addresses atomic recording of identity and mutation on the service side. Amazon Builders’ Library
Duplicate triggers are not necessarily extraordinary. Amazon SQS standard queues can deliver a message again, and AWS advises designing consumers accordingly. That is a documented queue behaviour, not evidence of a fault rate in your business. Amazon SQS: At-least-once delivery
A manual recovery screen should use the same action identity and rules as the automated worker. A separate “force create” route can defeat otherwise careful recovery controls.
Ask for failure evidence before accepting delivery
Give the implementer this test set for a sandbox using synthetic records. These are proposed acceptance tests, not tests already passed by your supplier.
| Test | Evidence to inspect |
|---|---|
| Destination succeeds; response is lost | One destination job, original key retained, local state eventually linked to that job |
| Two workers attempt the same action together | Destination result and logs show no duplicate business creation |
| Worker crashes after remote success, before saving locally | Restart resolves the original action without a new identity |
| Operator manually reruns an uncertain item | The recovery interface follows the same identity and authority rules |
| Same key is presented with changed input | No silent mutation or second creation; explicit conflict handling |
| Duplicate protection has expired | Reconciliation or a hold occurs before any new write |
| Creation succeeds; later work encounters uncertain delivery or changed approval | Test both a later effect whose acknowledgement is lost and approval changed or revoked before the next effect. Confirmed effects are skipped, uncertain effects are reconciled, and unauthorised effects are held |
| Evidence remains ambiguous | A named owner receives a bounded handoff; the system neither loops indefinitely nor reports success |
Inspect destination records as well as orchestration logs. A green dashboard can prove a workflow finished without proving that it created exactly one correct business outcome.
Webhook receipt is another boundary. Stripe documents duplicate deliveries and does not guarantee event order. Its guidance includes tracking processed event identities. Stripe: Webhooks For your design, receiving an event twice must not automatically mean fulfilling an order twice; confirm the current authorised business state before acting.
Where this model stops
The diagram is a design aid, not production code. It does not establish a cross-system “exactly once” guarantee. Key retention, concurrent processing, transaction boundaries, lookup consistency and downstream effects must be verified in the chosen stack.
A genuinely non-idempotent destination with no reliable lookup may require manual handling or a different integration design. A cancellation or refund is also a new consequential action with its own authority and failure modes, not proof that the original effect never happened.
An AI agent can help prepare an exception brief. It should not guess whether a delivery job exists or infer permission to repeat a write from a confident explanation. Keep those decisions tied to system evidence and explicit authority.
The useful acceptance question is therefore not “Does the automation retry?” It is:
After an uncertain result, can we explain what happened, recover without adding unintended work, and show who owns the unresolved cases?
If that answer is unclear, bring the workflow and its failure boundary to netlinkE’s Intelligent Workflow and Automation Infrastructure service. Request an Architecture Assessment to discuss the operating constraint. The initial submission starts a fit review; scope, fees and outputs are agreed separately. Share a sanitised example, not confidential customer records.
Sources and method
The provider statements above were checked on 12 September 2026. The scenario, decision table, diagram and supplier tests are original illustrative analysis grounded in the cited technical principles. No live provider experiment, customer result, industry statistic or firsthand incident is claimed. The historical editorial archive label does not imply the article or its research existed on that date.
