APIs and Webhooks Without Duplicates or Data Loss

Why Duplicates Are a Normal Scenario, Not an Exception

A user clicked “Submit,” but the browser did not receive a response in time and repeated the request. A payment service sent a webhook, did not receive an acknowledgment in time, and sent it again. A worker completed an operation in an external system but restarted before persisting the result. In all three cases, the network and applications are behaving as expected: they are trying to complete the action. The error appears when the business operation is designed as if every message arrives exactly once.

RFC 9110 defines an HTTP method as idempotent when multiple identical requests have the same intended effect as a single request. GET and PUT have this semantic by standard, while POST does not have it automatically. However, the method name does not protect the business from retries: a POST endpoint that creates a lead can be made idempotent with an application-level key, while a poorly implemented PUT can still produce unnecessary side effects.

Start With a Single Operation Identifier

Before choosing a queue or database, agree on what counts as one operation. For a client request, it may be an idempotency key generated on the client. For an incoming webhook, it may be the provider’s delivery identifier or a composite key made from the source, event type, and object identifier. For an import, it may be the row number combined with the file version. The key must remain the same when the same operation is retried and change when the user genuinely creates a new one.

• Store the key together with the owner or integration scope: identical values from different clients must not conflict.
• Store a fingerprint of the significant parameters. It is better to reject a repeated key with different content than to apply new data silently.
• Define the key retention period according to business risk and the maximum redelivery window, not according to how convenient it is to clean up the table.
• Pass a correlation ID through the API, queue, and log so that the complete operation can be found instead of isolated log fragments.

The same key means a retry of the same operation, not merely a similar request.

Agentix Labs design principle

How to Make a State-Changing API Idempotent

1. 01

   Accept the Key and Validate the Request

   The client sends an idempotency key separately from the business fields. The server validates authorization and the data format, then calculates a parameter fingerprint.

2. 02

   Claim the Right to Execute

   Within a transaction, create an operation record with a unique index on the owner and key. The database’s unique constraint closes the race between two requests arriving at the same time.

3. 03

   Apply the Business Change

   Create the lead, update the order, or enqueue a command within the same transactional boundary wherever possible. The operation status should clearly distinguish processing, success, and a controlled error.

4. 04

   Store a Stable Response

   Record the response code and useful response data. A repeated request with the same key and parameters receives the stored result instead of starting the action again.

Stripe documents a similar model: the client supplies a unique key, and the server associates it with the result of the first execution and returns that result on a retry. A custom system does not need to copy another provider’s retention periods and formats, but the principle is useful: the key, parameters, and result form a single verifiable record.

Receiving a Webhook Safely: Validate, Record, Acknowledge

A webhook endpoint sits on a trust boundary. It should validate the HTTPS connection at the infrastructure level, verify the signature against the raw request body, check the allowed timestamp, event type, and source. A secret must not be passed in a URL or written to an unprotected log. After validation, it is useful to store the original payload, service headers, and delivery identifier in an incoming log—the inbox.

Heavy work is better kept outside the HTTP request. The receiver persists the event and returns a successful response, while a separate worker applies the business logic. This prevents a temporarily slow CRM or document generation process from making the provider treat the delivery as failed. In its webhook recommendations, GitHub advises checking the event type and action, using a secret and a unique delivery identifier; when a webhook is redelivered manually, the GitHub identifier remains the same, making the retry recognizable.

Transactions and an Outbox Close the Gap Between the Database and the Queue

Even a well-designed inbox does not solve the opposite problem: the application wrote an order to the database but failed before sending a command to the CRM. Or it sent the command and then rolled back the local transaction. If the database and message broker do not support a shared transaction, a direct call between them leaves a window for loss or duplication.

The transactional outbox pattern places the business change and a record of the future event in a single database transaction. A separate publisher takes unpublished records and sends them to the queue. It can also send a message more than once, so the recipient applies its own inbox and idempotent business logic. As a result, a failure delays processing but does not leave the state impossible to recover.

• Do not mark an outbox record as sent before the actual transmission.
• Do not delete an event immediately after the first attempt: store its state, attempt count, and latest error.
• If ordering matters, define the partitioning key by business object and validate the event version.
• Keep every external call outside a long-running database transaction and design it to be safely repeatable.

Retries Must Be Limited and Meaningful

A retry is useful only for a temporary error: a network interruption, timeout, rate limit, or unavailable dependency. A schema error, invalid signature, missing required identifier, or forbidden state transition will not be fixed by a tenth attempt. Such events should move to a clear state for investigation instead of being returned to the queue indefinitely.

1. 01

   Classify the Error

   Separate temporary, permanent, and ambiguous errors. For an ambiguous result, first query the state of the external operation if the provider supports it.

2. 02

   Add a Delay and Random Jitter

   Exponential backoff with jitter reduces simultaneous retry requests after a service recovers.

3. 03

   Limit the Attempts

   After a defined threshold, move the event to a failed state or dead-letter queue while preserving the reason and the ability to restart it safely.

4. 04

   Make a Manual Retry Use the Same Event

   An operator action must preserve the original delivery ID or idempotency key; otherwise, resolving the error will itself create a duplicate.

How to Detect Losses as Well as Duplicates

Deduplication answers the question, “Have we processed this event?” It does not prove that every event arrived in the first place. Critical integrations therefore complement webhooks with periodic reconciliation: they request objects from the source that changed after the last confirmed cursor and compare them with the local log. The cursor is updated only after the range has been processed successfully; otherwise, some changes may be skipped.

• The number of received, unique, duplicate, successful, and failed events for each source.
• The age of the oldest unprocessed event and the queue size.
• The share of temporary errors and transitions to manual review, without publishing invented targets.
• Search by correlation ID from the incoming request to the business object and outgoing action.
• An alert when reconciliation reveals a discrepancy between the source and the local system.

The log must explain the state to a person: what was received, why something was skipped as a duplicate, which attempt is running, which external action has already been confirmed, and what may be retried safely. Masking tokens, personal data, and payment details must be designed before detailed logging is enabled.

What This Architecture Looks Like in a Real Product

In QRush, the event layer receives updates from the P2C flow, while processing rules account for the lead identifier, account state, limits, and protection against repeated processing. Isolated workers and shared operational state help keep trading sessions separate. This case shows why deduplication is part of the operation model rather than a row filter applied before an external call.

In TransferBot, a request passes through a Telegram dialog, an orchestrator, PostgreSQL, and a browser worker. The system must preserve the state of a long-running workflow, avoid starting a second active transfer, and correctly handle waiting, cancellation, an error, or expiration. Both projects support the same general approach: a central operation record, explicit statuses, isolated execution, and controlled retries.

Checklist Before Launching an Integration

• For every state-changing operation, the business identifier, key owner, and retention period are defined.
• Identical simultaneous requests encounter a unique constraint rather than passing through separate SELECT and INSERT operations.
• A repeated key with changed parameters is rejected and recorded for diagnostics.
• The webhook validates the signature against the raw body, checks the event type and delivery ID, then persists the event quickly.
• Business processing runs in a worker and can be retried safely after a restart.
• The data change and the creation of an outgoing event are linked through a transaction and an outbox.
• Errors are classified; retries are limited, and a manual restart preserves the original key.
• Periodic reconciliation with the source, correlation ID search, and a clear status for the operator are available.
• Tests reproduce a duplicate request, a concurrent request, a timeout after external success, a repeated webhook, and a missed delivery.

A practical way to begin implementation is to map one critical process, such as the path of a lead from a form to the CRM or the path of a payment status to result delivery. Mark the data owners, transaction boundaries, possible retries, points of unknown outcome, and the reconciliation method on the diagram. This makes it clear where a unique index is enough, where an inbox or outbox is required, and where the business process itself needs to change.