How to Build and Validate Webhooks: Retries, Signatures, and Idempotency

Overview

If you only remember one thing, remember this: a webhook consumer should assume every delivery may be duplicated, delayed, tampered with, or partially processed. A robust design does not try to make webhooks perfect. It makes them safe to receive repeatedly and easy to audit when something goes wrong.

A webhook is simply an outbound HTTP callback triggered by an event in another system. For example, a billing platform might POST an event when an invoice is paid. Your application exposes an endpoint, the provider sends the event, and your backend reacts.

The operational challenge is that webhooks are usually delivered with at-least-once semantics. That means the sender aims to deliver the event, but duplicates are possible. Some providers also do not guarantee strict ordering. Even when a sender documents reliable delivery, your own infrastructure can still introduce failure modes: slow database writes, timeouts, worker crashes, race conditions, and deploys during peak traffic.

A good webhook design usually includes five baseline requirements:

• Authentication of the sender through signature verification or another trusted mechanism.
• Validation of the payload so malformed or incomplete events fail safely.
• Fast acknowledgment so the sender does not keep retrying while your app performs slow work.
• Idempotent processing so duplicate deliveries do not create duplicate effects.
• Observability through logs, event IDs, timestamps, and replay workflows.

These patterns complement broader API security. If you want a high-level comparison of common auth approaches, see REST API Authentication Methods Compared: API Keys, OAuth, JWT, and Sessions. Webhook security is a narrower problem: you are not authenticating a human user, you are verifying that a machine-generated request really came from the expected provider and was not modified in transit.

Core framework

This section gives you a repeatable framework you can apply to most webhook integrations, regardless of language or provider.

1. Preserve the raw request body

Signature verification often depends on the exact raw bytes of the incoming request body. If your framework parses JSON before verification and then reserializes it, even small formatting changes can break the signature check.

As a rule, capture the raw body before any mutation. Then verify the signature against that exact payload. Only after verification should you parse the JSON into application objects.

This detail causes many avoidable bugs. It is especially common in Express, Fastify, Django, Flask, and serverless runtimes where body parsing middleware runs early. If your signature checks fail unexpectedly, the first thing to inspect is whether you still have access to the raw payload.

2. Verify the webhook signature

The most common pattern is an HMAC signature. The provider and your application share a secret. The provider computes a hash over the payload, often including a timestamp, and sends the result in a header. Your server recomputes the same hash and compares it using a constant-time comparison function.

A typical verification flow looks like this:

1. Read the raw request body.
2. Read the signature header and, if present, a timestamp header.
3. Compute the expected HMAC using your stored secret.
4. Compare the expected and received signatures using a timing-safe comparison.
5. Reject requests with missing, invalid, or stale signatures.

Why include a timestamp? It reduces replay risk. If an attacker captures a valid signed request, a timestamp window lets you reject old deliveries. Your system should allow some clock drift, but not accept arbitrarily old messages.

If you work with JWT-based webhook envelopes or signed tokens in related systems, the verification mindset is similar: validate integrity first, then inspect claims. For adjacent background reading, see JWT Decoder Guide: How to Read Tokens Safely and Validate Claims.

3. Validate required event fields

Signature verification tells you the request likely came from the sender. It does not tell you the payload is complete or usable. After authentication, validate a small set of required fields before doing any business logic. These commonly include:

• Event ID
• Event type
• Creation timestamp
• Object or resource ID
• Account, tenant, or environment identifier

Schema validation is useful here. Keep it strict enough to catch unexpected shapes, but avoid overfitting to fields you do not actually use. Providers evolve payloads over time. Your consumer should fail on missing critical fields, not on harmless additional fields.

4. Acknowledge quickly, process asynchronously

The safest webhook handler is usually thin. Verify the request, perform minimal validation, store the event durably, and return a success response as quickly as possible. Then process the event in a queue or background worker.

This pattern improves reliability in several ways:

• Your endpoint stays fast, reducing provider retries caused by timeouts.
• Business logic can be retried independently of delivery acceptance.
• Expensive downstream calls do not block the HTTP response.
• You get a clearer audit trail of accepted versus fully processed events.

A practical sequence is:

1. Receive request.
2. Verify signature.
3. Extract event ID and type.
4. Write the raw payload and metadata to an inbox table or queue.
5. Return 2xx.
6. Let a worker perform the actual side effects.

This is often called an inbox pattern on the receiving side. It is simple, durable, and easier to debug than performing everything inline.

5. Design for idempotency from the start

Idempotency means processing the same event more than once produces the same final result as processing it once. In webhook systems, this is not optional. It is a core safety property.

There are two common approaches:

• Track event IDs: store each provider event ID in a table with a unique constraint. If the same ID appears again, skip or short-circuit processing.
• Enforce business-level uniqueness: for example, only create one invoice record per external invoice ID, regardless of how many times the event is delivered.

In practice, the best systems use both. Event-level deduplication prevents repeat work. Business-level constraints protect you if the provider changes event IDs during replays or emits multiple event types for the same underlying action.

6. Plan retry behavior on both sides

Webhook retries happen in two places: the sender may retry delivery, and your own worker may retry processing after a transient failure. Treat them separately.

For incoming delivery retries, your HTTP endpoint should return status codes intentionally:

• 2xx: request accepted. Use this only when you have durably recorded the event or safely completed processing.
• 4xx: request is invalid and should not succeed by retrying, such as a bad signature or missing required header.
• 5xx: temporary problem on your side; the sender may retry later.

For internal worker retries, use bounded backoff and clear failure states. Not every error is retryable. A dead database connection may be retryable. A permanently invalid payload is not.

It helps to classify failures into three buckets: reject, retry, and escalate. This keeps your queue from filling with events that will never succeed.

7. Log for audit and replay

Webhook debugging is much easier when every delivery has traceable metadata. At minimum, log:

• Provider name
• Delivery ID, if available
• Event ID
• Event type
• Signature verification result
• Received timestamp
• Processing state
• Error category and message

Avoid logging secrets or sensitive payload fields unnecessarily. Store the raw payload only if you have a justified retention policy and understand the data sensitivity involved.

Practical examples

Here is a practical baseline you can adapt in Node.js or Python.

Example 1: Signature verification flow in Node.js

import crypto from 'node:crypto';

function verifySignature(rawBody, signatureHeader, timestampHeader, secret) {
  if (!signatureHeader || !timestampHeader) return false;

  const maxAgeSeconds = 300;
  const now = Math.floor(Date.now() / 1000);
  const timestamp = Number(timestampHeader);

  if (!Number.isFinite(timestamp)) return false;
  if (Math.abs(now - timestamp) > maxAgeSeconds) return false;

  const signedPayload = `${timestamp}.${rawBody}`;
  const expected = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  const received = signatureHeader.trim();
  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(received, 'utf8');

  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(a, b);
}

The exact header format varies by provider. Some include multiple signatures, version prefixes, or comma-separated values. The durable idea is the same: verify against the raw payload, enforce a timestamp tolerance, and compare safely.

Example 2: Idempotent event storage in SQL

CREATE TABLE webhook_events (
  id BIGSERIAL PRIMARY KEY,
  provider TEXT NOT NULL,
  event_id TEXT NOT NULL,
  event_type TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status TEXT NOT NULL DEFAULT 'received',
  payload JSONB NOT NULL,
  UNIQUE (provider, event_id)
);

With this schema, your handler can attempt an insert as soon as verification succeeds. If the unique constraint fails, you already know the event was seen before. That duplicate should usually return a 2xx response unless your provider specifies otherwise.

If you want to keep your SQL readable as these inbox tables grow more complex, a formatter can help keep team conventions consistent. See SQL Formatter Guide: How to Write More Readable Queries and Team Standards.

Example 3: Thin handler plus background worker

// Pseudocode
POST /webhooks/provider
  rawBody = readRawBody(request)
  if !verifySignature(rawBody, headers.signature, headers.timestamp, secret)
    return 400

  event = JSON.parse(rawBody)
  validateRequiredFields(event)

  inserted = saveToInbox(event)
  if inserted is duplicate
    return 200

  enqueue(event.event_id)
  return 200

The worker then performs the business logic:

worker(job)
  event = loadInboxEvent(job.event_id)
  if event.status == 'processed'
    return

  begin transaction
    applyBusinessChangeIdempotently(event)
    markEventProcessed(event.event_id)
  commit

This separation makes retries easier to reason about. Your HTTP endpoint cares about safe acceptance. Your worker cares about safe completion.

Example 4: Python verification helper

import hmac
import hashlib
import time


def verify_signature(raw_body: bytes, signature: str, timestamp: str, secret: str) -> bool:
    if not signature or not timestamp:
        return False

    try:
        ts = int(timestamp)
    except ValueError:
        return False

    if abs(int(time.time()) - ts) > 300:
        return False

    signed_payload = f"{ts}.".encode("utf-8") + raw_body
    expected = hmac.new(
        secret.encode("utf-8"),
        signed_payload,
        hashlib.sha256,
    ).hexdigest()

    return hmac.compare_digest(expected, signature.strip())

If you are building this in Python, isolate the environment and dependencies cleanly so deploys remain predictable. Python Virtual Environments Explained: venv, pipx, Poetry, and uv is a useful companion if your stack is still evolving.

Example 5: Business-level idempotency

Suppose you receive an event saying an external order was paid. Event-level deduplication is good, but the real safeguard may be a unique constraint on external_order_id in your payments table. That way, even if different event types or replay mechanisms hit your system, you still cannot create two payment records for the same external transaction.

Think of idempotency as layers, not a single flag.

Common mistakes

The fastest way to improve webhook reliability is to avoid a small set of recurring mistakes.

Verifying the parsed JSON instead of the raw body

This is one of the most common causes of failed webhook signature verification. Any change in whitespace, key ordering, or encoding may break the expected digest.

Doing too much work before returning 2xx

If your endpoint sends emails, writes to multiple services, and calls external APIs before acknowledging receipt, you are inviting retries and duplicate effects. Keep the receiver thin.

Assuming events arrive exactly once

They often do not. A provider retry plus your own partial failure is enough to produce duplicates. Build the deduplication path before your first production integration.

Trusting IP allowlists alone

IP filtering can help, but it is rarely enough by itself. Providers may change delivery ranges, use proxies, or document shared infrastructure. A cryptographic signature is generally the stronger primary control.

Returning 200 before durable storage

If you acknowledge an event before it is stored or safely processed, a crash between those steps can lose the event permanently. Return success only after you can prove you accepted responsibility for it.

Ignoring replay and clock issues

If you validate signatures but accept requests with any timestamp, you leave a replay gap. If your system clock is badly skewed, you may reject valid events. Monitor both.

Skipping documentation

Webhook integrations age better when you document headers, retry semantics, signing rules, sample payloads, and error handling. A short internal runbook saves time during incidents. For teams that maintain docs alongside code, Markdown Formatter and Linter Guide for Docs, READMEs, and Teams can help keep that documentation consistent.

When to revisit

Webhook integrations are not usually set-and-forget systems. Revisit your implementation when any of the underlying assumptions change.

• The provider changes its signing method, header format, hashing algorithm, or timestamp rules.
• You add new event types with different ordering or business constraints.
• Your throughput changes and synchronous handlers start timing out.
• You move infrastructure such as proxies, API gateways, serverless runtimes, or body parsing middleware.
• You discover duplicates or missed events in logs, billing, or support reports.
• Your compliance or retention requirements change, affecting what payload data you can store.

A practical review checklist looks like this:

1. Confirm you still have access to the raw request body in the current stack.
2. Rotate and test webhook secrets safely.
3. Verify the timestamp tolerance still makes sense for your environment.
4. Audit unique constraints and deduplication keys.
5. Replay a known event in staging and confirm the outcome is idempotent.
6. Review logs for events stuck in retry loops.
7. Check that your runbook explains how to inspect, replay, and recover failed deliveries.

If you are troubleshooting a browser-based integration around webhook dashboards or admin tools, cross-origin configuration sometimes becomes part of the path. For that adjacent problem space, CORS Errors Explained: Fix Common Cross-Origin Problems Fast is worth bookmarking.

To put this into action, aim for a minimal production standard: verify signatures against the raw body, store events durably, respond quickly, process asynchronously, and enforce idempotency at both the event and business levels. Those five habits solve most webhook problems before they become incident reports. They also make your integration easier to revisit when a provider updates its docs, your traffic grows, or your team inherits a webhook endpoint that no one has touched in months.