Webhook retries and replay

Your service will be unavailable at some point, and a webhook will arrive during it. Retries are what make that survivable rather than a lost event: a delivery is attempted five more times over roughly fifteen hours before it is given up on, and even then it can be replayed by hand once you have fixed whatever broke.

The retry schedule

Six attempts in total — the first, then five retries at increasing intervals. The spacing is deliberate: quick enough to recover from a brief blip, spread widely enough that a longer outage is still covered.

Attempt	When	Covers
`1`	Immediately	The normal case
`2`	After 1 minute	A restart or a momentary blip
`3`	After 5 minutes	A short outage
`4`	After 30 minutes	A deploy that went wrong
`5`	After 2 hours	A longer incident
`6`	After 12 hours	Something that needed a person

Total coverage is about fifteen hours from the first attempt. After the sixth, the delivery is marked failed and no further automatic attempts are made — it stays in the record and can still be replayed manually.

What triggers a retry

The response code your endpoint returns decides whether we try again. Getting this wrong in either direction is costly: a 4xx on a transient failure loses the event, and a 5xx on a permanent one produces six pointless attempts.

Your response	Retried	Meaning
`2xx`	No	Received. The delivery is marked succeeded.
`408 Request Timeout`	Yes	Treated as transient.
`425 Too Early`	Yes	Treated as transient.
`429 Too Many Requests`	Yes	You are rate limiting us. We back off.
`5xx`	Yes	Your service is having a problem that may clear.
`Other 4xx`	No	Your service has decided it will never accept this. Retrying is pointless.
`Timeout after 10 seconds`	Yes	No response arrived in time.
`Connection or TLS failure`	Yes	We could not reach you at all.

The ten-second budget is the one people trip over. Do the minimum synchronously — verify the signature, enqueue the payload, return 200 — and do the real work afterwards. A handler that charges a card inline will eventually exceed ten seconds, get retried, and charge it twice.

Delivery statuses

status	Meaning
`queued`	Accepted and waiting to be sent. Normally momentary.
`processing`	An attempt is in flight right now.
`retrying`	An attempt failed and another is scheduled. next_retry_at says when.
`succeeded`	Your endpoint returned 2xx. Done.
`failed`	All six attempts were used, or a non-retryable status was returned.
`cancelled`	Not delivered — the destination was disabled or deleted before it could be sent.

cancelled is worth noticing during an incident. Disabling a destination to stop the noise means the events that fire while it is off are not queued for later — they are dropped. Prefer fixing the handler over disabling the destination if the events matter.

Inspecting a delivery

Every attempt is recorded with what your endpoint returned, including an excerpt of the response body — which is often enough to diagnose the problem without touching your own logs.

One delivery, with its attempt history

GET /webhook-deliveries/0ec5af80-1125-43c8-9b72-04706b6da54a

{
  "data": {
    "id": "0ec5af80-1125-43c8-9b72-04706b6da54a",
    "event_id": "d59cf3d3-a20c-46bf-b187-c27cd6ff51e2",
    "event_type": "timer.completed",
    "timer_id": "550e8400-e29b-41d4-a716-446655440000",
    "destination_name": "Order automation",
    "webhook_rule_id": "264f4933-c839-47b5-a7ba-55a73ed56774",
    "replay_of_delivery_id": null,
    "status": "retrying",
    "attempt_count": 3,
    "response_status": 502,
    "response_excerpt": "<html><head><title>502 Bad Gateway</title>...",
    "last_error": "Upstream returned 502",
    "next_retry_at": "2030-01-01T15:30:00.000Z",
    "completed_at": null,
    "attempts": [
      { "attempt_number": 1, "response_status": 502 },
      { "attempt_number": 2, "response_status": 502 },
      { "attempt_number": 3, "response_status": 502 }
    ]
  },
  "request_id": "7b82b7f7-4d13-497b-9f20-58d46fd7a510"
}

response_excerpt is the underrated field here. A 502 with an nginx error page in it says something quite different from a 502 with your own JSON in it, and the excerpt is where you see which.

Finding what went wrong

Two queries answer most webhook questions. Run the first when something did not happen; run the second to see whether it is still coming.

Two useful queries

# What failed, and why
curl "https://countdownshare.com/api/v1/webhook-deliveries?status=failed&limit=100" \
  -H "Authorization: Bearer $COUNTDOWNSHARE_API_KEY"

# What is still in flight
curl "https://countdownshare.com/api/v1/webhook-deliveries?status=retrying" \
  -H "Authorization: Bearer $COUNTDOWNSHARE_API_KEY"

The diagnostic fork

If a delivery record exists, the rule matched and the problem is on the receiving side — read response_status and last_error. If no record exists, the rule never fired, which is a different investigation entirely and is covered under webhook rules.

Poll ?status=failed on a schedule and alert on anything non-empty. Failed deliveries are silent by nature — nobody notices the reservation that was never released until a customer does.

Replaying a delivery

Once you have fixed the handler, replay sends the same event again. It creates a new delivery carrying the same event ID — so a replayed delivery has its own delivery_id and a replay_of_delivery_id pointing back at the original.

Replay

curl -X POST \
  https://countdownshare.com/api/v1/webhook-deliveries/$DELIVERY_ID/replay \
  -H "Authorization: Bearer $COUNTDOWNSHARE_API_KEY"

# 202 Accepted — queued as a NEW delivery carrying the SAME event id.
# The rule is not re-evaluated; the original event is simply resent.

The rule is not re-evaluated. Replay resends the event exactly as it was recorded, so the data block still shows the remaining time at the moment the rule originally matched — not the timer's state now. That is what you want when reprocessing history, and worth knowing if you were expecting current values.

Because the event ID is unchanged, a handler that deduplicates on it will ignore a replay of something it already processed. That is correct behaviour, and it means replaying broadly is safe. See events and payload.

Replaying in bulk after an outage

The recovery pattern after a handler bug: fix it, deploy, confirm with a test delivery, then replay everything that failed while it was broken.

Replay a window of failures

// After fixing a handler bug, replay everything that failed
// while it was broken. Deduplication on your side makes this safe.
async function replayFailedSince(since) {
  let replayed = 0;

  for await (const delivery of allDeliveries({ status: "failed" })) {
    if (new Date(delivery.created_at) < since) break;   // list is newest-first

    await call(`/webhook-deliveries/${delivery.id}/replay`, { method: "POST" });
    replayed++;
  }

  return replayed;
}

Confirm the fix with POST /webhook-destinations/{destination_id}/test before replaying in bulk. Replaying into a still-broken handler burns through the failures again and tells you nothing new.

Building a handler that rarely fails

Acknowledge before working
Verify, enqueue, return 200. The ten-second budget is for accepting the delivery, not for completing the job.
Return 5xx for transient problems
A database that is momentarily down should produce a 5xx so the delivery is retried, not a 400 that discards it.
Deduplicate on the event id
Retries and replays both resend the same event. A unique constraint is enough.
Do not require ordering
Deliveries are independent. A "one hour left" event can arrive after "ended" if the first one was retried.
Monitor failed deliveries
Nothing else will tell you. Alert on a non-empty ?status=failed.

The ordering point catches people. Retries mean a delivery that failed can land after one that was sent later, so a handler that assumes it sees milestones in sequence will occasionally act on a stale one. Branch on what the event says, not on what you expected next.