Inspect and replay deliveries

Every attempt to deliver an event is recorded against its endpoint. When a receiver was down, slow or wrong, the log tells you what happened and a replay re-drives the delivery on a fresh attempt budget.

Delivery states#

Delivery rows are a summary: delivery_id, event_id, event_type, status, attempt_number, next_retry_at, last_response_status, created_at, updated_at and body_expires_at. They never carry the body, the URL or a secret. Timestamps on delivery rows are epoch milliseconds, as the API returns them.

Delivery rows are kept for 90 days and then expire, but the event body each one carried is kept only 7 days, because it holds message text. Replay within 7 days: after that the row still shows the status and attempts, body_expires_at has passed, and a replay is refused with replay_expired (409).

List an endpoint's deliveries#

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

const page = await client.webhooks.deliveries.list(process.env.WEBHOOK_ID!, { limit: 10 })
for (const d of page.deliveries) {
  console.log(d.delivery_id, d.event_type, d.status, 'attempt', d.attempt_number, 'last status', d.last_response_status)
}
console.log(page.deliveries.length, 'deliveries, next_cursor:', page.next_cursor)

Find what failed#

The summary route aggregates the failing rows server-side: by default the last 24 hours and the statuses dlq, failed, blocked and retrying. truncated is true when the examined-row cap was hit, so older failures may be missing; narrow since in that case.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

const failing = await client.webhooks.deliveries.summary(process.env.WEBHOOK_ID!, {
  since: Date.now() - 24 * 60 * 60 * 1000,
  status: ['dlq', 'failed', 'blocked', 'retrying'],
})
for (const d of failing.deliveries) console.log(d.delivery_id, d.status, 'last status', d.last_response_status)
console.log(failing.deliveries.length, 'failing deliveries', failing.truncated ? '(truncated)' : '')

Replay one delivery#

A replay resets the delivery to pending on a fresh attempt budget and re-sends the same event, with the same event id and a fresh signature timestamp. A delivery that already succeeded is refused with replay_conflict (409) unless you force it; so is one that is mid-attempt, which you can retry shortly. A delivery older than 7 days is refused with replay_expired (409): its event body is no longer kept.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

// force re-sends even a delivered event; the receiver sees the same event id again.
const result = await client.webhooks.deliveries.replay(process.env.WEBHOOK_ID!, process.env.DELIVERY_ID!, { force: true })
console.log('replay', result.delivery_id, result.status)

Replay everything in a window#

After an outage, replay every failing delivery between two times in one call. The batch is bounded and safe to run again: it returns matched, enqueued and skipped counts and a next_cursor when there is more, and in-flight deliveries are skipped. The default statuses are dlq and failed; including delivered requires force.

import { HookChat } from '@hookchat/node'

const client = new HookChat({ apiKey: process.env.HOOKCHAT_API_KEY!, baseUrl: process.env.HOOKCHAT_BASE_URL! })

let cursor: string | undefined
do {
  const result = await client.webhooks.deliveries.replayAll(process.env.WEBHOOK_ID!, {
    from: Date.now() - 24 * 60 * 60 * 1000,
    status: ['dlq', 'failed'],
    cursor,
  })
  console.log('matched', result.matched, 'enqueued', result.enqueued, 'skipped', result.skipped)
  cursor = result.next_cursor ?? undefined
} while (cursor)

The retry schedule and giving up#

A delivery that fails retryably is retried on a fixed backoff schedule: seven legs summing to about 33 hours, so eight attempts in total before it gives up.

attempt 1 fails  wait 5s    attempt 2
attempt 2 fails  wait 30s   attempt 3
attempt 3 fails  wait 5m    attempt 4
attempt 4 fails  wait 30m   attempt 5
attempt 5 fails  wait 2h    attempt 6
attempt 6 fails  wait 6h    attempt 7
attempt 7 fails  wait 24h   attempt 8
attempt 8 fails  gives up (status dlq)

• Retried: 408, 429 and 503 (a Retry-After up to 24 hours can make the wait longer, never shorter), any other 5xx, transport errors and timeouts.
• Final, never retried: a 2xx succeeds; any other 4xx is a client error to fix on your side (failed); an SSRF block, a redirect or a bad scheme is refused outright (blocked).

hookchat webhooks update "$WEBHOOK_ID" --enabled

Next#