Webhook Architecture at Scale: Building Reliable Event-Driven Systems That Never Miss an Event

"Our payment provider says the payment succeeded, but our database never updated."

"Customers received three confirmation emails for the same order."

"Our webhook endpoint returned 500 for five minutes, and we lost thousands of events."

These are not unusual production incidents. They are what happens when webhook systems are designed as simple HTTP endpoints instead of critical infrastructure.

A webhook isn't "just an API endpoint." It's the entry point of your event-driven architecture. It receives events from systems you don't control, at times you don't choose, in volumes you can't always predict.

Whether you're integrating with Stripe, GitHub, Slack, Shopify, Twilio, PayPal, Telebirr, or Chapa — the same architectural principles apply.

---

Table of Contents

1. What Is a Webhook?
2. Polling vs Webhooks
3. The Webhook Lifecycle
4. The Biggest Mistake Developers Make
5. The Golden Rule
6. Step 1 — Verify Authenticity
7. Step 2 — Validate the Payload
8. Step 3 — Store the Raw Event
9. Step 4 — Queue the Event
10. Idempotency Is Non-Negotiable
11. Event Ordering
12. Retry Strategies
13. Dead Letter Queues
14. Horizontal Scaling
15. Worker Architecture
16. Observability
17. Security Best Practices
18. Multi-Provider Architecture
19. Event Versioning
20. Common Mistakes
21. Production Architecture Example
22. Production Readiness Checklist

---

What Is a Webhook?

A webhook is an HTTP callback. Instead of your application repeatedly asking another system whether something has changed, that system pushes an event to you the moment something important happens.

Business event	Typical webhook
Payment completed	payment_intent.succeeded
Invoice paid	invoice.paid
User subscribed	customer.subscription.created
Repository updated	push
SMS delivered	message.status
Order shipped	fulfillment.create
Refund processed	charge.refunded

A webhook essentially says: "Something happened. Here's the information."

---

Polling vs Webhooks

Imagine checking your mailbox every minute.

Did I receive mail? No. Did I receive mail? No. Did I receive mail? No.

That's polling.

Now imagine the postal worker ringing your doorbell the moment a package arrives. That's a webhook.

	Polling	Webhooks
Latency	Seconds to minutes	Near real-time
Cost	High (mostly empty requests)	Low
Complexity	Simple client	Requires reliable receiver
Failure mode	Delayed data	Lost events if endpoint is down

Polling wastes resources. Webhooks reduce unnecessary requests and deliver events faster — but they shift reliability responsibility onto you.

---

The Webhook Lifecycle

Provider
   │  business event occurs
   ▼
Webhook created
   │  HTTPS request
   ▼
Load balancer
   ▼
Webhook API
   ├─ authentication
   ├─ validation
   └─ persistence
   ▼
Queue
   ▼
Workers
   ▼
Business logic ──► Database ──► Notifications

Notice something important: business logic doesn't execute immediately.

---

The Biggest Mistake Developers Make

Most applications process everything directly inside the webhook endpoint:

Receive webhook → update DB → send email → generate invoice
→ update inventory → call third-party APIs → return 200

Looks reasonable. Until production.

Suppose sending an email takes five seconds. Stripe expects a response within seconds. If your endpoint times out:

• Stripe retries.
• The same event arrives again.
• Your customer receives duplicate emails.
• Inventory updates twice.
• Chaos begins.

---

The Golden Rule

A webhook endpoint should do only four things: authenticate, validate, persist, return.

Everything else belongs in asynchronous processing.

❌  Webhook → Business Logic

✅  Webhook → Database → Message Queue → Workers → Business Logic

The webhook becomes extremely fast — usually under 100 milliseconds.

---

Step 1 — Verify Authenticity

Never trust an incoming request simply because it hit your webhook URL. An attacker can trivially send:

POST /webhooks/stripe
Content-Type: application/json

{ "payment": "successful" }

If your application blindly accepts that payload, you've shipped a serious vulnerability.

Most providers sign requests with a shared secret — Stripe signatures, GitHub X-Hub-Signature-256, Slack signing secrets, Shopify HMAC.

import { createHmac, timingSafeEqual } from 'crypto';

export function verifySignature(rawBody: string, header: string, secret: string) {
  const expected = createHmac('sha256', secret).update(rawBody).digest('hex');
  const a = Buffer.from(header);
  const b = Buffer.from(expected);
  return a.length === b.length && timingSafeEqual(a, b);
}

Two rules that catch teams out:

1. Sign the raw body, not the parsed JSON. Re-serializing changes bytes and breaks the HMAC.
2. Compare in constant time. === leaks timing information.

---

Step 2 — Validate the Payload

Authentication proves who sent the request. Validation proves the data is usable.

Verify required fields, event ID, event type, timestamp, schema version, and resource identifiers. Reject malformed requests early — before they poison your queue.

const WebhookEvent = z.object({
  id: z.string().min(1),
  type: z.string().min(1),
  created: z.number().int(),
  api_version: z.string().optional(),
  data: z.object({ object: z.record(z.unknown()) }),
});

---

Step 3 — Store the Raw Event

One of the most valuable practices in production systems is persisting the original payload before doing anything with it.

create table webhook_events (
  id             uuid primary key default gen_random_uuid(),
  provider       text        not null,
  event_id       text        not null,
  event_type     text        not null,
  payload        jsonb       not null,
  signature      text,
  status         text        not null default 'pending',
  attempts       int         not null default 0,
  last_error     text,
  received_at    timestamptz not null default now(),
  processed_at   timestamptz,
  unique (provider, event_id)
);

create index on webhook_events (status, received_at);

Why? Because if processing fails later, you can replay the event without asking the provider to resend it. The unique constraint on (provider, event_id) also gives you idempotency for free at the storage layer.

---

Step 4 — Queue the Event

Instead of executing business logic immediately, hand the event to a queue — RabbitMQ, Kafka, SQS, BullMQ, or Postgres-backed jobs.

Benefits:

• Fast responses
• Better scalability
• Retry support
• Load smoothing
• Fault isolation

@Post('stripe')
@HttpCode(200)
async receive(@Req() req: RawBodyRequest<Request>) {
  const raw = req.rawBody.toString('utf8');
  if (!verifySignature(raw, req.headers['stripe-signature'], this.secret)) {
    throw new UnauthorizedException();
  }

  const event = WebhookEvent.parse(JSON.parse(raw));

  const inserted = await this.events.insertIfNew('stripe', event);
  if (inserted) await this.queue.add('process-event', { id: inserted.id });

  return { received: true };   // ← under 100ms
}

---

Idempotency Is Non-Negotiable

Webhook providers retry aggressively — on timeouts, network interruptions, 5xx responses, and their own outages. Your system must assume duplicate deliveries.

Event arrives → already processed? ── yes ──► ignore, return 200
                       │
                       no
                       ▼
                  process once

Without idempotency you get duplicate invoices, duplicate shipments, duplicate emails, and — worst of all — duplicate refunds.

The cheapest reliable implementation is a unique index on the provider event ID, plus processing inside a transaction that flips status to processed atomically with the business write.

---

Event Ordering

Suppose these events occur in this order:

Payment created → Payment confirmed → Refund issued

Due to retries and network behavior, your system may receive:

Refund → Payment created → Payment confirmed

Out-of-order delivery is normal. Design consumers to tolerate it:

• Event versions and sequence numbers
• Aggregate timestamps (if event.created < row.updated_at, skip)
• Parking events whose dependencies haven't arrived
• Event sourcing, where the domain justifies it

Never assume arrival order equals creation order.

---

Retry Strategies

Failures are inevitable; retries are essential. Use exponential backoff, never a tight retry loop.

Attempt	Delay
1	1 second
2	5 seconds
3	30 seconds
4	2 minutes
5	10 minutes

Add jitter so a thousand failed jobs don't retry in the same millisecond and re-crash the dependency you were waiting on.

---

Dead Letter Queues

Eventually some events will never succeed — a malformed payload, a deleted resource, a bug.

Webhook → Queue → retry → retry → retry → Dead Letter Queue

The DLQ stores problematic events for manual investigation or automated recovery. Alert on DLQ growth, and build a replay tool early — you will need it at 2 a.m.

Never silently discard failed events.

---

Horizontal Scaling

A single webhook server won't handle millions of events.

Internet
   ▼
Load Balancer
   ├── Webhook Server 1
   ├── Webhook Server 2
   ├── Webhook Server 3
   └── Webhook Server 4
              ▼
        Shared Queue
              ▼
         Worker Pool

Because receivers only authenticate, validate, persist, and return, they are stateless — scaling them is a slider, not a project. Shared storage and queues maintain consistency.

---

Worker Architecture

Separate responsibilities by domain:

Webhook Queue
   ├── Payment Worker
   ├── Inventory Worker
   ├── Email Worker
   └── Analytics Worker

Each worker owns one concern. A failing email provider then degrades notifications — not payments.

---

Observability

Every webhook should be traceable end to end. Log the event ID, correlation ID, provider, event type, processing time, retry count, worker ID, and final status.

A single correlation ID should let you follow an event from receipt to completion across every service.

Dashboards worth having:

Metric	Why it matters
Events per minute	Traffic baseline and spike detection
p95 processing latency	Early warning of degradation
Authentication failures	Misconfiguration or attack
Duplicate event rate	Provider retry health
Queue depth	Workers falling behind
Retry count	Downstream instability
DLQ size	Events needing humans
Success rate	The number leadership asks about

Alert before customers notice.

---

Security Best Practices

• HTTPS only
• Signature verification on every request
• Secret rotation with dual-secret grace windows
• Timestamp validation to prevent replay attacks
• Rate limiting
• IP allowlists where the provider supports them
• Structured audit logs
• Least-privilege access to downstream services

Never expose debugging information in webhook responses. A stack trace in a 500 body is free reconnaissance.

---

Multi-Provider Architecture

As your platform grows you'll integrate multiple providers. Don't hardcode a branch per vendor:

Webhook Gateway
   ▼
Provider Detector
   ├── Stripe Handler
   ├── GitHub Handler
   ├── Slack Handler
   ├── Telebirr Handler
   └── Chapa Handler
              ▼
    Common Event Pipeline

Normalize provider-specific payloads into internal event models. This keeps business logic independent of external APIs — and makes swapping a provider a handler change rather than a rewrite.

---

Event Versioning

Providers evolve. A payload today may not match tomorrow's schema.

• Store the provider API version with every event
• Support multiple schema versions during migration
• Transform older events into a canonical internal format

Avoid tightly coupling business logic to raw provider payloads.

---

Common Mistakes

• Executing long-running work inside the webhook endpoint
• Trusting requests without signature verification
• Ignoring duplicate deliveries
• Assuming events always arrive in order
• Returning errors after business logic partially completed
• Failing to persist raw payloads
• Retrying forever without a DLQ
• Not monitoring queue growth
• Coupling webhook handling directly to domain logic
• Treating webhooks as synchronous APIs

---

Production Architecture Example

External Provider
        │
        ▼
Load Balancer
        │
        ▼
Webhook API
        ├── Verify Signature
        ├── Validate Payload
        ├── Store Raw Event
        └── Return HTTP 200
                 │
                 ▼
          Message Queue
                 │
        ┌────────┴────────┐
        ▼                 ▼
 Payment Worker     Notification Worker
        │                 │
        ▼                 ▼
 Business Logic      Business Logic
        │                 │
        └────────┬────────┘
                 ▼
            Database Update
                 │
                 ▼
        Metrics • Logs • Alerts

The endpoint stays simple and fast; the complexity lives in dedicated workers.

---

Production Readiness Checklist

• Signature verification implemented
• HTTPS enforced
• Payload validation with a schema
• Raw event persistence
• Queue-based asynchronous processing
• Idempotent event handlers
• Retry policy with exponential backoff and jitter
• Dead Letter Queue with alerting
• Correlation IDs for tracing
• Comprehensive monitoring and alerting
• Automated replay tools for failed events
• Documented operational procedures

---

Final Thoughts

Webhooks are often introduced as "just another endpoint." In reality they are the foundation of modern event-driven integrations.

The quality of your webhook architecture determines whether your system gracefully handles duplicate deliveries, provider outages, network failures, traffic spikes, schema changes, and processing delays.

A well-designed webhook system doesn't chase perfection. It assumes requests will be duplicated, networks will fail, queues will back up, and services will occasionally be unavailable — then builds mechanisms to recover automatically.

That's the difference between a webhook that works in development and a webhook architecture that reliably processes millions of events in production.

Great webhook systems aren't measured by how fast they process the happy path — they're measured by how predictably they recover when the unexpected happens.