Event-Driven Integrations in Regulated SaaS

As our healthcare platform expanded, we integrated with multiple external systems:

• AWS SNS for SMS notifications (including timezone-aware scheduling)
• Worldpay for payment webhooks
• Insurance authorization systems via webhook callbacks
• SignalWire for fax delivery
• Internal domain events across intake, billing, and authorization modules

Each integration introduced uncertainty: delayed responses, duplicate webhooks, transient failures, and third-party downtime.

In a regulated environment, missed events or double processing are not minor bugs — they are compliance and financial risks.

The challenge was to design event pipelines that assumed failure as normal behavior.

I adopted an event-driven pattern using:

• AWS SQS for queue-based decoupling
• Lambda workers for async processing
• Structured logging + DynamoDB for event persistence
• Correlation IDs for traceability
• Retry classification (temporary vs permanent failure)

The goal was not just asynchronous execution — but controlled reliability under uncertainty.

Design principle: external systems are unreliable; internal systems must be deterministic.

Core Patterns Implemented

• Validation-first webhook ingestion
• Persistence-before-processing
• Retry with exponential backoff
• Dead-letter queues for permanent failure
• Correlation IDs across all downstream logs

Duplicate Events

Both payment providers and insurance systems occasionally delivered duplicate webhooks.

Mitigation strategy:

• Deterministic event fingerprinting
• Unique constraint on event ID in DynamoDB
• Idempotent processing logic (enhancement pattern in progress)

Duplicate events were safely acknowledged without reprocessing.

Partial Failures

Failures were classified:

• Temporary — network issues, transient 5xx responses
• Permanent — validation failures, malformed payloads

Temporary failures triggered exponential backoff retries.
Permanent failures moved directly to Dead Letter Queue for manual review.

This avoided silent data loss.

// retry-classifier.ts — single source of truth for retry vs DLQ decisions.
type Decision =
  | { kind: "retry"; delayMs: number }
  | { kind: "dlq"; reason: string };

const TRANSIENT_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);

export function classify(err: unknown, attempt: number): Decision {
  if (err instanceof HttpError) {
    if (TRANSIENT_STATUS.has(err.status)) {
      return { kind: "retry", delayMs: backoff(attempt) };
    }
    if (err.status >= 400 && err.status < 500) {
      return { kind: "dlq", reason: `client error ${err.status}` };
    }
  }
  if (err instanceof ValidationError) {
    return { kind: "dlq", reason: "malformed payload" };
  }
  // Unknown shape — treat as transient; if retries exhaust, DLQ kicks in.
  return { kind: "retry", delayMs: backoff(attempt) };
}

// Capped exponential backoff with full jitter.
function backoff(attempt: number): number {
  const ceiling = Math.min(60_000, 500 * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}

Out-of-Order Events (Design Consideration)

While not yet fully implemented, the system is structured to support:

• Versioned state transitions on every domain entity
• Status reconciliation runs before any state mutation
• The persisted event store enables full event replay

This allows future resilience against out-of-order webhook delivery.

SMS via AWS SNS

Designed support for:

• Scheduled notifications across time zones
• Clinic-level timezone configuration
• Deferred message publishing
• Retry handling on SNS delivery failure

Scheduling logic normalized all timestamps to UTC internally, then converted per clinic configuration.

Fax via SignalWire

Fax integration followed the same async ingestion pattern:

• Submission request persisted
• Delivery status webhook processed
• Failure states reconciled against internal message record

Fax delivery, often slow and unpredictable, was treated as an eventually consistent event.

Reliability without observability is guesswork.

I implemented:

• Structured logs persisted to DynamoDB
• Correlation IDs per event
• CloudWatch log streaming
• Event-level tracing across Lambda execution chains

This allowed us to trace a payment webhook from ingestion → retry → final state mutation deterministically.

Alerting is not yet implemented, but structured logging makes future monitoring straightforward.

Even when not fully implemented, I designed the system with these failure modes in mind:

• Duplicate webhook delivery
• Partial processing failure mid-transaction
• Third-party downtime
• Message size limits (future guardrail consideration)
• Event replay for recovery
• Idempotent processors for safe reprocessing

Designing for these upfront reduced architectural rework later.

Why SQS instead of direct processing?

Direct webhook → DB write would have been simpler but fragile under burst traffic and retry storms. Queue decoupling provided buffering and controlled concurrency.

Why not Kafka?

Current scale does not justify the operational overhead. SQS + Lambda provides sufficient throughput and simpler maintenance.

Why persist before processing?

Persisting webhook payloads before business logic guarantees replay capability and forensic audit traceability.

• Reliable handling of duplicate webhooks
• Reduced operational incidents caused by transient failures
• Deterministic audit trails for compliance reviews
• Clean decoupling between external systems and core domain logic
• Extensible foundation for future event streaming evolution

The system assumes failure as normal — and remains predictable under it.

What worked

Persist-before-process and retry classification were the most impactful decisions. They turned unpredictable third-party behavior into deterministic internal workflows.

What I'd improve

• Formal idempotent event processor abstraction
• Replay tooling for selective event reprocessing
• Alerting on DLQ growth and retry exhaustion
• Message size validation guardrails

Future direction

• Introduce event versioning contracts
• Add DLQ monitoring and automated escalation
• Evaluate Kafka or EventBridge if event volume grows significantly
• Implement structured event schema registry