Understanding Event-Driven Architecture at Scale

When your monolith starts groaning under millions of events per second, the path forward isn't always a microservices rewrite. Sometimes the architecture you need is already buried inside the one you have — you just need to surface it.

Fig 1. — High-level architecture: producers publish to Kafka topics, consumers process independently.

Why events instead of direct calls?

Direct synchronous calls couple services tightly — the caller waits, latency compounds, and a single slow dependency can cascade. Events invert this: producers don't care who's listening, consumers process at their own pace, and the system degrades gracefully.

The three properties that matter

• Durability — events survive consumer restarts
• Ordering — per-partition ordering is preserved in Kafka
• Replayability — replay from offset to rebuild state or fix bugs

Setting up a Kafka producer in Node.js

Using kafkajs, wiring up a producer is straightforward. The key decisions are your partitioner strategy and whether you need idempotent delivery.

import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-service',
  brokers: ['broker1:9092', 'broker2:9092'],
});

const producer = kafka.producer({
  idempotent: true, // exactly-once semantics
});

await producer.connect();

await producer.send({
  topic: 'user-events',
  messages: [
    {
      key: userId,
      value: JSON.stringify({ type: 'signup', userId, timestamp: Date.now() }),
    },
  ],
});

↑ Replace the YouTube URL above with your own video link.

Results after migration

After moving our order-processing pipeline to an event-driven model, p99 latency dropped from 420ms to 38ms. Throughput increased 12×, and we gained the ability to replay events for backfills — something impossible with our old RPC approach.

Fig 2. — p99 latency before (orange) and after (green) the migration. Week 4 is cutover.

What to watch out for

Events aren't free. You trade request/response simplicity for eventual consistency, debugging complexity, and the operational burden of a message broker. Schema evolution is trickier than changing a function signature. Use a schema registry from day one.

# Register a schema with Confluent Schema Registry
curl -X POST http://schema-registry:8081/subjects/user-events-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schema": "{\"type\":\"record\",\"name\":\"UserEvent\",\"fields\":[{\"name\":\"userId\",\"type\":\"string\"},{\"name\":\"type\",\"type\":\"string\"}]}"
  }'

That's the core of it. Start small — one event type, one consumer — and expand from there. The patterns scale, but so does the complexity, so earn each step.