Event Sourcing with Kafka

There is a stage in the development of many software systems at which adding another API endpoint ceases to feel like implementing an isolated feature and begins to resemble a negotiation with every decision made before it.

A user changes a business object, an administrator changes it again, and the corresponding database row is updated. Another service needs to observe the change, so a webhook is introduced; subsequently, an email must be sent, an analytical model updated, a cache invalidated, an audit record retained and an external system notified. What began as a comprehensible request–response application gradually acquires a network of dependencies whose existence is rarely visible from the original operation.

This does not necessarily indicate that the system was poorly implemented. Rather, the architecture continues to treat state mutation as the primary fact, even though an increasing proportion of the business is interested in the action that produced the mutation.

That distinction was among the more important lessons to emerge from designing systems around event-driven architecture and event sourcing. Instead of asking only:

What is the current state?

it becomes necessary to ask:

What happened, and which parts of the system should react to it?

The change in wording is slight; the architectural consequences are not.

From state mutation to business events

Consider a conventional business application containing both customer-facing functionality and an administrative portal. A customer may create an order, cancel a booking, change an address, submit a document or accept an offer. An administrator may approve the same order, override a status, issue a refund, suspend an account or otherwise modify the same underlying business entity.

In a traditional application, these actions often terminate at the same technical boundary: an API handler validates the request, performs the relevant business logic and updates one or more database rows. As additional consequences accumulate, the implementation becomes increasingly procedural:

POST /cancel-order
    -> validate request
    -> update order
    -> update inventory
    -> send email
    -> write audit entry
    -> update analytics
    -> notify accounting

This structure is entirely reasonable while the operation remains bounded. The difficulty appears when every new consumer becomes another dependency of the original request path. Eventually, cancelling an order no longer means changing the order; it means synchronously coordinating a growing portion of the organisation.

An event-oriented model reverses this relationship. A customer action may produce:

OrderCancellationRequested

while a subsequent administrative decision produces:

OrderCancellationApproved

or, where the cancellation has already taken effect:

OrderCancelled

These distinctions are not cosmetic. A request, an approval and a completed cancellation represent different facts and therefore permit different failure and recovery semantics.

Once accepted, the relevant event enters shared event infrastructure and is processed according to what occurred rather than according to the screen or endpoint from which it originated:

User Application ─────┐
                      │
Admin Application ────┼──> Domain Events ───> Durable Event Log
                      │                          │
Internal Service ─────┘                          ├──> Order Projection
                                                 ├──> Notifications
                                                 ├──> Analytics
                                                 ├──> Audit
                                                 ├──> Billing
                                                 └──> External Integrations

The initiating interface becomes secondary. The domain event is the more durable abstraction.

Similar actions should converge on domain facts

Administrative and customer actions frequently belong in the same domain event stream. They may require different permissions, validation rules and command paths, but downstream systems often care principally about the resulting business fact.

Suppose that both a customer and an administrator can change an order’s delivery date. One implementation may create separate event types:

user.delivery-date-changed
admin.delivery-date-changed

This appears explicit, but it gradually reproduces the application’s interface hierarchy inside the event infrastructure. Every consumer must understand two event types representing substantially the same domain transition, and additional actor types produce additional duplication.

A more stable design preserves the distinction on the command side:

ChangeDeliveryDate
actor_type = USER

and:

ChangeDeliveryDate
actor_type = ADMIN

while recording the accepted fact as:

DeliveryDateChanged
{
    order_id,
    previous_date,
    new_date,
    actor_id,
    actor_type,
    occurred_at,
    ...
}

Each consumer can then determine whether the origin is relevant. The warehouse may care only about the new date, the notification service may vary its message according to the actor, and the audit system will ordinarily preserve the complete provenance. The fulfilment projection may simply apply the transition.

This produces an important architectural property: producers describe accepted reality, while consumers determine what that reality means within their own bounded responsibility.

Fan-out without producer coordination

Apache Kafka becomes attractive in this context not merely because it transports messages. Amazon Simple Queue Service (SQS), among other systems, already provides dependable asynchronous delivery. Kafka’s more distinctive property is that an event can become part of an ordered and durable log which several independent consumer groups read at their own pace.

An event such as:

CustomerAccountSuspended

may be processed independently by authentication, billing, notifications, risk, analytics, customer-support projections and audit. The account service does not need to call each consumer directly; it records the accepted fact, after which the log distributes that fact to interested consumers.

Without a shared log, the dependency structure may resemble:

Account Service
 ├──> Authentication Service
 ├──> Billing Service
 ├──> Notification Service
 ├──> Analytics Service
 ├──> CRM
 └──> Audit Service

With a log, it becomes:

Account Service
        │
        ▼
AccountSuspended
        │
        ▼
      Kafka
   ┌────┼────┬────┬────┐
   ▼    ▼    ▼    ▼    ▼
 Auth Billing CRM Audit Analytics

The difference is not merely diagrammatic. It changes the ownership of orchestration. Producer complexity remains comparatively stable as new reactions are introduced, provided that the event contract itself remains coherent.

This does not mean that all workflows should be implemented as uncoordinated event choreography. Business processes requiring explicit deadlines, compensating actions or an intelligible overall state may still benefit from an orchestrator or process manager. The useful distinction is that the producer should not become the accidental coordinator of every downstream consequence simply because it originated the first action.

Centralised control without centralised execution

The design objective can initially sound contradictory: centralise control of the event history without centralising all business execution in one service.

In this model, the log acts as a control plane. It records what occurred, the order in which related events occurred, the entity to which they apply, their schema version and the progress of each consumer. Execution remains distributed, with individual consumers materialising the state and initiating the side effects required for their respective workloads.

The result is a coherent historical backbone without a corresponding requirement for one enormous application service.

Leslie Lamport famously described distributed computing as follows:

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.”

The observation is amusing because it captures a common operational reality. Distributed architectures become fragile when synchronous dependencies are both numerous and poorly visible. A durable log does not eliminate network partitions, partial failure, duplicated delivery or inconsistent timing; however, it can make many dependencies explicit, asynchronous and inspectable.

The log is therefore not a mechanism for making distribution disappear. It is a mechanism for controlling and observing part of the complexity that distribution introduces.

Event-driven architecture and event sourcing are not synonymous

Event-driven architecture and event sourcing overlap, but they should not be conflated.

An event-driven application may update its authoritative database and subsequently publish an event:

update database
      ↓
publish event

The database remains the source of truth, while the event communicates a completed change to other systems. This model is widely useful, although the apparent sequence conceals a dual-write problem: the database update may succeed while publication fails, or publication may occur before the transaction is committed. A transactional outbox, change-data-capture mechanism or equivalent atomic publication strategy is ordinarily required where losing the event would violate correctness.

A genuinely event-sourced system reverses the authority:

append event
     ↓
derive state

The accepted event history becomes authoritative, and current state is calculated as a projection of that history. For example:

OrderCreated
ItemAdded
ItemAdded
DeliveryAddressChanged
DiscountApplied
PaymentCaptured
OrderSubmitted

can be reduced into the current representation of an order. The corresponding order table is then more accurately understood as a materialised view than as the original source of the business facts.

This produces a useful conceptual separation:

• the event history answers, “What was accepted as having happened?”;
• the projection answers, “Given those events and this projection logic, what state do we currently derive?”

The qualification matters. An event log is authoritative only if the surrounding system deliberately provides the retention, integrity, backup, ordering and recovery guarantees required of a system of record. Merely publishing transient Kafka messages does not make an application event sourced.

Replay changes the recovery model

Replay initially appears to be a useful operational convenience. In practice, it can become one of the principal reasons for retaining a durable event history.

Suppose an order stream contains three years of activity and a new fraud model is introduced. In a state-oriented system, the information required by the new model must be reconstructed from current tables, historical tables and whatever audit data happens to remain. The original context may already have been overwritten.

With a sufficiently complete event history, a new consumer can be created:

fraud-model-v2

and started from the beginning of the retained stream. It processes the historical events until it reaches the live position:

event 1
event 2
event 3
...
event 492,783,102

A new implementation has effectively been evaluated against historical operational reality.

The same mechanism can repair a projection. If a defect caused a balance consumer to calculate customer balances incorrectly, the recovery procedure can be:

Historical Event Log
         │
         ▼
    Corrected Consumer
         │
         ▼
    Rebuilt Projection

The faulty projection is replaced or rebuilt, and the corrected consumer reprocesses the relevant history. A hand-authored migration does not need to infer the intended state from already corrupted values because the accepted event sequence remains available.

This changes the interpretation of certain defects. A corrupted projection is not necessarily corrupted history; it may be an incorrect interpretation of an intact history.

Replay nevertheless has costs. Reprocessing hundreds of millions of events may require substantial time and compute, while old side-effecting consumers must not resend emails, recharge cards or repeat external operations. Snapshots may be required to avoid rebuilding long-lived aggregates from their first event, and consumers need explicit replay modes or side-effect boundaries. Replay is therefore an architectural capability that must be exercised, not a comforting property inferred from the presence of a Kafka topic.

Events must survive the software that produced them

Placing messages in Kafka does not by itself create useful replay. Events must contain enough semantic information to remain intelligible independently of the application version that created them.

A payload such as:

{
  "id": "123",
  "status": 4
}

may be obvious to its author today, yet indecipherable two years later. Similarly:

{
  "customer_id": "123"
}

is of limited historical value if understanding the event requires querying a customer row whose contents have subsequently changed.

Useful events describe domain facts with sufficient contemporaneous context:

{
  "event_type": "SubscriptionCancelled",
  "subscription_id": "sub_123",
  "customer_id": "customer_456",
  "previous_plan": "professional",
  "effective_at": "2026-08-27T03:14:00Z",
  "reason": "customer_requested",
  "actor_type": "customer",
  "schema_version": 2
}

The appropriate level of enrichment is conditional. Events should not become indiscriminate copies of entire database records, since doing so increases coupling, storage, privacy exposure and schema complexity. However, they require enough information for a future consumer to understand the accepted fact without reconstructing the complete database as it existed at that moment.

Designing events for present software is straightforward. Designing events that remain useful to software which does not yet exist is materially harder.

The migration path should be earned

Despite the advantages of a durable log, a modest application should rarely begin with a substantial Kafka estate merely because event sourcing appears architecturally sophisticated. The progression towards a log-oriented system is more defensible when it follows demonstrated workload and recovery requirements.

In our case, the legacy system coupled application behaviour closely to state mutation. As functionality moved onto Amazon Web Services (AWS), an effective intermediate architecture resembled:

API Gateway
    │
    ▼
 Lambda
    │
    ├──> RDS
    │
    └──> SQS
            │
            ▼
         Lambda

This remains an excellent architecture for a surprisingly broad class of systems. SQS provides durable asynchronous processing, retries, dead-letter queues, straightforward identity and access management integration, consumption-based pricing and almost no infrastructure management. AWS Lambda provides a similarly useful scaling model: when no work exists, little compute runs; when a burst arrives, many workers can be instantiated.

For intermittent and moderate workloads, that economic and operational simplicity is difficult to surpass.

The broader lesson is that architectural sophistication should be earned by workload characteristics and business requirements. It is easy to design for the organisation one hopes to become rather than the system presently being operated. Running a distributed streaming platform for 500 events per hour because the application may someday process 500,000 per second is ordinarily architecture theatre.

A serverless queue-based system can remain elastic:

03:00:
traffic ≈ 0
compute ≈ 0

and, during a burst:

traffic ↑
Lambda concurrency ↑

before returning to its resting state. Where demand is irregular, this is a highly desirable property.

When a queue becomes a log-shaped requirement

The motivation to move beyond SQS generally emerges through requirements rather than throughput alone. Suppose the same order event must now be consumed independently by billing, inventory, recommendations, analytics, risk, search, notifications, warehouse systems and a machine-learning pipeline. Queue-based fan-out can be implemented with Amazon Simple Notification Service (SNS) and SQS:

                   ┌──> SQS Billing
                   ├──> SQS Analytics
Event ──> SNS ─────┼──> SQS Inventory
                   ├──> SQS Notifications
                   └──> SQS Risk

There is nothing inherently deficient about this arrangement; it may remain the correct architecture. The pressure towards a log tends to arise when the questions change:

• Can a new service consume the preceding six months of activity?
• Can a projection be rebuilt from accepted history?
• Can independent teams process the same stream according to different models?
• Can ordering be preserved for all events affecting one account or order?
• Can the historical stream be inspected as a first-class dataset?

These requirements are less naturally represented by a collection of work queues. They begin to describe a durable shared log, at which point Kafka starts to earn its operational cost.

Scaling into Kafka without architectural ideology

A later-stage architecture may therefore resemble:

                         ┌──────────────┐
User Actions ───────────>│              │
                         │ Event Gateway│
Admin Actions ──────────>│              │
                         └──────┬───────┘
                                │
                                ▼
                              Kafka
                                │
             ┌──────────────────┼──────────────────┐
             │                  │                  │
             ▼                  ▼                  ▼
        Consumer Group     Consumer Group     Consumer Group
           Orders            Analytics          Billing
             │                  │                  │
             ▼                  ▼                  ▼
            EC2                EC2               EC2

The migration need not be ideological. Kafka does not require the abandonment of serverless execution, nor does its introduction require every asynchronous interaction to move onto Kafka:

Kafka
 ├──> Lambda consumer
 ├──> EC2 consumer fleet
 ├──> ECS workload
 ├──> analytical pipeline
 └──> long-running stream processor

Kafka should assume responsibility for the parts of the system in which ordered retention, independent consumption and replay provide genuine value. Existing queues and functions can remain where their simpler semantics better match the workload.

The objective is not to produce “the Kafka architecture”. It is to assign each workload to the mechanism whose guarantees and operating model correspond to its actual requirements.

Why long-running consumers remain useful

Moving from Lambda consumers to consumers hosted on Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS) or Kubernetes can initially appear regressive. It is not; the two approaches optimise for different execution models.

Kafka consumers are naturally long-lived. A consumer joins a group, receives one or more partitions and advances its offsets continuously:

Partition 0 ──> Worker A
Partition 1 ──> Worker B
Partition 2 ──> Worker C
Partition 3 ──> Worker D

When workers are added or removed, Kafka rebalances the partitions across the remaining group members. The relevant scaling unit is therefore not necessarily one invocation per message, but available consumer capacity per partition.

For sustained workloads, long-running workers can be operationally and economically appropriate. They maintain connection pools, local caches, warmed runtime state, batching buffers, model state and persistent stream-processing contexts. In exchange, the system accepts less native elasticity and greater responsibility for provisioning, health management and scaling.

Neither execution model is universally superior. Lambda is compelling for bursty or low-volume processing with short-lived work; persistent consumers are often preferable where throughput is sustained, batching is valuable, state must remain warm or partition ownership is central to processing.

Scaling down is an architectural requirement

Distributed-system diagrams are ordinarily drawn towards growth: more users, partitions, workers and throughput. Designing how the system contracts is equally important.

Serverless systems obscure this problem because scaling towards zero is intrinsic to their operating model. Persistent Kafka consumer fleets require explicit consideration of partition count, consumer-group rebalancing, startup time, minimum healthy capacity, backlog recovery and the signals used for autoscaling.

Central processing unit utilisation is frequently an inadequate scaling metric. A consumer may use only 20% of a processor while falling several hours behind the event stream. The more relevant quantity is often consumer lag.

Where:

producer rate > consumer rate

lag increases and additional capacity may be required. Where:

consumer rate > producer rate

lag decreases, and capacity can be reduced once the backlog has been recovered.

This connects scaling to the operational property that matters: how far the consumer’s derived state has fallen behind accepted reality. That is ordinarily more informative than asking how busy a particular machine happens to be.

Partitioning encodes domain ordering

Kafka guarantees ordering within a partition rather than globally across a topic. Consequently, selection of the partition key is part of the domain architecture rather than a minor infrastructure setting.

For example:

key = customer_id

preserves ordering among events associated with one customer, while:

key = order_id

preserves the sequence of transitions for an individual order. Either may be correct depending on which aggregate requires coherent ordering.

The choice also establishes scaling constraints. A disproportionately active customer or order class may create a hot partition, while changing the strategy later can complicate replay and ordering semantics. Partition count places an upper bound on active consumers within a group for a given topic, and increasing that count does not retroactively repartition history without consequences.

Partitioning therefore encodes an answer to a domain question: which sequence of events must remain intelligible as a single ordered history?

Idempotency is part of correctness

At-least-once delivery means that consumers must ordinarily assume the same event may be presented more than once. A naïve balance update is therefore dangerous:

PaymentCaptured
      ↓
add $100 to balance

If the event is processed twice, the balance is increased by $200. The system needs an idempotency strategy, commonly based on a stable event identifier:

event_id = 018f9...

A consumer records which event identifiers have already affected its projection and applies the state change atomically with that record. Where possible, naturally idempotent assignments are preferable:

set payment.status = CAPTURED

rather than:

increment captured_payment_count

although even apparently idempotent updates require care when surrounding side effects are not idempotent.

Once retries, replay and asynchronous processing become fundamental, idempotency is not a performance optimisation. It is a correctness property.

Dead-letter queues retain failures; they do not resolve them

SQS makes dead-letter queues (DLQs) straightforward to configure. This is valuable, but it can encourage the mistaken model:

failed message → DLQ → problem solved

A dead-letter queue is merely a durable location in which unresolved failures accumulate. An operationally mature system still needs to determine who monitors the queue, which diagnostic context accompanies the failure, whether the message can be replayed safely, how poison data is distinguished from a transient dependency failure and when processing should halt rather than skip an event whose absence would invalidate subsequent state.

Kafka introduces analogous decisions through retry topics, parking-lot topics and offset management. The infrastructure differs, but the operational responsibility remains.

Observability must follow causality

Once execution fans out asynchronously, conventional request tracing becomes less intuitive. An HTTP request may return milliseconds after appending an event, while the corresponding business operation continues for another thirty seconds across several consumers.

Event envelopes should therefore carry stable correlation metadata:

{
  "event_id": "...",
  "correlation_id": "...",
  "causation_id": "...",
  "actor_id": "...",
  "occurred_at": "...",
  "schema_version": 3
}

The event_id identifies the individual fact, the correlation_id groups events belonging to the broader business operation, and the causation_id identifies the event or command that caused the present event.

These relationships allow the system to reconstruct a causal graph:

UserSubmittedOrder
        │
        ▼
OrderCreated
        │
        ├──> InventoryReserved
        │         │
        │         └──> WarehouseTaskCreated
        │
        ├──> PaymentRequested
        │         │
        │         └──> PaymentCaptured
        │
        └──> ConfirmationSent

The observability system is no longer merely collecting log lines emitted by individual processes. It is reconstructing the causal history of a business operation.

Event schemas are public contracts

Event schemas should be treated with comparable seriousness to externally consumed APIs. Once several consumers depend on:

OrderSubmitted.v1

changing its payload is no longer a local refactor. Consumers may be deployed at different versions, historical events may remain in the log, and replay may expose years-old payloads to code written yesterday.

Schema evolution consequently requires deliberate rules. New fields should generally be additive and optional where compatibility requires it; semantic meaning should not change silently; consumers should tolerate fields they do not understand; and versioning or upcasting strategies should be explicit.

One of the least durable event designs is the direct serialisation of an object-relational mapping (ORM) entity:

Order ORM object
      ↓
JSON
      ↓
Kafka

Although convenient, this couples the organisation’s historical contract to the present database representation. A column rename, persistence optimisation or relationship change then appears to be a change in business history.

Domain events should represent domain facts, not implementation objects.

Commands express intent; events record accepted facts

Commands and events also require distinct semantics. A command requests an operation:

Please perform this action.

An event records an accepted fact:

This action occurred.

For example:

CancelOrder

is a command and may be rejected. By contrast:

OrderCancelled

is a historical fact within the system. A representative flow is therefore:

HTTP request
     │
     ▼
CancelOrder command
     │
     ▼
business rules
     │
     ├── rejected
     │
     └── accepted
           │
           ▼
    OrderCancelled
           │
           ▼
         Kafka

The log should ordinarily contain accepted facts rather than unfulfilled wishes. Where requests themselves must be retained—for example, because an approval process is part of the domain—they should be modelled as facts such as OrderCancellationRequested, not as commands masquerading as completed events.

The event log is not a universal query engine

Another common error is attempting to make the log answer every application query. Kafka is well suited to retaining and distributing ordered event streams; it is not the natural serving layer for a request such as:

GET /customers/123/orders?status=pending

Consumers should instead construct projections appropriate to their access patterns:

Kafka
  │
  ├──> PostgreSQL operational projection
  ├──> OpenSearch search projection
  ├──> Redis low-latency projection
  ├──> S3 analytical archive
  └──> ClickHouse analytical projection

The resulting architecture is intentionally polyglot. The retained history provides the authoritative sequence of accepted facts, while databases provide useful views of those facts for particular workloads.

This initially appears redundant because related information exists in several places. The copies, however, serve different purposes. They are not competing sources of authority; they are indexes and interpretations constructed from the same history. Their disposable nature depends on whether they can genuinely be rebuilt, which is another reason replay must be tested rather than assumed.

Begin with use cases, not infrastructure primitives

Kafka, event sourcing and serverless architecture can encourage premature reasoning in terms of topics, partitions, queues, consumer groups, functions and instances. This reverses the appropriate order of design.

The useful starting point is the business use case. Consider:

An administrator refunds an order.

The operation may decompose into:

RefundRequested
      ↓
RefundValidated
      ↓
RefundIssued

after which the accepted refund produces several independent consequences:

RefundIssued
 ├──> accounting adjusts balance
 ├──> notification sends email
 ├──> analytics records refund
 ├──> customer history updates
 └──> risk engine evaluates account

Only then should infrastructure questions be asked. Does the operation require ordered processing? Must it be replayable? Can its consequences occur asynchronously? Can a consumer remain offline for six hours? Is the history required five years later? What throughput and acceptable lag apply? Which failures require an orchestrated compensation rather than independent reactions?

The answers determine whether the appropriate mechanism is a database transaction, function call, Lambda invocation, SQS queue, Kafka topic, long-running worker or some combination thereof.

Architecture follows the use case, not the reverse.

What I would establish earlier

If approaching the same architectural evolution again, I would put disproportionate effort into several foundations.

Event semantics would be defined before topic names. Stable event identifiers, correlation identifiers, causation identifiers and schema versions would be present from the beginning. Every asynchronous consumer would be idempotent by default, while replay would be treated as a tested recovery requirement rather than a theoretical capability inferred from retention settings.

Monitoring would be organised around lag, failed business outcomes and incomplete workflows rather than infrastructure utilisation alone. Event publication would be made atomic with the originating state transition where the database remained authoritative. Privacy, deletion and retention obligations would be considered before immutable histories accumulated sensitive data that could not be governed safely.

Most importantly, I would resist premature distribution. There is nothing inherently sophisticated about operating forty services. A well-structured monolith that publishes a small number of meaningful domain events may be considerably more evolvable than dozens of services exchanging poorly specified messages.

The objective is not distribution. It is control over change.

The architecture being constructed

At first glance, the progression appears technological:

Legacy Application
        ↓
AWS Lambda + SQS
        ↓
Kafka + scalable consumer infrastructure

The deeper progression concerns the system’s model of history and responsibility.

The legacy application says:

Change the database and notify every dependent component.

The asynchronous serverless application says:

Perform the operation and allow background workers to execute its consequences.

The event-driven system says:

Record what occurred and permit independent consumers to react.

The event-sourced system goes further:

The accepted event history is authoritative; current state is one projection of that history.

Under the appropriate guarantees, Kafka then becomes more than a high-throughput queue. It becomes a durable boundary between facts and reactions, allowing producers to remain independent of their consumers, new functionality to process old events, broken projections to be rebuilt and workloads to scale according to the processing they actually require.

None of these capabilities is free. The architecture inherits explicit concerns around ordering, idempotency, schema evolution, eventual consistency, retention, replay, consumer lag, privacy, observability and operational recovery. Kafka may also introduce complexity that a smaller system did not previously possess, which is why its adoption should be justified rather than ceremonial.

Nevertheless, many of the difficulties exposed by a log-oriented architecture are not created by the log. They already existed in the business process, in hidden dependencies and in assumptions embedded within mutable state. The architecture simply makes those difficulties visible and gives them durable representations.

That is perhaps the central lesson in moving from a legacy application, through serverless queues, towards a log-oriented architecture:

A well-designed event-driven system does not make a complicated business simple. It makes the relevant complexity explicit, replayable and controllable.