Beyond CRUD: Building Reliable Software Systems · Part 8
New to this series? Start with Part 1
The Outbox Pattern Explained: Publishing Events Without Losing Data
“A database transaction can guarantee your data is correct. It cannot guarantee the rest of your system knows about it.”
Imagine you’re building an e-commerce platform. A customer places an order, and your application begins processing the request. Inside a database transaction, it creates the order, deducts the purchased items from inventory, records the payment, and commits the transaction successfully. From the perspective of the database, everything has gone exactly as planned. Every change has been saved, every business rule has been respected, and the transaction completes without error.
If this were a monolithic application, the story might end there, but modern software rarely consists of a single application. The moment an order is created, several other systems need to react. A shipping service must prepare the package, an email service needs to send an order confirmation, an analytics platform wants to record another sale, a loyalty service may award reward points, and an accounting system might need to generate journal entries. Rather than constantly querying the orders table looking for changes, these services usually rely on events published through a message broker such as Kafka, RabbitMQ, Azure Service Bus, or Amazon SQS.
A straightforward implementation seems almost obvious: once the transaction commits successfully, publish an OrderCreated event.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
BEGIN TRANSACTION
↓
Create Order
↓
Reduce Inventory
↓
Record Payment
↓
COMMIT
↓
Publish OrderCreated Event
At first glance, there’s nothing wrong with this approach, until something fails. Suppose the database transaction commits successfully, permanently saving the customer’s order. A fraction of a second later, however, Kafka becomes temporarily unavailable. Perhaps RabbitMQ disconnects, or maybe the application crashes before it can publish the event.
The order now exists in the database, the customer’s payment has been processed, and inventory has already been reduced, yet none of the downstream services know the purchase ever happened. The warehouse never receives instructions to prepare the shipment, the customer never receives a confirmation email, the analytics dashboard quietly reports incorrect sales figures, and the accounting system never records the transaction.
Nothing inside the database is inconsistent. The inconsistency exists between systems.
This subtle failure is one of the most common reliability problems in distributed architectures. It’s known as the Dual-Write Problem, and it’s surprisingly easy to introduce because writing to a database and publishing an event are two completely independent operations. One can succeed while the other fails, leaving different parts of the system with different versions of reality.
In our previous articles, we’ve explored how transactions protect database operations, how isolation levels govern concurrent access to data, and how distributed locks coordinate work across multiple application servers. The Outbox Pattern builds on those concepts by solving a different challenge: ensuring that once your database commits a change, the rest of your system will eventually learn about it as well.
Before we look at the solution, let’s first understand why this seemingly simple problem is much harder than it appears.
The Dual-Write Problem
The Dual-Write Problem occurs whenever an application attempts to update two independent systems as part of a single business operation. One system is usually your database; the other is typically a message broker, search index, cache, analytics platform, or another external service. Although these updates feel like one logical operation, they are technically two completely separate actions.
Consider an online banking application. When a customer transfers money, the application updates account balances inside the database. Afterward, it publishes a MoneyTransferred event so other services can respond. A fraud detection system may inspect the transaction, a notification service may send an SMS, and an accounting platform may update its ledgers.
Conceptually, all of this represents one business event, but technically, it looks more like this:
1
2
3
4
5
Update Database
↓
Publish Event
The problem is that neither system knows anything about the other. Your database has no idea whether Kafka accepted the message, and Kafka has no knowledge of whether your SQL transaction committed successfully. They’re completely independent, and that independence creates four possible outcomes.
| Database | Event | Result |
|---|---|---|
| ✅ Success | ✅ Success | Everything works correctly. |
| ❌ Failure | ❌ Failure | Nothing happens, which is acceptable. |
| ❌ Failure | ✅ Success | Downstream systems react to an event for data that doesn’t exist. |
| ✅ Success | ❌ Failure | The database is correct, but no other service knows the change occurred. |
The first two outcomes are easy to reason about, but it’s the final two that create serious problems. If an event is published for data that never commits, downstream services begin processing information that doesn’t actually exist. Equally dangerous is the opposite scenario, where the database commits successfully but the event is never published. From that moment onward, every service in the architecture has a different understanding of reality. This isn’t merely an inconvenience; it’s a consistency problem that becomes increasingly difficult to detect as systems grow.
Why Transactions Can’t Solve This
A natural question follows: why not simply include the message broker inside the database transaction? Unfortunately, that’s not how database transactions work. Transactions provide atomicity because every operation is coordinated by the same database engine. The database controls when the transaction begins, when it commits, and when it rolls back. Every SQL statement participates in that process because they’re all executed by the same system.
A message broker lives outside that boundary. Kafka doesn’t participate in your PostgreSQL transaction, RabbitMQ doesn’t know your SQL Server transaction exists, and PostgreSQL has no mechanism for asking Kafka whether a message was published successfully before committing the transaction. In other words, there is no shared transaction manager coordinating both systems.
Years ago, technologies such as Two-Phase Commit (2PC) attempted to solve this problem by coordinating transactions across multiple systems. Although theoretically appealing, they introduced significant complexity, increased latency, reduced availability, and often became bottlenecks in distributed environments.
As microservices became more popular, the industry gradually moved toward simpler, more resilient approaches. Rather than trying to make two systems commit simultaneously, engineers began asking a different question:
What if we only committed to one system first and made the second system eventually consistent?
That shift in thinking led to one of the most influential patterns in modern backend architecture: the Outbox Pattern.
What Is the Outbox Pattern?
The Outbox Pattern solves the Dual-Write Problem by changing the order in which work is performed. Instead of attempting to update the database and publish an event as part of the same operation, the application first commits everything it needs to the database, including the event itself. That last part is the key. Rather than sending the event directly to Kafka or RabbitMQ, the application writes the event into a dedicated database table commonly known as the outbox.
Because the business data and the outbox record are written inside the same database transaction, they succeed or fail together. If the transaction commits, both the business data and the event are safely stored; if it rolls back, neither exists. Only after the transaction has completed does another process read events from the outbox table and publish them to the message broker.
Conceptually, the workflow changes from this:
1
2
3
4
5
Update Database
↓
Publish Event
to this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
BEGIN TRANSACTION
↓
Update Business Data
↓
Insert Event Into Outbox
↓
COMMIT
↓
Background Publisher Reads Outbox
↓
Publish Event
↓
Mark Event As Published
At first glance, this may seem like a small adjustment, but in reality, it completely changes the reliability characteristics of your application. The application is no longer trying to coordinate two independent systems within the same request. Instead, it commits to a single source of truth, the database, and lets another process handle communication with external systems afterward. If the message broker is temporarily unavailable, nothing is lost. The event is already safely stored inside the database and simply waits until the publisher can deliver it.
Understanding the Outbox Table
The outbox itself is usually nothing more than a normal database table. A simplified version might look like this:
| id | event_type | payload | created_at | published_at |
|---|---|---|---|---|
| 1 | OrderCreated | {…} | 2026-07-01 10:15 | NULL |
| 2 | PaymentReceived | {…} | 2026-07-01 10:16 | NULL |
| 3 | UserRegistered | {…} | 2026-07-01 10:18 | 2026-07-01 10:18 |
Each row represents an event that should eventually be delivered. Notice the published_at column: rows where this value is NULL haven’t yet been published, while rows with a timestamp have already been successfully delivered.
The outbox isn’t intended to become a permanent event store. Instead, it acts as a reliable staging area between your transactional database and your messaging infrastructure. Once an event has been published successfully, and your retention policy allows, it can be archived or deleted.
A Complete Walkthrough
Let’s revisit our e-commerce example. A customer purchases a laptop. Inside a single transaction, the application performs three operations: first, it creates the order; second, it deducts one item from inventory; and finally, instead of publishing an event immediately, it inserts a new row into the outbox table describing what happened.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
BEGIN TRANSACTION
↓
INSERT Order
↓
UPDATE Inventory
↓
INSERT Outbox Event
↓
COMMIT
At this point, the customer’s purchase has been committed successfully, and just as importantly, the event describing that purchase has also been committed. Suppose Kafka goes offline immediately afterward: the request still succeeds, nothing has been lost, and the application doesn’t need to roll back the order because the event hasn’t disappeared. It is sitting safely inside the outbox table waiting to be delivered.
A separate publisher service periodically checks the outbox. When Kafka becomes available again, it publishes the event and updates the record to indicate that delivery succeeded.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Outbox Publisher
↓
Read Unpublished Events
↓
Publish To Kafka
↓
Success?
│
Yes │ No
│
Mark Published
│
Retry Later
Notice something important: the user’s request is no longer responsible for talking to Kafka. Its only responsibility is ensuring that the event is safely recorded, and publishing becomes a separate concern. That separation dramatically improves reliability because temporary failures in external systems no longer affect the original business transaction.
Why This Works
The elegance of the Outbox Pattern comes from the fact that it reduces a distributed systems problem to a database problem. Earlier in this series, we spent considerable time discussing transactions and learned that they guarantee a group of related database operations either all succeed or all fail together. The Outbox Pattern deliberately takes advantage of that guarantee: instead of asking a database and a message broker to commit simultaneously, a task they were never designed to perform, it asks only the database to commit. Since both the business data and the outbox record live inside the same database, the transaction naturally guarantees their consistency.
Everything after that becomes a delivery problem rather than a consistency problem. Even if the publisher crashes, Kafka becomes unavailable, or the network experiences intermittent failures, the event remains safely stored. The publisher will eventually retry, and the event will eventually be delivered. The system moves from requiring immediate consistency between the database and the message broker to achieving eventual consistency in a reliable and predictable way. That single shift in mindset is what has made the Outbox Pattern one of the most widely adopted reliability patterns in modern distributed systems.
Why Not Publish Immediately After the Commit?
Some developers look at the Outbox Pattern and ask a reasonable question: “If the transaction has already committed successfully, why not simply publish the event right afterward and retry if it fails?” The problem is that retries only work if the application survives long enough to perform them.
Imagine the following sequence of events:
1
2
3
4
5
6
7
8
9
COMMIT Transaction
↓
Application Crashes
↓
Restart
The order exists, the event was never published, and the application has no memory that it still owes Kafka an event because that information was never persisted anywhere. With an outbox table, the event survives the crash because it was committed alongside the business data. When the publisher starts again, it simply resumes reading unpublished events from the database. The application doesn’t have to remember what happened because the database already does.
This is one of the reasons the Outbox Pattern is so resilient. It relies on durable storage rather than application memory, making it naturally tolerant of crashes, restarts, and temporary infrastructure failures.
How Events Leave the Outbox
By now, we’ve established that the application’s responsibility ends once the business data and the corresponding event have been committed to the database. The next challenge is equally important: How do those events actually reach Kafka, RabbitMQ, or another messaging system?
Broadly speaking, there are two common approaches. The first is a Polling Publisher, where a background worker periodically checks the outbox table for unpublished events. Every few seconds, or even every few milliseconds, it queries the database, publishes any pending events, and marks them as successfully delivered.
Conceptually, the process looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Background Publisher
↓
Query Outbox
↓
Find Unpublished Events
↓
Publish Event
↓
Success?
│
Yes │ No
│
Mark Published
│
Retry Later
This approach is simple to understand, easy to implement, and works well for many applications. Since the publisher operates independently of the original request, temporary failures in Kafka or RabbitMQ don’t affect users. If publishing fails, the worker simply retries later.
The second approach is Change Data Capture (CDC). Instead of periodically querying the outbox table, a CDC tool monitors the database’s transaction log and automatically detects newly inserted outbox records. As soon as a new event appears, the tool publishes it to the message broker.
One of the most popular CDC solutions is Debezium, which integrates with databases such as PostgreSQL, MySQL, and SQL Server. Rather than polling the database repeatedly, Debezium continuously streams database changes, reducing unnecessary queries while providing near real-time event publication. Both approaches solve the same problem. Polling is generally simpler and perfectly adequate for many systems, while CDC becomes attractive when applications process very large numbers of events or require lower publishing latency.
Real-World Examples
The Outbox Pattern appears in many more places than developers often realize. Once you begin recognizing the Dual-Write Problem, you start seeing it almost everywhere.
E-Commerce
A customer places an order. The transaction creates the order, updates inventory, and inserts an OrderCreated event into the outbox. Later, the publisher delivers that event to Kafka, and other services respond independently:
- Shipping prepares the package.
- Email sends the confirmation.
- Analytics records the sale.
- Loyalty awards points.
Every service receives exactly the same event without the original request needing to coordinate them.
Loan Management Systems
Suppose a borrower makes a repayment. Inside a transaction, the application updates the loan balance, records the payment, recalculates outstanding interest, and inserts a LoanRepaymentReceived event into the outbox. Once published, other systems can react independently. The accounting service posts journal entries, the notification service sends an SMS receipt, reporting dashboards update repayment statistics, and credit scoring services refresh customer profiles. The repayment itself remains fast because none of these downstream systems participate in the original transaction.
User Registration
A customer creates a new account. The application stores the user’s information and writes a UserRegistered event into the outbox. Later, other services consume the event. An email service sends a welcome message, a CRM platform creates a customer profile, a marketing system subscribes the user to onboarding campaigns, and a recommendation engine begins generating personalized suggestions. The registration endpoint doesn’t need to know anything about those systems; its only responsibility is recording that the registration occurred.
Payment Processing
Imagine a payment gateway confirms a successful transaction. The payment service updates account balances, records the payment, and inserts a PaymentCompleted event. Even if Kafka becomes unavailable immediately afterward, the payment itself isn’t lost. Once messaging resumes, the event is delivered and downstream systems continue exactly where they left off.
Benefits of the Outbox Pattern
One of the reasons the Outbox Pattern has become so widely adopted is that it solves several problems simultaneously. First, it provides reliability. Once an event has been written into the outbox, it becomes durable. Temporary network failures, message broker outages, or application crashes no longer result in permanently lost events. Second, it simplifies application code. Rather than forcing every request to coordinate database updates and message publication, the request focuses exclusively on committing business data, and event publication becomes the responsibility of a dedicated background process. Third, it improves scalability. Because event publication happens asynchronously, user requests complete more quickly, and slow downstream systems no longer delay the original transaction. Finally, it naturally embraces eventual consistency. Instead of requiring every service to update simultaneously, the system guarantees that every interested service will eventually receive the event once communication becomes available.
Common Mistakes
Like every architectural pattern, the Outbox Pattern can be implemented incorrectly.
One common mistake is assuming that publishing an event means the work is finished. Publishing only guarantees that the message reached the broker. Consumers may still fail, messages may still be retried, and downstream services must therefore remain resilient and, where appropriate, idempotent.
Another common mistake is forgetting to clean up the outbox table. If events remain forever, the table will continue growing until queries become unnecessarily expensive. Most production systems archive or remove published events after an appropriate retention period.
Developers also sometimes attempt to perform expensive business logic inside the publisher. The publisher should remain intentionally simple. Its responsibility is publishing events, not recalculating business rules or modifying application state.
Finally, don’t assume every database change requires an event. Publishing unnecessary events creates noise, increases infrastructure costs, and makes systems more difficult to understand. Good events represent meaningful business occurrences, not individual SQL statements.
Outbox Pattern vs Distributed Transactions
At first glance, the Outbox Pattern and distributed transactions appear to solve the same problem. Both attempt to coordinate work across multiple systems, but the difference lies in how they approach consistency. Distributed transactions attempt to make every participating system commit or roll back together. The Outbox Pattern accepts that this is often impractical in distributed architectures. Instead, it commits business data first and guarantees that events will eventually be delivered. This approach sacrifices immediate consistency in exchange for simplicity, resilience, and availability. For modern cloud-native applications, that trade-off is often the better engineering decision.
Outbox Pattern vs Event Sourcing
The Outbox Pattern is also frequently confused with Event Sourcing. Although both involve events, they solve entirely different problems. With the Outbox Pattern, the database remains the primary source of truth, and events simply communicate that something has happened. With Event Sourcing, events are the source of truth. Instead of storing the current state of an order or account, the system stores every event that led to its current state, and the application reconstructs state by replaying those events.
An outbox event might say:
Order Created.
An event-sourced system would permanently record every event throughout the order’s lifetime:
- Order Created
- Payment Authorized
- Inventory Reserved
- Shipment Prepared
- Order Delivered
Both patterns involve events, but only Event Sourcing uses them to reconstruct application state.
Best Practices
If you’re considering adopting the Outbox Pattern, several practices consistently lead to reliable implementations.
Keep the outbox in the same database as your business data so that both participate in the same transaction.
Design your publisher to be idempotent wherever possible. Network failures and retries are inevitable, and duplicate publication attempts should never produce incorrect results.
Monitor the outbox table. A growing backlog of unpublished events is often the earliest warning sign that something is wrong with your messaging infrastructure.
Treat event schemas as part of your public contract. Once other services depend on them, changing them carelessly becomes just as risky as changing a public API.
Finally, remember that publishing an event doesn’t guarantee it has been processed. Downstream services should always be designed to handle retries, duplicates, and temporary failures gracefully.
Bringing It All Together
At this point in the series, we’ve explored several techniques for building reliable software systems. Each one addresses a different question.
| Concept | Question It Answers |
|---|---|
| Idempotency | What if the same request is sent twice? |
| Race Conditions | What if multiple requests modify the same data simultaneously? |
| Database Transactions | What if part of my database operation fails? |
| Isolation Levels | What should concurrent transactions be allowed to see? |
| Distributed Locks | How do multiple servers coordinate work? |
| Outbox Pattern | How do I reliably notify other systems after committing data? |
Notice the progression: the concepts don’t replace one another, they build upon one another. Modern backend systems are reliable precisely because they combine multiple patterns, each solving a different class of failure.
Final Thoughts
One of the defining characteristics of distributed systems is that communication eventually fails. Networks become unreliable, applications restart, and message brokers experience outages. Trying to eliminate these failures entirely is unrealistic.
The Outbox Pattern embraces this reality by changing the problem. Instead of attempting to guarantee that two independent systems succeed simultaneously, it guarantees that business data is safely committed first and that communication will eventually catch up. This seemingly small architectural decision dramatically improves reliability because it removes timing from the equation. Your application no longer depends on Kafka, RabbitMQ, or another messaging system being available at the exact moment a customer submits a request. Instead, it relies on something your database already does exceptionally well: storing data reliably.
The next time you find yourself writing code that updates a database and immediately publishes an event, pause for a moment and ask yourself:
“What happens if my database succeeds but my message broker doesn’t?”
If the answer is “my systems become inconsistent,” you’ve probably found a place where the Outbox Pattern belongs.
What’s Next?
So far in this series, we’ve focused on making individual operations reliable. But what happens when a single business process spans multiple microservices, each with its own database and transaction?
Imagine placing an order that requires the payment service, inventory service, shipping service, and notification service to all complete successfully. What happens if one of those services fails halfway through?
In the next article, we’ll explore the Saga Pattern, one of the most widely used approaches for coordinating long-running business transactions across distributed systems.