Beyond CRUD: Building Reliable Software Systems · Part 6
New to this series? Start with Part 1
Distributed Locks Explained: Coordinating Work Across Multiple Servers
“Database locks protect rows. Distributed locks protect systems.”
Imagine your application has been wildly successful. What started as a simple web application running on a single server now runs across multiple machines behind a load balancer. Requests are shared between application instances, background workers process jobs independently, and scheduled tasks run on every server.
From the outside, everything looks better than ever: pages load faster, traffic scales effortlessly, and users are happy. Then one morning, your finance department calls. Every customer has received four monthly invoices.
Nothing appears wrong with the code. The invoice generation job is scheduled to run once every month, so why did it execute four times? The answer is surprisingly simple: you now have four application servers. At midnight, every server woke up, checked the scheduler, and independently decided that it was responsible for generating invoices. From each server’s perspective, everything was perfectly correct; collectively, however, they created a costly mistake.
Now imagine a different scenario. Your payment gateway sends a webhook confirming a successful payment. The webhook is delivered to one of your load-balanced servers, but a retry occurs because the payment provider doesn’t receive a response quickly enough, so another server processes the same webhook. Both servers begin updating balances, both create accounting entries, and both generate receipts.
You’ve already learned how idempotency protects against duplicate requests and how transactions ensure database operations succeed together.
But neither of those concepts answers a new question:
How do multiple application servers agree that only one of them should perform a particular piece of work?
That problem is solved by distributed locks.
As applications grow beyond a single server, distributed locks become one of the most important coordination mechanisms in modern software engineering.
Why Database Locks Are No Longer Enough
Earlier in this series, we explored database transactions and row-level locking.
Suppose two transactions attempt to update the same bank account.
Using a statement such as:
1
2
3
4
SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;
the database ensures only one transaction can modify that row at a time.
This works beautifully because both transactions are coordinating through the same database.
Now consider a different problem.
Suppose you have four application servers running the exact same code, and each server has a scheduler responsible for calculating monthly loan interest. Midnight arrives, and every server starts the same scheduled job. None of them are updating the same database row immediately. Instead, they’re deciding whether to begin an entire business process. The database has nothing to lock yet, and by the time one server begins writing data, the others have already started processing. The result is duplicated work.
Database locks are excellent at protecting individual records, but they are not designed to coordinate entire applications spread across multiple machines. This is the gap distributed locks were created to fill.
What Is a Distributed Lock?
A distributed lock is a coordination mechanism that allows multiple independent servers to agree that only one of them may perform a particular operation at a given time. Instead of protecting a single database row, a distributed lock protects an entire business activity.
Imagine a conference room with a single key. Anyone can use the room, but only the person holding the key may enter, and everyone else must wait until the key is returned. A distributed lock works in much the same way: before performing an operation, a server first attempts to acquire the lock. If the lock is available, the server proceeds; if another server already owns the lock, the operation waits, retries, or exits.
The important point is that every server asks the same central authority for permission before beginning work. That authority might be Redis, ZooKeeper, etcd, or another distributed coordination system.
A Simple Example
Imagine four application servers.
1
2
3
4
5
6
7
Load Balancer
│
┌──────────┼──────────┐
│ │ │
Server A Server B Server C
│ │ │
Server D
At midnight, each server checks whether it’s time to generate invoices.
Without a distributed lock:
1
2
3
4
5
6
7
Server A → Generate invoices ✅
Server B → Generate invoices ✅
Server C → Generate invoices ✅
Server D → Generate invoices ✅
Four executions, four invoices, and one very unhappy finance department.
With a distributed lock:
1
2
3
4
5
6
7
Server A → Acquire Lock ✅
Server B → Lock Exists
Server C → Lock Exists
Server D → Lock Exists
Only Server A proceeds, and everyone else exits. The invoices are generated exactly once.
Real-World Examples
Distributed locks appear in far more places than most developers realize. Whenever multiple application instances could accidentally perform the same work, a distributed lock becomes a potential solution.
Scheduled Jobs
Suppose your loan management platform calculates accrued interest every night at midnight. Without coordination, every application server performs the calculation independently, interest may be applied multiple times, and a distributed lock ensures exactly one server performs the calculation.
Payment Processing
Imagine receiving a payment webhook from M-Pesa. Network retries cause multiple servers to receive the same notification, and without coordination, several servers may attempt to update balances simultaneously. A distributed lock allows only one server to process the payment while the others simply exit.
Inventory Management
Only one laptop remains in stock, and two servers receive purchase requests simultaneously. Each attempts to reserve the item. Although transactions protect database consistency, a distributed lock can coordinate reservation workflows across multiple application instances before they even begin modifying inventory.
Sending Emails
Marketing decides to send a promotional email to one million subscribers, and your email scheduler is deployed across five worker nodes. Without coordination, every worker starts sending the campaign and customers receive the same email five times. With a distributed lock, only one worker initiates the campaign while the others remain idle.
Report Generation
Generating annual financial reports may take several minutes. Without coordination, multiple servers might begin generating the exact same report simultaneously, wasting CPU time and increasing database load. A distributed lock ensures only one report generation process is active.
When Should You Consider a Distributed Lock?
A useful rule of thumb is to ask yourself a simple question:
What would happen if two servers performed this operation at exactly the same time?
If the answer is:
- Duplicate invoices
- Duplicate payments
- Duplicate notifications
- Duplicate accounting entries
- Duplicate interest calculations
then the operation is probably a candidate for a distributed lock.
Not every feature needs one. Most HTTP requests don’t, reading data doesn’t, and serving web pages doesn’t. Distributed locks are primarily useful for shared background work and critical business processes where duplicate execution would produce incorrect results.
Distributed Locks Are About Coordination
One misconception worth addressing early is that distributed locks replace database transactions.
They don’t. Transactions guarantee consistency inside the database; distributed locks coordinate between application servers. The two solve different problems.
In practice, many enterprise systems use both together. A server first acquires a distributed lock, then begins a database transaction. When the transaction completes successfully, the server releases the distributed lock. The lock ensures only one server performs the work, and the transaction ensures the database remains consistent while that work is being performed. Together, they provide a powerful foundation for building reliable distributed systems.
At this point, we’ve answered why distributed locks exist.
The next question is equally important:
How do multiple servers actually agree on who owns the lock?
How Distributed Locks Work
At a high level, every distributed lock follows the same basic workflow.
Before performing a critical operation, an application asks a shared coordination service for permission to proceed. If the lock is available, the application acquires it and begins its work; if another server already owns the lock, the application waits, retries later, or simply exits. Once the work has been completed, the lock is released, allowing another server to acquire it.
Although different technologies implement this process differently, the underlying idea remains remarkably simple.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Application Server
│
Request Lock
│
───────────────
Lock Service
───────────────
Lock Available?
│ │
Yes No
│ │
Acquire Wait / Retry / Exit
│
Perform Work
│
Release Lock
The important detail is that every application instance talks to the same lock service. Without a shared source of truth, each server would simply believe it owned the lock.
Acquiring a Lock
Imagine four servers attempting to generate monthly invoices.
Each server sends a request to Redis asking for a lock called:
1
invoice-generation
Redis receives the requests almost simultaneously. The first request succeeds, and Redis stores something similar to:
1
2
3
4
5
invoice-generation
Owner: Server A
Expires: 30 seconds
When the remaining servers ask for the same lock, Redis responds that the lock already exists. Only Server A continues. Servers B, C, and D either wait, retry after a short delay, or abandon the operation altogether.
The beauty of distributed locks lies in their simplicity: instead of every server making its own decision, they all trust a single coordinator.
Holding the Lock
Once a server has successfully acquired the lock, it proceeds with the protected operation.
This might involve:
- Generating invoices.
- Processing a payment.
- Calculating loan interest.
- Sending reminder emails.
- Synchronizing inventory.
During this period, every other server attempting the same operation sees that the lock is already owned. Rather than performing duplicate work, those servers simply back off. The lock effectively becomes a reservation saying that someone is already doing this work and others should wait.
Releasing the Lock
When the protected work completes successfully, the server releases the lock.
1
2
3
4
5
6
7
8
9
Acquire Lock
↓
Process Work
↓
Release Lock
Once Redis removes the lock, another server is free to acquire it if necessary. Releasing the lock is just as important as acquiring it. A lock that is never released eventually blocks every future attempt to perform that operation.
The Problem with Permanent Locks
Now consider something less pleasant.
Server A acquires the invoice-generation lock, but halfway through generating invoices, the server crashes. Perhaps the machine loses power, perhaps Kubernetes terminates the container, or perhaps someone accidentally restarts the application. The important point is that Server A never gets the opportunity to release its lock.
If the lock remained permanent, every future invoice generation attempt would fail because the system would forever believe Server A still owned the lock. This is one of the biggest differences between traditional application locks and distributed locks: distributed systems must always assume that servers can disappear without warning.
Lock Expiration (TTL)
To solve this problem, distributed locks almost always include an expiration time, often called a Time-To-Live (TTL).
Instead of storing only the lock name, the lock service stores something like:
1
2
3
4
5
6
7
8
9
10
11
Lock:
invoice-generation
Owner:
Server A
Expires:
30 seconds
If Server A completes successfully, it releases the lock before those thirty seconds expire. If Server A crashes, Redis automatically deletes the lock after the TTL expires, allowing another server to continue the work instead of waiting forever.
Think of it like borrowing a meeting room: rather than reserving it indefinitely, your booking automatically expires after one hour. If you forget to leave, the reservation eventually disappears and someone else can use the room. TTL prevents abandoned locks from permanently blocking the system.
Choosing the Right TTL
Choosing a lock duration isn’t as straightforward as it might seem.
Suppose generating invoices normally takes ten seconds, and a thirty-second TTL provides plenty of room for occasional delays. But what if one month the process unexpectedly takes forty-five seconds? The lock expires after thirty seconds, another server acquires it, and now both servers are generating invoices simultaneously. You’ve accidentally recreated the very problem the lock was supposed to prevent.
On the other hand, choosing an extremely long TTL isn’t ideal either. If a server crashes while holding a lock that expires after thirty minutes, every other server must wait half an hour before continuing. Finding the right TTL therefore requires understanding how long your operation normally takes, while leaving enough room for occasional delays.
Some distributed lock implementations even allow servers to periodically renew the TTL while they’re still actively working. This approach, often called a heartbeat, keeps long-running operations alive without requiring excessively long expiration times.
What Happens If Two Servers Ask at the Same Time?
One question naturally arises: what happens if two servers request the lock at exactly the same millisecond? The answer depends on the lock service.
Redis, ZooKeeper, etcd, and similar systems perform lock acquisition atomically. That means checking whether the lock exists and creating it happen as a single indivisible operation. There is never a moment when both servers successfully acquire the same lock: one request succeeds and the other fails. This atomicity is exactly what makes distributed locks reliable; without it, two servers could both believe they owned the lock, defeating the entire purpose.
Lock Ownership Matters
Imagine Server A acquires a lock. Before finishing its work, the lock expires because the TTL was too short, and Server B now acquires the same lock. Moments later, Server A finally finishes and attempts to release it. If the lock service simply deleted the lock without checking ownership, Server A would accidentally remove Server B’s lock, Server C could now acquire it, and suddenly two servers are working simultaneously again.
To prevent this, distributed lock implementations associate every lock with a unique owner identifier. When releasing a lock, the application must prove that it is still the owner; if ownership has already changed, the release request is ignored. This simple verification prevents one server from accidentally deleting another server’s lock.
Distributed Locks Aren’t Magic
It’s important to understand that distributed locks don’t eliminate failures. Servers can still crash, networks can still become partitioned, and Redis instances can still fail. Distributed locks simply provide a coordinated way for multiple application instances to make decisions despite those realities. They reduce duplicate work, improve consistency, and coordinate critical business operations, but like every distributed systems technique, they must be implemented carefully and combined with other reliability mechanisms such as transactions, retries, idempotency, and monitoring.
In the next section, we’ll explore the most common technologies used to implement distributed locks, including Redis, Redlock, ZooKeeper, etcd, and Consul, along with the strengths and weaknesses of each approach.