Beyond CRUD: Building Reliable Software Systems · Part 7
New to this series? Start with Part 1
Distributed Locks Explained: Technologies That Implement Distributed Locks
Now that we understand how distributed locks work conceptually, the next question becomes:
Where do these locks actually live?
Unlike database transactions, distributed locks cannot rely on the memory of a single application server. Remember, our application might be running on five, ten, or even hundreds of machines, and every server must consult the same source of truth before deciding whether it may perform a particular operation.
Over the years, several technologies have emerged to solve this coordination problem. Although they all provide distributed locking capabilities, they were designed with slightly different goals in mind. Let’s look at the most common ones.
Redis
For most web applications, Redis is by far the most popular choice. Originally designed as an in-memory data store, Redis is incredibly fast, making it an excellent candidate for lightweight coordination tasks.
Acquiring a lock in Redis is surprisingly straightforward. A server attempts to create a key using an atomic command that says, in effect:
“Create this key only if it doesn’t already exist.”
If Redis successfully creates the key, the server owns the lock; if the key already exists, another server is already performing the work. Because Redis executes this operation atomically, two servers can never successfully create the same lock at the same time.
A simplified example looks like this:
1
2
3
4
5
6
7
SET invoice-generation
Server-A
NX
EX 30
The command says:
- Create the key only if it doesn’t already exist (
NX). - Automatically expire it after thirty seconds (
EX 30).
In one atomic operation, Redis both acquires the lock and ensures it won’t remain forever if the server crashes. For many applications, this is all that’s needed.
Why Redis Is So Popular
Redis has become the default choice for distributed locks because it satisfies three important requirements. First, it’s extremely fast: since Redis stores data in memory rather than on disk, lock acquisition usually takes only a few milliseconds. Second, many applications already use Redis for caching, sessions, queues, or rate limiting, so adding distributed locking often requires little additional infrastructure. Finally, Redis has mature client libraries for virtually every programming language, with Laravel, Django, Spring Boot, .NET, Node.js, and Go all providing excellent Redis support.
For the vast majority of business applications, Redis offers an excellent balance between simplicity, performance, and reliability.
The Challenge with a Single Redis Instance
Suppose your entire application depends on one Redis server. Everything works perfectly, then Redis crashes. Suddenly, no application server can acquire new locks, and even worse, if Redis loses its in-memory state during a restart, locks may disappear unexpectedly.
This introduces a new challenge: the coordinator itself has become a single point of failure. For many applications, this risk is acceptable. For others, particularly financial systems or globally distributed services, it isn’t. This challenge led to one of the most discussed topics in distributed systems: the Redlock algorithm.
The Redlock Algorithm
Redlock was proposed by Redis creator Salvatore Sanfilippo as a way to make Redis-based distributed locks more resilient. Instead of relying on one Redis server, Redlock uses multiple independent Redis instances.
Imagine five Redis servers.
1
2
3
4
5
6
7
8
9
Redis 1
Redis 2
Redis 3
Redis 4
Redis 5
When acquiring a lock, the application attempts to obtain it from all five servers, and the lock is considered successful only if a majority agree.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
Acquire Lock
↓
Redis 1 ✅
Redis 2 ✅
Redis 3 ✅
Redis 4 ❌
Redis 5 ❌
Three out of five succeeded, so the application proceeds. If only two servers grant the lock, the operation fails because a majority wasn’t reached. This approach significantly reduces the impact of individual Redis failures.
However, Redlock is also one of the most debated algorithms in distributed systems. Some engineers argue that it’s sufficient for many practical systems, while others, including Martin Kleppmann, have published detailed critiques explaining situations where Redlock may not provide the guarantees developers expect. The important lesson isn’t that Redlock is good or bad; it’s that distributed systems involve trade-offs, and understanding those trade-offs matters more than memorizing algorithms.
ZooKeeper
Long before Redis became popular for distributed locking, many large distributed systems relied on Apache ZooKeeper. ZooKeeper was designed specifically for coordination. Rather than functioning as a cache, it provides services such as:
- Distributed locks
- Leader election
- Configuration management
- Service discovery
Think of ZooKeeper as a highly reliable coordinator for distributed applications. Its primary goal isn’t speed, it’s correctness. Large systems such as Apache Kafka, Hadoop, and HBase have historically relied on ZooKeeper to coordinate clusters of machines. If your application requires complex distributed coordination rather than simple locking, ZooKeeper remains an excellent choice.
etcd
If you’ve worked with Kubernetes, you’ve already encountered etcd, even if you didn’t realize it. Every Kubernetes cluster stores its configuration inside etcd. Like ZooKeeper, etcd is a distributed key-value store designed for coordination rather than caching. It provides:
- Distributed locks
- Leader election
- Configuration storage
- Consensus
- Service coordination
Unlike Redis, etcd prioritizes consistency over raw performance. Its API is also designed around long-lived leases, making lock management particularly elegant. Modern cloud-native applications frequently choose etcd when they already operate within Kubernetes ecosystems.
Consul
HashiCorp Consul occupies a similar space. Although many developers know Consul for service discovery, it also provides distributed locking capabilities through sessions. Organizations using Consul often rely on it for:
- Service registration
- Health checks
- Distributed configuration
- Leader election
- Distributed locks
Like ZooKeeper and etcd, Consul focuses on reliable coordination across distributed infrastructure.
Which Technology Should You Choose?
There isn’t a universal answer. Instead, the right choice depends on your application’s requirements. If you’re building a typical web application that already uses Redis, implementing distributed locks with Redis is usually the simplest and most practical solution. If you’re coordinating hundreds of services across a Kubernetes cluster, etcd may integrate more naturally with your infrastructure. If your organization already uses Consul or ZooKeeper for service coordination, leveraging those existing systems often makes more sense than introducing Redis solely for locking.
Choosing a technology isn’t just about features. It’s also about operational complexity, existing infrastructure, and the guarantees your business requires. The important thing to remember is this: distributed locks are a concept, while Redis, ZooKeeper, etcd, and Consul are simply different tools for implementing that concept. Understanding the underlying idea is far more valuable than becoming attached to a specific technology.
Do You Always Need a Distributed Lock?
Reading this article, it might be tempting to conclude that distributed locks are the solution to every concurrency problem. They’re not. In fact, many applications never need them. If your application runs on a single server, ordinary in-memory locks are often sufficient, and if your database transaction already guarantees correctness, introducing a distributed lock may simply add unnecessary complexity.
Distributed locks are powerful, but they should be introduced only when multiple independent application instances genuinely need to coordinate shared work. Like every distributed systems technique, they solve a very specific class of problems, and the best engineering decision is often knowing when not to use them.
Common Mistakes When Using Distributed Locks
Like many distributed systems concepts, distributed locks appear deceptively simple: acquire a lock, perform some work, release the lock. In practice, however, there are several subtle mistakes that can introduce bugs that are even harder to diagnose than the problem the lock was intended to solve. Understanding these pitfalls is just as important as understanding distributed locks themselves.
Assuming a Lock Lasts Forever
One of the most common mistakes is forgetting that distributed locks usually have an expiration time. Suppose a server acquires a lock with a TTL of thirty seconds, and the developer assumes the operation will always finish within that window. Months later, a new feature makes the operation take forty-five seconds. The lock expires while the first server is still working, another server acquires the same lock and begins executing the exact same task, and suddenly, duplicate work appears again.
Choosing an appropriate TTL, and renewing it for long-running tasks when necessary, is essential.
Forgetting to Release the Lock
Although expiration protects against permanent deadlocks, applications should still release locks as soon as the protected work finishes. Holding a lock longer than necessary reduces concurrency and delays other servers waiting to perform legitimate work.
A good rule is simple:
Hold the lock only for the work that genuinely requires exclusive access.
Everything else should happen outside the lock whenever possible.
Protecting Too Much Code
Developers sometimes wrap entire workflows inside a distributed lock. For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Acquire Lock
↓
Call External Payment API
↓
Generate PDF
↓
Upload File
↓
Send Email
↓
Release Lock
This means every other server waits while network requests, file generation, and email delivery are taking place. Often, only a small portion of the workflow actually requires exclusive access. A better approach is to keep the critical section as short as possible.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Acquire Lock
↓
Update Shared Resource
↓
Release Lock
↓
Generate PDF
↓
Send Email
The shorter the lock, the better the system scales.
Distributed Locks vs Database Locks
At first glance, distributed locks and database locks appear very similar. Both prevent concurrent operations and both coordinate access to shared resources, but they operate at completely different levels.
A database lock protects data inside the database. For example, when two transactions attempt to update the same customer record, the database can lock that row until one transaction completes. Everything happens within the database engine itself.
A distributed lock protects work performed by application servers. Instead of preventing two transactions from updating the same row, it prevents two application instances from starting the same business process. Consider generating monthly invoices: before any invoice rows even exist in the database, every server must first decide whether it should begin the job. That decision happens outside the database, and a distributed lock coordinates that decision.
An easy way to remember the difference is:
Database locks protect data. Distributed locks protect business operations.
In many systems, you’ll use both together. A distributed lock ensures only one server begins generating invoices, and database transactions then ensure every invoice is written consistently.
Distributed Locks vs Optimistic Concurrency
Another concept frequently confused with distributed locks is optimistic concurrency. Optimistic concurrency assumes conflicts are relatively rare. Instead of preventing multiple users from editing the same record, it detects whether someone else changed the data before saving.
Suppose two employees open the same customer profile, and each begins editing. The system stores a version number alongside the record. When Employee A saves, the version changes from 5 to 6. When Employee B later attempts to save, the application notices that the version has already changed. Rather than silently overwriting Employee A’s work, it rejects the update and asks Employee B to refresh the page. No locking was required; the conflict was simply detected before committing.
Distributed locks take a different approach. Instead of detecting conflicts afterward, they prevent conflicting work from starting in the first place. Neither technique is universally better: optimistic concurrency works well when conflicts are uncommon, while distributed locks work best when duplicate execution would be expensive or dangerous.
Best Practices
As with most distributed systems techniques, simplicity is your friend. If you’re considering introducing distributed locks into your application, the following guidelines will help you avoid many common problems.
Keep Critical Sections Small
Acquire the lock immediately before modifying shared resources, and release it immediately afterward. The less work performed while holding the lock, the better your system scales.
Always Use Lock Expiration
Servers crash, containers restart, and networks fail. Never assume your application will always release its lock correctly. Expiration protects the rest of the system from waiting forever.
Verify Lock Ownership
Before releasing a lock, ensure your application still owns it. Ownership checks prevent one server from accidentally deleting another server’s lock after an expiration or retry.
Design for Failure
Distributed systems should always assume that something will eventually fail. Ask yourself:
- What happens if Redis becomes unavailable?
- What happens if the server crashes?
- What happens if the network partitions?
- What happens if the lock expires unexpectedly?
Thinking through failure scenarios early often prevents painful production incidents later.
Combine Reliability Patterns
Distributed locks are rarely used in isolation. Production systems often combine multiple reliability techniques. For example:
- Idempotency prevents duplicate requests.
- Transactions guarantee atomic database updates.
- Isolation Levels provide predictable views of data.
- Distributed Locks coordinate multiple application servers.
Each technique solves a different problem. Together, they create systems that continue behaving correctly even under heavy load and unexpected failures.
Bringing It All Together
At this point in the series, we’ve explored several concepts that all contribute to building reliable backend systems. Each one addresses a different question.
| Concept | Question It Answers |
|---|---|
| Idempotency | What if the same request is sent twice? |
| Race Conditions | What if multiple requests modify the same data simultaneously? |
| Database Transactions | What if part of my operation fails? |
| Isolation Levels | What should transactions be allowed to see while others are running? |
| Distributed Locks | How do multiple servers agree who should perform a task? |
Notice how these concepts complement one another. None replaces the others; reliable systems are built by combining the right tools for the right problems.
Final Thoughts
As applications grow, the biggest challenges often aren’t writing business logic. They’re coordinating work across multiple users, multiple requests, multiple transactions, and eventually multiple servers.
Distributed locks exist because modern applications are no longer confined to a single machine. They’re deployed across clusters, containers, cloud regions, and worker nodes that all need to cooperate without constantly stepping on each other’s toes.
Like transactions and isolation levels, distributed locks aren’t something you’ll use for every feature. But when you do need them, they can be the difference between a system that behaves predictably and one that quietly creates duplicate invoices, repeated payments, or inconsistent business data.
The next time you design a background job, scheduled task, or critical workflow, ask yourself one simple question:
“What happens if two servers try to do this at exactly the same time?”
If the answer is “something bad,” you’ve probably found a place where a distributed lock belongs.
What’s Next?
We’ve now covered:
- Contract Testing
- Idempotency
- Race Conditions
- Database Transactions
- Database Concurrency
- Database Isolation Levels
- Distributed Locks
So far, every concept has focused on keeping operations consistent while they’re happening. But there’s another challenge waiting. Imagine you’ve successfully updated your database inside a transaction and now need to publish an event to Kafka, RabbitMQ, or another message broker. What happens if the database commit succeeds, but publishing the event fails? Or worse, what if the event is published but the transaction rolls back?
This problem has caused countless production incidents in distributed systems. In the next article, we’ll explore The Outbox Pattern Explained: Publishing Events Without Losing Data, one of the most widely used patterns for ensuring your database and message broker stay in sync.