Distributed Locking Explained: Redis Redlock, ZooKeeper, and Why It’s Tricky

Learn distributed locking with Redis SET NX, Redlock, ZooKeeper ephemeral znodes, GC pause risks, and fencing tokens for safe concurrency.

1. Introduction

Imagine two servers trying to do the same job at the same time. One deducts money from an account. The other does the exact same deduction a split second later, because it never knew the first one was running. The customer gets charged twice. This is the kind of mess that distributed locking is meant to prevent.

On a single machine, this problem is easy. Your language gives you a lock, a mutex, or a synchronized block, and the operating system makes sure only one thread enters a critical section at a time. But the moment your system runs on many machines, that cozy guarantee disappears. There is no shared memory across servers. There is no single operating system holding everyone back. Each node thinks it is alone in the world.

So we reach for a lock that lives outside all the servers, in a shared place every node can see. Usually that shared place is Redis or ZooKeeper. The idea sounds simple. In practice, distributed locking is one of those topics that looks easy on the whiteboard and quietly breaks in production. Networks pause. Processes freeze. Clocks drift. Any one of these can hand the same lock to two servers who both believe they are the only holder.

In this article we will build the concept up slowly. First, why a normal lock does not work across machines. Then two popular ways to build a distributed lock, using Redis and using ZooKeeper. After that, the famous debate about whether Redis locks are truly safe, and finally the one trick, fencing tokens, that makes locks actually correct. By the end you will understand not just how to build one, but why so many of them are subtly wrong.

2. Why a Normal Lock Breaks Across Machines

Let us start with what you already know. Inside one program, a lock is a small flag in memory. A thread flips it to “taken”, does its work, and flips it back to “free”. Other threads wait their turn. The operating system coordinates everything, and it all works because every thread shares the same memory and the same clock.

Now split that program across five servers behind a load balancer. Each server runs its own copy. Each has its own memory. A flag set in one server’s memory means nothing to the other four. They cannot even see it. So a plain in-memory lock protects a thread from its siblings on the same box, and does absolutely nothing about the four other boxes doing the same work.

2.1 A Quick Scenario

Picture a scheduled job that sends invoice emails. You deploy three instances of your service for safety. At midnight, all three wake up and run the same job. Without coordination, every customer gets three identical invoices. Embarrassing, and in a payments system, dangerous.

What you actually want is this: all three may wake up, but only one should win the right to run the job. The other two should notice someone else is working and step aside. That shared “right to run” is the lock, and because it must be visible to every instance, it has to live somewhere all of them can reach.

2.2 What We Really Need From a Distributed Lock

Before picking a tool, it helps to write down what a good lock must give us. These are the properties people quietly assume and then forget to check.

• Mutual exclusion: at most one client holds the lock at any moment. This is the whole point.
• Deadlock freedom: if a lock holder crashes, the lock must eventually be released, or everyone waits forever.
• Fault tolerance: the lock service itself should keep working even if some of its nodes go down.
• No false confidence: a client must never believe it holds a lock when it actually does not.

That last point is where almost every tricky bug lives. Holding a lock is easy. Being sure you still hold it, at the exact moment you use it, is the hard part.

3. Building a Lock With Redis

Redis is a natural first choice. It is fast, it is in memory, and almost every backend already has it around for caching. The basic idea is to represent the lock as a single key. If the key exists, the lock is taken. If it does not, the lock is free. Whoever manages to create the key first becomes the holder.

3.1 The Naive First Attempt

A beginner’s first instinct is to check if the key exists, and if not, set it. Something like “if the lock is free, grab it”. But that is two separate steps, and between those two steps another client can slip in. Both check, both see it free, both grab it. Two holders. The lock is broken before we even started.

The fix is to make grabbing the lock a single atomic operation. Redis gives us exactly that with SET … NX. The NX flag means “only set this key if it does not already exist”. One command, no gap for anyone to sneak through.

3.2 SET NX With a Timeout

There is a second problem to handle. What if the client that holds the lock crashes before releasing it? The key would sit there forever, and nobody could ever get the lock again. That is a deadlock. To avoid it, we attach an expiry time to the key. If the holder dies, the key expires on its own after a few seconds, and the lock frees up automatically.

Acquiring a lock safely in Redis with SET NX PX

# NX  = set only if the key does not exist (atomic grab)
# PX  = expire after 30000 ms, so a dead holder can't block forever
# The value is a random, unique token for THIS client only.

SET lock:invoice-job  "c9f3-7b21-uuid"  NX  PX 30000

# Reply is OK   -> you got the lock
# Reply is nil  -> someone else holds it, back off and retry

Notice the value is a random token unique to this client. That detail matters more than it looks, and we will see why in a second.

3.3 Releasing the Lock Without Stealing It

Releasing seems trivial: just delete the key. But there is a sneaky bug here. Suppose client A grabs the lock with a 30 second timeout. Client A then freezes, maybe a long garbage collection pause. The 30 seconds pass, the key expires, and the lock is now free. Client B grabs it. Then client A wakes up, finishes its work, and deletes the key, which now belongs to B. Client A just released a lock it no longer owned, and a third client can now grab it while B is still working.

This is why the value is a unique token. On release, the client must check that the token stored in Redis still matches its own. Only then should it delete. And that check-and-delete must be atomic too, so we run it as a tiny Lua script.

Safe release: delete only if the token is still mine (Lua, runs atomically)

-- KEYS[1] is the lock key, ARGV[1] is my unique token
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
else
    return 0   -- not my lock anymore, do not touch it
end

3.4 The Single Redis Problem

So far we have used one Redis server. That one server is now a single point of failure. If it goes down, nobody can acquire or release locks, and your whole system stalls. The obvious fix is to add a replica. But replication in Redis is asynchronous, and that opens a nasty hole.

Here is the failure. Client A acquires the lock on the master. The master has not yet copied that key to the replica. Right then, the master crashes. The replica gets promoted to master. But the new master never heard about A’s lock, so as far as it knows, the lock is free. Client B asks for it and gets it. Now A and B both hold the same lock. Mutual exclusion is gone, and nothing in the system even noticed.

4. Redlock: Redis Locking Across Many Nodes

To fix the single-master weakness, the creator of Redis proposed an algorithm called Redlock. The idea is to stop trusting any single Redis server. Instead, run several completely independent Redis masters, with no replication between them, and require a client to win the lock on a majority of them.

4.1 How Redlock Works, Step by Step

The classic setup uses five independent Redis nodes, usually on different machines. A client that wants the lock does the following.

• Record the current time as the start. This is needed to measure how long the whole thing took.
• Try to acquire the lock on all five nodes one after another, using the same SET NX PX trick with the same token. Each attempt gets a tiny timeout, like 10 milliseconds, so a dead node does not stall the client.
• Count how many nodes said yes. The lock counts as acquired only if the client won on at least three of the five, a majority.
• Check the time spent. Subtract the elapsed time from the lock’s expiry. When enough validity time is left, the lock is truly held. Otherwise, treat it as a failure.
• Should the client miss a majority, or run out of time, it must release the lock on every node right away, even the ones that said yes.

The majority rule is what gives Redlock its resilience. Because the nodes are independent, one or two can die and a client can still win on three. And since two different clients cannot both win a majority of the same five nodes at the same time, you get mutual exclusion without any replication at all.

4.2 Why Timing Is Baked Into the Algorithm

Redlock does not just count node votes. It also watches the clock, and that is deliberate. Acquiring across five nodes takes real time. If it took so long that the lock is about to expire anyway, holding it is pointless. So the usable lock time is the original expiry minus the time already spent acquiring, minus a small safety margin for clock drift between machines.

This timing dependency is elegant, and it is also exactly where critics started asking hard questions. Because once an algorithm’s correctness depends on time and timeouts, you have to ask what happens when time itself misbehaves.

5. Building a Lock With ZooKeeper

ZooKeeper comes at the problem from a very different direction. Redis is a fast cache that we bent into a lock. ZooKeeper was built from day one as a coordination service for distributed systems, and locking is one of the things it was designed to do well. It is slower than Redis, but it trades that speed for much stronger guarantees.

5.1 Znodes, Ephemeral and Sequential

ZooKeeper stores data in a tree of small nodes called znodes, a bit like files in folders. Two special kinds of znode make locking clean.

• Ephemeral znodes exist only while the client that created them stays connected. If that client crashes or its session times out, ZooKeeper deletes the znode automatically. This solves the crashed-holder problem for free, with no manual expiry to tune.
• Sequential znodes get an ever-increasing number appended by ZooKeeper when they are created. The first request gets 0001, the next gets 0002, and so on. ZooKeeper assigns these numbers, so they are reliable and ordered.

Put the two together and you get an ephemeral sequential znode. It disappears if its owner dies, and it carries a number that records the order in which it was created. That combination is the heart of a ZooKeeper lock.

5.2 The Lock Recipe

The standard ZooKeeper lock works like a well-behaved queue. Everyone lines up, and the person at the front holds the lock.

• To request the lock, a client creates an ephemeral sequential znode under a shared lock folder. It receives a number, say lock-0005.
• The client then lists all children of the folder. If its own znode has the smallest number, it holds the lock. It can start its work.
• If it is not the smallest, someone ahead is holding the lock. The client sets a watch on exactly the one znode right before its own, and then waits quietly.
• When the holder finishes, it deletes its znode. ZooKeeper fires the watch on the next client in line, which rechecks, finds itself smallest, and takes the lock.
• If a holder crashes, its ephemeral znode vanishes on its own, and the line moves forward just the same.

The lock folder as a queue of ephemeral sequential znodes

/locks/invoice-job/
    lock-0003   <-- smallest number, this client HOLDS the lock
    lock-0004   <-- watches lock-0003, waiting
    lock-0005   <-- watches lock-0004, waiting

# When lock-0003 is deleted (work done OR client crashed),
# ZooKeeper notifies lock-0004 only. It becomes smallest -> its turn.

5.3 Why Watch Only the Predecessor

You might wonder why each client watches just the one znode ahead of it, instead of watching the whole folder. The reason is a classic scaling trick. If every waiting client watched the lock holder, then the moment the holder released, ZooKeeper would have to wake up every single waiter at once. They would all rush to recheck, and all but one would go back to sleep. This stampede is called the herd effect, and it hammers the server for no reason.

By watching only its immediate predecessor, each client is woken exactly once, and only when it is genuinely its turn. The line advances one step at a time, calmly. It is a small design choice that makes a huge difference under load, and interviewers love to ask about it.

6. Why Distributed Locking Is Genuinely Tricky

Now we reach the heart of the matter. Both approaches above look solid. Yet distributed locking has a famous reputation for being unsafe, and the reason is not bad code. It is that the world underneath the lock refuses to behave. Three things in particular ruin the party: process pauses, network delays, and clock drift.

6.1 The Garbage Collection Pause That Breaks Everything

Here is the scenario that started a well-known debate in the systems community. A client acquires a lock with a 30 second lease. It begins its work. Then, right in the middle, the process freezes. Maybe a long garbage collection pause in a Java service, maybe the machine was overloaded, maybe it got paused by the hypervisor. Whatever the cause, the client is frozen and has no idea.

While it is frozen, the 30 second lease expires. The lock service, quite correctly, gives the lock to another client. That second client starts working on the same resource. Then the first client unfreezes. It does not know any time has passed. It believes it still holds the lock, and it marches on to write to the resource. Now two clients write at once. The lock did its job perfectly and the system still broke.

The key lesson is uncomfortable. A lease that expires does not stop a frozen client. The client only finds out it lost the lock if it checks, and a frozen client checks nothing. No amount of clever locking on the Redis or ZooKeeper side can fix this alone, because the problem is on the client that went to sleep holding a knife.

6.2 Networks and Clocks Make It Worse

The same shape of bug appears in other costumes. A network delay can hold up a client’s “I still have the lock” message so long that the lock expires in the meantime. A clock that jumps forward, because of a bad time sync, can make a lock expire earlier than anyone expected. Redlock in particular leans on timing, so if the clocks on the Redis machines drift apart, its safety reasoning weakens.

• Process pause: the holder freezes, the lease expires, two clients end up active.
• Network delay: messages arrive late, so a client acts on stale lock information.
• Clock drift: timeouts fire at the wrong moment, so leases expire sooner or later than planned.

6.3 The Redlock Debate in Short

This is exactly what sparked the Redlock debate. A well-known researcher argued that Redlock relies too heavily on well-behaved clocks and timing, and that under real pauses and drift it cannot guarantee mutual exclusion for correctness-critical work. The author of Redis replied that in practice it is safe enough for many uses and that the timing assumptions are reasonable. Both sides actually agreed on one thing: if you need true correctness, the lock by itself is not enough. You need something more.

7. Fencing Tokens: The Fix That Actually Works

The missing piece is a fencing token. The idea is beautifully simple. Every time the lock is granted, the lock service also hands out a number that always goes up. First holder gets token 33, next holder gets 34, and so on. The token never decreases and is never reused.

The client then carries this token with every write it makes to the protected resource. And here is the trick: the resource itself remembers the highest token it has seen, and rejects any write that carries a smaller one. So even if a frozen old client wakes up and tries to write with its stale token 33, the resource has already accepted token 34 from the new holder and simply refuses the old write.

The resource rejects stale writes using fencing tokens

Client A acquires lock -> gets token 33
Client A freezes (long GC pause)...
Lock expires. Client B acquires lock -> gets token 34
Client B writes with token 34 -> resource stores lastToken = 34  (accepted)

Client A wakes up, still thinks it holds the lock
Client A writes with token 33 -> resource sees 33 < 34 -> REJECTED

Result: the stale write is fenced off. Only one real update wins.

This is why fencing tokens are such a big deal. They move the final safety check from the lock service to the resource being protected. The lock can be wrong. A client can be confused about whether it still holds the lock. None of that matters, because the resource makes the final call and only ever moves forward. The old write literally cannot land.

7.1 Who Gives You Tokens Easily

ZooKeeper makes fencing tokens almost free. Remember the sequential number on each znode? That number only ever increases, so it works directly as a fencing token. This is one more reason ZooKeeper is favored for correctness-critical coordination. A plain Redis lock does not give you an ordered token out of the box, which is part of the Redlock criticism. You can build one, but you have to do the work yourself.

8. Redis Versus ZooKeeper: Which to Pick

Neither tool is simply better. They sit at different points on the speed-versus-safety line, and the right choice depends on what breaks if the lock fails. A good way to decide is to ask one blunt question: if two clients held this lock at once for a moment, would it be annoying, or would it be a disaster?

If the honest answer is “annoying but survivable”, a Redis lock is usually fine and much faster. If the answer is “a disaster, money moves or data corrupts”, lean toward ZooKeeper with fencing, or add fencing tokens on top of whatever you use.

In real systems many teams start with a simple Redis lock because it is already there and it is quick. That is a reasonable default for jobs like “make sure only one worker runs this cron”. They move to ZooKeeper, or bolt on fencing tokens, only once they have a resource where a double write would be truly costly. There is no shame in starting simple, as long as you know the limits of what you started with.

9. Common Mistakes and Interview Insights

Most distributed locking bugs come from a short list of mistakes that keep repeating across teams. Knowing them saves you from learning each one the hard way in production.

• Forgetting the expiry. A lock with no timeout turns one crash into a permanent deadlock for everyone.
• Deleting someone else’s lock. Releasing with a plain delete can wipe a lock that already expired and belongs to a new holder. Always check your own token first.
• Setting the timeout too short. If the lease is shorter than the work, the lock expires mid-task and a second client jumps in. Size it generously.
• Trusting the lock for correctness without fencing. If a double-hold would corrupt data, the lock alone is not enough. You need fencing tokens.
• Using a single Redis as if it were highly available. One master is a single point of failure; a replica brings the async replication hole we saw earlier.

9.1 How to Talk About This in an Interview

Interviewers rarely want you to recite an algorithm. They want to see that you know a lock can lie to you. So when the topic comes up, show the full arc. Start simple, admit the weakness, then reach for the real fix.

A strong path sounds like this. Begin with a single SET NX PX lock and explain the token-and-Lua release. Point out the single-point-of-failure problem and mention Redlock’s majority approach as the answer. Then raise the GC pause scenario yourself, before the interviewer does, and explain that timing-based locks cannot fully guarantee correctness. Finish with fencing tokens and note that ZooKeeper’s sequential znodes give them for free. Walking that road, from naive to safe, signals real experience far better than naming tools ever could.

10. FAQ’s on distributed locking

11. Key Takeaways

Distributed locking is one of those areas where the simple version works in the demo and fails at 3 a.m. under load. The concepts are not hard, but the failure modes are quiet, and that is what makes the topic tricky. Keep a few ideas with you into any design.

• A single-machine lock cannot coordinate many machines, so the lock must live in a shared service all nodes can reach.
• To be safe, a Redis lock needs three things: an atomic grab with SET NX, an expiry so crashes self-heal, and a unique token with an atomic release.
• Redlock spreads the lock across several independent Redis nodes and needs a majority, removing the single point of failure.
• ZooKeeper uses ephemeral sequential znodes, giving automatic crash cleanup, fair ordering, and a built-in fencing token.
• Process pauses, network delays, and clock drift can all make a client believe it holds a lock it has already lost.
• Fencing tokens are the real fix for correctness: the resource rejects any write carrying an older token than the newest it has seen.

Master this progression, from a naive key to a fenced write, and you will understand distributed locking better than most engineers who have shipped it. More importantly, you will know when a simple lock is good enough, and when you must reach for something stronger. That judgment is the real skill.

12. Further Reading

• CAP Theorem and PACELC
• Consistency Models in System Design
• ReentrantLock in Java
• Martin Kleppmann: How to do distributed locking
• Redis documentation: Distributed Locks with Redis