How to Write Deterministic Unit Tests for Concurrent Code

A test that fails one run in fifty is worse than no test at all. It teaches your team to hit re-run, and eventually to ignore the red build entirely. Concurrency is the usual culprit: two threads, one shared queue, and an assertion that quietly depends on which of them got there first. The answer isn't faster hardware or longer timeouts. It's writing tests where the interleaving is something you choose rather than something the operating system hands you.

Why concurrent tests flake

Almost every flaky concurrency test has one of three root causes. It sleeps and hopes. It asserts on elapsed time. Or it reaches into a global it can't control — a thread pool, a system clock, a singleton queue.

Sleeping is the most common. You start a worker, sleep for 200 milliseconds, then check the result. On a quiet laptop that works. On a CI machine running twelve other jobs, the worker hasn't finished. So you raise the sleep to two seconds, the suite gets slower, and it still fails occasionally.

Aim for something narrower than "test the concurrency". Make the test deterministic: same result on every run, on any machine, in milliseconds. Then decide separately whether you also need a stress test to catch races that no single interleaving will reveal.

Inject the sources of non-determinism

Dependency injection is usually described as a general testability pattern. For concurrency it is more specific: it means pulling the non-deterministic parts of the system out to the edge of your code, to a place where a test can swap them.

The usual suspects:

• A clock — anything that reads wall-clock time or sleeps.
• An executor or task scheduler — thread pools, dispatch queues, timers.
• A source of randomness — inject a seeded generator rather than calling a global one.
• A queue or channel, when the production code creates its own internally.
• An ID or sequence generator, if workers stamp work as it passes through.

Take a retry policy with exponential backoff. The naive version calls sleep between attempts, so any test of it takes seconds and proves nothing about the logic. Give it a small Clock interface with now() and sleep(duration), and let the test supply a fake that records durations and returns immediately. Now you can assert the sequence 100ms, 200ms, 400ms in a test that finishes instantly, and you can also assert what happens when the clock crosses a deadline.

Fake schedulers: take control of when work runs

If the code hands work to an executor, give it one you can drive. Two fakes are useful, and they answer different questions.

An inline executor runs each submitted task immediately, on the calling thread. Every task finishes before the next is submitted, so there is no interleaving at all. That is ideal for testing pipeline logic — does the handler run, is the result posted to the right place, does an error propagate — with no thread scheduling involved. Be honest about the trade-off: an inline executor proves nothing about your locking, and it will happily pass a design that deadlocks in production.

A virtual-time scheduler keeps a priority queue of tasks ordered by their due time and advances only when the test tells it to. Call runUntilIdle(), or advance by thirty seconds, and it executes everything due in that window, in order, on one thread. Timeouts, heartbeats and retry delays all become deterministic, and a test covering an hour of scheduled work finishes in microseconds. Kotlin's coroutine test library ships a scheduler like this, and .NET offers TimeProvider with a fake implementation for testing. Where your platform provides one, use it before writing your own.

Latches and gates to force an interleaving

Some properties only exist when two threads are genuinely in flight. "A second writer blocks until the first releases the lock" cannot be tested on a single thread. This is where latches and gates earn their place.

The pattern is to hold a thread at a known point until the test releases it, then assert. Give the production code one small hook — a callback invoked at the start of the critical section — and keep it to one hook per interesting boundary.

1. Create a latch meaning "thread A is inside the section" and a second meaning "the test says go".
2. Start thread A. It signals the first latch, then blocks on the second.
3. Wait for the first latch with a generous timeout. Now you know the exact state of the world.
4. Start thread B and assert it is waiting, either via a flag or a tryLock-style call that returns false.
5. Release the second latch, join both threads, and assert on the final state.

CountDownLatch, CyclicBarrier, WaitGroup, channel, event: the names differ, the shape doesn't. What matters is that the test controls the moment of release rather than guessing at it. And if a timeout is what makes a test pass, the test is lying to you.

Assert on outcomes, not elapsed time

An assertion like "the operation finished in under 50 milliseconds" is a performance test wearing a unit test's clothes, and it will fail on a loaded build agent. Assert on state instead: the counter equals 1,000, the queue is empty, exactly one worker claimed the job. Where order matters, append events to a list and assert either an exact sequence or one of the permitted ones.

Then run the test in a loop. A deterministic test repeated a thousand times should still take under a second, and the repetition catches ordering assumptions that a single run misses.

Keep stress tests separate. Some bugs only appear under real threads and real volume, and that is worth knowing — but put those in a suite that runs nightly with many iterations, not on every commit. Flaky tests in the pull request gate teach people to ignore failures, which is precisely the wrong lesson.

Where to start

You don't need to refactor everything at once. Pick the flakiest concurrent test in the suite and find the sleep inside it. Replace the clock, the executor or the latch with something the test drives. Then do the next one.

A short checklist for the next concurrent test you write:

• Name the interleaving the test depends on, and make it explicit in the code.
• Put a hard timeout on every wait, so a deadlock fails the test instead of hanging the build.
• Prefer final state over timing.
• Build shared helpers for fake schedulers and latches, so the next person doesn't reinvent the flakiness.

Fast, repeatable tests change how you write concurrent code. You stop avoiding the tricky paths because they are awkward to test, and start proving them instead.

Photo: Pavel Danilyuk / Pexels