Linux TLB Shootdowns and Their Performance Cost
A TLB shootdown is the process Linux uses to make multiple CPU cores discard cached virtual-to-physical address translations after a page-table change. It usually involves interrupting other cores, asking them to invalidate matching entries, and waiting until they have done so.
The problem: cached address translations
Programs use virtual addresses: addresses that belong to a process's private view of memory. Hardware translates each virtual address into a physical address, the location in RAM, using a page table maintained by the operating system.
A page table is made of entries. A page-table entry (PTE) says, in effect, "this virtual page maps to this physical page" and also records permissions such as readable, writable, or executable.
Walking the page table for every load and store would be expensive. CPUs therefore keep recently used translations in a small cache called the translation lookaside buffer (TLB). Once a core has cached a translation, it can translate subsequent addresses without consulting the page table again.
That optimization creates a consistency problem. Suppose a process has this mapping:
virtual page A → physical page X, writable
The process accesses it, and several cores cache that translation in their private TLBs. Linux then unmaps the page, changes its permissions, or replaces the mapping:
virtual page A → physical page Y, or no mapping at all
Updating the PTE changes memory, but it does not automatically erase copies already held in other cores' TLBs. A core with the old entry could continue accessing physical page X, perhaps with permissions Linux has revoked.
Without a way to remove those stale entries, an munmap, mprotect, process exit, copy-on-write operation, or memory reclamation could leave CPUs using an obsolete view of memory. That is both a correctness problem and, in some cases, a protection failure.
Why another core must be interrupted
Each CPU core generally has its own TLB. Core 0 cannot simply edit core 1's TLB as if it were ordinary shared memory. The usual way to make core 1 perform a local invalidation is to send it an interprocessor interrupt (IPI): a hardware interrupt directed from one CPU to another.
The high-level sequence looks like this:
- Linux changes the relevant page-table entry, with the required ordering so other CPUs see a consistent update.
- It identifies CPUs that might currently be using that process's address space.
- It invalidates the affected translation on the CPU making the change.
- It sends those other CPUs an IPI requesting a TLB flush.
- Each receiving CPU briefly enters the kernel, invalidates the requested address or range in its local TLB, and records that it has finished.
- The initiating CPU waits for the necessary acknowledgements before treating the change as complete.
The exact invalidation instruction depends on the processor architecture. Some CPUs can invalidate one virtual address; others can invalidate a range or an entire address-space context. Some architectures also provide hardware facilities for broadcasting invalidations. Linux hides those differences behind architecture-specific memory-management code, but the coordination problem remains the same.
The target is not necessarily every CPU in the machine. Linux tracks which CPUs may have run with a particular process address space. A CPU that has never used that address space cannot have a stale TLB entry for it, and a CPU currently running a different process may not need an immediate flush. Context-switch mechanisms can also make old entries harmless or mark them for later invalidation. These optimizations reduce unnecessary IPIs, but they do not remove the need for shootdowns when another CPU could still use the old translation.
What the pause costs
A shootdown is expensive because it is a distributed synchronization event on a machine whose normal work is highly parallel.
- The IPI interrupts useful work. The target core stops what it was doing, enters the kernel, performs the invalidation, and resumes.
- The initiating core may wait. If several CPUs must respond, the operation is limited by the slowest response and by interrupt-handling delays.
- Caches and pipelines are disturbed. Entering the kernel and changing translation state can evict useful instructions or data and reduce the benefit of warm CPU state.
- The cost grows with participants. A change affecting many active CPUs requires more messages and acknowledgements.
- Frequent changes amplify the overhead. An application that repeatedly maps, unmaps, or changes permissions can spend substantial time coordinating rather than doing application work.
A flush is not required for every page-table write. For example, Linux can often build new page-table structures privately before publishing them. The expensive case is a change to a translation or permission that another CPU may already have cached. The kernel therefore tries to batch invalidations, flush a range in one operation, and avoid sending shootdowns to CPUs that cannot have relevant entries.
Large pages illustrate the trade-off. One TLB entry can cover much more memory, improving translation performance, but changing part of that mapping may require more careful invalidation or breaking the large mapping into smaller ones. Either way, the invalidation must cover every stale translation that could be used.
Where you may encounter the term
The phrase often appears in performance investigations rather than application errors. Linux profiling and tracing tools may report TLB flush or shootdown activity, while an incident report may correlate high CPU usage, scheduling delays, or poor scaling with heavy mapping churn.
The key distinction is that the page-table update itself is usually not the whole cost. The cost comes from making all relevant cores agree that their cached translations are no longer valid. On a lightly loaded system, that coordination may be barely visible. On a large, busy machine with many threads repeatedly changing memory mappings, it can become a measurable limit on performance.