arthrop0d

Brief articles on varied topics.

Windows I/O Completion Ports and Worker Scheduling

microsoftconcurrency

Windows I/O completion ports (IOCPs) are kernel-managed queues that receive a notification when an asynchronous I/O operation finishes. Worker threads remove those notifications and process the results, while the port controls how many workers are allowed to run at once.

The problem IOCP solves

A synchronous read or socket operation occupies its calling thread until the operating system finishes it. That is simple, but it scales poorly when many connections spend most of their time waiting for network data: one thread can be stuck per connection.

Windows also supports overlapped I/O, its name for operations that can continue after the initiating function returns. The caller supplies an OVERLAPPED structure, starts the operation, and does other work while the kernel performs the I/O. The operation might finish immediately, or the function might return with ERROR_IO_PENDING, meaning completion is still in progress.

Overlapped I/O removes the need for the initiating thread to wait, but the program still needs a reliable way to learn that the operation finished. It could poll each operation, use one event per operation, or create a callback arrangement. Those approaches become awkward and expensive with thousands of outstanding reads and writes.

An IOCP provides one shared completion queue and a controlled group of workers.

Registering handles and starting operations

The usual setup has three steps:

  1. Create a completion port with CreateIoCompletionPort.
  2. Associate supported handles—commonly sockets or files—with that port, optionally attaching a completion key to each handle.
  3. Start overlapped operations such as ReadFile, WriteFile, WSARecv, or WSASend.

The OVERLAPPED structure passed to an operation must remain valid until that operation completes. Applications commonly embed it in a larger per-operation object containing the buffer, operation type, and other state.

A simplified shape looks like this:

struct IoRequest {
    OVERLAPPED overlapped;
    char buffer[4096];
    enum { Read, Write } kind;
};

// The socket was previously associated with an IOCP.
// Start an overlapped receive using request->overlapped.
// If the call returns WSA_IO_PENDING, completion will arrive later.

The exact operation call differs between files and sockets, but the important part is the shared OVERLAPPED object. The program does not create a waiting thread for this individual request.

Completion becomes a queued packet

When the kernel finishes the operation, it places a completion packet on the associated port. The packet contains, among other things:

A worker retrieves a packet by calling GetQueuedCompletionStatus. That call can block while the port is empty. Once it returns, the worker uses the completion key and the OVERLAPPED pointer to identify the connection and operation, checks the result, and usually starts another overlapped operation.

The port does not call an application callback for you. It is a queue and a wake-up mechanism; your worker code dispatches the packet to the appropriate connection or operation handler.

A failed I/O operation can also produce a completion. Therefore, worker code must inspect both the return status from GetQueuedCompletionStatus and the error information it reports, rather than treating every dequeued packet as success.

The same mechanism can represent application messages too. PostQueuedCompletionStatus lets a program place its own packet on the port, which is useful for asking workers to stop or for sending internal work through the same scheduling path.

Why there is not one thread per operation

The crucial distinction is between outstanding operations and active workers.

An application might have 10,000 receives in progress, but those receives consume no user-mode thread while the network or storage device is working. Their eventual results wait in the IOCP queue. The application might use only a small worker pool to process those results:

10,000 overlapped operations
          ↓
   one completion queue
          ↓
   a limited worker pool

When creating the port, the application supplies a concurrency value. This tells Windows how many threads associated with that port should be allowed to execute concurrently. A value of zero selects a system-based default, generally the number of processors.

Workers call GetQueuedCompletionStatus and sleep when there is no packet. When a completion arrives, Windows makes an appropriate waiting worker runnable. It does not need to wake every worker to announce the same event. As workers finish processing and ask for more packets, they can continue consuming queued work.

This limit is not a fixed number of threads and not a promise that exactly that many threads exist. It is a limit on how many associated workers Windows lets run concurrently for the port. An application may create somewhat more worker threads than the limit so that one blocked worker does not prevent progress, but the port's scheduling policy avoids allowing all of them to run simultaneously during normal processing.

That distinction matters because a worker can accidentally block while handling a completion—for example, by waiting on another lock or performing a synchronous operation. Windows can schedule another eligible worker, while the concurrency setting still bounds normal parallel execution. Applications should nevertheless keep completion handlers short and avoid blocking where possible.

What this looks like in practice

A log line mentioning GetQueuedCompletionStatus, ERROR_IO_PENDING, an OVERLAPPED pointer, or an “I/O completion port” usually describes this division of labor: the kernel performed the wait, queued the result, and a worker later processed it.

IOCP therefore combines two ideas. Completion queuing lets many asynchronous operations share one result-delivery mechanism. Concurrency-controlled workers let the process handle those results without allocating a thread for every operation. The result is a model suited to servers with many mostly-idle connections: outstanding I/O scales independently from the number of threads actively executing application code.

← All articles