wslc: Switch port relay AcceptThread from WaitForMultipleObjects to IO completion ports by benhillis · Pull Request #40042 · microsoft/WSL

benhillis · 2026-03-30T21:19:33Z

Replace the WaitForMultipleObjects-based wait loop in relay::MultiHandleWait with an IO completion port (IOCP), removing the MAXIMUM_WAIT_OBJECTS (64) handle limit for all callers (IORelay, container IO, socket accept, etc.).

Switch the port relay AcceptThread from WaitForMultipleObjects to a direct IOCP, removing the 63-port mapping limit.

relay::MultiHandleWait changes (relay.hpp/cpp)

Add unique_registered_wait RAII type for RegisterWaitForSingleObject handles
Add Register() pure virtual to OverlappedIOHandle, implemented by all subclasses
Handles that support IOCP (pipes, sockets) register directly; handles that don't (console handles, events) use a RegisterWaitForSingleObject bridge
Convert Run() to use GetQueuedCompletionStatus instead of WaitForMultipleObjects
Cancel() posts a key=0 completion to wake Run(); key=0 sets m_cancel on dequeue
Fix member order: m_iocp declared before m_handles for correct destruction

Port relay changes (localhost.cpp)

Associate listen sockets directly with an IOCP via CreateIoCompletionPort
AcceptEx completions go straight to the IOCP — no events or thread pool waits
Stop signal via PostQueuedCompletionStatus key=0
Remove AcceptEvent from PortRelay (no longer needed)
Remove the MAXIMUM_WAIT_OBJECTS port limit check

Copilot

Pull request overview

This PR updates the WSLC localhost port relay to use an I/O completion port (IOCP) for AcceptEx completions instead of WaitForMultipleObjects, removing the 64-handle wait limitation and enabling more than 63 concurrent port mappings.

Changes:

Replace the accept loop with an IOCP-driven AcceptThread and associate each listen socket with a shared completion port.
Remove the relay’s per-port accept event and route both sync/async AcceptEx completions through IOCP.
Update Windows tests to validate mapping/unmapping 100 ports instead of enforcing the prior 63-port limit.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`src/windows/wslrelay/localhost.cpp`	Switch accept scheduling/completions to IOCP; add stdin `MessageReader` to coordinate exit + message readiness.
`test/windows/WSLCTests.cpp`	Update port-mapping test to validate >63 mappings (now 100).

src/windows/wslrelay/localhost.cpp

test/windows/WSLCTests.cpp

OneBlue

I think this is a good idea, but given that the relay code is not our long term plan, I would recommend making this change at the relay::MultiHandleWait level so it will also solve the 64 handles issue in other places in the code (specifically the container IO thread, which could go over the limit if many Logs() / Exec() calls are made)

If we need to, we can then move the relay to use MultiHandleWait and match the rest of the code

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

src/windows/wslrelay/localhost.cpp

benhillis · 2026-04-01T15:45:54Z

I think this is a good idea, but given that the relay code is not our long term plan, I would recommend making this change at the relay::MultiHandleWait level so it will also solve the 64 handles issue in other places in the code (specifically the container IO thread, which could go over the limit if many Logs() / Exec() calls are made)

If we need to, we can then move the relay to use MultiHandleWait and match the rest of the cod

That's a good call, let me look at that.

…letion ports Replace the WaitForMultipleObjects-based accept loop in the WSLC port relay with an IO completion port (IOCP). This removes the MAXIMUM_WAIT_OBJECTS (64) handle limit, allowing unlimited port mappings. Changes: - Associate each listen socket with a shared IOCP using the PortRelay pointer as the completion key. - Rewrite AcceptThread to use GetQueuedCompletionStatus instead of WaitForMultipleObjects. On shutdown, cancel all pending I/O and drain completions before returning. - Remove the AcceptEvent from PortRelay (no longer needed with IOCP). - Change ScheduleAccept from bool to void since both sync and async AcceptEx completions are now delivered through the IOCP. - Add a MessageReader that reads stdin on a dedicated thread, allowing the main loop to WaitForMultipleObjects on the exit event and message ready event simultaneously. - Remove the 63-port limit and update tests to validate 100 port mappings.

… AcceptThread - Add InheritHandle() for m_vmTerminatingEvent so the relay child process receives a valid exit event handle via PROC_THREAD_ATTRIBUTE_HANDLE_LIST. - Close parent pipe handles with .reset() instead of .release() so ReadFile in MapRelayPort detects relay process death via pipe EOF. - Move AcceptThread I/O drain into wil::scope_exit so pending accepts are always cancelled even if ScheduleAccept throws during initialization. - Wrap initial ScheduleAccept loop in try/catch to prevent a single port failure from killing the entire accept thread. - Track accept thread liveness with std::atomic<bool> to avoid posting stale IOCP exit signals when the thread has already exited. - Make PortRelay destructor cancel pending I/O instead of __fastfail. - Add <atomic> to precomp.h.

…IO completion ports Replace the WaitForMultipleObjects-based wait loop in relay::MultiHandleWait with an IO completion port (IOCP), removing the MAXIMUM_WAIT_OBJECTS (64) handle limit for all callers. Changes to relay::MultiHandleWait (relay.hpp/cpp): - Add unique_registered_wait RAII type for RegisterWaitForSingleObject handles - Add Register() pure virtual to OverlappedIOHandle, implemented by all subclasses - Handles that support IOCP (pipes, sockets) register directly; handles that don't (console handles, events) use a RegisterWaitForSingleObject bridge - Convert Run() to use GetQueuedCompletionStatus instead of WaitForMultipleObjects - Cancel() posts a key=0 completion to wake Run(); key=0 sets m_cancel on dequeue - Fix member order: m_iocp declared before m_handles for correct destruction Changes to port relay (localhost.cpp): - Replace AcceptThread's WaitForMultipleObjects with a direct IOCP: associate listen sockets via CreateIoCompletionPort, AcceptEx completions go straight to the IOCP, stop signal via PostQueuedCompletionStatus key=0 - Remove the MAXIMUM_WAIT_OBJECTS port limit check - Remove AcceptEvent from PortRelay (no longer needed with direct IOCP)

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

Copilot · 2026-04-01T21:37:38Z

src/windows/common/relay.cpp

+    if (Iocp != nullptr && !WaitHandle)
+    {
+        THROW_IF_WIN32_BOOL_FALSE(
+            RegisterWaitForSingleObject(&WaitHandle, Handle.Get(), &EventHandle::WaitCallback, this, INFINITE, WT_EXECUTEONLYONCE));


RegisterWaitForSingleObject expects a PHANDLE for the returned wait handle, but WaitHandle is a wil::unique_any wrapper. Passing &WaitHandle is not a valid output parameter type and should be replaced with the wrapper’s dedicated out-parameter helper (e.g., .put()/.addressof() depending on the WIL type) so this compiles and the previous registration is correctly overwritten.

Suggested change

RegisterWaitForSingleObject(&WaitHandle, Handle.Get(), &EventHandle::WaitCallback, this, INFINITE, WT_EXECUTEONLYONCE));

RegisterWaitForSingleObject(WaitHandle.put(), Handle.Get(), &EventHandle::WaitCallback, this, INFINITE, WT_EXECUTEONLYONCE));

Copilot · 2026-04-01T21:37:38Z

src/windows/common/relay.cpp

+        if (!RegisteredWithIocp && Iocp != nullptr && !WaitBridge)
+        {
+            THROW_IF_WIN32_BOOL_FALSE(RegisterWaitForSingleObject(
+                &WaitBridge, Event.get(), &ReadHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));


RegisterWaitForSingleObject expects a PHANDLE output parameter, but WaitBridge is a wil::unique_any wrapper. Passing &WaitBridge is an invalid type for the API and will not compile reliably; use the wrapper’s out-parameter helper so the returned handle is stored correctly.

Suggested change

&WaitBridge, Event.get(), &ReadHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

WaitBridge.put(), Event.get(), &ReadHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

Copilot · 2026-04-01T21:37:38Z

src/windows/common/relay.cpp

+        if (!RegisteredWithIocp && Iocp != nullptr && !WaitBridge)
+        {
+            THROW_IF_WIN32_BOOL_FALSE(RegisterWaitForSingleObject(
+                &WaitBridge, Event.get(), &SingleAcceptHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));


RegisterWaitForSingleObject expects a PHANDLE output parameter, but WaitBridge is a wil::unique_any wrapper. Passing &WaitBridge is the wrong type for this API; use the wrapper’s out-parameter helper to receive the registration handle.

Suggested change

&WaitBridge, Event.get(), &SingleAcceptHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

WaitBridge.put(), Event.get(), &SingleAcceptHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

Copilot · 2026-04-01T21:37:38Z

src/windows/common/relay.cpp

+        if (!RegisteredWithIocp && Iocp != nullptr && !WaitBridge)
+        {
+            THROW_IF_WIN32_BOOL_FALSE(RegisterWaitForSingleObject(
+                &WaitBridge, Event.get(), &WriteHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));


RegisterWaitForSingleObject expects a PHANDLE output parameter, but WaitBridge is a wil::unique_any wrapper. Passing &WaitBridge is an invalid output argument type; use the wrapper’s out-parameter helper so the registration handle is captured correctly.

Suggested change

&WaitBridge, Event.get(), &WriteHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

WaitBridge.put(), Event.get(), &WriteHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

Copilot · 2026-04-01T21:37:39Z

src/windows/common/relay.cpp

+        // Try to associate the handle with the IOCP. Not all handle types support this
+        // (e.g., some socket types, console handles). Fall back to event-based mode on failure.
+        auto result = CreateIoCompletionPort(Handle.Get(), iocp, reinterpret_cast<ULONG_PTR>(completionTarget), 0);
+        if (result != nullptr)
+        {
+            SetFileCompletionNotificationModes(Handle.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);
+
+            // Clear the event from OVERLAPPED — completions now go to the IOCP.
+            Overlapped.hEvent = nullptr;
+            RegisteredWithIocp = true;
+        }
+        // else: fall back to event-based mode (Overlapped.hEvent remains set)


After associating a handle with an IOCP, the code relies on FILE_SKIP_COMPLETION_PORT_ON_SUCCESS to avoid a completion packet being queued for synchronous I/O (which would later trigger Collect() while the handle is not in the pending state). SetFileCompletionNotificationModes can fail; currently its return value is ignored and RegisteredWithIocp is still set true. Handle the failure explicitly (e.g., throw, or fall back to event-based mode and keep Overlapped.hEvent intact) to avoid double-completion/incorrect state transitions.

Suggested change

// Try to associate the handle with the IOCP. Not all handle types support this

// (e.g., some socket types, console handles). Fall back to event-based mode on failure.

auto result = CreateIoCompletionPort(Handle.Get(), iocp, reinterpret_cast<ULONG_PTR>(completionTarget), 0);

if (result != nullptr)

{

SetFileCompletionNotificationModes(Handle.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

// Clear the event from OVERLAPPED — completions now go to the IOCP.

Overlapped.hEvent = nullptr;

RegisteredWithIocp = true;

}

// else: fall back to event-based mode (Overlapped.hEvent remains set)

//

// First, attempt to configure the handle so that successful synchronous I/O

// does not queue a completion to the IOCP. If this fails, continue using

// event-based completion (Overlapped.hEvent remains set) and do not

// associate the handle with the IOCP.

//

if (!SetFileCompletionNotificationModes(Handle.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS))

{

// Fall back to event-based mode on failure.

// Overlapped.hEvent is left intact and RegisteredWithIocp stays false.

LOG_LAST_ERROR();

return;

}

// Try to associate the handle with the IOCP. Not all handle types support this

// (e.g., some socket types, console handles). Fall back to event-based mode on failure.

auto result = CreateIoCompletionPort(Handle.Get(), iocp, reinterpret_cast<ULONG_PTR>(completionTarget), 0);

if (result != nullptr)

{

// Clear the event from OVERLAPPED — completions now go to the IOCP.

Overlapped.hEvent = nullptr;

RegisteredWithIocp = true;

}

// else: fall back to event-based mode (Overlapped.hEvent remains set, mode flag is harmless)

Copilot · 2026-04-01T21:37:39Z

src/windows/common/relay.cpp

+            SetFileCompletionNotificationModes(ListenSocket.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);
+            Overlapped.hEvent = nullptr;
+            RegisteredWithIocp = true;


Same issue as other IOCP registrations: SetFileCompletionNotificationModes(..., FILE_SKIP_COMPLETION_PORT_ON_SUCCESS) return value is ignored even though the state machine assumes it succeeded (otherwise synchronous AcceptEx could still queue an IOCP packet and lead to double-processing / Collect() assertions). Consider checking the call and falling back to event-bridge mode (or failing the registration) if it isn’t supported.

Suggested change

SetFileCompletionNotificationModes(ListenSocket.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

Overlapped.hEvent = nullptr;

RegisteredWithIocp = true;

BOOL notificationModesSet = SetFileCompletionNotificationModes(ListenSocket.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

LOG_LAST_ERROR_IF(!notificationModesSet);

if (notificationModesSet)

{

Overlapped.hEvent = nullptr;

RegisteredWithIocp = true;

}

Copilot · 2026-04-01T21:37:39Z

src/windows/common/relay.cpp

+        auto result = CreateIoCompletionPort(Handle.Get(), iocp, reinterpret_cast<ULONG_PTR>(completionTarget), 0);
+        if (result != nullptr)
+        {
+            SetFileCompletionNotificationModes(Handle.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);


Same issue as other IOCP registrations: SetFileCompletionNotificationModes(..., FILE_SKIP_COMPLETION_PORT_ON_SUCCESS) is unchecked even though the code assumes synchronous I/O will not queue a completion packet. If this call fails, you can get a queued completion for an operation already handled synchronously, causing incorrect state/Collect() assertions. Handle failure (throw or fall back to event mode) before setting RegisteredWithIocp = true.

Suggested change

SetFileCompletionNotificationModes(Handle.Get(), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

THROW_IF_WIN32_BOOL_FALSE(SetFileCompletionNotificationModes(

Handle.Get(),

FILE_SKIP_COMPLETION_PORT_ON_SUCCESS));

Copilot · 2026-04-01T21:37:39Z

src/windows/common/relay.cpp

+        // Find the top-level handle in m_handles that matches the completion target.
+        auto* completionTarget = reinterpret_cast<OverlappedIOHandle*>(completionKey);
+
+        auto it = std::ranges::find_if(
+            m_handles, [completionTarget](const auto& entry) { return entry.second.get() == completionTarget; });
+
+        if (it == m_handles.end())
+        {
+            // Handle was already removed (e.g., by CancelOnCompleted). Discard the completion.
+            continue;
+        }


MultiHandleWait::Run does a linear search through m_handles for every IOCP completion to map completionKey back to the owning handle. Since this change explicitly removes the 64-handle cap, this can become an O(n) bottleneck with larger handle sets. Consider keeping an auxiliary map from OverlappedIOHandle* (completion key) to index/iterator (and updating it on erase) to make completion dispatch O(1).

Copilot · 2026-04-01T21:37:40Z

src/windows/wslrelay/localhost.cpp

+                    THROW_LAST_ERROR_IF_NULL(CreateIoCompletionPort(
+                        reinterpret_cast<HANDLE>(e.second->ListenSocket.get()), iocp.get(), reinterpret_cast<ULONG_PTR>(e.second.get()), 0));
+
+                    SetFileCompletionNotificationModes(reinterpret_cast<HANDLE>(e.second->ListenSocket.get()), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);


SetFileCompletionNotificationModes(..., FILE_SKIP_COMPLETION_PORT_ON_SUCCESS) return value is ignored, but the accept loop assumes synchronous AcceptEx completions won’t also enqueue an IOCP packet (otherwise you risk double-processing an accept). Please check/handle failure explicitly (throw, or avoid relying on skip-on-success when it isn’t supported) before proceeding.

Suggested change

SetFileCompletionNotificationModes(reinterpret_cast<HANDLE>(e.second->ListenSocket.get()), FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

BOOL completionModesSet = SetFileCompletionNotificationModes(

reinterpret_cast<HANDLE>(e.second->ListenSocket.get()),

FILE_SKIP_COMPLETION_PORT_ON_SUCCESS);

THROW_LAST_ERROR_IF(!completionModesSet);

benhillis requested a review from a team as a code owner March 30, 2026 21:19

Copilot AI review requested due to automatic review settings March 30, 2026 21:19

Copilot started reviewing on behalf of benhillis March 30, 2026 21:20 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

src/windows/wslrelay/localhost.cpp Outdated Show resolved Hide resolved

src/windows/wslrelay/localhost.cpp Outdated Show resolved Hide resolved

src/windows/wslrelay/localhost.cpp Show resolved Hide resolved

test/windows/WSLCTests.cpp Show resolved Hide resolved

OneBlue reviewed Mar 31, 2026

View reviewed changes

benhillis force-pushed the user/benhill/fix-port-relay-handle-limit branch from 7e40403 to 8370b5a Compare April 1, 2026 15:31

Copilot AI review requested due to automatic review settings April 1, 2026 15:34

benhillis force-pushed the user/benhill/fix-port-relay-handle-limit branch from 8370b5a to 87b2cd8 Compare April 1, 2026 15:34

Copilot started reviewing on behalf of benhillis April 1, 2026 15:36 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

src/windows/wslrelay/localhost.cpp Show resolved Hide resolved

src/windows/wslrelay/localhost.cpp Show resolved Hide resolved

benhillis marked this pull request as draft April 1, 2026 15:46

Ben Hillis added 4 commits April 1, 2026 14:26

formatting

337a116

benhillis force-pushed the user/benhill/fix-port-relay-handle-limit branch from 87b2cd8 to 6b9bbbb Compare April 1, 2026 21:26

benhillis marked this pull request as ready for review April 1, 2026 21:28

Copilot AI review requested due to automatic review settings April 1, 2026 21:28

Copilot started reviewing on behalf of benhillis April 1, 2026 21:29 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

	RegisterWaitForSingleObject(&WaitHandle, Handle.Get(), &EventHandle::WaitCallback, this, INFINITE, WT_EXECUTEONLYONCE));
	RegisterWaitForSingleObject(WaitHandle.put(), Handle.Get(), &EventHandle::WaitCallback, this, INFINITE, WT_EXECUTEONLYONCE));

	&WaitBridge, Event.get(), &ReadHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));
	WaitBridge.put(), Event.get(), &ReadHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

	&WaitBridge, Event.get(), &SingleAcceptHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));
	WaitBridge.put(), Event.get(), &SingleAcceptHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

	&WaitBridge, Event.get(), &WriteHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));
	WaitBridge.put(), Event.get(), &WriteHandle::WaitBridgeCallback, this, INFINITE, WT_EXECUTEONLYONCE));

Conversation

benhillis commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

relay::MultiHandleWait changes (relay.hpp/cpp)

Port relay changes (localhost.cpp)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OneBlue left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

benhillis commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benhillis commented Mar 30, 2026 •

edited

Loading