Continuation Preemption is a double edge sword

[franz1981]

Since currently loom support cooperative scheduling, having CPU taxing tasks can hog a single event loop scheduler at the point to cause HOL behaviours, affecting tail latency of subsequent tasks.
There are several solutions to this problem:

1. Work stealing, assuming the other schedulers have spare CPU cycles
2. Preemptive continuations (manual, via Thread::yield or from runtime - still not implemented in hotspot)

Sometime the former is not possible e.g. too few cores containers (1,2) or presence of N CPU taxing tasks altogether - making preemptive continuations more appealing.
But, preemption can be dangerous since sharing not linearizable concurrent primitives between a continuation and its carrier, can cause unexpected deadlocks e.g.:

1. JCTools mpsc offer split the offering into increasing atomically a producer sequence and setting an item in the right array's slot
2. JCTools mpsc poll first check the producer sequence and than, if it expect an item to eventually become visible, spin wait on the expected slot, that the item is set

This behaviour can cause a carrier scheduler to break, for example, while dealing with the mpsc task queue:

1. The carrier run a continuation which offer another continuation to the task queue, but is preempted right before placing the item in the right slot
2. The carrier can observe a progressed producer sequence without an item slot set; if there are no other elements in the queue, it is going to spin wait forever the element to be set

This is just an example and I have missed the important detail of what happen once a continuation is preempted; if i assume that it will behave similar to Thread::yield (i.e. unmounting the continuation and calling Executor::execute as carrier, to resubmit the preempted continuation) it is safe to assume that the previous scenario can still happen, even in the case where the re-submit enqueue the task in the mpsc q, because the producer sequence has been already altered.

A similar problem can happen if a virtual thread will offer a WriteTask' Runnable to the event loop carrier's Netty taskQueue, since it uses an mpsc queue under the hood.