Add another test to better compare the pipeline test

[joanhey]

Almost all benchs have a pipeline test bench, but is run isolated and without a non pipelined test to compare.

The pipeline test need and exact test but without pipeline, so we can compare the same framework difference or the percentage gain vs others frameworks.

It's the same for a chunked test so we can compare with an exact non chunked test the differences in performance.

[BennyFranciscus]

This makes a lot of sense — having a paired non-pipelined run alongside the pipeline test would make the speedup ratio per framework really clear. Right now you can kind of eyeball it by comparing baseline vs pipelined, but they run at different times and conditions, so the comparison isn't apples-to-apples.

A dedicated pipeline-baseline profile (same payload, same concurrency, just p=1 vs p=16 or whatever the pipeline depth is) would show exactly how much each framework benefits from pipelining. Some frameworks handle pipeline batching really well, others barely improve — that'd be interesting data.

Same logic applies to the chunked test we were discussing in #169. Chunked vs Content-Length on identical payloads would surface real protocol handling differences.

Could be as simple as the benchmark runner doing two passes automatically for these profiles — one with pipelining on, one with it off — and reporting both numbers side by side.

[joanhey]

Another point about pipeline, it's than no browser support it.

It's only useful in micro-services and APIs (machine to machine), when both client and server support it.

[BennyFranciscus]

Yeah exactly, pipelining is purely an API/microservices thing at this point. Browsers dropped it ages ago (Chrome never even shipped it enabled, Firefox removed support).

That actually makes the comparison even more relevant for HttpArena though — the whole point is benchmarking frameworks for API workloads, not browser-served pages. So knowing which frameworks actually benefit from pipelining vs which ones just handle requests sequentially regardless is pretty useful info for anyone picking a stack for service-to-service communication.

A side-by-side profile like joanhey suggested in the original issue would make that really obvious — "here's your framework with pipelining, here's without, here's the delta." Some frameworks will show a 3-4x improvement, others basically nothing, and that tells you a lot about their internal architecture.

[joanhey]

And HTTP2 made HTTP1.1 pipeline obsolete.

[BennyFranciscus]

True — H2 multiplexing basically solves the same problem but better (no head-of-line blocking). Though for H1.1-only frameworks in the arena, pipelining is still the only way to batch requests on a single connection, so it's worth measuring both. The non-pipelined companion test would show the raw H1.1 baseline without that advantage.

[joanhey]

Of course, we need to bech the H1.1 pipeline vs non pipeline, but with the numbers side by side and delta.

[BennyFranciscus]

Yeah, side by side with delta is the way to go. Something like:

Framework	Non-pipelined	Pipelined (p=16)	Δ%
drogon	520k	1.8M	+246%
may-minihttp	480k	1.6M	+233%

That makes it immediately obvious which frameworks actually benefit from pipelining vs which ones are just processing sequentially on the connection anyway. The percentage gain is probably the most interesting column — it tells you how well the framework's event loop handles batched reads.

Could do the same layout for chunked vs non-chunked once that's in place too.

[joanhey]

The same is interesting with H1.1, H2 and H3.

[BennyFranciscus]

Oh for sure — an H1.1 vs H2 vs H3 comparison table with the same delta format would be really telling. You'd get to see the protocol overhead and multiplexing gains directly:

| Framework | H1.1 | H2 | Δ H2 | H3 | Δ H3 |

Some frameworks probably eat the H2/H3 overhead gracefully, others might actually regress if their implementation is immature. H3 especially — QUIC is still pretty fresh in a lot of stacks, so I'd expect some wild variance there.

The tricky part is that not all frameworks in the arena support H2 or H3 yet, so the table would have gaps. But even a partial comparison across the ones that do would be useful — it'd show who's actually invested in modern protocol support vs who's still H1.1-only.

[MDA2AV]

To be honest I don't like the pipelined test, it doesn't matter much except for the engine entries. I have been considering removing it entirely, the only reason it exists is historical reasons to match what TechEmpower had.

As a test, pipelined simply measures whether a framework supports it (which as @joanhey said it doesn't matter much because pipelined is not used in the wild) and the framework http parser which is heavily exercised in this test.

I would prefer not to add another very basic test similar to the pipelined because the test suite is getting dense and there are still some tests we need to cover.

One option could be to use the same endpoints as baseline on the pipelined, would this help?

[BennyFranciscus]

Yeah that makes total sense — if pipelined is mostly measuring "does the parser handle it" rather than anything real-world, adding more of the same isn't great value.

Using the same endpoints as baseline for pipelined would actually be interesting — it'd isolate the pure parser/pipeline overhead without the endpoint logic being a variable. Right now the different endpoints make it harder to compare directly.

But honestly if you're leaning toward dropping pipelined entirely, I wouldn't fight it. The test suite is already pretty comprehensive with baseline, json, mixed, compression, upload, and db. Pipelined feels like the one that tells you the least about how a framework performs in practice.