[DB] This test needs to be completely redone!

[remittor]

The current implementation of the DB test is no different from the JSON test. Both tests perform synchronous execution, with a relatively short duration (and are nearly equivalent in execution time).

That is, in the current state of the project, there is no practical reason to launch the current implementation of the DB test (the JSON test does the same thing).

My opinion is that the DB test should primarily check how quickly the engine's asynchronous subsystem copes with switching between executions of long asynchronous tasks.

Therefore, it is worth abandoning the use of the sqlite3 in this DB test (it is synchronous) and switching to a database that has a full-fledged asynchronous API. I recommend using Postgres.

And the results of this DB test need to be shown separately, and not just in the mixed version!

In this DB test, the engines that will lead are those that have a very high-quality implementation of the asynchronous subsystem for processing tasks that generated by HTTP requests.

[remittor]

@BennyFranciscus , tell me your opinion on this topic.

[BennyFranciscus]

Yeah, you're raising a really valid point here. The current DB test with sqlite3 is basically just JSON serialization with an extra step — there's no real async I/O happening, so it doesn't differentiate frameworks in any meaningful way beyond what the JSON test already covers.

Switching to Postgres makes a lot of sense. That would actually stress the async machinery — connection pooling, query pipelining, how the event loop handles I/O wait vs compute. That's where you'd start seeing real separation between frameworks that have well-tuned async subsystems and ones that don't.

A few thoughts:

• Postgres over a network (even localhost TCP) adds realistic latency that sqlite3 completely avoids. That's where async scheduling actually matters.
• Connection pool management becomes a real factor — frameworks that can efficiently multiplex queries across a pool would shine.
• Showing DB results separately is a good call. Mixing them in obscures what's actually being measured.

The one thing to think through is the test setup complexity — spinning up a Postgres instance adds infrastructure overhead for contributors who want to run benchmarks locally. But honestly, a Docker container for Postgres is pretty standard these days, so it's probably fine.

I'm on board with this direction. It would make the DB test actually test something distinct from JSON.

[MDA2AV]

Yes it is synchronous, with SQLite + mmap, the query is essentially a CPU-bound memory read — there's no disk I/O to await. So yes, async frameworks don't get to show off their async I/O scheduling. But that's intentional — the test measures query processing + JSON serialization under CPU load, not async I/O. The synchronized lock issue we found in quarkus, thread-local connection patterns, and the full table scan design all create real differentiation.

I agree though we need to exercise async I/O load, the challenge is that every framework will need access to an async driver for SQL queries, no ORM can be allowed to avoid benchmarking the ORM instead of the framework and some frameworks only have access to a sync one (this is why we went with a sync approach to the DB test and why there isn't really a separate test for it since it doesn't add much value).

My proposal is that we create a new DB test that exercises async I/O, doesn't even need to be DB related, as long as there is async I/O involved it is fine. The option with a separate docker for the postgres is an option

[BennyFranciscus]

That's a fair point — I was too quick to dismiss the current DB test. The synchronized lock differentiation and thread-local connection patterns are real things that matter, especially for JVM frameworks where that stuff shows up clearly. So yeah, keeping it as-is makes sense.

A separate async I/O test is a great idea. Doesn't have to be DB-related — could be something like fetching from an external service (another container running a simple HTTP endpoint with configurable latency?). That would exercise connection pooling, event loop efficiency, and how frameworks handle I/O wait under concurrency.

The nice thing about that approach is it avoids the sync-only driver problem you mentioned — every framework can do HTTP client requests, and most have async HTTP clients available. No ORM overhead to worry about either.

[MDA2AV]

That's a fair point — I was too quick to dismiss the current DB test. The synchronized lock differentiation and thread-local connection patterns are real things that matter, especially for JVM frameworks where that stuff shows up clearly. So yeah, keeping it as-is makes sense.

A separate async I/O test is a great idea. Doesn't have to be DB-related — could be something like fetching from an external service (another container running a simple HTTP endpoint with configurable latency?). That would exercise connection pooling, event loop efficiency, and how frameworks handle I/O wait under concurrency.

The nice thing about that approach is it avoids the sync-only driver problem you mentioned — every framework can do HTTP client requests, and most have async HTTP clients available. No ORM overhead to worry about either.

aren't httpclients typically crap across all languages though? plus they usually have a very slow first request making the first out of 3 runs basically useless.

[BennyFranciscus]

Honestly yeah, you're right — HTTP clients in most languages have a lot of overhead. Connection setup, TLS handshake, DNS resolution... the first request is always going to be garbage, and even warmed up they add a ton of noise that has nothing to do with the framework itself.

Maybe a raw TCP socket approach would be cleaner? Like a simple echo service in another container — the framework makes a TCP connection, sends N bytes, waits for N bytes back, then responds. Minimal protocol overhead, still exercises the async I/O path. You could even add configurable latency on the echo service side to simulate real-world wait times.

That way you're measuring how the framework handles I/O wait and connection multiplexing without also benchmarking whatever HTTP client library happens to be available. And the first-run cold start issue mostly goes away since TCP connections are trivial to warm up compared to HTTP.

[MDA2AV]

Honestly yeah, you're right — HTTP clients in most languages have a lot of overhead. Connection setup, TLS handshake, DNS resolution... the first request is always going to be garbage, and even warmed up they add a ton of noise that has nothing to do with the framework itself.

Maybe a raw TCP socket approach would be cleaner? Like a simple echo service in another container — the framework makes a TCP connection, sends N bytes, waits for N bytes back, then responds. Minimal protocol overhead, still exercises the async I/O path. You could even add configurable latency on the echo service side to simulate real-world wait times.

That way you're measuring how the framework handles I/O wait and connection multiplexing without also benchmarking whatever HTTP client library happens to be available. And the first-run cold start issue mostly goes away since TCP connections are trivial to warm up compared to HTTP.

Well, I actually prefer the async I/O postgres - this test should be only required to framework level entries, engines wouldn't need to implement and nowadays if your framework does not have access to an async raw sql driver for postgres then it should be penalized.

[BennyFranciscus]

That makes a lot of sense actually — PostgreSQL async drivers are the real-world use case anyway. If your framework doesn't have a proper async pg driver in 2026, that's a legitimate thing to measure.

And limiting it to framework-level entries keeps it clean — engines like nginx/h2o don't need to implement HTTP-to-database flows, that's not what they're for.

Would you want the test to use something like pgbouncer in front of postgres, or direct connections? Direct would be simpler but pgbouncer might better reflect production setups where connection pooling is external.

[remittor]

Maybe a raw TCP socket approach would be cleaner? Like a simple echo service in another container — the framework makes a TCP connection, sends N bytes, waits for N bytes back, then responds. Minimal protocol overhead, still exercises the async I/O path. You could even add configurable latency on the echo service side to simulate real-world wait times.

I agree. Not a bad option!

There's another way. Create a Docker container (or choose an existing one) that uses a file system that doesn't cache files in memory. Or allow the file to be cached in memory.
And the engine must calculate the CRC32 hash for a very large file.
I think this is comparable to a long database call over the network stack.
All file operations have had asynchronous implementations for a long time now.

[BennyFranciscus]

The file I/O idea is interesting — CRC32 over a large uncached file would definitely stress async disk I/O. And with something like O_DIRECT to bypass page cache, you'd get consistent, measurable latency per operation.

That said, I think MDA2AV's direction with async Postgres makes the most sense here. It's the closest to what frameworks actually do in production, and it tests the full async I/O chain — event loop scheduling, connection pooling, result parsing — all at once. Plus it solves the "does your ecosystem even have a real async driver" question, which is honestly a meaningful differentiator.

The file approach could be cool as a separate test down the road though, especially for engines that don't talk to databases but do handle file serving. Different axis of async performance.

[remittor]

And the engine must calculate the CRC32 hash for a very large file.

The CRC32 algorithm can be replaced with a simple count of the number of zero bytes (0x00) in the very large file.

[BennyFranciscus]

Yeah, counting zero bytes would be even simpler to implement and verify — no CRC library dependency to worry about, just a tight loop over the buffer. Same async I/O pressure on the read side.

Still think the Postgres route is the priority for the framework-level test, but this could work well as a lightweight async I/O benchmark for engines that don't need database connectivity.