Static, Static TLS: Entries do not serve the same amount of data (case in point Bun)

[uNetworkingAB]

If you look at fulmine and Bun, fulmine has higher composite score even though Bun beats fulmine in everything except static, static TLS. And then you look into why and you see exactly the same pitfall as the uWS.js entry had:

HttpArena/frameworks/bun/server.ts

Line 143 in 652b778

const file = Bun.file("/data/static/" + name);

It does not pick the .bz compressed file version. And so it ends up scoring HALF of what it should.

You probably need to make this a failure so that this is no longer a silent failure that massively penalizes those who do not pick the .bz files. It's not even remotely close to apples-apples if you compare non-compressed file RPS against precompressed file RPS.

[uNetworkingAB]

This entry is standard, so the .br/.gz files the
// harness leaves on disk are left alone: hand-rolled suffix lookup is a tuned
// technique and Bun.serve has no documented pre-compressed static API.

This is nonsense. Your whole test basically boils down to:

those who know the secret trick - to serve compressed files - are guaranteed to win the benchmark because it's entirely impossible to win by sending the full uncompressed file.

You're not benchmarking servers, you're benchmarking how well the server know your secret trick. Either change the static benchmark to measure GB/sec instead of req/sec or remove those servers who don't know the trick.

It's pointless to participate in a benchmark where the competition is having its perf. halved for bureaucratic reasons.

[uNetworkingAB]

Is this such a complex intricate thing to ask to frameworks

Well, the problem is that you're assuming a filename X with extension .bz is the same contents as the compressed file without that .bz file name, and that servers know this. That's a decent convention but it's not standard, so essentially no one other than you follow this rule.

Maybe this is well known in .NET but it's still not standard. A file with extension .bz can be entirely different from the same name without it. For instance, maybe the compressed variant is a snapshot from a few versions back? Etc, etc.

Edit: "maybe the compressed variant is a snapshot from a few versions back?" -> this is actually something I do myself often, now that I think of it.

[uNetworkingAB]

What is then a test that suits to Bun, a plain text Hello World?

That's kind of the problem with these tests - they often overengineer their purpose just because (for instance by adding database tests which have nothing with the web server at all to do) something that basically is quite slim:

You can really only reliably test a few basic things:

1. Standards compliance (you have good tests here)
2. Fragmentation (I have good tests here)
3. A simple hello world benchmark without any pipelining, but with various URL endpoints so it's not just hitting the same all the time
4. The same with pipelining (maybe)
5. Static file serving GB/sec
6. Query parsing is a good test because it makes it somewhat dynamic but still entirely HTTP
7. JSON parsing/serialization is kiiiiiiiiiiiind of stretching it but maybe still within its purpose as a web server (given how everything is JSON/REST)
8. Responding with compression is also kiiiiind of a stretch but I can take it with the same motivation as the JSON. However, here it's important that the test enforces all solutions to use the same exact compression algo and not allow them to miss (which is the cause of this very issue report)
9. TLS is the same, it belongs with the server (even by standard) but again here it's important to enforce all to use the same ciphers and TLS version and enforce that.

But that's about it, adding more than this really only overengineers the test "just because" and causes these strange edge cases where what's normal for you is not for others

Also - think about correcting some shotcircuits in for instance your JSON test one could easily partition the data vector into its different offsets and just serve those offsets statically based on the "count" query - try and redesign that test so it forces dynamic behavior

[uNetworkingAB]

Secret trick.. it's quite clearly stated in the implementation guidelines.

That's an objectively flawed argument.

1. You wrote the Bun entry yourself (skipping the trick with an arbitrary motivation referring to your own rule)
2. You wrote the implementation guideline
3. Your trick is at best a convention, not an expected standard

What you have in your repo here, does not imply everyone else will automatically agree and change for you. You can have whatever kinds of rules in your own repo, it does not dictate the ecosystem.

This is what I meant with your bias - you have your own entry, your own arbitrary rules, your own opinions about other entries and your own motivations to why all of a sudden one entry gets half its performance because "yeah that makes sense according to my rule".

It's not a charming trait and it will get you nowhere

[uNetworkingAB]

By enabling that trick, Bun jumped 4 positions up. But nothing about Bun itself changed. So by common sense and critical thinking, it was never a true benchmark of Bun. That's the problem, a benchmark cannot arbitrarily cap random entries because of arbitrary rules - that's how you ruin credibility of your test. If the test presents apples-to-steamrollers as if they are equal, it does not matter what some README file in some obscure part of a repo says. People will not read it, they will just look at the most prominent table and leave. So the table must be enough on its own.

[MDA2AV]

1. Standards compliance (you have good tests here)
2. Fragmentation (I have good tests here)

• True

3. A simple hello world benchmark without any pipelining, but with various URL endpoints so it's not just hitting the same all the time.

• Yes that is baseline test, not hello world but a sum of the query params with body chunked and CL both exercised.

4. The same with pipelining (maybe)

• It is unscored because pipelined is not used.

5. Static file serving GB/sec -

• This will benefit serving uncompressed and this was the original static test, it led to problems because users complained it was "not a realistic scenario, you always serve pre compressed files" also if we want to test output bandwidth we dont need a static test, could be something a lot more simple serving some basic 200KB static payload and in a way JSON test already covers it because the uncompressed variant has as 10-20KB payload, similar to average static files size, so we would just remove the static test or make it unscored for those who want to show off their static serve module.

6. Query parsing is a good test because it makes it somewhat dynamic but still entirely HTTP -

• Done in the baseline test

7. JSON parsing/serialization is kiiiiiiiiiiiind of stretching it but maybe still within its purpose as a web server (given how everything is JSON/REST)
8. Responding with compression is also kiiiiind of a stretch but I can take it with the same motivation as the JSON. However, here it's important that the test enforces all solutions to use the same exact compression algo and not allow them to miss (which is the cause of this very issue report) -

• JSON is the core of rest APIs and the json tests are basically testing serialization, tls, compression (CPU intensive) and I/O because payload is large.

I will add that a database test is important not because of the database but the async workload, with all the previous tests you can create a sync epoll or io_uring basic loop that will be more efficient that one that needs to address asynchronous workloads, specially when you consider that most modern frameworks are based on work stealing model, specialized for async. It doesnt need to be a database test.. just a forcefully async one so.. any I/O really.. it just happens that a postgres driver is probably the closest we have to a common ground between all these frameworks

[MDA2AV]

So this would lead to:

• Make static tests unscored, probably remove static and keep static-tls only too.
• CRUD is also a very opinionated test and a candidate to be unscored.
• Async workload test can be revisioned, an async test is vital to see how a framework responds under high load where a purely sync framework would block and die.

[MDA2AV]

Also on JSON Compression, we can't guarantee every framework uses same level of compression or even same algorithm.. again that would be the same biasing pitfall so the test does penalize higher bandwidth (weaker compression)

[uNetworkingAB]

static file serving is not the same as just doing 200kb many times. static file serving has completely different ratio of parsing overhead per byte, vs streaming a big file. so keeping the static file serving but measuring GB/sec is still valuable (some servers have terrible IO) Den sön 23 aug. 2026 18:31Diogo Martins ***@***.***> skrev:

…

*MDA2AV* left a comment (MDA2AV/HttpArena#1275) <#1275 (comment)> Also on JSON Compression, we can't guarantee every framework uses same level of compression or even same algorithm.. again that would be the same biasing pitfall so the test does penalize higher bandwidth (weaker compression) — Reply to this email directly, view it on GitHub <#1275?email_source=notifications&email_token=A2NMOMJCSZVSD66CKWCD2ML5LML6XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZYG4YDOMZZGMZKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5387073932>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2NMOMN77YFUSNQMA7UDHZL5LML6XAVCNFSNUABGKJSXA33TNF2G64TZHMYTCNZSGY3DIMRQG45US43TOVSTWNJSGI2DEMJSHA3DJILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/A2NMOMPKOHYYR4A5DTMBA4D5LML6XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZYG4YDOMZZGMZKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/A2NMOMPQ32VNHY7ITCX2Y4L5LML6XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZYG4YDOMZZGMZKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[MDA2AV]

The static file test needs to stay as it is, allowing pre compression of files because this test is used in h2, h3 and proxies tests, testing the check header for compression, etc. We can make it unscored for "origin" frameworks in h1 tests and think about something else to test IO for in and out.

After hardening the TCP frag and static tests now 30% frameworks are failing validation https://github.com/MDA2AV/HttpArena/actions/runs/32654549340/job/97231206019?pr=1285 , a purge is coming anyway.