Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions benchmarks/content-length/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.work/
94 changes: 94 additions & 0 deletions benchmarks/content-length/README.md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the Hono version this benchmark affects is v4.13 with honojs/hono#5122, or later. So it's better to update the lockfile and devDependencies of hono (it uses 4.12.8)

Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Content-Length benchmarks

Compare the original caller-header mutation, a copy of plain header records, and
an HTTP/1 `_contentLength` candidate with compatibility guards. Dependencies are
resolved from the repository's unchanged frozen lockfile (currently Hono 4.12.8).
The harness records the installed Hono version without requiring a particular
release.

Use the same Node.js release and frozen dependencies for all variants in a
comparison. Repeat the measurements with other Node releases as needed.
The existing Node 20.x, 22.x and 24.x CI matrix runs the compatibility tests
as part of the full suite, checking native serialization and fallback paths.

```sh
pnpm install --frozen-lockfile
node benchmarks/content-length/prepare.mjs 82ba34e6b19da49ca500d4cac95b5fb25ee48cc8
node benchmarks/content-length/run.mjs micro
node benchmarks/content-length/run.mjs pipeline
BENCH_BOMBARDIER=/path/to/bombardier node benchmarks/content-length/run.mjs http
node benchmarks/content-length/summarize.mjs benchmarks/content-length/.work/{micro,pipeline,http}-v*.jsonl
```

Run each measurement with the desired Node executable, sequentially on an idle
machine. Build preparation uses the current working tree for the candidate and
Git snapshots for the baseline/copy, without changing the checkout. Rebuild after
changing the candidate. Generated snapshots, bundles, metadata and results go in
`.work/`. Choose a new `BENCH_RESULTS` path when repeating a run; existing results are
never overwritten.
Preparation records the candidate commit, uncommitted source diff and bundle
hashes. Each result includes the commit and measured bundle hash so committed
and experimental candidates remain identifiable. The runner verifies these
hashes before timing and rejects missing or modified prepared variants, or a
Hono version changed since preparation.

To compare against a baseline that already uses native length, set
`BENCH_SKIP_COPY=1` for both preparation and measurement. This omits the
historical copy variant, which requires the original mutation implementation:

```sh
BENCH_SKIP_COPY=1 node benchmarks/content-length/prepare.mjs BASE_REF
BENCH_SKIP_COPY=1 node benchmarks/content-length/run.mjs pipeline
```

- `micro`: lightweight Response creation, native ServerResponse creation, cache
handling and Node's real header serializer; `end()` is a no-op. Each process
warms up for 300,000 iterations, then reports the median of five 200,000-iteration
samples. Natural GC is included; forced GC is disabled by default because it
can invalidate optimized code before a timed sample.
- `pipeline`: the same, including the incoming request adapter, Hono dispatch,
context creation and `c.json()`/`c.text()`. Awaits each listener call so
resolved Promise adoption jobs are included and do not accumulate between samples.
- `http`: real TCP keep-alive traffic using Bombardier v2.0.2, 64 connections,
`GOMAXPROCS=2`, one second of warm-up and four seconds measured per process.
Records the server's user + system CPU time as well as RPS. Client and server
share the same machine. CPU timing includes client startup/shutdown boundaries.

Each mode defaults to five rounds and rotates variant order. `BENCH_ROUNDS` and
`BENCH_SECONDS` override those settings. HTTP cases are a small `c.json()` response,
JSON with seven additional headers, and `c.text()` with no custom headers as
controls. Pipeline and HTTP results record the actual header representation
before timing, without materializing `response.headers`. Ordinary `c.json()`
uses `Headers` in Hono 4.12.8 and a plain record in 4.13.8, so these versions
exercise different adapter paths. The `micro` cases exercise plain records
directly, independently of Hono's response construction. Summaries separate
results by Hono version and recorded header representation.
Validate response status/body before loading the endpoint; failures in load
generation are errors, not successful benchmark samples.

For nanoseconds lower is better; for RPS higher is better. The reported ranges
are observed run-to-run ranges, not confidence intervals. Microbenchmarks cannot
establish end-to-end throughput. Small RPS differences on a shared machine do
not establish either a speedup or an absence of regression.

The GC mode and warm-up count are recorded in each microbenchmark row.
`BENCH_GC=forced BENCH_WARMUP=100000` reproduces the original exploratory method;
use it only for comparison, since forced GC can invalidate optimized code.

For the one-variable-at-a-time investigation (after preparation):

```sh
ABLATION_NODE=/path/to/node node benchmarks/content-length/ablation.mjs
ABLATION_GC=natural ABLATION_NODE=/path/to/node node benchmarks/content-length/ablation.mjs
```

The first command reproduces the forced-GC experiment and compares isolated
`json8` with `json` followed by `json8`. The second measures both cases with
natural GC and longer warm-up. Both default to five process-level rounds and
refuse to overwrite previous output.

To compare cloning with property addition in isolation:

```sh
BENCH_RESULTS=/tmp/clone-cost.jsonl node benchmarks/content-length/clone-cost.mjs /path/to/node20 /path/to/node24
```
94 changes: 94 additions & 0 deletions benchmarks/content-length/ablation.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
// Diagnostic variants only: run prepare.mjs before this script.
import { spawnSync } from 'node:child_process'
import { appendFileSync, mkdirSync, readFileSync, writeFileSync, existsSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { createHash } from 'node:crypto'
const work = new URL('./.work/', import.meta.url)
const baseline = readFileSync(new URL('dist/baseline/index.mjs', work), 'utf8')
const guarded = readFileSync(new URL('dist/guarded/index.mjs', work), 'utf8')
const original = `\tif (!hasContentLength) {
\t\tif (typeof body === "string") header["Content-Length"] = Buffer.byteLength(body);
\t\telse if (body instanceof Uint8Array) header["Content-Length"] = body.byteLength;
\t\telse if (body instanceof Blob) header["Content-Length"] = body.size;
\t}`
if (baseline.split(original).length !== 2)
throw new Error('Unexpected baseline; regenerate and inspect it')
const local = `\tif (!hasContentLength) {
\t\tif (typeof body === "string") { let length = Buffer.byteLength(body); header["Content-Length"] = length; }
\t\telse if (body instanceof Uint8Array) { let length = body.byteLength; header["Content-Length"] = length; }
\t\telse if (body instanceof Blob) { let length = body.size; header["Content-Length"] = length; }
\t}`
const merged = `\tif (!hasContentLength) {
\t\tlet length;
\t\tif (typeof body === "string") length = Buffer.byteLength(body);
\t\telse if (body instanceof Uint8Array) length = body.byteLength;
\t\telse if (body instanceof Blob) length = body.size;
\t\tif (length !== undefined) header["Content-Length"] = length;
\t}`
const variants = {
baseline,
'baseline-repeat': baseline,
'let-local': baseline.replace(original, local),
'let-merged': baseline.replace(original, merged),
guarded,
}
for (const [name, code] of Object.entries(variants)) {
mkdirSync(new URL(`dist/${name}/`, work), { recursive: true })
writeFileSync(new URL(`dist/${name}/index.mjs`, work), code)
}
const naturalGC = process.env.ABLATION_GC === 'natural'
let worker = readFileSync(new URL('./pipeline.mjs', import.meta.url), 'utf8')
.replace('`./.work/dist/${variant}/index.mjs`', '`./dist/${variant}/index.mjs`')
.replace("'./header-path.mjs'", "'../header-path.mjs'")
.replace('Object.entries(cases)', "process.argv[3].split(',').map(kind => [kind, cases[kind]])")
worker = worker
.replace("const forceGC = process.env.BENCH_GC === 'forced'", `const forceGC = ${!naturalGC}`)
.replace(
'const warmupIterations = Number(process.env.BENCH_WARMUP || 300000)',
`const warmupIterations = ${naturalGC ? 300000 : 100000}`
)
writeFileSync(new URL('ablation-worker.mjs', work), worker)
writeFileSync(
new URL('ablation-manifest.json', work),
JSON.stringify(
Object.fromEntries(
Object.entries(variants).map(([name, code]) => [
name,
{ sha256: createHash('sha256').update(code).digest('hex'), length: code.length },
])
),
null,
2
)
)
if (process.argv[2] === '--prepare-only') process.exit(0)
const node = process.env.ABLATION_NODE || process.execPath
const rounds = Number(process.env.ABLATION_ROUNDS || 5)
const output = new URL(
`ablation-${node.split('/').slice(-3, -1).join('-') || 'node'}${naturalGC ? '-natural-gc' : ''}.jsonl`,
work
)
if (existsSync(output)) throw new Error(`Output exists: ${output}`)
const names = Object.keys(variants)
for (let round = 0; round < rounds; round++) {
for (const order of naturalGC
? ['json,json8']
: round % 2
? ['json,json8', 'json8']
: ['json8', 'json,json8']) {
for (let j = 0; j < names.length; j++) {
const variant = names[(j + round) % names.length]
const result = spawnSync(
node,
['--expose-gc', fileURLToPath(new URL('ablation-worker.mjs', work)), variant, order],
{ encoding: 'utf8' }
)
if (result.status !== 0) throw new Error(result.stderr)
for (const line of result.stdout.trim().split('\n')) {
const row = { round, order, gc: naturalGC ? 'natural' : 'forced', ...JSON.parse(line) }
appendFileSync(output, JSON.stringify(row) + '\n')
console.log(JSON.stringify(row))
}
}
}
}
54 changes: 54 additions & 0 deletions benchmarks/content-length/clone-cost.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
'use strict'
const variant = process.argv[2]
const make =
variant === 'update'
? () => ({ 'Content-Type': 'application/json', 'Content-Length': 0 })
: () => ({ 'Content-Type': 'application/json' })
const ops = {
identity: (h) => h,
clone: (h) => ({ ...h }),
add: (h) => {
h['Content-Length'] = 27
return h
},
copyadd: (h) => {
const c = { ...h }
c['Content-Length'] = 27
return c
},
update: (h) => {
h['Content-Length'] = 27
return h
},
}
const op = ops[variant]
if (!op) throw Error('Unknown variant')
const batch = 4096
const input = new Array(batch)
const output = new Array(batch)
let sink = 0
function run(count) {
let ns = 0n
for (let start = 0; start < count; start += batch) {
for (let i = 0; i < batch; i++) input[i] = make()
const before = process.hrtime.bigint()
for (let i = 0; i < batch; i++) output[i] = op(input[i])
ns += process.hrtime.bigint() - before
for (let i = 0; i < batch; i++)
sink += output[i]['Content-Type'].length + (output[i]['Content-Length'] || 0)
}
return Number(ns) / (Math.ceil(count / batch) * batch)
}
run(500000)
const samples = Array.from({ length: 7 }, () => run(1000000)).sort((a, b) => a - b)
console.log(
JSON.stringify({
node: process.version,
v8: process.versions.v8,
variant,
medianNs: samples[3],
samples,
batch,
sink,
})
)
26 changes: 26 additions & 0 deletions benchmarks/content-length/clone-cost.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { execFileSync } from 'node:child_process'
import { appendFileSync, existsSync } from 'node:fs'
import { fileURLToPath } from 'node:url'

// Pass fixed Node executables to compare engine versions. Each case gets a
// fresh process, with rotating case order and alternating version order.
const nodes = process.argv.slice(2)
if (!nodes.length) nodes.push(process.execPath)
const output =
process.env.BENCH_RESULTS || fileURLToPath(new URL('./.work/clone-cost.jsonl', import.meta.url))
if (existsSync(output)) throw new Error(`Refusing to overwrite ${output}`)
const worker = fileURLToPath(new URL('./clone-cost.cjs', import.meta.url))
const variants = ['identity', 'clone', 'add', 'copyadd', 'update']
for (let round = 0; round < 5; round++) {
for (const node of round % 2 ? [...nodes].reverse() : nodes) {
for (let k = 0; k < variants.length; k++) {
const variant = variants[(k + round) % variants.length]
const row = {
round,
...JSON.parse(execFileSync(node, [worker, variant], { encoding: 'utf8' })),
}
appendFileSync(output, JSON.stringify(row) + '\n')
}
}
console.log(`Completed round ${round + 1}`)
}
11 changes: 11 additions & 0 deletions benchmarks/content-length/header-path.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
// Inspect the cache without materializing response.headers or changing the measured path.
export const headerPath = (response) => {
const cache =
response[Object.getOwnPropertySymbols(response).find((key) => key.description === 'cache')]
if (!cache) return 'uncached'
const headers = cache[2]
if (!headers) return 'default'
if (headers instanceof Headers) return 'Headers'
if (Array.isArray(headers)) return 'tuples'
return 'plain'
}
62 changes: 62 additions & 0 deletions benchmarks/content-length/micro.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import { ServerResponse } from 'node:http'
const forceGC = process.env.BENCH_GC === 'forced'
const warmupIterations = Number(process.env.BENCH_WARMUP || 300000)
if (!Number.isSafeInteger(warmupIterations) || warmupIterations < 1)
throw new Error('BENCH_WARMUP must be a positive integer')
const variant = process.argv[2]
const { responseViaCache, LightweightResponse } = await import(`./.work/dist/${variant}/index.mjs`)
class MeasuredResponse extends ServerResponse {
end() {
return this
}
}
const request = { method: 'GET', httpVersionMajor: 1, httpVersionMinor: 1 }
const factories = {
plain1: () => ({ 'content-type': 'text/plain' }),
plain8: () => ({
'content-type': 'text/plain',
'x-a': '1',
'x-b': '2',
'x-c': '3',
'x-d': '4',
'x-e': '5',
'x-f': '6',
'x-g': '7',
}),
noheaders: () => undefined,
}
let sink = 0
for (const [kind, makeHeaders] of Object.entries(factories)) {
const run = (count) => {
const start = process.hrtime.bigint()
for (let i = 0; i < count; i++) {
const headers = makeHeaders()
const outgoing = new MeasuredResponse(request)
responseViaCache(new LightweightResponse('hello', { headers }), outgoing)
sink += outgoing._header.length
}
return Number(process.hrtime.bigint() - start) / count
}
run(warmupIterations)
const samples = []
for (let i = 0; i < 5; i++) {
if (forceGC) global.gc()
samples.push(run(200000))
}
samples.sort((a, b) => a - b)
console.log(
JSON.stringify({
node: process.version,
variant,
gc: forceGC ? 'forced' : 'natural',
warmupIterations,
iterationsPerSample: 200000,
sampleCount: 5,
kind,
medianNs: +samples[2].toFixed(1),
minNs: +samples[0].toFixed(1),
maxNs: +samples.at(-1).toFixed(1),
sink,
})
)
}
Loading
Loading