Skip to content

performance.md

Latest commit

 

History

History
242 lines (198 loc) · 10.3 KB

performance.md

File metadata and controls

242 lines (198 loc) · 10.3 KB
Raw

Performance Benchmarks

This document has two parts:

  1. Standalone benchmark — MessageFrame on its own, via benchmarks/benchmark.cpp: how the library scales with the parameter count and what the FrameConfig sizing hint buys you.
  2. Cross-format comparison — MessageFrame vs protobuf, nlohmann/json, and msgpack-cxx on the same telemetry message, via the reproducible harness in benchmarks/cross_format.

Both were measured on: Intel Core 7 240H, Ubuntu 22.04 (x64, GCC, Release build). Run-to-run variance on this hardware is roughly ±10%. Each part lists its exact measurement mode and commands.

Standalone benchmark (benchmarks/benchmark.cpp)

The figures below reflect the complete end-to-end lifecycle (parameter addition + serialization + deserialization). Iteration counts differ per scenario to keep total run time reasonable; each scenario lists the exact command used.

MessageFrame adapts its storage to the parameter count: up to 128 parameters it uses a flat, allocation-free std::vector; beyond that it switches to a tsl::robin_map, which can additionally be pre-sized via FrameConfig (see architecture.md).

Scenario A: small frame (4 parameters)

--iterations 1000000 --params 4. Header + 4 parameters, no attachment.

Metric Value
Avg time per message 0.678 us
Throughput 1,473,936 messages/sec (119.3 MB/sec)
Avg packed size 84 bytes
add / serialize / deserialize 0.10 us / 0.16 us / 0.32 us

Scenario B: peak vector streaming (127 parameters)

--iterations 1000000 --params 127. Header + 127 parameters — right at the ceiling of vector-mode storage, without entering the hash map.

Metric Value
Avg time per message 10.41 us
Throughput 96,009 messages/sec (190.07 MB/sec)
Avg packed size 2,075 bytes
add / serialize / deserialize 2.81 us / 2.66 us / 4.54 us

Scenario C: large frame (150 parameters)

--iterations 1000000 --params 150. Header + 150 parameters — past SMALL_CAPACITY, so the container has switched to hash-map mode.

Metric Value
Avg time per message 22.03 us
Throughput 45,402 messages/sec (107.8 MB/sec)
Avg packed size 2,488 bytes
add / serialize / deserialize 8.47 us / 3.45 us / 8.98 us

Scenario D: large frame with sizing hint (1024 parameters, --reserve 1024)

--iterations 200000 --params 1024, run once with --reserve 0 and once with --reserve 1024. This isolates the cost of the vector-to-map migration that a sizing hint lets you skip.

Metric Without hint (--reserve 0) With hint (--reserve 1024) Change
Avg time per message 219.43 us 173.72 us -21%
Throughput 82.63 MB/sec 104.37 MB/sec +26%
Avg packed size 19,012 bytes 19,012 bytes unchanged
sum_add (parameter insertion) 88.14 us 41.09 us -53%
sum_find (worst case: last-inserted key) 0.06 us 0.06 us unchanged
sum_serialize 28.10 us 27.37 us roughly unchanged
sum_deserialize 85.03 us 85.10 us roughly unchanged

The improvement is concentrated in sum_add: with the hint, the container starts directly in map mode sized for 1024 entries, so it never fills a vector to SMALL_CAPACITY and migrates it. serialize/deserialize cost is unaffected either way, since it depends only on the final parameter count, not on how the container got there. Point lookups stay at ~60 ns regardless of the hint, since tsl::robin_map's contiguous bucket layout gives O(1) lookups either way.

Cross-format comparison (benchmarks/cross_format)

The build uses the repository defaults for Release — -O3 -march=native plus LTO (CMAKE_INTERPROCEDURAL_OPTIMIZATION_RELEASE). Single thread. Each scenario is a full per-message cycle — parameter addition + serialization + deserialization. Every result is the min of 3 passes to reduce timer noise. The complete, reproducible harness lives in benchmarks/cross_format — run it yourself before drawing conclusions.

This comparison pits MessageFrame against three alternatives on the same logical telemetry message:

  • protobuf (map<string, TelemetryValue>, TelemetryValue is a oneof of double / int64 / bool / string; one bytes field for the attachment). Schema shown below. Standard heap API — no Arena.
  • nlohmann/json — the same key/value dictionary as a JSON object; the binary attachment is base64-encoded (the honest price of a text format for binary payloads).
  • msgpack-cxx — raw MessagePack packing of the same dictionary and a bin attachment, unpacked back into a std::map. It has no in-memory document, typed value model, header, or two-part key addressing, so it is a baseline floor for the wire format, not an equivalent API.

Scenario definition

Every backend receives the identical input:

  • N parameters. Key d{dev}.p{k} where dev = k % 8, value type cycles as double -> int64 -> bool -> string, values depend only on the key index (never on the iteration counter), so wire sizes stay constant between runs.
  • Optionally one 1 MiB binary attachment (0xAB-filled, representing an IQ capture).
Scenario params attachment iterations
Small frame 4 — 200,000
Medium frame 32 — 100,000
Vector ceiling 127 — 50,000
Hash-map mode 150 — 50,000
Large frame 1,024 — 10,000
1 MiB attachment 4 1 MiB 2,000

protobuf schema

syntax = "proto3";
package bench;

message TelemetryValue {
  oneof v {
    double d = 1;
    int64 i = 2;
    bool b = 3;
    string s = 4;
  }
}

message TelemetryFrame {
  map<string, TelemetryValue> params = 1;
  bytes attachment = 2;
}

This is the closest structural equivalent of MessageFrame's two-part key addressing (device -> parameter -> value). The flat key "d0.p0" in protobuf/json/msgpack corresponds to MessageFrame::add("d0", "p0", ...). MessageFrame stores its flat key with the internal \x1F separator, which accounts for the small wire-size difference between it and raw msgpack.

Results — time per message (µs)

Scenario MessageFrame MessageFrame + hint protobuf nlohmann/json msgpack-cxx
Small frame (4 params) 0.59 — 0.58 1.64 0.28
Medium frame (32 params) 3.04 — 4.76 10.56 1.04
Vector ceiling (127 params) 11.28 — 19.52 47.09 3.57
Hash-map mode (150 params) 24.01 19.85 22.76 64.94 4.18
Large frame (1,024 params) 196.50 157.46 303.52 471.40 27.22
1 MiB attachment (4 params) 189.76 — 276.07 14,159 138.86

MessageFrame beats protobuf by 1.5–2x on large frames (32/127/1,024 params) and is statistically tied with it at 4 and 150 params; the sizing hint (FrameConfig::initial_reserve) pulls it ahead at 150 and 1,024. Note that the hash-map-mode row (150 params) is the vector->hash-map migration point: without the hint MessageFrame pays the one-time migration cost of moving 128 entries, which is why it lands at parity with protobuf there.

Per-phase breakdown

Scenario Backend add (µs) serialize (µs) deserialize (µs)
4 params MessageFrame 0.13 0.17 0.29
4 params protobuf 0.12 0.16 0.30
4 params nlohmann/json 0.22 0.36 1.06
4 params msgpack-cxx 0.02 0.10 0.16
1,024 params MessageFrame 82.80 36.96 76.07
1,024 params MessageFrame + hint 52.78 35.03 70.54
1,024 params protobuf 94.02 59.39 152.70
1,024 params nlohmann/json 145.32 62.08 264.21
1,024 params msgpack-cxx 0.03 14.07 13.13
1 MiB attachment MessageFrame 25.93 44.74 116.04
1 MiB attachment protobuf 116.12 113.90 46.06
1 MiB attachment nlohmann/json 1,457 3,725 8,977
1 MiB attachment msgpack-cxx 0.03 92.57 46.11

Wire size

Scenario MessageFrame protobuf nlohmann/json msgpack-cxx
4 params 73 B 66 B 68 B 53 B
127 params 1,816 B 2,237 B 1,986 B 1,550 B
1,024 params 15,297 B 18,858 B 16,795 B 13,237 B
1 MiB attachment 1,048,662 B 1,048,646 B 1,398,188 B 1,048,645 B

Fairness caveats

  1. msgpack-cxx is not an equivalent API. It has no in-memory document, so its "add" phase is near-zero by design and the cost is fully inside serialize. Treat it as the floor of the wire format, not as a competing high-level library.
  2. protobuf uses the standard heap API, no Arena. Arena would reduce its allocation overhead on large messages. The schema was written to be the closest structural equivalent of MessageFrame's model and was not tuned further.
  3. JSON pays for text twice: base64 encoding of the attachment on the way in and the full text parse on the way out. The 1 MiB scenario is dominated by that cost — this is the real-world price of text formats for binary data.
  4. MessageFrame copies attachments on deserialize (building a fresh std::vector): visible in the 1 MiB scenario (116.04 µs vs 46.06 µs for protobuf's in-place bytes). This is the documented zero-copy trade-off of the library, not an oversight.
  5. Deterministic keys and values keep wire sizes stable; iteration counts are high enough that timer overhead is negligible on all rows.
  6. GCC LTO (the repo Release default) slows raw msgpack-cxx's packer by ~2x (e.g. 1,024-param serialize: 14.1 µs with CMAKE_INTERPROCEDURAL_OPTIMIZATION vs 5.9 µs without). Reproduced across runs. MessageFrame, protobuf, and nlohmann/json numbers do not move with the LTO setting. We publish the LTO numbers since that is what building this repository produces by default.

Reproducing

git clone --recursive https://github.com/stubcpp/MessageFrame.git
cd MessageFrame
cmake -B build -DCMAKE_BUILD_TYPE=Release -DMSGFRAME_BUILD_CROSS_BENCHMARK=ON
cmake --build build -j
./build/mf_crossbench

The harness lives in benchmarks/cross_format: one shared scenario definition (bench.h), one backend implementation per format (backend_mf.cpp, backend_pb.cpp, backend_json.cpp, backend_msgpack.cpp), and a single main.cpp that runs min-of-3 passes.