feat: add benchmark evaluator and regression gates

[niteshpurohit]

• Introduced a new benchmark evaluator script to assess performance metrics against defined thresholds.
• Created a regression gates file to specify workload metrics and allowed regression percentages.
• Enhanced the JSON report generation in the benchmark runner to include sample data and additional build information.
• Updated CMake configuration to include the new evaluator and ensure proper artifact generation.
• Added tests to validate the behavior of the benchmark evaluator against various scenarios, including baseline and candidate comparisons.
• Improved error handling and messaging for better debugging during evaluation runs.

closes: #82

[Copilot]

🟡 Changes recommended

Three moderate findings in bench/evaluator.cpp remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a benchmark evaluator, regression-gate scaffolding, richer benchmark reports, CMake integration, and validation tests.

Changes:

• Adds evaluator CLI/wrapper with bootstrap regression checks.
• Extends reports with samples and build metadata.
• Integrates evaluator targets, gate configuration, and tests.

File summaries

File	Summary
scripts/benchmark-evaluate	Evaluator build and invocation wrapper
CMakeLists.txt	Registers evaluator verification
cmake/LaghuBuildIdentity.cmake	Includes evaluator inputs in build identity
cmake/LaghuBenchmarks.cmake	Configures evaluator target and validation
cmake/ExpectBenchmarkRunner.cmake	Validates expanded reports
cmake/ExpectBenchmarkEvaluator.cmake	Provides evaluator fixtures and scenarios
bench/runner.cpp	Emits samples and build metadata
bench/regression-gates.tsv	Defines regression-gate configuration format
bench/evaluator.cpp	Parses artifacts and evaluates regressions; has three moderate findings (1, 3, and 1 votes) concerning unavailable CPU identity, unsupported metrics, and zero-valued bootstrap samples

Review details

Suppressed comments (2)

bench/evaluator.cpp:550

• The runner deliberately emits a schema-valid cpu.description with status unavailable when host identification fails, and the runner validation accepts both statuses. Requiring available plus value here makes those valid artifacts unusable for --environment and for comparisons on such hosts; handle the unavailable form consistently (or explicitly reject only hardware-gated evaluations with a separate policy/error).

  if (!required_object(artifact.document, root, "cpu", cpu) ||
      !required_object(artifact.document, cpu, "description", description) ||
      !required_string(artifact.document, description, "status", cpu_status) ||
      cpu_status != "available" ||
      !required_string(artifact.document, description, "value", artifact.cpu_description)) {

bench/evaluator.cpp:856

• The benchmark runner permits end == begin and only marks throughput unavailable when the total duration is zero, so zero-valued latency samples can be emitted. This path rejects any bootstrap resample with a zero mean because regression_ppm cannot divide by zero, causing an otherwise schema-valid artifact to exit 65 with reason=bootstrap_invalid rather than producing a gate result. Define zero-metric handling or reject such artifacts during validation with a specific diagnostic.

  if (baseline == 0U || candidate == 0U) {
    return false;
  }

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate review findings must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

cmake/LaghuBenchmarks.cmake:22

• This changes the benchmark artifact shape (new required build fields and per-interval sample arrays) but leaves LAGHU_BENCHMARK_SCHEMA_VERSION at laghu-benchmark-v1. A baseline produced by the previous runner still advertises v1 yet is rejected by load_artifact because these fields/samples are missing, so mixed-version baseline/candidate evaluation cannot work. Bump the artifact schema version and update the evaluator/fixtures, or explicitly retain a v1 reader.

"// SPDX-License-Identifier: AGPL-3.0-only\n#pragma once\n\n#include <string_view>\n\nnamespace laghu::benchmark::internal {\ninline constexpr std::string_view schema_version = \"${LAGHU_BENCHMARK_SCHEMA_VERSION}\";\ninline constexpr std::string_view build_id = \"${LAGHU_BUILD_ID}\";\ninline constexpr std::string_view build_profile = \"${LAGHU_BUILD_PROFILE}\";\ninline constexpr std::string_view compiler_id = \"${CMAKE_CXX_COMPILER_ID}\";\ninline constexpr std::string_view compiler_version = \"${CMAKE_CXX_COMPILER_VERSION}\";\ninline constexpr std::string_view sanitizer_profile = \"${LAGHU_SANITIZER_PROFILE}\";\ninline constexpr std::string_view standard_library_id = \"${LAGHU_STANDARD_LIBRARY_ID}\";\ninline constexpr std::string_view standard_library_version = \"${LAGHU_STANDARD_LIBRARY_VERSION}\";\ninline constexpr std::string_view target_architecture = \"${CMAKE_SYSTEM_PROCESSOR}\";\ninline constexpr std::string_view target_os = \"${CMAKE_SYSTEM_NAME}\";\ninline constexpr std::string_view features_json = R\"laghu(${LAGHU_BUILD_IDENTITY_FEATURES_JSON})laghu\";\ninline constexpr std::string_view dependencies_json = R\"laghu(${LAGHU_BUILD_IDENTITY_DEPENDENCIES_JSON})laghu\";\n}  // namespace laghu::benchmark::internal\n")

• Files reviewed: 9/9 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Unresolved schema compatibility, build-configuration, sample-count, and bootstrap edge-case issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

Previously missed (3) — in code that hasn't changed since the last review.

bench/evaluator.cpp:22

• minimum_samples is 10, but the repository's benchmark command and validation test deliberately generate reports with --intervals 5 (cmake/ExpectBenchmarkRunner.cmake:10). Passing those otherwise valid runner artifacts to this evaluator is rejected as field=parameters before any threshold runs. Align the evaluator's minimum with the runner contract, or enforce and document a >=10 artifact contract in the runner and wrapper.
  bench/evaluator.cpp:1064
• This runs regression_ppm for every bootstrap replicate. With valid sparse count data (for example, many zero allocation or syscall intervals), a resample can have baseline_mean == 0 while the candidate mean is nonzero, causing zero_baseline_undefined and aborting the entire evaluation with exit 65 even though the original baseline is nonzero. Zero-denominator bootstrap replicates need an explicit policy, such as paired sampling or a metric-specific CI, rather than invalidating the artifact.
  bench/runner.cpp:381
• These additions make the report contract incompatible with the previous laghu-benchmark-v1: the evaluator now requires profile/standard-library metadata and per-metric sample arrays. Because the emitted schema identifier remains v1, a baseline generated by the previous runner is accepted as v1 but then rejected as field=build or field=metrics, so cross-version regression comparisons cannot work. Bump the report schema or add an explicit legacy compatibility path.

bench/evaluator.cpp:601

• The report shape has been expanded with required build fields and per-metric sample arrays, but the loader still accepts and identifies it as laghu-benchmark-v1. Reports emitted by the previous v1 runner lack these fields and are rejected as invalid, which breaks the baseline-artifact workflow while claiming schema compatibility. Bump the benchmark schema version (and fixtures/expectations) or implement an explicitly backward-compatible reader.

  if (!required_string(artifact.document, root, "schema_version", schema) ||
      schema != "laghu-benchmark-v1") {

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Unresolved sample-count compatibility and regression calculation defects block approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

bench/evaluator.cpp:1009

• bit iterates powers of two, but 1000000U is a decimal mask, so this branch is enabled only for the binary bits that happen to be set in 1,000,000. Together with the preceding carry step (which can make carry equal to 2), this does not compute the fractional part of difference / baseline scaled to ppm, so regressions near a threshold can be misclassified. Replace this with a proper overflow-safe fixed-point division.

    if ((bit & 1000000U) != 0U) {

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Lite