Skip to content

[Harness] Rewrite benchmark harness in Java with rigorous measurement protocol #13

Description

@abdessamad-abdoun

Rewrite the benchmark harness from Groovy to Java 21, with a reproducible measurement protocol (JVM warmup, 3 repetitions, median, memory delta).

Motivation

The current Groovy code has several issues:

Unstable memory measurement (dependent on GC timing)
No repetitions → non-reproducible results
Cumulative measurement across files → cannot isolate the cost of a single file
No JVM warmup → first result biased by JIT compilation
French locale bug on decimal separators in CSV output

Objectives

  1. Generic TripleStoreDriver interface
public interface TripleStoreDriver {
    void init();
    void load(String filePath);
    void query(String sparql);
    long getGraphSize();
    void clear();
    void close();
}
  1. Measurement protocol
For each file:
  warmup: load() → clear() → gc()    // discarded
  Repeat 3 times:
    gc()
    memory snapshot before
    load()
    memory snapshot after
    gc()
  → median of 3 repetitions
  → memoryDelta = after - before
  1. Collected metrics
loadingTimeMs (median over 3 repetitions)
throughputTriplesPerSec
memoryDeltaMb (real delta, not global heap)
cpuPercent (via OperatingSystemMXBean)
  1. JSON output format

{
  "metadata": {
    "generatedAt": "...",
    "protocol": { "repetitions": 3, "warmup": true }
  },
  "runs": [
    {
      "engine": "corese.4.6.5",
      "dataset": "dbpedia",
      "file": "category_labels.nt",
      "triples": 565108,
      "loadingTimeMs": 12310,
      "throughputTriplesPerSec": 45890,
      "memoryDeltaMb": 424.8,
      "cpuPercent": 78.2
    }
  ]
}

Files to delete

src/main/groovy/*.groovy
python-utils/plot-compare.py
python-utils/template.html
public/
Files to create

src/main/java/fr/inria/corese/benchmark/driver/TripleStoreDriver.java
src/main/java/fr/inria/corese/benchmark/harness/BenchmarkHarness.java
src/main/java/fr/inria/corese/benchmark/metrics/MetricsCollector.java
src/main/java/fr/inria/corese/benchmark/metrics/BenchmarkResult.java
src/main/java/fr/inria/corese/benchmark/Main.java
Update build.gradle for Java 21, remove Groovy

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

featureNew feature or capability

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions