diff --git a/.gitlab/BENCHMARKS.md b/.gitlab/BENCHMARKS.md new file mode 100644 index 000000000..efcec2fdc --- /dev/null +++ b/.gitlab/BENCHMARKS.md @@ -0,0 +1,30 @@ +# Benchmarks + +This describes the GitLab CI configuration for the benchmarks that run on the +[Benchmarking Platform](https://datadoghq.atlassian.net/wiki/spaces/APMINT/pages/2419261562/Benchmarking+Platform). + +## Layout + +- `benchmarks.yml`: Google Benchmark microbenchmarks. + - `benchmarks` clones `benchmarking-platform` (`dd-trace-cpp` branch), runs the benchmark, + converts and analyzes results, uploads and comments on the PR. + - `check-big-regressions` fails on regressions above the threshold defined on + `bp-runner.fail-on-regression.yml` in the `dd-trace-cpp` branch of + [benchmarking-platform](https://github.com/DataDog/benchmarking-platform). + +## Marking a Benchmark as Flaky + +Add it to `FLAKY_BENCHMARKS_REGEX` in the `benchmarks` job's `variables` in `benchmarks.yml`. + +The benchmark still runs and reports, but doesn't fail the gate. + +- The regex matches anywhere in the scenario name. + - `BM_Trace` quarantines every scenario containing it, like `BM_TraceTinyCCSource`. + - Anchor with `^...$` to target one scenario. + +```yaml +FLAKY_BENCHMARKS_REGEX: "^BM_TraceTinyCCSource$" +``` + +Open a ticket to fix or remove it. See +[Flaky Benchmarks Monitoring](https://datadoghq.atlassian.net/wiki/x/dAKLrgE) for more details. diff --git a/.gitlab/benchmarks.yml b/.gitlab/benchmarks.yml index 8e5184467..14250c46d 100644 --- a/.gitlab/benchmarks.yml +++ b/.gitlab/benchmarks.yml @@ -32,6 +32,8 @@ benchmarks: UPSTREAM_PROJECT_NAME: $CI_PROJECT_NAME # "dd-trace-cpp" UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME # The branch or tag name for which project is built. UPSTREAM_COMMIT_SHA: $CI_COMMIT_SHA # The commit revision the project is built for. + # See https://datadoghq.atlassian.net/wiki/x/dAKLrgE for more details. + FLAKY_BENCHMARKS_REGEX: "" # This repository is using PR-level performance quality gates. # Verify that the check-big-regressions CI job has passed. If any regression happened, merging this PR will be blocked.