ci: declare FLAKY_BENCHMARKS_REGEX and document benchmarks CI

.gitlab/BENCHMARKS.md

@@ -0,0 +1,30 @@
+# Benchmarks
+
+This describes the GitLab CI configuration for the benchmarks that run on the
+[Benchmarking Platform](https://datadoghq.atlassian.net/wiki/spaces/APMINT/pages/2419261562/Benchmarking+Platform).
+
+## Layout
+
+- `benchmarks.yml`: Google Benchmark microbenchmarks.
+    - `benchmarks` clones `benchmarking-platform` (`dd-trace-cpp` branch), runs the benchmark,
+      converts and analyzes results, uploads and comments on the PR.
+    - `check-big-regressions` fails on regressions above the threshold defined on
+      `bp-runner.fail-on-regression.yml` in the `dd-trace-cpp` branch of
+      [benchmarking-platform](https://github.com/DataDog/benchmarking-platform).
+
+## Marking a Benchmark as Flaky
+
+Add it to `FLAKY_BENCHMARKS_REGEX` in the `benchmarks` job's `variables` in `benchmarks.yml`.
+
+The benchmark still runs and reports, but doesn't fail the gate.
+
+- The regex matches anywhere in the scenario name.
+    - `BM_Trace` quarantines every scenario containing it, like `BM_TraceTinyCCSource`.
+    - Anchor with `^...$` to target one scenario.
+
+```yaml
+FLAKY_BENCHMARKS_REGEX: "^BM_TraceTinyCCSource$"
+```
+
+Open a ticket to fix or remove it. See
+[Flaky Benchmarks Monitoring](https://datadoghq.atlassian.net/wiki/x/dAKLrgE) for more details.

.gitlab/benchmarks.yml

@@ -32,6 +32,8 @@ benchmarks:
     UPSTREAM_PROJECT_NAME: $CI_PROJECT_NAME # "dd-trace-cpp"
     UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME # The branch or tag name for which project is built.
     UPSTREAM_COMMIT_SHA: $CI_COMMIT_SHA # The commit revision the project is built for.
+    # See https://datadoghq.atlassian.net/wiki/x/dAKLrgE for more details.
+    FLAKY_BENCHMARKS_REGEX: ""
 
 # This repository is using PR-level performance quality gates.
 # Verify that the check-big-regressions CI job has passed. If any regression happened, merging this PR will be blocked.