ci: declare FLAKY_BENCHMARKS_REGEX and document benchmarks CI

.gitlab/BENCHMARKS.md

@@ -0,0 +1,32 @@
+# Benchmarks
+
+This describes the GitLab CI configuration for the benchmarks that run on the
+[Benchmarking Platform](https://datadoghq.atlassian.net/wiki/x/egAzk).
+
+## Layout
+
+- `benchmarks.yml`: k6 load tests against an Nginx server with the Datadog module.
+    - `baseline` and `only-tracing` run the `normal_operation` and `high_load` scenarios via the
+      `.benchmarks` template.
+    - `check-slo-breaches` gates releases based on SLOs defined on `bp-runner.fail-on-breach.yml`.
+    - Steps live in the `cpp/nginx` branch of
+      [benchmarking-platform](https://github.com/DataDog/benchmarking-platform).
+
+## Marking a Benchmark as Flaky
+
+Add it to `FLAKY_BENCHMARKS_REGEX` in `.benchmarks` in `benchmarks.yml`.
+
+The benchmark still runs and reports, but doesn't fail the `check-slo-breaches` performance
+quality gate (SLO-based).
+
+- The regex matches anywhere in the scenario name.
+    - `normal_operation` quarantines every `normal_operation` scenario, across both
+      `baseline` and `only-tracing`.
+    - Anchor with `^...$` to target one scenario.
+
+```yaml
+FLAKY_BENCHMARKS_REGEX: "^high_load--only-tracing--nginx-utilization$"
+```
+
+Open a ticket to fix or remove it. See
+[Flaky Benchmarks Monitoring](https://datadoghq.atlassian.net/wiki/x/dAKLrgE).

.gitlab/benchmarks.yml

@@ -70,6 +70,9 @@ build-nginx-module:
     DD_TRACE_DEBUG: "false"
     DD_RUNTIME_METRICS_ENABLED: "true"
 
+    # See https://datadoghq.atlassian.net/wiki/x/dAKLrgE for more details.
+    FLAKY_BENCHMARKS_REGEX: ""
+
     DD_BENCHMARKS_NGINX_IMAGE_TAG: amd64-${NGINX_VERSION}
 
     K6_OPTIONS_NORMAL_OPERATION_RATE: 1000