diff --git a/.gitlab/BENCHMARKS.md b/.gitlab/BENCHMARKS.md new file mode 100644 index 00000000..3d801320 --- /dev/null +++ b/.gitlab/BENCHMARKS.md @@ -0,0 +1,32 @@ +# Benchmarks + +This describes the GitLab CI configuration for the benchmarks that run on the +[Benchmarking Platform](https://datadoghq.atlassian.net/wiki/x/egAzk). + +## Layout + +- `benchmarks.yml`: k6 load tests against an Nginx server with the Datadog module. + - `baseline` and `only-tracing` run the `normal_operation` and `high_load` scenarios via the + `.benchmarks` template. + - `check-slo-breaches` gates releases based on SLOs defined on `bp-runner.fail-on-breach.yml`. + - Steps live in the `cpp/nginx` branch of + [benchmarking-platform](https://github.com/DataDog/benchmarking-platform). + +## Marking a Benchmark as Flaky + +Add it to `FLAKY_BENCHMARKS_REGEX` in `.benchmarks` in `benchmarks.yml`. + +The benchmark still runs and reports, but doesn't fail the `check-slo-breaches` performance +quality gate (SLO-based). + +- The regex matches anywhere in the scenario name. + - `normal_operation` quarantines every `normal_operation` scenario, across both + `baseline` and `only-tracing`. + - Anchor with `^...$` to target one scenario. + +```yaml +FLAKY_BENCHMARKS_REGEX: "^high_load--only-tracing--nginx-utilization$" +``` + +Open a ticket to fix or remove it. See +[Flaky Benchmarks Monitoring](https://datadoghq.atlassian.net/wiki/x/dAKLrgE). diff --git a/.gitlab/benchmarks.yml b/.gitlab/benchmarks.yml index dc9b6e0f..3574d765 100644 --- a/.gitlab/benchmarks.yml +++ b/.gitlab/benchmarks.yml @@ -70,6 +70,9 @@ build-nginx-module: DD_TRACE_DEBUG: "false" DD_RUNTIME_METRICS_ENABLED: "true" + # See https://datadoghq.atlassian.net/wiki/x/dAKLrgE for more details. + FLAKY_BENCHMARKS_REGEX: "" + DD_BENCHMARKS_NGINX_IMAGE_TAG: amd64-${NGINX_VERSION} K6_OPTIONS_NORMAL_OPERATION_RATE: 1000