PredictOps

ML-based predictive incident detection for distributed systems.

Traditional observability detects incidents after they happen. PredictOps asks:

Can telemetry predict an upcoming incident?

---

Prediction task

Primary formulation (Synthetic V2):

10 minutes of telemetry
        ↓
predict whether checkout-service has an incident
in the next 5 minutes

Incident definition (unchanged across V1/V2): a 1-minute checkout bucket with ≥10 spans and failure_rate > 10%.

V1 formulation (baseline, preserved): same 10-minute observation window, but a 15-minute prediction horizon and an earlier synthetic generator.

V2 moved to 5 minutes as a controlled forecasting setup after real-data analysis showed weak short-horizon precursors—not because RCAEval/AIOps proved a universal 5-minute precursor exists.

---

Datasets

Dataset	Role
Synthetic V1	Original baseline (span telemetry, N=10 / T=15, 12 features)
Synthetic V2	Real-data-informed controlled forecasting environment (N=10 / T=5, 38 features)
RCAEval RE1	Realism reference — feature trends / AT–POST signatures
AIOps 2020	Realism reference — continuous KPIs / fault timestamps

Important distinction:

Real datasets  = realism reference (distributions, relationships, AT/POST)
V2 synthetic   = controlled predictive environment (adds PRE-fault trajectories)

RCAEval and AIOps 2020 showed strong AT/POST signatures and weak/rare PRE signals. V2 intentionally adds noisy, varied precursors so early-warning learning can be studied under plausible assumptions.

---

Synthetic V2 (summary)

• 6 fault scenarios: cpu_pressure, memory_pressure, dependency_delay, disk_io, queue_load, cascade_payment
• 3 precursor strengths: weak / moderate / strong (different amplitude, noise, duration)
• Resource KPIs: CPU, memory, load (+ span latency/errors/throughput)
• Cross-service / dependency propagation
• Realistic noise, non-monotonic ramps, harmless spikes on negatives
• 38 temporal features (mean/std/latest/delta/slope, cross-service)
• 618 balanced windows (309 / 309), 160 fault episodes
• Episode-aware train/test split

Details: docs/synthetic-v2.md, realism audit: docs/v2-realism-audit.md.

---

Model

Primary comparison uses the same XGBoost architecture for V1 and V2 (isolates dataset/feature effects).

Optional Experiment C uses a mildly richer XGBoost on V2 only.

---

Results

Exp	Dataset	Features	Model	Precision	Recall	F1	ROC-AUC	PR-AUC	Lead Time
A	V1	12	baseline XGB	0.635	0.653	0.644	0.589	0.693	n/a (15m task)
B	V2	38	same XGB	0.673	0.532	0.595	0.640	0.695	~3 min median
C	V2	38	richer XGB	0.679	0.581	0.626	0.639	0.691	~3 min median

V2 F1 is lower than V1. That is expected: V2 is a harder, shorter-horizon, balanced forecasting task with measurable lead time. Do not prefer V1 solely because F1 is higher.

---

Limitations (honest)

• Data is synthetic (V1/V2).
• V2 precursors are controlled assumptions, not copies of real PRE signals.
• Real datasets (RCAEval, AIOps 2020) do not establish that 5-minute precursors are universal.
• Limited episode count; scenario×strength cells can be small.
• Accuracy alone is not the objective—lead time and temporal validity matter.

---

Repository structure

PredictOps/
├── README.md
├── CONTRIBUTING.md
├── docs/
│   ├── synthetic-v2.md              # V2 design & results
│   ├── v2-realism-audit.md          # final realism audit
│   ├── flow.md                      # end-to-end pipeline
│   ├── decisions.md                 # major project decisions
│   ├── rcaeval-feature-trends.md    # RCAEval analysis
│   └── aiops2020-precursor-comparison.md
├── telemetary/app/                  # simulators / V2 generator
├── ingestion/                       # process_telemetry.py (+ _v2)
├── model/                           # train / experiments / plots
│   └── data/v1|v2/                  # frozen processed CSVs
├── versions/v1|v2/                  # metrics, plots, V2 telemetry
├── tests/                           # leakage & pipeline tests
└── external/                        # optional real-data downloads (large; not required to train V1/V2)

---

Reproducibility

# V1 baseline train (uses frozen model/data/v1)
telemetary/.venv/bin/python model/train_model.py
# or: telemetary/.venv/bin/python model/run_experiments.py   # runs A/B/C

# V2 regenerate → process → evaluate → tests → plots
telemetary/.venv/bin/python telemetary/app/generate_v2_dataset.py
telemetary/.venv/bin/python ingestion/process_telemetry_v2.py
telemetary/.venv/bin/python model/run_experiments.py
telemetary/.venv/bin/python tests/test_v2_pipeline.py
telemetary/.venv/bin/python model/make_v2_plots.py

Fixed seeds: V2 generator 2026; models 42.

---

Docs map

Doc	Contents
docs/flow.md	Technical pipeline
docs/decisions.md	Why each major choice was made
docs/synthetic-v2.md	V2 design
docs/v2-realism-audit.md	Realism verdict & evidence
docs/rcaeval-feature-trends.md	Real-data RE1
docs/aiops2020-precursor-comparison.md	Real-data AIOps 2020