Production-style API testing and diagnosis platform with Java execution authority, Python Agent workflows, a guarded Tool Gateway, tracing/evaluation, and a React console.
中文说明 · Documentation · HR demo · Benchmark results
- Java owns deterministic execution, security, and authoritative
TestReportfacts. - Python AgentLab owns TestCase generation, diagnosis, tool orchestration, tracing, and runtime evaluation.
- Agents cannot bypass Java-owned resources; resource access crosses the guarded Java Tool Gateway.
- The Java ↔ Python boundary is verified with real cross-process E2E tests.
- The public release includes 14 complete APIOps Bench runs, with failures, unknowns and execution modes preserved.
Java owns deterministic execution and security boundaries. Python owns Agent orchestration, diagnosis, tracing, and evaluation.
OpenAPI → Metadata → TestCase DSL → Async Runner → Assertion → TestReport
→ Diagnosis → Evidence / Tool → DiagnosisReport → Trace / Evaluation
LLM-generated TestCases are candidates. Java validates the DSL, runs the test, and publishes the authoritative TestReport; AgentLab consumes that report and controlled evidence to produce diagnosis and evaluation outputs.
| Fact / Responsibility | Authority |
|---|---|
| OpenAPI metadata | Java Platform |
| DSL validation | Java Platform |
| HTTP execution | Java Runner |
| Assertions | Java Platform |
| TestReport | Java Platform |
| Tool authorization | Java Tool Gateway |
| Agent workflow | Python AgentLab |
| Diagnosis reasoning | Python AgentLab |
| Agent trace | Python AgentLab |
| Runtime evaluation | Python AgentLab |
LLM output is a candidate or inference. Java TestReport is execution truth. Python ALLOW is not Java ALLOW, and Human Approval is not Java Authorization.
- OpenAPI metadata and project-scoped API access
- TestCase DSL validation
- HTTP Runner and asynchronous batch execution
- Assertion Engine and authoritative TestReport
- JWT/project authorization
- RAG retrieval and Context Pack construction
- Structured Agent generation and diagnosis integration
- Tool Gateway, guards, sanitization, and audit
- TestCase Generation workflow
- Diagnosis workflow with structured outputs
- Context engineering and Java evidence consumption
- Tool planning and guarded tool-use workflow
- HITL approval flow
- Trace recording and redaction
- Runtime evaluator, metrics, judge, and reporting
- Overview
- API Studio
- Runs
- Diagnosis Studio and Diagnosis execution
- Traces
- Evaluation
- Settings
The Console supports Chinese and English, URL-based navigation, run cancellation and reruns, redacted HTTP snapshots, report export, persisted diagnosis history and approval recovery. Automatic diagnosis triggering and browser-based Java diagnosis are not implemented. Diagnosis locking and recovery target a single-machine SQLite deployment.
APIOps-Bench 105 is a project-specific evaluation across generation, diagnosis, tool safety, RAG evidence and end-to-end API operations. The public bundle contains 14 complete runs and 1,470 task results: 12 real-model adapter runs and two mixed fixture baselines.
| Run | PASS | FAIL | UNKNOWN |
|---|---|---|---|
| Portfolio v5 | 95 | 9 | 1 |
| Diagnosis Contract v2 | 94 | 8 | 3 |
These runs use different evaluation contracts and must not be combined or treated as a controlled model comparison. The v5 pass rate is 95/105 (90.48%). Historical failures and unavailable metrics remain visible. This is not a general model leaderboard or a production reliability estimate.
The Console reads saved results and starts no evaluations. See result interpretation, publication policy and the published bundles. This release copies existing evidence; it does not call models, rerun tasks or change scores.
The HR demo setup creates a dedicated project, presenter/viewer
accounts with random local passwords, and real HTTP success/failure examples. After
configuring the services, run scripts/setup-hr-demo.ps1 -SkipStart on Windows.
The prepared Windows launcher is also included.
Walk through API Studio, a successful run, an intentional assertion failure, the diagnosis entry point and Benchmark history. The existing examples remain contract fixtures; they are not presented as new runtime evidence.
Python tool selection is orchestration, not authorization. Python Guard is defense-in-depth; HITL is workflow consent, not Java resource authorization. Java Tool Gateway performs final authorization, guarding, execution, sanitization, and audit. Java DENY is terminal; Python has no raw-resource fallback.
| Capability | Implementation | Verification |
|---|---|---|
| OpenAPI metadata | OpenAPI metadata controller | Controller test |
| TestCase DSL validation | DSL validator | Validation tests |
| HTTP execution | Run execution service | Runner service tests |
| Assertions | Assertion Engine | Assertion tests |
| Async test batches | Batch controller | Batch controller tests |
| TestReport | Report controller | Report controller tests |
| Tool Gateway | Tool Gateway | Gateway tests |
| RAG / evidence | Context Pack builder | Context Pack tests |
| Python generation | Generation workflow | Generation workflow tests |
| Python diagnosis | Diagnosis workflow | Diagnosis workflow tests |
| Tracing | Trace recorder | Trace recorder tests |
| Runtime evaluation | Evaluator | Evaluator tests |
| Java ↔ Python integration | Java API client | Cross-process E2E test |
| Console context trail | Console shell | Context-trail verification |
The public integration surface is project-scoped:
| Capability | Method | Path |
|---|---|---|
| OpenAPI metadata | GET |
/api/v1/projects/{projectId}/openapi/apis/{apiId} |
| Validate an API's TestCase DSL | POST |
/api/v1/projects/{projectId}/openapi/apis/{apiId}/testcases:validate |
| Submit test batch | POST |
/api/v1/projects/{projectId}/test-batches |
| Read test report | GET |
/api/v1/projects/{projectId}/test-runs/{runId}/report |
| Call a guarded tool | POST |
/api/v1/projects/{projectId}/tool-calls |
Java also validates the DSL at Agent and Runner boundaries. The Python read-only
Benchmark endpoint is GET /api/v1/benchmark/results.
Run each verification block from the repository root.
# Java Platform
cd java-apiops-platform
./mvnw clean verify# Python AgentLab
cd python-apiops-agentlab
uv run ruff check .
uv run pytest
uv run python ../scripts/validate-schemas.py# React Console
cd apiops-console
npm ci
npm run lint
npm run test:context
npm run verify:benchmark
npm run buildOn Windows, use ./mvnw.cmd clean verify for the Java command.
# Python API
cd python-apiops-agentlab
uv run uvicorn app.main:app --reload# Console
cd apiops-console
npm run devThe Console and AgentLab connect to their configured Java/Python services. docker-compose.dev.yml provides local MySQL, Redis, and RabbitMQ infrastructure; it is not a one-command application deployment.
See development setup for configuration, proxy ports and the complete Console verification commands. Real provider credentials and personal deployment endpoints must be supplied locally.
The public source and evidence workflow runs the Python suite, Ruff, shared-schema checks, Console checks and immutable evidence verification. Java CI runs the default Maven verification with Docker. Use the commands above to reproduce the checks locally.
Environment-gated live tests may be skipped; skipped checks are not successful live validation. Historical model-run results are independent of these code checks. The public source inventory records publication work with no benchmark execution or model calls.
agentic-apiops-public
├── .github
│ └── assets
├── java-apiops-platform
├── python-apiops-agentlab
├── apiops-console
├── shared-schemas
├── examples
├── artifacts/benchmark
├── docs
├── scripts
└── README.md
Java is responsible for execution, security, audit, and report truth. Python is responsible for Agent workflow, tool orchestration, tracing, and runtime evaluation. The projects meet through shared schemas, REST contracts, and the Java Tool Gateway.
This public portfolio includes executable source, ordinary tests, synthetic fixtures, shared schemas, selected documentation, demo initialization scripts and the reviewed Benchmark publication and selected regression evidence. Unrelated raw experiments, local databases, service logs, private development notes and credentials are excluded. Historical provenance paths remain as source identifiers; available public result files are listed in the manifest. Machine-local paths in selected regression copies are normalized and recorded in the evidence inventory; the published run bundles remain byte-identical to their sources.
Configuration templates are provided as .env.example files; real credentials are never committed.