Skip to content

fix(run_orchestrator): preserve multiple_results in MAD_CONTAINER_IMAGE synthetic manifest - #166

Merged
coketaste merged 1 commit into
developfrom
coketaste/fix-local-image-multiple-results
Aug 12, 2026
Merged

fix(run_orchestrator): preserve multiple_results in MAD_CONTAINER_IMAGE synthetic manifest#166
coketaste merged 1 commit into
developfrom
coketaste/fix-local-image-multiple-results

Conversation

@coketaste

Copy link
Copy Markdown
Collaborator

Summary

  • _create_manifest_from_local_image() builds a synthetic manifest for MAD_CONTAINER_IMAGE (local-image) runs, but the built_models entry it constructs omitted the multiple_results field
    from models.json.
  • Without it, ContainerRunner never sets MAD_OUTPUT_CSV, never copies the perf CSV out of the container, and falls back to regex-scraping the run log for a performance: NUMBER METRIC
    line — reporting FAILURE (no performance metrics) even when the model produced a fully valid perf CSV.
  • Discovered while debugging a MAD Uber-storefront run that completed a full correctness + perf sweep successfully but was marked FAILURE in perf.csv/perf_super.json solely because it
    ran via local-image mode.

Fix

  • run_orchestrator.py: add "multiple_results": model.get("multiple_results", "") to the synthetic built_models entry, matching how every other models.json field is already carried
    through.

Test plan

  • Added TestCreateManifestFromLocalImage in tests/unit/test_orchestration.py:
    • test_multiple_results_field_is_preserved — a model with multiple_results set retains it in the generated manifest.
    • test_multiple_results_defaults_to_empty_string — a model without it still gets the key defaulted to "", matching existing field-default conventions.
  • pytest tests/unit tests/integration/test_orchestrator_workflows.py — 562 passed.

MAD_CONTAINER_IMAGE (local image) mode built a synthetic manifest that
omitted the models.json `multiple_results` field. Without it,
ContainerRunner never sets MAD_OUTPUT_CSV, never copies the perf CSV
out of the container, and falls back to scraping the run log for a
"performance: NUMBER METRIC" line -- reporting FAILURE even when the
model produced valid perf-CSV results.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@coketaste coketaste self-assigned this Aug 10, 2026
Copilot AI lite review requested due to automatic review settings August 10, 2026 22:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes local-image (MAD_CONTAINER_IMAGE) execution by ensuring the synthetic build_manifest.json produced by RunOrchestrator._create_manifest_from_local_image() preserves the multiple_results field from models.json, which ContainerRunner relies on to set MAD_OUTPUT_CSV and correctly export perf CSV results.

Changes:

  • Preserve multiple_results when constructing synthetic built_models entries for local-image runs.
  • Add unit tests verifying multiple_results is preserved when provided and defaults to "" when absent.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/madengine/orchestration/run_orchestrator.py Adds multiple_results to the synthetic manifest’s built_models entries so local-image runs follow the same manifest schema as built-image runs.
tests/unit/test_orchestration.py Adds unit coverage to prevent regressions for multiple_results propagation and defaulting behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@coketaste
coketaste merged commit 4b739b6 into develop Aug 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants