Skip to content

Vector-valued outputs as fzd objectives (part 2/2) - #76

Closed
yannrichet-asnr wants to merge 3 commits into
feat/vector-outputs-fzr-fzofrom
feat/vector-outputs-fzd
Closed

Vector-valued outputs as fzd objectives (part 2/2)#76
yannrichet-asnr wants to merge 3 commits into
feat/vector-outputs-fzr-fzofrom
feat/vector-outputs-fzd

Conversation

@yannrichet-asnr

Copy link
Copy Markdown
Member

Summary

Part 2 of vector-output support — part 1 is #75 (fzr/fzo). This PR is stacked on top of it (base branch: feat/vector-outputs-fzr-fzo) since the docs cross-reference each other; it'll auto-retarget to main once #75 merges.

fzd's algorithms (sampling, optimization, ...) always need a single scalar objective per case, but the underlying model output can now be a vector (e.g. a time series, per #75). This PR makes output_expression — the place where that reduction happens — actually usable for it.

Changes

  • fz/algorithms.py: evaluate_output_expression() gains sum(), len(), sorted(), mean(), median(), stdev(), variance() in its safe eval namespace, alongside the pre-existing math functions and indexing/slicing (which already worked, e.g. series[-1], max(series)). Referencing a vector-valued output without reducing it used to fail with a bare float() argument must be a string or a real number, not 'list' TypeError; it now raises a ValueError naming the offending output(s) and suggesting a concrete reduction (e.g. mean(T_series)). fzd's per-case error handling already treats any evaluate_output_expression failure as a failed point (output value None) rather than aborting the run, so this is a message-quality fix, not a behavior change there.
  • tests/test_fzd_vector_outputs.py (new, 14 tests): unit coverage of the new reduction helpers (including a realistic "RMS over a vector output" expression) and the improved error message (names the vector-valued key(s), leaves scalar outputs out of the blame list, keeps the old undefined-name error unchanged); plus two end-to-end fzd() tests with a real vector-output model (python://json_file time series) — one with a correctly reduced output_expression, verified against the manually computed mean for every sampled point, and one with a deliberately unreduced expression, verifying the run completes with every point reported as a failed evaluation rather than crashing.
  • Docs: doc/core-functions.md and examples/fzd_example.md get a new "Vector-valued outputs as objectives" subsection (with a working RMS example: sqrt(sum(v**2 for v in T_series) / len(T_series))); README.md and skills/fz/reference.md get matching notes; doc/INDEX.md links added; NEWS.md entry.

Scope note

fzd's own objective is still a single scalar per case — this is about reducing a vector-valued model output to that scalar, not multi-objective/vector-objective optimization (out of scope here).

Testing

  • New suite: pytest tests/test_fzd_vector_outputs.py — 14/14 passed.
  • No regressions: full test_fzd.py (35/35, including the slower subprocess-based TestFzdIntegration class), test_algorithm_options.py, test_algorithm_resolution.py, test_algorithm_installation.py, test_algorithm_plugins.py, test_r_algorithms.py, test_examples_scripts.py, test_readme_snippets.py, test_skill_static.py, plus a broad sweep of the rest of the suite (same exclusions CI itself applies).

🤖 Generated with Claude in Cowork mode.

@yannrichet-asnr

Copy link
Copy Markdown
Member Author

Pushed a follow-up commit checking (and confirming) that `output_expression` also supports objectives built from more than one vector output, not just a reduction of a single one:

  • Concatenate then reduce (already worked, no code change — just verified and documented): plain `+` on two lists concatenates them, e.g. `mean(a + b)` pools both series before averaging.
  • Combine two independent reductions: `mean(a) - mean(b)`, etc. — already worked.
  • Element-wise combination via a new `zip()` helper: e.g. an RMSE/residual between a simulated and a reference series — `sqrt(sum((x - y) ** 2 for x, y in zip(sim, ref)) / len(sim))`.

While adding the `zip()` case I found a real, separate bug it exposed: `evaluate_output_expression()` called `eval(expr, {"builtins": {}}, safe_dict)` — a split globals/locals dict. Any generator-expression or comprehension body resolves names through the globals dict only (never the separately-passed locals dict), so calling a helper function like `abs()` inside a genexp body (e.g. `max(abs(x - y) for x, y in zip(a, b))`) raised a spurious `name 'abs' is not defined` — even though the exact same call works fine outside a genexp. Fixed by evaluating with a single combined globals dict.

Added TestEvaluateOutputExpressionCombiningDifferentOutputs (6 tests) plus an end-to-end fzd() RMSE test to tests/test_fzd_vector_outputs.py (20/20 passing, was 14/14), and updated all the doc surfaces accordingly.

…reductions

Part 2 of vector-output support (part 1: #75, fzr/fzo). fzd's algorithms
(sampling, optimization, ...) always need a single scalar objective per
case, but the underlying model output can now be a vector (e.g. a time
series, per #75). This change makes output_expression -- the place where
that reduction happens -- actually usable for it.

- fz/algorithms.py: evaluate_output_expression() gains sum(), len(),
  sorted(), mean(), median(), stdev(), variance() in its safe eval
  namespace, alongside the pre-existing math functions and indexing/
  slicing (which already worked, e.g. "series[-1]", "max(series)").
  Referencing a vector-valued output without reducing it used to fail with
  a bare "float() argument must be a string or a real number, not 'list'"
  TypeError; it now raises a ValueError naming the offending output(s) and
  suggesting a concrete reduction (e.g. "mean(T_series)"). fzd's per-case
  error handling already treats any evaluate_output_expression failure as
  a failed point (output value None) rather than aborting the run, so
  this is a message-quality fix, not a behavior change there.
- tests/test_fzd_vector_outputs.py (new, 14 tests): unit coverage of the
  new reduction helpers (including a realistic "RMS over a vector output"
  expression) and the improved error message (names the vector-valued
  key(s), leaves scalar outputs out of the blame list, keeps the old
  undefined-name error unchanged); plus two end-to-end fzd() tests with a
  real vector-output model (python://json_file time series) -- one with a
  correctly reduced output_expression, verified against the manually
  computed mean for every sampled point, and one with a deliberately
  unreduced expression, verifying the run completes with every point
  reported as a failed evaluation rather than crashing.
- Docs: doc/core-functions.md and examples/fzd_example.md get a new
  "Vector-valued outputs as objectives" subsection (with a working RMS
  example: "sqrt(sum(v**2 for v in T_series) / len(T_series))"); README.md
  and skills/fz/reference.md get matching notes; doc/INDEX.md links added;
  NEWS.md entry.

fzd's own objective is still a single scalar per case -- this is about
reducing a vector-valued model *output* to that scalar, not multi-
objective/vector-objective optimization (out of scope here).
…scoping bug

Follow-up to the previous commit, prompted by explicitly checking that
output_expression supports objectives built from more than one vector
output, not just a reduction of a single one.

- fz/algorithms.py: add zip() to evaluate_output_expression()'s safe eval
  namespace, so two *different* vector outputs can be combined
  element-wise (e.g. an RMSE/residual between a simulated and a reference
  series: "sqrt(sum((x - y) ** 2 for x, y in zip(sim, ref)) / len(sim))").
  Concatenating two vector outputs instead needs no new helper: plain "+"
  on two lists already concatenates them (e.g. "mean(a + b)" pools both
  series before reducing) -- confirmed working, just newly documented.
- Fixed a real bug this uncovered: evaluate_output_expression() called
  eval(expr, {"__builtins__": {}}, safe_dict) -- a split globals/locals
  dict. Any generator-expression or comprehension body (e.g. the zip()
  pattern above, or even "sum(v**2 for v in series)" from the previous
  commit if it called a function like abs() inside the body) executes in
  its own nested scope, which Python resolves through the *globals* dict
  only, never through a separately-passed locals dict -- so referencing
  any output variable or helper function from inside such a body raised a
  spurious "name '...' is not defined". Fixed by evaluating with a single
  combined globals dict instead.
- tests/test_fzd_vector_outputs.py: new
  TestEvaluateOutputExpressionCombiningDifferentOutputs class covering
  concatenation-then-reduce, combining two independent reductions,
  element-wise zip() combination, the exact genexp-with-function-call
  pattern that exposed the scoping bug, and a three-output expression;
  plus a new end-to-end fzd() integration test computing an RMSE between a
  simulated and a reference vector output, verified point-by-point against
  a manual computation. 20/20 passing (was 14/14).
- Docs: doc/core-functions.md, examples/fzd_example.md, README.md and
  skills/fz/reference.md all get the concatenation/zip() patterns;
  NEWS.md entry updated.
@yannrichet-asnr
yannrichet-asnr force-pushed the feat/vector-outputs-fzd branch from cf1e188 to 5192ac6 Compare July 22, 2026 17:07
…e algorithm

Part 3 of the vector-output series (#75: fzr/fzo vector outputs; #76:
reductions of vector outputs to a scalar objective). #76's scope note
explicitly leaves multi-objective optimization out; this commit adds it:

- fzd()'s output_expression now also accepts a list of expressions; each
  case yields a list of scalars (one per expression, same order) passed
  as-is to get_next_design()/get_analysis(). A plain string keeps the
  legacy single-scalar behavior byte-for-byte unchanged.
- New evaluate_output_expressions() (fz/algorithms.py); log formatting,
  XY DataFrame and Y_<iteration>.csv gain one column per objective;
  function-model mode supported; errors in any expression fail the whole
  point (None), consistent with existing per-case error handling.
- examples/algorithms/nsga2.py: NSGA-II (Deb 2002) at the fzd plugin
  format. Batch-parallel generations (population returned as a batch so
  fzd spreads it across calculators), SBX + polynomial mutation, (mu+
  lambda) elitist selection, failed cases treated as dominated. Pareto
  front written to nsga2_pareto.csv and returned in analysis data.
- tests/test_fzd_multiobjective.py (8 tests): unit coverage of
  evaluate_output_expressions; end-to-end fzd+NSGA-II on the Binh-Korn
  problem validated against its analytic Pareto front in objective space
  (normalized deviation < 3%); scalar-expression backward-compat; partial
  objective failure -> whole point None, run completes.
- Docs: doc/core-functions.md 'Multi-objective (vector) objectives'
  subsection, skills/fz/reference.md note, NEWS.md entry.

No regressions: test_fzd.py, test_fzd_vector_outputs.py,
test_algorithm_options.py, test_algorithm_plugins.py -> 85 passed.
@yannrichet-asnr
yannrichet-asnr deleted the branch feat/vector-outputs-fzr-fzo July 23, 2026 07:10
@yannrichet-asnr
yannrichet-asnr deleted the feat/vector-outputs-fzd branch July 23, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant