Vector-valued outputs as fzd objectives (part 2/2) - #76
Conversation
|
Pushed a follow-up commit checking (and confirming) that `output_expression` also supports objectives built from more than one vector output, not just a reduction of a single one:
While adding the `zip()` case I found a real, separate bug it exposed: `evaluate_output_expression()` called `eval(expr, {"builtins": {}}, safe_dict)` — a split globals/locals dict. Any generator-expression or comprehension body resolves names through the globals dict only (never the separately-passed locals dict), so calling a helper function like `abs()` inside a genexp body (e.g. `max(abs(x - y) for x, y in zip(a, b))`) raised a spurious `name 'abs' is not defined` — even though the exact same call works fine outside a genexp. Fixed by evaluating with a single combined globals dict. Added |
…reductions Part 2 of vector-output support (part 1: #75, fzr/fzo). fzd's algorithms (sampling, optimization, ...) always need a single scalar objective per case, but the underlying model output can now be a vector (e.g. a time series, per #75). This change makes output_expression -- the place where that reduction happens -- actually usable for it. - fz/algorithms.py: evaluate_output_expression() gains sum(), len(), sorted(), mean(), median(), stdev(), variance() in its safe eval namespace, alongside the pre-existing math functions and indexing/ slicing (which already worked, e.g. "series[-1]", "max(series)"). Referencing a vector-valued output without reducing it used to fail with a bare "float() argument must be a string or a real number, not 'list'" TypeError; it now raises a ValueError naming the offending output(s) and suggesting a concrete reduction (e.g. "mean(T_series)"). fzd's per-case error handling already treats any evaluate_output_expression failure as a failed point (output value None) rather than aborting the run, so this is a message-quality fix, not a behavior change there. - tests/test_fzd_vector_outputs.py (new, 14 tests): unit coverage of the new reduction helpers (including a realistic "RMS over a vector output" expression) and the improved error message (names the vector-valued key(s), leaves scalar outputs out of the blame list, keeps the old undefined-name error unchanged); plus two end-to-end fzd() tests with a real vector-output model (python://json_file time series) -- one with a correctly reduced output_expression, verified against the manually computed mean for every sampled point, and one with a deliberately unreduced expression, verifying the run completes with every point reported as a failed evaluation rather than crashing. - Docs: doc/core-functions.md and examples/fzd_example.md get a new "Vector-valued outputs as objectives" subsection (with a working RMS example: "sqrt(sum(v**2 for v in T_series) / len(T_series))"); README.md and skills/fz/reference.md get matching notes; doc/INDEX.md links added; NEWS.md entry. fzd's own objective is still a single scalar per case -- this is about reducing a vector-valued model *output* to that scalar, not multi- objective/vector-objective optimization (out of scope here).
…scoping bug
Follow-up to the previous commit, prompted by explicitly checking that
output_expression supports objectives built from more than one vector
output, not just a reduction of a single one.
- fz/algorithms.py: add zip() to evaluate_output_expression()'s safe eval
namespace, so two *different* vector outputs can be combined
element-wise (e.g. an RMSE/residual between a simulated and a reference
series: "sqrt(sum((x - y) ** 2 for x, y in zip(sim, ref)) / len(sim))").
Concatenating two vector outputs instead needs no new helper: plain "+"
on two lists already concatenates them (e.g. "mean(a + b)" pools both
series before reducing) -- confirmed working, just newly documented.
- Fixed a real bug this uncovered: evaluate_output_expression() called
eval(expr, {"__builtins__": {}}, safe_dict) -- a split globals/locals
dict. Any generator-expression or comprehension body (e.g. the zip()
pattern above, or even "sum(v**2 for v in series)" from the previous
commit if it called a function like abs() inside the body) executes in
its own nested scope, which Python resolves through the *globals* dict
only, never through a separately-passed locals dict -- so referencing
any output variable or helper function from inside such a body raised a
spurious "name '...' is not defined". Fixed by evaluating with a single
combined globals dict instead.
- tests/test_fzd_vector_outputs.py: new
TestEvaluateOutputExpressionCombiningDifferentOutputs class covering
concatenation-then-reduce, combining two independent reductions,
element-wise zip() combination, the exact genexp-with-function-call
pattern that exposed the scoping bug, and a three-output expression;
plus a new end-to-end fzd() integration test computing an RMSE between a
simulated and a reference vector output, verified point-by-point against
a manual computation. 20/20 passing (was 14/14).
- Docs: doc/core-functions.md, examples/fzd_example.md, README.md and
skills/fz/reference.md all get the concatenation/zip() patterns;
NEWS.md entry updated.
cf1e188 to
5192ac6
Compare
…e algorithm Part 3 of the vector-output series (#75: fzr/fzo vector outputs; #76: reductions of vector outputs to a scalar objective). #76's scope note explicitly leaves multi-objective optimization out; this commit adds it: - fzd()'s output_expression now also accepts a list of expressions; each case yields a list of scalars (one per expression, same order) passed as-is to get_next_design()/get_analysis(). A plain string keeps the legacy single-scalar behavior byte-for-byte unchanged. - New evaluate_output_expressions() (fz/algorithms.py); log formatting, XY DataFrame and Y_<iteration>.csv gain one column per objective; function-model mode supported; errors in any expression fail the whole point (None), consistent with existing per-case error handling. - examples/algorithms/nsga2.py: NSGA-II (Deb 2002) at the fzd plugin format. Batch-parallel generations (population returned as a batch so fzd spreads it across calculators), SBX + polynomial mutation, (mu+ lambda) elitist selection, failed cases treated as dominated. Pareto front written to nsga2_pareto.csv and returned in analysis data. - tests/test_fzd_multiobjective.py (8 tests): unit coverage of evaluate_output_expressions; end-to-end fzd+NSGA-II on the Binh-Korn problem validated against its analytic Pareto front in objective space (normalized deviation < 3%); scalar-expression backward-compat; partial objective failure -> whole point None, run completes. - Docs: doc/core-functions.md 'Multi-objective (vector) objectives' subsection, skills/fz/reference.md note, NEWS.md entry. No regressions: test_fzd.py, test_fzd_vector_outputs.py, test_algorithm_options.py, test_algorithm_plugins.py -> 85 passed.
Summary
Part 2 of vector-output support — part 1 is #75 (fzr/fzo). This PR is stacked on top of it (base branch:
feat/vector-outputs-fzr-fzo) since the docs cross-reference each other; it'll auto-retarget tomainonce #75 merges.fzd's algorithms (sampling, optimization, ...) always need a single scalar objective per case, but the underlying model output can now be a vector (e.g. a time series, per #75). This PR makesoutput_expression— the place where that reduction happens — actually usable for it.Changes
fz/algorithms.py:evaluate_output_expression()gainssum(),len(),sorted(),mean(),median(),stdev(),variance()in its safe eval namespace, alongside the pre-existing math functions and indexing/slicing (which already worked, e.g.series[-1],max(series)). Referencing a vector-valued output without reducing it used to fail with a barefloat() argument must be a string or a real number, not 'list'TypeError; it now raises aValueErrornaming the offending output(s) and suggesting a concrete reduction (e.g.mean(T_series)).fzd's per-case error handling already treats anyevaluate_output_expressionfailure as a failed point (output valueNone) rather than aborting the run, so this is a message-quality fix, not a behavior change there.tests/test_fzd_vector_outputs.py(new, 14 tests): unit coverage of the new reduction helpers (including a realistic "RMS over a vector output" expression) and the improved error message (names the vector-valued key(s), leaves scalar outputs out of the blame list, keeps the old undefined-name error unchanged); plus two end-to-endfzd()tests with a real vector-output model (python://json_filetime series) — one with a correctly reducedoutput_expression, verified against the manually computed mean for every sampled point, and one with a deliberately unreduced expression, verifying the run completes with every point reported as a failed evaluation rather than crashing.doc/core-functions.mdandexamples/fzd_example.mdget a new "Vector-valued outputs as objectives" subsection (with a working RMS example:sqrt(sum(v**2 for v in T_series) / len(T_series)));README.mdandskills/fz/reference.mdget matching notes;doc/INDEX.mdlinks added;NEWS.mdentry.Scope note
fzd's own objective is still a single scalar per case — this is about reducing a vector-valued model output to that scalar, not multi-objective/vector-objective optimization (out of scope here).Testing
pytest tests/test_fzd_vector_outputs.py— 14/14 passed.test_fzd.py(35/35, including the slower subprocess-basedTestFzdIntegrationclass),test_algorithm_options.py,test_algorithm_resolution.py,test_algorithm_installation.py,test_algorithm_plugins.py,test_r_algorithms.py,test_examples_scripts.py,test_readme_snippets.py,test_skill_static.py, plus a broad sweep of the rest of the suite (same exclusions CI itself applies).🤖 Generated with Claude in Cowork mode.