Vector-valued outputs as fzd objectives, plus multi-objective (NSGA-II) support - #77
Merged
Merged
Conversation
…reductions Part 2 of vector-output support (part 1: #75, fzr/fzo). fzd's algorithms (sampling, optimization, ...) always need a single scalar objective per case, but the underlying model output can now be a vector (e.g. a time series, per #75). This change makes output_expression -- the place where that reduction happens -- actually usable for it. - fz/algorithms.py: evaluate_output_expression() gains sum(), len(), sorted(), mean(), median(), stdev(), variance() in its safe eval namespace, alongside the pre-existing math functions and indexing/ slicing (which already worked, e.g. "series[-1]", "max(series)"). Referencing a vector-valued output without reducing it used to fail with a bare "float() argument must be a string or a real number, not 'list'" TypeError; it now raises a ValueError naming the offending output(s) and suggesting a concrete reduction (e.g. "mean(T_series)"). fzd's per-case error handling already treats any evaluate_output_expression failure as a failed point (output value None) rather than aborting the run, so this is a message-quality fix, not a behavior change there. - tests/test_fzd_vector_outputs.py (new, 14 tests): unit coverage of the new reduction helpers (including a realistic "RMS over a vector output" expression) and the improved error message (names the vector-valued key(s), leaves scalar outputs out of the blame list, keeps the old undefined-name error unchanged); plus two end-to-end fzd() tests with a real vector-output model (python://json_file time series) -- one with a correctly reduced output_expression, verified against the manually computed mean for every sampled point, and one with a deliberately unreduced expression, verifying the run completes with every point reported as a failed evaluation rather than crashing. - Docs: doc/core-functions.md and examples/fzd_example.md get a new "Vector-valued outputs as objectives" subsection (with a working RMS example: "sqrt(sum(v**2 for v in T_series) / len(T_series))"); README.md and skills/fz/reference.md get matching notes; doc/INDEX.md links added; NEWS.md entry. fzd's own objective is still a single scalar per case -- this is about reducing a vector-valued model *output* to that scalar, not multi- objective/vector-objective optimization (out of scope here).
…scoping bug
Follow-up to the previous commit, prompted by explicitly checking that
output_expression supports objectives built from more than one vector
output, not just a reduction of a single one.
- fz/algorithms.py: add zip() to evaluate_output_expression()'s safe eval
namespace, so two *different* vector outputs can be combined
element-wise (e.g. an RMSE/residual between a simulated and a reference
series: "sqrt(sum((x - y) ** 2 for x, y in zip(sim, ref)) / len(sim))").
Concatenating two vector outputs instead needs no new helper: plain "+"
on two lists already concatenates them (e.g. "mean(a + b)" pools both
series before reducing) -- confirmed working, just newly documented.
- Fixed a real bug this uncovered: evaluate_output_expression() called
eval(expr, {"__builtins__": {}}, safe_dict) -- a split globals/locals
dict. Any generator-expression or comprehension body (e.g. the zip()
pattern above, or even "sum(v**2 for v in series)" from the previous
commit if it called a function like abs() inside the body) executes in
its own nested scope, which Python resolves through the *globals* dict
only, never through a separately-passed locals dict -- so referencing
any output variable or helper function from inside such a body raised a
spurious "name '...' is not defined". Fixed by evaluating with a single
combined globals dict instead.
- tests/test_fzd_vector_outputs.py: new
TestEvaluateOutputExpressionCombiningDifferentOutputs class covering
concatenation-then-reduce, combining two independent reductions,
element-wise zip() combination, the exact genexp-with-function-call
pattern that exposed the scoping bug, and a three-output expression;
plus a new end-to-end fzd() integration test computing an RMSE between a
simulated and a reference vector output, verified point-by-point against
a manual computation. 20/20 passing (was 14/14).
- Docs: doc/core-functions.md, examples/fzd_example.md, README.md and
skills/fz/reference.md all get the concatenation/zip() patterns;
NEWS.md entry updated.
…e algorithm Part 3 of the vector-output series (#75: fzr/fzo vector outputs; #76: reductions of vector outputs to a scalar objective). #76's scope note explicitly leaves multi-objective optimization out; this commit adds it: - fzd()'s output_expression now also accepts a list of expressions; each case yields a list of scalars (one per expression, same order) passed as-is to get_next_design()/get_analysis(). A plain string keeps the legacy single-scalar behavior byte-for-byte unchanged. - New evaluate_output_expressions() (fz/algorithms.py); log formatting, XY DataFrame and Y_<iteration>.csv gain one column per objective; function-model mode supported; errors in any expression fail the whole point (None), consistent with existing per-case error handling. - examples/algorithms/nsga2.py: NSGA-II (Deb 2002) at the fzd plugin format. Batch-parallel generations (population returned as a batch so fzd spreads it across calculators), SBX + polynomial mutation, (mu+ lambda) elitist selection, failed cases treated as dominated. Pareto front written to nsga2_pareto.csv and returned in analysis data. - tests/test_fzd_multiobjective.py (8 tests): unit coverage of evaluate_output_expressions; end-to-end fzd+NSGA-II on the Binh-Korn problem validated against its analytic Pareto front in objective space (normalized deviation < 3%); scalar-expression backward-compat; partial objective failure -> whole point None, run completes. - Docs: doc/core-functions.md 'Multi-objective (vector) objectives' subsection, skills/fz/reference.md note, NEWS.md entry. No regressions: test_fzd.py, test_fzd_vector_outputs.py, test_algorithm_options.py, test_algorithm_plugins.py -> 85 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces #76 (accidentally auto-closed by GitHub when I deleted its stacked
base branch
feat/vector-outputs-fzr-fzoafter merging #75 — sorry aboutthat; same head branch
feat/vector-outputs-fzd, no commits lost). Nowtargets
maindirectly since #75 is merged.Part 2 of vector-output support (part 1, #75, is merged).
fzd's algorithms(sampling, optimization, ...) always need a scalar objective per case, but
the underlying model output can now be a vector (e.g. a time series, per
#75). This PR makes
output_expression— the place where that reductionhappens — actually usable for it, and additionally adds native
multi-objective support (see "Multi-objective" section below, added on top
of the original #76 scope).
Changes
fz/algorithms.py:evaluate_output_expression()gainssum(),len(),sorted(),mean(),median(),stdev(),variance()in itssafe eval namespace, alongside the pre-existing math functions and
indexing/slicing (e.g.
series[-1],max(series)). Referencing avector-valued output without reducing it used to fail with a bare
TypeError; it now raises aValueErrornaming the offending output(s)and suggesting a concrete reduction (e.g.
mean(T_series)).fzd'sper-case error handling already treats any evaluation failure as a failed
point (
None) rather than aborting the run, so this is a message-qualityfix, not a behavior change there.
tests/test_fzd_vector_outputs.py(14 tests): unit coverage of thereduction helpers and the improved error message; two end-to-end
fzd()tests with a real vector-output model.
doc/core-functions.md,examples/fzd_example.md,README.md,skills/fz/reference.md,doc/INDEX.md,NEWS.md.Multi-objective (vector) objectives — added on top of #76's original scope
#76's scope note explicitly left multi-objective optimization out; thisadds it:
output_expressionnow also accepts a list of expressions: each caseyields a list of scalars (one per expression, same order), passed as-is to
the algorithm's
get_next_design()/get_analysis(). A plain string keepsthe legacy single-scalar behavior byte-for-byte unchanged.
evaluate_output_expressions()(fz/algorithms.py); log formatting,the
XYDataFrame andY_<iteration>.csvgain one column per objective;function-model mode supported.
examples/algorithms/nsga2.py: NSGA-II (Deb 2002) as an fzd plugin —batch-parallel generations, SBX + polynomial mutation, (mu+lambda) elitist
selection, Pareto front written to
nsga2_pareto.csvand returned in theanalysis
data(pareto_X/pareto_F).tests/test_fzd_multiobjective.py(8 tests): unit coverage ofevaluate_output_expressions; end-to-endfzd+ NSGA-II on theBinh-Korn problem validated against its analytic Pareto front (objective-
space deviation < 3%); scalar-expression backward-compat; partial
objective failure -> whole point
None.Testing
pytest tests/test_fzd_vector_outputs.py— 14/14 passed.pytest tests/test_fzd_multiobjective.py— 8/8 passed.test_fzd.py,test_fzd_vector_outputs.py,test_algorithm_options.py,test_algorithm_plugins.py— 85 passed, 1skipped.