Proxy env-bound primitives at eval time so blackbox can grant them

[yixuanhuang98]

Summary

build_primitives now refuses env-bound primitives under blackbox (PR #176), which
takes blackbox × low_level off the table. Restoring it means proxying env-bound
primitives through the env server at evaluation time, the way the sandbox already
does during synthesis.

Why it was refused

Env-bound primitives (ENV_DEPENDENT_PRIMITIVES = check_action_collision,
bilevel_models) are bound as a partial over the live env. The sandbox is careful
about this: blackbox_primitive_manifest (primitive_specs.py:96) emits a
host_proxy spec, and env_client.BlackboxEnv.make_primitives (env_client.py:653)
reconstructs it as a call over the wire, so the env object never enters the container.

Evaluation does not go through the manifest. AgenticApproach._load_generated
(agentic_base.py:320-323) hands the generated program self._primitives — the
host-side dict from build_primitives (run_experiment.py:133) — and the scored env
sits in that closure. A program granted check_action_collision reaches it as
primitives["check_action_collision"].args[0].

PR #176 wraps that env in a read-only view, which stops writes but passes reads
through. That is the right trade under whitebox, where the agent has the env source
anyway, and the wrong one under blackbox, where the mode exists to withhold it.

Reads cannot be closed off in-process. partial exposes fn.args; a plain closure
exposes fn.__closure__[0].cell_contents; an instance attribute exposes __dict__.
Whatever the binding, a callable that closes over the env in a shared process hands
the env to whoever holds the callable. So the combination was refused rather than
served unsafely.

What would fix it

Give evaluation the same host-proxy treatment the sandbox gets: when blackbox is set,
build env-dependent primitives as proxies that call the host across the env-server
boundary, instead of partials over the in-process env. The generated program then
holds a client, not an environment, and the blackbox boundary holds at scoring time
as well as during synthesis.

Sketch of the work:

• Run an env server for the evaluation env under blackbox (today the server is only
  stood up for the sandbox, in agentic_base around the env_server_running block).
• Build the eval-time primitives dict from blackbox_primitive_manifest rather than
  binding to the live env, so the two paths share one definition of what a black-box
  program may hold.
• Decide what a proxied check_action_collision costs per call: it is a per-step
  callable, so a round trip per query may be too slow for long episodes and may need
  batching or a coarser interface.
• Drop the blackbox guard in build_primitives and the low_level half of the
  constraints.py exclusion once the above holds.

Acceptance

• blackbox × low_level runs end to end, and the generated program cannot reach an
  env object through any primitive it is granted.
• A test in the spirit of tests/utils/test_generated_approach_examples.py: a program
  that tries to pull the env out of a primitive's closure gets a proxy with no env
  behind it, rather than the scored env.

Related

• PR Guard the scored env at the object; follow imports for the planner check #176 — added the guard this issue proposes lifting.
• Support bilevel primitives in blackbox #148 — bilevel_models under blackbox. Adjacent but separate: that one is about
  exposing the models at all (they leak env structure the agent would otherwise have
  to discover), this one is about the eval-time closure. bilevel_models needs both
  before it can run blackbox; check_action_collision needs only this one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bx7uCXEqziaAxTKbdkmp1g