Add OpenDoor benchmark and repair the atomic-action benchmark suite by Yuan-Xinyi · Pull Request #619 · DexForce/EmbodiChain

Yuan-Xinyi · 2026-09-12T09:14:45Z

Description

Adds an OpenDoor benchmark — the first articulated-skill benchmark in scripts/benchmark/atomic_action/ — and repairs the five existing atomic-action benchmarks, which could not run as shipped.

New: OpenDoor benchmark

open_door_benchmark.py plans the tutorial microwave-door opening across hinge targets (30°/60°/90°), physically replays the compiled trajectory, and gates success on the replayed hinge angle reaching the target within 0.15 rad — not merely on plan success. Registered in run_benchmark.py, so embodichain benchmark atomic-action --action open_door dispatches it, and it emits the three required Markdown tables (Time & Memory / Success & Metrics / Leaderboard).

Full-sample result (UR5 + PGI gripper, n_sample=10000, CUDA):

case	plan time	hinge error	opened
open_60	7.74 s	0.022 rad (1.3°)	✅

(Planning time is dominated by antipodal grasp-pose generation + NMS on the handle; the smoke profile uses n_sample=1000 and is a "runs without crashing" gate, matching the other benchmarks' smoke semantics.)

Prerequisite repairs to the existing suite

All five shipped benchmarks (move_joints, move_end_effector, pick_up, place, move_held_object) failed to run; OpenDoor reuses the same helpers, so these were fixed first:

Import failures — the tutorial modules never exported create_robot / initialize_simulation (and pickup/place/move_held_object also referenced compute_pick_close_end_step / make_pre_pick_eef_pose), so every benchmark raised ImportError at startup. Added them, plus a shared initialize_benchmark_simulation that adapts a benchmark argument namespace to create_tutorial_simulation.
Missing planning context — move_joints / move_end_effector called compile() without a context and hit IK interpolation requires explicit interpolation_dt; they now pass initial_context(control_dt=sim.sim_config.physics_dt).
Leaked SimulationManager — no benchmark released its sim, so every run (pass or fail) hung the process at exit; entry points now wrap main() in run_tutorial for deterministic top-level teardown.
Stale signature — pickup / place / move_held_object called the old two-arg get_hand_open_close_qpos(robot, device).
Off-by-epsilon tolerance — move_end_effector's 0.01 m success tolerance sat exactly on the resampled-endpoint error (0.0100 m) and failed by micrometres; widened to 0.015 m with a comment.

After the repairs, move_joints passes end-to-end (316 ms plan, success, report written, clean exit); the mesh-object benchmarks import, plan, replay, and report correctly.

Dependencies: none. (pick_up and grasp-sampling skills still require cuRobo for the full profile, unchanged.)

Type of change

New feature (non-breaking change which adds functionality)

Screenshots

N/A

Checklist

I have run the black . command to format the code base.
I have made corresponding changes to the documentation (benchmark scripts; no routed API docs affected)
Public API changes are reflected in the API docs (none)
I have added tests that prove my feature works (benchmark scripts are the executable artifact; validated by smoke + full runs, results above)
Dependencies have been updated, if applicable (none)

Validation

python -m scripts.benchmark.atomic_action.open_door_benchmark --smoke                          -> report written
python -m scripts.benchmark.atomic_action.open_door_benchmark --door_cases open_60 \
        --n_sample 10000 --device cuda                                                          -> opened, 0.022 rad
python -m scripts.benchmark.atomic_action.run_benchmark --action open_door --smoke              -> dispatches
python -m scripts.benchmark.atomic_action.move_joints_benchmark --smoke                         -> success, clean exit
black .                                                                                         -> clean

Add scripts/benchmark/atomic_action/open_door_benchmark.py, the first articulated-skill benchmark in the suite: it plans the microwave-door opening across hinge targets, physically replays the trajectory, and gates success on the replayed hinge angle reaching the target within 0.15 rad. Registered in run_benchmark.py so `embodichain benchmark atomic-action --action open_door` dispatches it. Full-sample run (Franka/PGI UR5, n_sample=10000, CUDA): plan 7.7 s, door opened with 0.022 rad hinge error. The five existing atomic-action benchmarks could not run as shipped; fixed as prerequisites (open_door reuses the same helpers): - The tutorial modules never exported create_robot / initialize_simulation (and pickup/place/move_held_object also needed compute_pick_close_end_step, make_pre_pick_eef_pose), so every benchmark failed at import. Added them plus a shared initialize_benchmark_simulation that adapts a benchmark namespace to create_tutorial_simulation. - move_joints / move_end_effector compiled without a planning context; they now pass initial_context(control_dt=sim.sim_config.physics_dt). - No benchmark released its SimulationManager, so every run (success or failure) hung the process at exit; entry points now wrap main() in run_tutorial for deterministic top-level teardown. - pickup / place / move_held_object called the old get_hand_open_close_qpos(robot, device) two-arg signature. - move_end_effector's 0.01 m success tolerance sat exactly on the resampled-endpoint error (0.0100 m) and failed by micrometres; widened to 0.015 m with a comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

greptile-apps · 2026-09-12T09:17:52Z

Confidence Score: 3/5

The PR is not safe to merge until PickUp physical validation uses the real compiled segment boundary and OpenDoor supports aggregate in-process dispatch.

Findings

Unreachable PickUp Boundary ▶
In-Process Dispatch Crashes ▶
Focused Tests Are Missing ▶
Public API Lacks Annotation ▶
Public Exports Are Incomplete ▶
Duplicate Helper Import ▶

Fix with agent prompt

### Issue 1
scripts/tutorials/atomic_action/tutorial_utils.py:1206
The fallback claims to return the end of PickUp's close segment, but `120 + 12` is beyond the entire 120-waypoint trajectory. With the benchmark's `sample_count=120` and `hand_interp_steps=12`, the compiled lift segment starts at waypoint 77. Both callbacks using this fallback therefore never call `obj.clear_dynamics()`, so physical PickUp validation and recorded replays omit the intended post-grasp dynamics transition and can produce invalid results. Pass the available compiled result to this helper or derive the boundary using PickUp's segment construction.

### Issue 2
scripts/benchmark/atomic_action/open_door_benchmark.py:279
Aggregate in-process dispatch constructs a child namespace without `door_cases`, but this benchmark unconditionally reads `args.door_cases`. Running `embodichain benchmark atomic-action --action open_door --in_process`, or reaching OpenDoor through `--action all --in_process`, therefore raises `AttributeError` before writing a report. Add the OpenDoor default to `_make_child_args` or apply the normal default when this attribute is absent.

### Issue 3
scripts/benchmark/atomic_action/open_door_benchmark.py:258
This feature and the accompanying benchmark repairs add no focused automated tests, despite the repository requirement that new fixes and features include tests proving their behavior. Before merging, add coverage for OpenDoor case selection and physical hinge success, aggregate in-process dispatch, and the new PickUp segment-boundary behavior. Executable benchmark scripts and manually reported smoke runs do not provide regression coverage.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 4
scripts/tutorials/atomic_action/tutorial_utils.py:1169
The new public helper leaves `args` unannotated, contrary to the repository requirement to fully annotate public APIs. This requirement must be satisfied before merging by adding an appropriate namespace or protocol annotation. The same new unannotated adapter parameters occur in `move_held_object.py` and `pickup.py`, with the pattern also present in the other tutorial adapters.

### Issue 5
scripts/tutorials/atomic_action/tutorial_utils.py:1169
The module's explicit public surface omits both newly added helpers, contrary to the repository requirement to define `__all__` for public modules. This requirement must be satisfied before merging by exporting `initialize_benchmark_simulation` and `compute_pick_close_end_step`. The new OpenDoor benchmark also lacks `__all__`, and the newly exposed tutorial adapter APIs follow the same pattern elsewhere.

### Issue 6
scripts/tutorials/atomic_action/move_held_object.py:62-66
`make_eef_pose_at` is imported in both adjacent import blocks introduced by this change. This redundant binding increases maintenance noise and can trigger duplicate-import lint failures. Remove the second occurrence.

```suggestion
from scripts.tutorials.atomic_action.tutorial_utils import (
    compute_pick_close_end_step,
    initialize_benchmark_simulation,
)
```

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds 30°, 60°, and 90° OpenDoor cases with hinge-angle-based success reporting.
Adds benchmark simulation and tutorial adapter helpers.
Supplies explicit planning contexts and wraps benchmark entry points in deterministic simulation teardown.
Contains two blocking benchmark defects: the PickUp dynamics boundary is unreachable, and OpenDoor crashes under aggregate in-process dispatch.
Also requires focused regression tests and public-API cleanup under repository conventions.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    CLI[Atomic-action benchmark CLI] --> Mode{Dispatch mode}
    Mode -->|isolated subprocess| Module[python -m action benchmark]
    Module --> Cleanup[run_tutorial lifecycle]
    Mode -->|--in_process| ChildArgs[_make_child_args]
    ChildArgs --> Existing[Existing action benchmarks]
    ChildArgs --> OpenDoor[OpenDoor run_all_benchmarks]
    OpenDoor --> Missing[Missing door_cases attribute]
    Existing --> Plan[Compile and replay trajectory]
    Plan --> Pickup{PickUp validation}
    Pickup --> Boundary[Compute post-grasp boundary]
    Boundary --> Wrong[Fallback returns waypoint 132]
    Wrong --> Replay[120-waypoint replay never clears dynamics]

_{Reviews (1) · Last reviewed commit: "bench(atomic): add OpenDoor benchmark an..."}

greptile-apps · 2026-09-12T09:17:55Z

+    """
+    if compiled is not None:
+        return int(compiled.segment(invocation_index, "lift").start)
+    return 120 + 12


Unreachable PickUp Boundary

The fallback claims to return the end of PickUp's close segment, but 120 + 12 is beyond the entire 120-waypoint trajectory. With the benchmark's sample_count=120 and hand_interp_steps=12, the compiled lift segment starts at waypoint 77. Both callbacks using this fallback therefore never call obj.clear_dynamics(), so physical PickUp validation and recorded replays omit the intended post-grasp dynamics transition and can produce invalid results. Pass the available compiled result to this helper or derive the boundary using PickUp's segment construction.

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/tutorials/atomic_action/tutorial_utils.py Line: 1206 Comment: **Unreachable PickUp Boundary** The fallback claims to return the end of PickUp's close segment, but `120 + 12` is beyond the entire 120-waypoint trajectory. With the benchmark's `sample_count=120` and `hand_interp_steps=12`, the compiled lift segment starts at waypoint 77. Both callbacks using this fallback therefore never call `obj.clear_dynamics()`, so physical PickUp validation and recorded replays omit the intended post-grasp dynamics transition and can produce invalid results. Pass the available compiled result to this helper or derive the boundary using PickUp's segment construction. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-12T09:17:55Z

+        initialize_benchmark_simulation,
+    )
+
+    cases = _select_cases(args.door_cases)


In-Process Dispatch Crashes

Aggregate in-process dispatch constructs a child namespace without door_cases, but this benchmark unconditionally reads args.door_cases. Running embodichain benchmark atomic-action --action open_door --in_process, or reaching OpenDoor through --action all --in_process, therefore raises AttributeError before writing a report. Add the OpenDoor default to _make_child_args or apply the normal default when this attribute is absent.

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/benchmark/atomic_action/open_door_benchmark.py Line: 279 Comment: **In-Process Dispatch Crashes** Aggregate in-process dispatch constructs a child namespace without `door_cases`, but this benchmark unconditionally reads `args.door_cases`. Running `embodichain benchmark atomic-action --action open_door --in_process`, or reaching OpenDoor through `--action all --in_process`, therefore raises `AttributeError` before writing a report. Add the OpenDoor default to `_make_child_args` or apply the normal default when this attribute is absent. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-12T09:17:55Z

+    return perf_rows, metric_rows
+
+
+def run_all_benchmarks(args: argparse.Namespace | None = None) -> Path:


Focused Tests Are Missing

This feature and the accompanying benchmark repairs add no focused automated tests, despite the repository requirement that new fixes and features include tests proving their behavior. Before merging, add coverage for OpenDoor case selection and physical hinge success, aggregate in-process dispatch, and the new PickUp segment-boundary behavior. Executable benchmark scripts and manually reported smoke runs do not provide regression coverage.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/benchmark/atomic_action/open_door_benchmark.py Line: 258 Comment: **Focused Tests Are Missing** This feature and the accompanying benchmark repairs add no focused automated tests, despite the repository requirement that new fixes and features include tests proving their behavior. Before merging, add coverage for OpenDoor case selection and physical hinge success, aggregate in-process dispatch, and the new PickUp segment-boundary behavior. Executable benchmark scripts and manually reported smoke runs do not provide regression coverage. **Context Used:** CLAUDE.md ([source](https://github.com/dexforce/embodichain/blob/main/CLAUDE.md)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

greptile-apps · 2026-09-12T09:17:55Z

 ]
+
+
+def initialize_benchmark_simulation(args) -> "SimulationManager":


Public API Lacks Annotation

The new public helper leaves args unannotated, contrary to the repository requirement to fully annotate public APIs. This requirement must be satisfied before merging by adding an appropriate namespace or protocol annotation. The same new unannotated adapter parameters occur in move_held_object.py and pickup.py, with the pattern also present in the other tutorial adapters.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/tutorials/atomic_action/tutorial_utils.py Line: 1169 Comment: **Public API Lacks Annotation** The new public helper leaves `args` unannotated, contrary to the repository requirement to fully annotate public APIs. This requirement must be satisfied before merging by adding an appropriate namespace or protocol annotation. The same new unannotated adapter parameters occur in `move_held_object.py` and `pickup.py`, with the pattern also present in the other tutorial adapters. **Context Used:** AGENTS.md ([source](https://github.com/dexforce/embodichain/blob/main/AGENTS.md)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-12T09:17:55Z

 ]
+
+
+def initialize_benchmark_simulation(args) -> "SimulationManager":


Public Exports Are Incomplete

The module's explicit public surface omits both newly added helpers, contrary to the repository requirement to define __all__ for public modules. This requirement must be satisfied before merging by exporting initialize_benchmark_simulation and compute_pick_close_end_step. The new OpenDoor benchmark also lacks __all__, and the newly exposed tutorial adapter APIs follow the same pattern elsewhere.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/tutorials/atomic_action/tutorial_utils.py Line: 1169 Comment: **Public Exports Are Incomplete** The module's explicit public surface omits both newly added helpers, contrary to the repository requirement to define `__all__` for public modules. This requirement must be satisfied before merging by exporting `initialize_benchmark_simulation` and `compute_pick_close_end_step`. The new OpenDoor benchmark also lacks `__all__`, and the newly exposed tutorial adapter APIs follow the same pattern elsewhere. **Context Used:** AGENTS.md ([source](https://github.com/dexforce/embodichain/blob/main/AGENTS.md)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-12T09:17:55Z

+from scripts.tutorials.atomic_action.tutorial_utils import (
+    compute_pick_close_end_step,
+    initialize_benchmark_simulation,
+    make_eef_pose_at,
+)


Duplicate Helper Import

make_eef_pose_at is imported in both adjacent import blocks introduced by this change. This redundant binding increases maintenance noise and can trigger duplicate-import lint failures. Remove the second occurrence.

Suggested change

from scripts.tutorials.atomic_action.tutorial_utils import (

compute_pick_close_end_step,

initialize_benchmark_simulation,

make_eef_pose_at,

)

from scripts.tutorials.atomic_action.tutorial_utils import (

compute_pick_close_end_step,

initialize_benchmark_simulation,

)

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/tutorials/atomic_action/move_held_object.py Line: 62-66 Comment: **Duplicate Helper Import** `make_eef_pose_at` is imported in both adjacent import blocks introduced by this change. This redundant binding increases maintenance noise and can trigger duplicate-import lint failures. Remove the second occurrence. ```suggestion from scripts.tutorials.atomic_action.tutorial_utils import ( compute_pick_close_end_step, initialize_benchmark_simulation, ) ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Yuan-Xinyi · 2026-09-12T09:21:27Z

拆分说明：这个 PR 里「修复既有 benchmark 套件可运行性」的部分已单独拆为 #620（纯 bug 修复，先行合入）。本 PR 保留新增的 OpenDoor benchmark，转为 draft 暂挂——待团队关于原子技能 benchmark 的详细指引出来后，按最终约定重做并 rebase 到 #620 之上。

Yuan-Xinyi added the enhancement New feature or request label Sep 12, 2026

greptile-apps Bot reviewed Sep 12, 2026

View reviewed changes

Yuan-Xinyi marked this pull request as draft September 12, 2026 09:21

Navigation Menu

Uh oh!

Add OpenDoor benchmark and repair the atomic-action benchmark suite - #619

Add OpenDoor benchmark and repair the atomic-action benchmark suite#619
Yuan-Xinyi wants to merge 1 commit into
mainfrom
xinyi/atomic02

Yuan-Xinyi commented Sep 12, 2026

Uh oh!

greptile-apps Bot commented Sep 12, 2026 •

edited

Loading

Diagram

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

greptile-apps Bot Sep 12, 2026

Uh oh!

Yuan-Xinyi commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		return perf_rows, metric_rows


		def run_all_benchmarks(args: argparse.Namespace \| None = None) -> Path:

		]


		def initialize_benchmark_simulation(args) -> "SimulationManager":

Uh oh!

Conversation

Yuan-Xinyi commented Sep 12, 2026

Description

New: OpenDoor benchmark

Prerequisite repairs to the existing suite

Type of change

Screenshots

Checklist

Validation

Uh oh!

greptile-apps Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

Yuan-Xinyi commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Sep 12, 2026 •

edited

Loading