Skip to content

Make the atomic-action benchmark suite runnable - #620

Open
Yuan-Xinyi wants to merge 1 commit into
mainfrom
xinyi/atomic-bench-fix
Open

Make the atomic-action benchmark suite runnable#620
Yuan-Xinyi wants to merge 1 commit into
mainfrom
xinyi/atomic-bench-fix

Conversation

@Yuan-Xinyi

Copy link
Copy Markdown
Collaborator

Description

The five shipped atomic-action benchmarks (move_joints, move_end_effector, pick_up, place, move_held_object) could not run as delivered. This PR fixes each blocker so the suite is runnable; it adds no new benchmark.

# Problem Fix
1 Tutorial modules never exported create_robot / initialize_simulation (pickup/place/move_held_object also referenced compute_pick_close_end_step, make_pre_pick_eef_pose) — every benchmark raised ImportError at startup Added those symbols, plus a shared initialize_benchmark_simulation that adapts a benchmark argument namespace to create_tutorial_simulation
2 move_joints / move_end_effector compiled without a planning context → IK interpolation requires explicit interpolation_dt Pass initial_context(control_dt=sim.sim_config.physics_dt)
3 No benchmark released its SimulationManager → every run (success or failure) hung the process at exit Entry points wrap main() in run_tutorial for deterministic top-level teardown
4 pickup / place / move_held_object called the old two-arg get_hand_open_close_qpos(robot, device) Updated to the current keyword signature
5 move_end_effector's 0.01 m success tolerance sat exactly on the resampled-endpoint error (0.0100 m) and failed by micrometres Widened to 0.015 m with an explanatory comment

After the fixes, move_joints runs end to end (≈0.1–0.3 s plan, success, report written, clean exit); the mesh-object benchmarks import, plan, replay, and report correctly.

Dependencies: none. (pick_up and grasp-sampling skills still require cuRobo for the full profile, unchanged.)

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Screenshots

N/A

Checklist

  • I have run the black . command to format the code base.
  • I have made corresponding changes to the documentation (benchmark/tutorial scripts; no routed API docs affected)
  • Public API changes are reflected in the API docs (none)
  • I have added tests that prove my fix works (benchmark scripts are the executable artifact; validated by move_joints_benchmark --smoke running to a written report with a clean exit)
  • Dependencies have been updated, if applicable (none)

Validation

python -m scripts.benchmark.atomic_action.move_joints_benchmark --smoke   -> success, report written, clean exit
black .                                                                    -> clean

The five shipped atomic-action benchmarks (move_joints, move_end_effector,
pick_up, place, move_held_object) could not run as delivered. Each fix:

- The tutorial modules never exported create_robot / initialize_simulation
  (pickup/place/move_held_object also referenced compute_pick_close_end_step
  and make_pre_pick_eef_pose), so every benchmark failed at import. Added
  those symbols plus a shared initialize_benchmark_simulation that adapts a
  benchmark argument namespace to create_tutorial_simulation.
- move_joints / move_end_effector compiled without a planning context and
  hit "IK interpolation requires explicit interpolation_dt"; they now pass
  initial_context(control_dt=sim.sim_config.physics_dt).
- No benchmark released its SimulationManager, so every run (success or
  failure) hung the process at exit; entry points now wrap main() in
  run_tutorial for deterministic top-level teardown.
- pickup / place / move_held_object called the old two-arg
  get_hand_open_close_qpos(robot, device) signature.
- move_end_effector's 0.01 m success tolerance sat exactly on the
  resampled-endpoint error (0.0100 m) and failed by micrometres; widened to
  0.015 m with an explanatory comment.

After the fixes move_joints runs end to end (316 ms plan, success, report
written, clean exit) and the mesh-object benchmarks import, plan, replay,
and report correctly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 12, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

The PR is not yet safe to merge because the pickup phase boundary can corrupt physical-validation results, and the new public APIs must satisfy the repository’s annotation requirement.

Fix All in CodexFindings

  1. P1 Incorrect pickup phase boundary
  2. P2 Public APIs lack annotations
Fix with agent prompt
### Issue 1
scripts/tutorials/atomic_action/tutorial_utils.py:1206
The no-argument fallback returns step 132, but the configured 120-waypoint PickUp trajectory has 65 approach, 12 close, and 43 lift waypoints, so lifting begins at step 77. The pickup, place, and move-held-object benchmarks call this helper without passing their available compiled result. Their dynamics-clearing callback therefore runs too late-or never runs for a 120-waypoint trajectory-making the physical-validation measurements unreliable. Use the compiled `"lift"` segment boundary at these call sites instead of the fixed calculation.

### Issue 2
scripts/tutorials/atomic_action/tutorial_utils.py:1169
The new public `initialize_benchmark_simulation` function leaves `args` unannotated. The same omission affects `compiled` in `compute_pick_close_end_step` and parameters in the new tutorial wrapper APIs. This violates the repository directive that public APIs must be fully annotated, with circular imports guarded by `TYPE_CHECKING` where needed. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

  • Adds shared benchmark-to-tutorial simulation initialization and tutorial compatibility exports.
  • Supplies an explicit control timestep for joint and end-effector compilation.
  • Wraps standalone benchmark entry points in deterministic simulation teardown.
  • Updates gripper helper calls and adjusts the end-effector success tolerance.
  • The new pickup boundary fallback is inconsistent with the actual segmented trajectory and can invalidate physical-validation metrics.
  • Newly introduced public helper APIs require complete parameter annotations.

Diagram

sequenceDiagram
    participant B as Benchmark
    participant T as Tutorial adapter
    participant E as AtomicActionEngine
    participant P as PickUp planner
    participant S as Simulation
    B->>T: initialize_simulation(args)
    T->>S: create tutorial simulation
    B->>E: compile(invocation, context)
    E->>P: build approach/close/lift trajectory
    P-->>E: compiled trajectory and segment metadata
    E-->>B: result
    B->>B: determine dynamics-clear waypoint
    loop trajectory waypoints
      B->>S: replay waypoint
    end
Loading

Reviews (1) · Last reviewed commit: "fix(bench): make the atomic-action bench..."

"""
if compiled is not None:
return int(compiled.segment(invocation_index, "lift").start)
return 120 + 12

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Incorrect pickup phase boundary

The no-argument fallback returns step 132, but the configured 120-waypoint PickUp trajectory has 65 approach, 12 close, and 43 lift waypoints, so lifting begins at step 77. The pickup, place, and move-held-object benchmarks call this helper without passing their available compiled result. Their dynamics-clearing callback therefore runs too late—or never runs for a 120-waypoint trajectory—making the physical-validation measurements unreliable. Use the compiled "lift" segment boundary at these call sites instead of the fixed calculation.

Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/tutorials/atomic_action/tutorial_utils.py
Line: 1206

Comment:
**Incorrect pickup phase boundary**

The no-argument fallback returns step 132, but the configured 120-waypoint PickUp trajectory has 65 approach, 12 close, and 43 lift waypoints, so lifting begins at step 77. The pickup, place, and move-held-object benchmarks call this helper without passing their available compiled result. Their dynamics-clearing callback therefore runs too late—or never runs for a 120-waypoint trajectory—making the physical-validation measurements unreliable. Use the compiled `"lift"` segment boundary at these call sites instead of the fixed calculation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

]


def initialize_benchmark_simulation(args) -> "SimulationManager":

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Public APIs lack annotations

The new public initialize_benchmark_simulation function leaves args unannotated. The same omission affects compiled in compute_pick_close_end_step and parameters in the new tutorial wrapper APIs. This violates the repository directive that public APIs must be fully annotated, with circular imports guarded by TYPE_CHECKING where needed. This repository requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/tutorials/atomic_action/tutorial_utils.py
Line: 1169

Comment:
**Public APIs lack annotations**

The new public `initialize_benchmark_simulation` function leaves `args` unannotated. The same omission affects `compiled` in `compute_pick_close_end_step` and parameters in the new tutorial wrapper APIs. This violates the repository directive that public APIs must be fully annotated, with circular imports guarded by `TYPE_CHECKING` where needed. This repository requirement must be satisfied before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/dexforce/embodichain/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code

@Yuan-Xinyi

Copy link
Copy Markdown
Collaborator Author

CI 的 build / test 失败发生在各自 setup 步骤的 pip install -e ".[gensim]"(经私有源 pyp.open3dv.site + blender 源),不是脚本本身:

  • 唯一不安装 gensim 的 lint job 通过;两个安装 gensim 的 job 同时失败,且 pytest 运行在安装之后
  • 本 PR 只改 scripts/ 下的 benchmark/tutorial 脚本,不涉及 docs 源、公共 API 或依赖清单。
  • 本地在 py3.11 + dexsim 0.5.0 环境验证了直接依赖本 PR 改动的全部测试:tests/sim/atomic_actions/test_tutorial_utils.py 70 passed;tests/benchmark/motion_generation/{test_atomic_pose_config,test_motion_generation_benchmark}.py 62 passed / 1 skipped。

已重跑失败 job。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant