Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
11f5f0c
Align the doc, test, and client with the newest .env setup (image seg…
MuziWong Aug 10, 2026
4c68adf
Add image generation client + update .env + update doc + test files
MuziWong Aug 11, 2026
88de161
Reformat the code, add scene edit basic framework, notice that the do…
MuziWong Aug 11, 2026
139850b
Basic design of scene graph
MuziWong Aug 12, 2026
ecaacda
i
MuziWong Aug 12, 2026
51e1c64
Fixed the prompt of scne understanding: delete the location in descri…
MuziWong Aug 12, 2026
64ca9f1
modify the prompt of scene understanding, to avoid a very long descri…
MuziWong Aug 12, 2026
4236f9d
finish the llm understand scene edit
MuziWong Aug 12, 2026
e52db87
finish the scene edit plan -> updated scene graph
MuziWong Aug 12, 2026
95f4759
move client try catch outside the understand_scene
MuziWong Aug 12, 2026
a72f101
finish part of the scene edit: from user prompt to simreadyed asset, …
MuziWong Aug 13, 2026
a2398eb
finish basic simready real world scale + semantic-based rotation
MuziWong Aug 13, 2026
ccab0f3
Moved the table support infos into simready.
MuziWong Aug 14, 2026
9b27be2
Finished scene edit
MuziWong Aug 14, 2026
78e01a3
fix some bug before pr
MuziWong Aug 14, 2026
3cd9147
fix a real-world size bug, using z-up mesh
MuziWong Aug 14, 2026
69431c2
update image-to-scene scene understanding: let vlm give orientation s…
MuziWong Aug 15, 2026
083d614
fix big-font problem in asset_masks_with_ids; add scene graph based c…
MuziWong Aug 15, 2026
c5e9087
run black
MuziWong Aug 15, 2026
4779f01
run black to tests/
MuziWong Aug 15, 2026
4a792f0
replace hard-code bottle-z-up with VLM auto-analyze
MuziWong Aug 18, 2026
d3f4365
make assets group layout optimizer more rubust
MuziWong Aug 18, 2026
1d5e367
delete the hard-coded spatial check in assets' names and descriptionsa
MuziWong Aug 18, 2026
9eb4ca1
replace jiange's pca heuristic method with VLM operate rotation tools…
MuziWong Aug 18, 2026
ccb4a94
ignore the fixed 2d aabb overlap when doing the on-table optimization
MuziWong Aug 19, 2026
031b5f8
finished the table as root layout optimization (but still needs to be…
MuziWong Aug 19, 2026
ae4095e
finish a simple on-relationship optimizer
MuziWong Aug 19, 2026
73d3c7f
add the heuristic attaches in collision optimization
MuziWong Aug 19, 2026
a03cda1
replace the assets gravity settler with gravity settler
MuziWong Aug 19, 2026
6f8e26e
use gravity settler in parent surface layout
MuziWong Aug 20, 2026
a84be90
Updated docs
MuziWong Aug 20, 2026
654165c
fix(scene-engine): validate edit categories and preserve batched plan…
MuziWong Aug 20, 2026
e92e978
Outside-table-2d-region checker now does not check those who did not …
MuziWong Aug 20, 2026
ad23063
fix(scene-engine): delete the function export
MuziWong Aug 24, 2026
301d7eb
finished scene-graph based on realtion addtion in image as input pipe…
MuziWong Aug 24, 2026
86ef235
Delete the debug code for coarse layout from server output
MuziWong Aug 25, 2026
0865b90
support lying object VLM-based simready
MuziWong Aug 25, 2026
3f8cba3
add: 1. articulated or not 2. segmented RGBA image for each asset
MuziWong Aug 25, 2026
a9afa66
1. VLM judge lying or standing 2. VLM give a yaw 3. updated render to…
MuziWong Aug 27, 2026
3aeede4
add dongge's client, but failed
MuziWong Aug 27, 2026
acd4387
VLM-auto gives a yaw, but still have a small bug in coarse layout: so…
MuziWong Aug 28, 2026
342f151
Transform each asset into the SimReady table frame to preserve its co…
MuziWong Aug 28, 2026
d4dd211
finished articulated replace: need to be tested
MuziWong Aug 28, 2026
6e5f2bd
Merge remote-tracking branch 'origin/main' into muzi/update_image_input
MuziWong Aug 28, 2026
6217e79
fix the preview problem after added usdc format articulated object
MuziWong Aug 28, 2026
6c15627
fix bug in scene engine preview, delete the wrong-import
MuziWong Aug 28, 2026
245d716
add articulation affordance | add turn knob
Aug 12, 2026
3ec392c
update
Aug 14, 2026
647b406
update
Aug 17, 2026
9d3c9cc
update press button
Aug 17, 2026
8c5d6e7
fix test
Aug 17, 2026
deb94c4
add pull push articulation part
Aug 18, 2026
91d7137
update
Aug 18, 2026
c0962b6
fix module import
Aug 18, 2026
6926307
rename
Aug 19, 2026
1b51dd0
Add unified policy evaluation for RL checkpoints (#510)
acrlw Aug 19, 2026
a4636fa
Revert "Add unified policy evaluation for RL checkpoints (#510)"
acrlw Aug 19, 2026
2953d0b
stash
Aug 19, 2026
ccfb7a7
fix
Aug 19, 2026
34178ca
add pour action
Aug 20, 2026
bd3c0de
fix docs
Aug 24, 2026
adfe065
feat(scene-engine): add versioned authoring API
skywhite1024 Aug 24, 2026
838b167
feat(atomic-actions): preserve GenSim execution contracts
skywhite1024 Aug 24, 2026
168ed64
feat(gen-sim): migrate action engine onto axis baseline
skywhite1024 Aug 24, 2026
9f28ef0
feat(gen-sim): migrate task orchestration onto scene baseline
skywhite1024 Aug 24, 2026
33f570d
feat(action-engine): add body-grasp planning and verified cleanup for…
skywhite1024 Aug 25, 2026
e8a724a
feat(action-engine): add body-grasp planning and verified cleanup for…
skywhite1024 Aug 25, 2026
063cd5e
feat(gen-sim): add failure continuation and task video archival
skywhite1024 Aug 25, 2026
0219d5a
feat(action-engine): implement single-arm E3 pour workflow
skywhite1024 Aug 25, 2026
ebdb66a
feat(action-engine): add contract-driven payload-aware coordinated ma…
skywhite1024 Aug 25, 2026
a371e85
feat(gen-sim): add configurable PGI and Robotiq gripper profiles
skywhite1024 Aug 25, 2026
62ab860
fix(gen-sim): preserve source videos and reduce recording frequency
skywhite1024 Aug 26, 2026
f611c0f
fix(action-engine): align E2 upright orientation to the local z-axis
skywhite1024 Aug 26, 2026
fa26c34
feat(gen-sim): add configurable planner modes and runtime diagnostics
skywhite1024 Aug 26, 2026
8caf7f7
feat(action-engine): extend E4 handover with hold and placement outcomes
skywhite1024 Aug 27, 2026
ad283d5
fix(action-engine): harden E2 and E5 post-action cleanup safety
skywhite1024 Aug 27, 2026
d8c5d2c
refactor(action-engine): use stable entity IDs for object semantics
skywhite1024 Aug 27, 2026
7def96b
refactor(action-engine): use stable entity IDs for object semantics
skywhite1024 Aug 27, 2026
7304a92
feat(gen-sim): add IK solver profiles and E5 grasp diagnostics
skywhite1024 Aug 27, 2026
5f5dbce
fix(gen-sim): harden coordinated pickment safety
skywhite1024 Aug 27, 2026
1921e9e
temp
skywhite1024 Aug 28, 2026
8899abc
refactor(action-engine): implement E2 upright placement with pickup a…
skywhite1024 Aug 31, 2026
e4468a8
Merge remote-tracking branch 'origin/muzi/update_image_input' into lj…
skywhite1024 Aug 31, 2026
e8c86b3
feat(gen-sim): support articulated objects in scene generation and ex…
skywhite1024 Aug 31, 2026
2beaf92
temp
skywhite1024 Sep 1, 2026
8eeb6e8
fix(graspkit): switch convex collision decomposition to V-HACD
skywhite1024 Sep 1, 2026
9125f34
feat(task-engine): add --open-window simulator execution option
skywhite1024 Sep 1, 2026
36ad975
Merge branch 'main' into refactor/gen-sim-task-program-phase1
yuecideng Sep 1, 2026
9edbd8d
Merge remote-tracking branch 'origin/main' into sync/action-engine-re…
yuecideng Sep 3, 2026
503026b
Merge remote-tracking branch 'origin/main' into ljd/action_engine_ref…
yuecideng Sep 3, 2026
35c3351
refactor(task-engine): execute GenSim semantic graphs via Task Progra…
yuecideng Sep 8, 2026
cdd3be4
Merge origin/main into codex/sync-action-engine-main-20260908
skywhite1024 Sep 8, 2026
058cd7e
feat(task-engine): add --open-window simulator execution option
skywhite1024 Sep 8, 2026
c414b8d
fix(sim): support DexSim 0.4.3 without DLSSConfig
skywhite1024 Sep 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions agent_context/topics/simulation-system/simulation-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,12 @@ affect FastRT/OfflineRT windows; hybrid and offscreen cameras derive internal
size from their own output and quality. DexSim initializes DLSS lazily on a
rendered frame, so config tests do not qualify GPU/NGX support.

On legacy engines without `dexsim.DLSSConfig` (including 0.4.3), render
configuration validates the DLSS settings but warns and skips their native
conversion. Renderer selection, sampling, OptiX denoising and tone mapping
still apply. This fallback does not enable DLSS on the legacy engine or change
the direct `DLSSCfg.to_dexsim_cfg()` API's native-support requirement.

`gym/utils/gym_utils.py:config_to_cfg()` decodes task `render_cfg.dlss`
mappings into `DLSSCfg` before constructing `RenderCfg`. DLSS switches require
booleans; ratio/exposure settings require real numbers, excluding booleans.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
embodichain.gen_sim.task_engine
================================

The Task Engine turns instructions and generated-scene evidence into immutable
semantic task graphs. Its execution workflow delegates Semantic Calls to the
canonical Task Program runtime; it does not ground atomic goals or issue robot
commands itself.

Public facade
-------------

.. automodule:: embodichain.gen_sim.task_engine
:members:
:imported-members:
:no-index:

Interpretation and planning contracts
-------------------------------------

.. automodule:: embodichain.gen_sim.task_engine.agent
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.cli
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.config
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.contracts
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.interpretation
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.ontology
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.state_machine
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.workflow_contracts
:members:
:no-index:

Scene orchestration
-------------------

These modules preserve canonical scene identity while separating generated
authoring data, conservative planning evidence, and live simulator bindings.

.. automodule:: embodichain.gen_sim.task_engine.orchestration
:members:
:imported-members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.artifacts
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.contracts
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.coordinator
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.legacy_scene
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.scene_adapter
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.orchestration.scene_source
:members:
:no-index:

Scene analysis and inspection
-----------------------------

.. automodule:: embodichain.gen_sim.task_engine.scene
:members:
:imported-members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene.conservative_graph
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene.contracts
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene.feasibility
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene.final_inspection
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene.scene_engine_v1
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.scene_backend
:members:
:no-index:

Execution workflow
------------------

The workflow owns run directories, bounded orchestration attempts, and
tensor-free reports. Physical execution remains behind the configured Task
Program subprocess boundary.

.. automodule:: embodichain.gen_sim.task_engine.run_directory
:members:
:no-index:

.. automodule:: embodichain.gen_sim.task_engine.workflow
:members:
:no-index:
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ embodichain.lab.task_program.compiler

CompiledArticulationJointPositionValidator
CompiledBarrier
CompiledObjectNearRelativeTargetValidator
CompiledObjectNearTargetValidator
CompiledParallelBlock
CompiledParallelBranch
Expand All @@ -26,4 +27,3 @@ embodichain.lab.task_program.compiler
CompiledTaskProgramValidator
TaskProgramCompileError
TaskProgramCompiler

Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ embodichain.lab.task_program
BarrierCfg
WaitStablePostCfg
ObjectNearTargetValidatorCfg
ObjectNearRelativeTargetValidatorCfg
ArticulationJointPositionValidatorCfg
TaskProgramCompiler
CompiledTaskProgram
Expand Down Expand Up @@ -98,6 +99,9 @@ Schema
.. autoclass:: ObjectNearTargetValidatorCfg
:members:

.. autoclass:: ObjectNearRelativeTargetValidatorCfg
:members:

.. autoclass:: ArticulationJointPositionValidatorCfg
:members:

Expand Down
1 change: 1 addition & 0 deletions docs/source/api_reference/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,5 @@ documentation. CI runs this same checker after style checks and before tests.
.. toctree::
:maxdepth: 1

embodichain/embodichain.gen_sim.task_engine
public_api
182 changes: 182 additions & 0 deletions docs/source/api_reference/public_api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,125 @@ embodichain.gen_sim.gradio_ui.gradio_app

main

embodichain.gen_sim.task_engine.orchestration.grounding
--------------------------------------------------------

Task-conditioned grounding selects canonical scene UIDs from a redacted
semantic inventory. It does not construct physical goals or action
invocations.

.. currentmodule:: embodichain.gen_sim.task_engine.orchestration.grounding

.. autosummary::

GroundingCaller
GroundingResult
ground_scene_references

embodichain.gen_sim.task_engine.orchestration.scene_assets
-----------------------------------------------------------

Generated GLB assets are normalized into renderer-safe runtime geometry while
the source files and their provenance remain unchanged.

.. currentmodule:: embodichain.gen_sim.task_engine.orchestration.scene_assets

.. autosummary::

normalize_scene_assets

embodichain.gen_sim.task_engine.orchestration.scene_inventory
--------------------------------------------------------------

The structural inventory preserves source semantics and validates only
explicit task/scene compatibility. It deliberately avoids language matching
and physical goal grounding.

.. currentmodule:: embodichain.gen_sim.task_engine.orchestration.scene_inventory

.. autosummary::

SceneEntity
SceneInventory
validate_source_compatibility
validate_target_compatibility

embodichain.gen_sim.task_engine.orchestration.source_scene
-----------------------------------------------------------

Source-scene utilities resolve supported exports and normalize their stable
identities, paths, transforms, and conservative physics metadata for Task
Engine planning and simulator loading.

.. currentmodule:: embodichain.gen_sim.task_engine.orchestration.source_scene

.. autosummary::

PreparedScene
ResolvedSceneSource
is_prompt2scene_export
prepare_scene
resolve_gym_config_path
resolve_source_scene

embodichain.gen_sim.task_engine.reporting
------------------------------------------

Tensor-free Task Engine reports retain the canonical Task Program runtime
result and its per-environment semantic outcomes without reconstructing
physical execution state.

.. currentmodule:: embodichain.gen_sim.task_engine.reporting

.. autosummary::

EXECUTION_REPORT_FILENAME
TASK_PROGRAM_EXECUTION_REPORT_SCHEMA
TaskProgramExecutionReport
validate_execution_report
write_execution_report

embodichain.gen_sim.task_engine.semantic_graph
-----------------------------------------------

Semantic task graphs are immutable, JSON-safe planning artifacts whose nodes
contain canonical Semantic Calls and exclude grounded action data.

.. currentmodule:: embodichain.gen_sim.task_engine.semantic_graph

.. autosummary::

SEMANTIC_TASK_GRAPH_SCHEMA
SemanticTaskGraph
semantic_task_graph_hash
validate_semantic_task_graph

embodichain.gen_sim.task_engine.semantic_planner
-------------------------------------------------

The semantic planner lowers validated Task Engine candidates and canonical
scene bindings into provider-free semantic task graphs.

.. currentmodule:: embodichain.gen_sim.task_engine.semantic_planner

.. autosummary::

SemanticTaskPlanner
UnsupportedSemanticCapabilityError

embodichain.gen_sim.task_engine.task_program_bundle
----------------------------------------------------

Bundle generation materializes and preflights a fingerprint-bound configured
Task Program deployment from a semantic task graph and prepared scene.

.. currentmodule:: embodichain.gen_sim.task_engine.task_program_bundle

.. autosummary::

TaskProgramBundlePaths
generate_task_program_bundle

embodichain.gen_sim.scene_engine.core.scene_edit_plan
-----------------------------------------------------

Expand All @@ -238,6 +357,58 @@ embodichain.gen_sim.scene_engine.core.scene_edit_plan
SceneEditOperation
SceneEditPlan

embodichain.gen_sim.scene_engine.errors
---------------------------------------

.. currentmodule:: embodichain.gen_sim.scene_engine.errors

Scene service failures preserve typed preparation and materialization errors
across the Task Engine boundary.

.. autosummary::

SceneServiceError

embodichain.gen_sim.scene_engine.pipeline
-----------------------------------------

.. currentmodule:: embodichain.gen_sim.scene_engine.pipeline

The public authoring boundary separates deterministic scene analysis from
side-effecting materialization for generated and edited scenes.

.. autosummary::

SCENE_BLUEPRINT_SCHEMA
SCENE_EDIT_BLUEPRINT_SCHEMA
SceneBlueprintPackage
SceneEditBlueprintPackage
SceneMaterialization
analyze_edit
analyze_image
materialize_blueprint
materialize_edit

embodichain.gen_sim.scene_engine.pipeline.api
---------------------------------------------

.. currentmodule:: embodichain.gen_sim.scene_engine.pipeline.api

Versioned blueprint artifacts and analyze/materialize operations provide the
implementation-level Scene Engine authoring contract.

.. autosummary::

SCENE_BLUEPRINT_SCHEMA
SCENE_EDIT_BLUEPRINT_SCHEMA
SceneBlueprintPackage
SceneEditBlueprintPackage
SceneMaterialization
analyze_edit
analyze_image
materialize_blueprint
materialize_edit

embodichain.gen_sim.scene_engine.pipeline.editing.scene_edit_asset_preparation
-------------------------------------------------------------------------------

Expand Down Expand Up @@ -567,6 +738,7 @@ embodichain.lab.sim.atomic_actions.control
GRASP_COMMAND
JointPositionCommand
OPEN_COMMAND
PARK_COMMAND

embodichain.lab.sim.atomic_actions.core
---------------------------------------
Expand Down Expand Up @@ -1248,6 +1420,16 @@ embodichain.lab.sim.motion.solvers.pink_solver
PinkSolver
PinkSolverCfg

embodichain.lab.sim.motion.solvers.pytorch_solver
-------------------------------------------------

.. currentmodule:: embodichain.lab.sim.motion.solvers.pytorch_solver

.. autosummary::

PytorchSolverCfg
PytorchSolver

embodichain.lab.sim.motion.solvers.srs_solver
---------------------------------------------

Expand Down
Loading
Loading