[Proposal] Define joint and EEF action representations for expert and LeRobot datasets

[yuecideng]

Proposal

Define an explicit action-representation contract for expert datasets, including LeRobot exports: when should a dataset store joint targets, end-effector (EEF) pose targets, or both?

Keep the existing joint-target representation as the default for joint-controlled demonstrations. Evaluate opt-in Cartesian action representations for policies deployed through an EEF controller, while distinguishing requested Cartesian targets from measured EEF poses and executed joint commands.

This issue records a design proposal, not a request to make EEF pose a mandatory action field or immediately introduce a new trajectory abstraction.

Motivation

Related: #595 introduces timed-trajectory velocity targets and explicit expert position/velocity action schemas.

A demonstration may pass through several representations:

Cartesian goal -> IK / motion planning -> timed joint targets -> controller -> observed motion

These values serve different purposes. A Cartesian goal supplied once per segment is not automatically a per-control-step action, and an EEF pose computed from observed joints is not the commanded target. Mixing them produces ambiguous training labels and replay behavior.

We need to support:

• Joint-space policies trained on qpos or qpos + qvel targets.
• Cartesian policies trained on absolute EEF targets or pose increments, with an explicit deployment controller.
• Auxiliary EEF observations and target labels for analysis, supervision, and trajectory visualization.
• Future cross-embodiment experiments, without assuming Cartesian actions alone guarantee transfer across robots.

Additional context

Proposed direction

1. Match the training action to the policy's deployment interface. Joint-controlled datasets should retain joint targets by default. Cartesian actions should be opt-in and require a defined Cartesian controller/IK contract. Record both representations where useful, but explicitly identify which is the training action and which is an auxiliary target or executed command.
2. Separate observations from commands. Store measured/FK-derived EEF poses as observations. Store requested EEF targets separately when available. Do not silently infer a commanded EEF target from the observed pose.
3. Use an explicit schema. Evaluate how to express primary actions and auxiliary numeric fields in the supported LeRobot version, including compatibility with existing loaders and training pipelines. Avoid silently concatenating joint and Cartesian targets into one ambiguous action vector.
4. Preserve temporal alignment. Define the relationship between observation at step t, command at step t, and the subsequent transition. Specify how segment-level goals and resampled targets align with control-step records.
5. Keep dataset ownership separate from planning results. Assess whether existing PlanResult fields and dataset-boundary adapters are sufficient before adding or expanding trajectory container types. A planner's EEF path is not automatically a dataset action sequence.

Schema questions to resolve

• Action mode: joint position, joint position + velocity, absolute EEF pose, or EEF pose delta.
• Command reference frame: world, robot base, or local tool frame; include frame and TCP identities.
• Delta semantics: translation frame, rotation composition order, and whether increments are relative to the measured or previous target pose.
• Pose encoding, quaternion ordering, units, and normalization conventions.
• Control period, interpolation/hold behavior, and sample timestamps.
• Gripper command representation and multi-arm endpoint ordering.
• Controller/IK configuration and version, including relevant limits and mappings needed to interpret or replay the commands.
• Joint names, ordering, and schema version for backward-compatible loading.

Acceptance criteria

• Document when EEF pose belongs in the primary action, an observation, or an auxiliary target field.
• Define schemas for the chosen joint and Cartesian representations, including frame, rotation, gripper, and timing semantics.
• Verify that the proposed fields are compatible with the supported LeRobot export and training/loading interfaces.
• Provide a small joint-controlled and Cartesian-controlled example showing observation/action alignment and the deployment controller boundary.
• Add focused encoding/loading tests if implemented, including multi-arm ordering and a frame or delta-semantics mismatch case.
• Preserve the existing joint-action dataset default and document any migration required for new representations.

Checklist

• I have checked that there is no similar issue in the repo (required)