fix(ci): downgrade contents permission to read in claude.yml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.github/CLAUDE.md

@@ -0,0 +1,86 @@
+# LeRobot — Claude Code Instructions
+
+You are a senior robotics ML engineer reviewing code for **LeRobot**, a PyTorch framework for real-world robot learning.
+Apply these principles to every PR review, fix, or task.
+
+---
+
+## Core Abstractions
+
+These are the load-bearing types. Handle them with care — breaking changes here affect every user.
+
+| Type             | Location                     | Role                                                         |
+| ---------------- | ---------------------------- | ------------------------------------------------------------ |
+| `LeRobotDataset` | `src/lerobot/datasets/`      | Streaming replay buffer; HF Hub integration                  |
+| `Policy`         | `src/lerobot/policies/`      | Base class for all learning agents (ACT, Diffusion, SARM, …) |
+| `Robot`          | `src/lerobot/robots/`        | Hardware abstraction; carries `_output_pipeline`             |
+| `Teleoperator`   | `src/lerobot/teleoperators/` | Leader-side hardware abstraction; carries `_output_pipeline` |
+| `Env`            | `src/lerobot/envs/`          | Gym-like robotics environments                               |
+| `Processor`      | `src/lerobot/processor/`     | Data transformation pipelines attached to robots/teleops     |
+
+**Never break their public APIs without a migration note and explicit user approval.**
+
+---
+
+## Engineering Principles
+
+### Code quality
+
+- Explicit over magic — no hidden control flow, no implicit state.
+- No deep inheritance trees. Prefer composition.
+- No decorative comment separators (`===`, `---`, etc.).
+- Add comments only where the logic is non-obvious.
+- No over-engineering. YAGNI applies strictly.
+
+### Type safety
+
+- All new and modified Python code must be fully typed (PEP 484).
+- `mypy --strict` must pass on changed files.
+- Do not widen or weaken existing type signatures.
+
+### Backwards compatibility
+
+- Public API changes require migration notes.
+- Additive changes are preferred over modifications.
+- `so100_follower` / `so101_follower` are aliases — never bleed changes there unintentionally.
+
+### HF ecosystem
+
+- Use `push_to_hub()`, HF Hub dataset streaming, and `evaluate` scripts.
+- Dataset changes must preserve streaming compatibility.
+- Prefer reusing HF primitives over rolling custom solutions.
+
+---
+
+## PR Review Checklist
+
+Before approving or marking P1 issues resolved, verify:
+
+- [ ] `pre-commit run -a` would pass (ruff, mypy, typos, zizmor, bandit)
+- [ ] All new/modified code is typed and passes `mypy --strict`
+- [ ] New features have unit tests; no silent behavioral changes
+- [ ] Public APIs of `LeRobotDataset`, `Policy`, `Robot`, `Teleoperator`, `Env` are unchanged (or migration note present)
+- [ ] HF Hub streaming still works for dataset changes
+- [ ] No unnecessary abstractions introduced
+- [ ] No breaking changes to training scripts (`lerobot-train`, `lerobot-eval`, `lerobot-record`)
+
+---
+
+## ML-Specific Checks
+
+Flag these as **P1** if found:
+
+- **Data leakage**: train and val/test splits must be constructed before any normalization or augmentation that uses train statistics.
+- **Loss function errors**: verify reduction mode (`mean` vs `sum`), correct masking, correct shape alignment.
+- **Gradient flow**: new modules must have gradients flowing (check `requires_grad`, no detached tensors in the loss path by accident).
+- **Distributed training**: operations on tensors must be DDP-safe; no in-place ops on parameters; batch norm needs `SyncBatchNorm` if used.
+- **Memory leaks**: no accumulation of tensors outside the training loop; `optimizer.zero_grad()` called correctly.
+
+---
+
+## What to Skip
+
+- Don't flag style nitpicks on unchanged surrounding code.
+- Don't propose refactors outside the PR's scope.
+- Don't add docstrings or comments to code the PR didn't touch.
+- Don't suggest speculative future features (YAGNI).

.github/workflows/claude-code-review.yml

@@ -0,0 +1,49 @@
+name: Claude Code Review
+
+on:
+  pull_request:
+    types: [opened, synchronize, ready_for_review, reopened]
+
+jobs:
+  claude-review:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: read
+      id-token: write
+      actions: read
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: Run Claude Code Review
+        id: claude-review
+        uses: anthropics/claude-code-action@26ddc358fe3befff50c5ec2f80304c90c763f6f8 # v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          use_sticky_comment: true
+          prompt: |
+            Read `.github/CLAUDE.md` for lerobot-specific conventions, then review this PR.
+            Provide structured, actionable feedback.
+
+            Focus areas (in priority order):
+            1. **Correctness**: Logic errors, off-by-ones, wrong tensor shapes, incorrect loss functions
+            2. **Type safety**: All new/modified Python code must pass `mypy --strict`; check for missing annotations
+            3. **Backwards compatibility**: Does this break `LeRobotDataset`, `Policy`, `Robot`, `Teleoperator`, `Env`, or `Processor` public APIs?
+            4. **Tests**: New features must have tests; no silent behavioral changes
+            5. **Code style**: Explicit over magic, no unnecessary abstractions, no decorative comments
+            6. **HF integration**: Dataset streaming, `push_to_hub`, HF Hub compatibility preserved?
+            7. **pre-commit**: Would `pre-commit run -a` pass? (ruff, mypy, typos, zizmor)
+
+            Format findings as P1 (must fix) / P2 (should fix) / P3 (nice to have).
+            Skip P3 if the PR is already high quality.
+          claude_args: '--model claude-opus-4-6'
+          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
+          # or https://code.claude.com/docs/en/cli-reference for available options

.github/workflows/claude.yml

@@ -0,0 +1,58 @@
+name: Claude Code
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  issues:
+    types: [opened, assigned]
+  pull_request_review:
+    types: [submitted]
+
+jobs:
+  claude:
+    if: |
+      (github.event_name == 'issue_comment' &&
+       contains(github.event.comment.body, '@claude') &&
+       (github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'COLLABORATOR')) ||
+      (github.event_name == 'pull_request_review_comment' &&
+       contains(github.event.comment.body, '@claude') &&
+       (github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'COLLABORATOR')) ||
+      (github.event_name == 'pull_request_review' &&
+       contains(github.event.review.body, '@claude') &&
+       (github.event.review.author_association == 'OWNER' || github.event.review.author_association == 'MEMBER' || github.event.review.author_association == 'COLLABORATOR')) ||
+      (github.event_name == 'issues' &&
+       (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
+       (github.event.issue.author_association == 'OWNER' || github.event.issue.author_association == 'MEMBER' || github.event.issue.author_association == 'COLLABORATOR'))
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+      id-token: write
+      actions: read
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: Run Claude Code
+        id: claude
+        uses: anthropics/claude-code-action@26ddc358fe3befff50c5ec2f80304c90c763f6f8 # v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          use_sticky_comment: true
+
+          # This is an optional setting that allows Claude to read CI results on PRs
+          additional_permissions: |
+            actions: read
+
+          claude_args: '--system-prompt "Read .github/CLAUDE.md for lerobot-specific conventions before responding."'
+          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
+          # or https://code.claude.com/docs/en/cli-reference for available options

.github/workflows/docker_publish.yml

@@ -12,8 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# This workflow handles nightly testing & docker images publishing.
-name: Nightly
+# This workflow handles Docker image publishing & testing.
+name: Docker Publish & Test
 permissions:
   contents: read
 
@@ -39,8 +39,8 @@ concurrency:
 
 jobs:
   # This job builds a CPU image for testing & distribution
-  build-docker-cpu-nightly:
-    name: Build CPU Docker for Nightly
+  build-docker-cpu:
+    name: Build CPU Docker
     runs-on:
       group: aws-general-8-plus
     if: github.repository == 'huggingface/lerobot'
@@ -74,8 +74,8 @@ jobs:
           tags: ${{ env.DOCKER_IMAGE_NAME_CPU }}
 
   # This job builds a GPU image for testing & distribution
-  build-docker-gpu-nightly:
-    name: Build GPU Docker for Nightly
+  build-docker-gpu:
+    name: Build GPU Docker
     runs-on:
       group: aws-general-8-plus
     if: github.repository == 'huggingface/lerobot'
@@ -109,9 +109,9 @@ jobs:
           tags: ${{ env.DOCKER_IMAGE_NAME_GPU }}
 
   # This job runs the E2E tests + pytest with all extras in the CPU image
-  nightly-cpu-tests:
-    name: Nightly CPU Tests
-    needs: [build-docker-cpu-nightly]
+  cpu-tests:
+    name: CPU Tests
+    needs: [build-docker-cpu]
     runs-on:
       group: aws-g6-4xlarge-plus
     env:
@@ -121,7 +121,7 @@ jobs:
       TRITON_CACHE_DIR: /home/user_lerobot/.cache/triton
       HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
     container:
-      image: ${{ needs.build-docker-cpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-cpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
       options: --shm-size "16gb"
       credentials:
         username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
@@ -142,9 +142,9 @@ jobs:
         run: make test-end-to-end
 
   # This job runs the E2E tests + pytest with all extras in the GPU image
-  nightly-gpu-tests:
-    name: Nightly GPU Tests
-    needs: [build-docker-gpu-nightly]
+  gpu-tests:
+    name: GPU Tests
+    needs: [build-docker-gpu]
     runs-on:
       group: aws-g6-4xlarge-plus
     env:
@@ -154,7 +154,7 @@ jobs:
       TRITON_CACHE_DIR: /home/user_lerobot/.cache/triton
       HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
     container:
-      image: ${{ needs.build-docker-gpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-gpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
       options: --gpus all --shm-size "16gb"
       credentials:
         username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
@@ -175,9 +175,9 @@ jobs:
         run: make test-end-to-end
 
   # This job runs multi-GPU training tests with 4 GPUs
-  nightly-multi-gpu-tests:
-    name: Nightly Multi-GPU Tests
-    needs: [build-docker-gpu-nightly]
+  multi-gpu-tests:
+    name: Multi-GPU Tests
+    needs: [build-docker-gpu]
     runs-on:
       group: aws-g4dn-12xlarge  # Instance with 4 GPUs
     env:
@@ -188,7 +188,7 @@ jobs:
       CUDA_VISIBLE_DEVICES: "0,1,2,3"
       HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
     container:
-      image: ${{ needs.build-docker-gpu-nightly.outputs.image_tag }} # zizmor: ignore[unpinned-images]
+      image: ${{ needs.build-docker-gpu.outputs.image_tag }} # zizmor: ignore[unpinned-images]
       options: --gpus all --shm-size "16gb"
       credentials:
         username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}

.github/workflows/documentation-upload-pr.yml

@@ -33,7 +33,7 @@ jobs:
       github.event.workflow_run.event == 'pull_request' &&
       github.event.workflow_run.conclusion == 'success' &&
       github.repository == 'huggingface/lerobot'
-    uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@main
+    uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
     with:
       package_name: lerobot
     secrets:

.github/workflows/documentation.yml

@@ -55,7 +55,7 @@ jobs:
       github.repository == 'huggingface/lerobot'
     permissions:
       contents: read
-    uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main
+    uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
     with:
       commit_sha: ${{ github.sha }}
       package: lerobot
@@ -78,7 +78,7 @@ jobs:
     permissions:
       contents: read
       pull-requests: write
-    uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main
+    uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@90b4ee2c10b81b5c1a6367c4e6fc9e2fb510a7e3  # main
     with:
       commit_sha: ${{ github.event.pull_request.head.sha }}
       pr_number: ${{ github.event.number }}

.github/workflows/fast_tests.yml

@@ -27,6 +27,7 @@ on:
       - "tests/**"
       - ".github/workflows/**"
       - "pyproject.toml"
+      - "uv.lock"
       - "Makefile"
   push:
     branches:
@@ -36,6 +37,7 @@ on:
       - "tests/**"
       - ".github/workflows/**"
       - "pyproject.toml"
+      - "uv.lock"
       - "Makefile"
 
 permissions:
@@ -63,7 +65,7 @@ jobs:
       HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
       HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
           lfs: true
@@ -81,14 +83,14 @@ jobs:
           libusb-1.0-0-dev speech-dispatcher libgeos-dev portaudio19-dev
 
       - name: Setup uv and Python
-        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
         with:
           enable-cache: true
           version: ${{ env.UV_VERSION }}
           python-version: ${{ env.PYTHON_VERSION }}
 
       - name: Install lerobot with test extras
-        run: uv sync --extra "test"
+        run: uv sync --locked --extra "test"
 
       - name: Login to Hugging Face
         if: env.HF_USER_TOKEN != ''

.github/workflows/full_tests.yml

@@ -29,6 +29,7 @@ on:
       - "tests/**"
       - ".github/workflows/**"
       - "pyproject.toml"
+      - "uv.lock"
       - "Makefile"
 
 permissions:
@@ -62,7 +63,7 @@ jobs:
       HF_LEROBOT_HOME: /mnt/cache/.cache/huggingface/lerobot
       HF_USER_TOKEN: ${{ secrets.LEROBOT_HF_USER }}
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           lfs: true
           persist-credentials: false
@@ -79,14 +80,14 @@ jobs:
           speech-dispatcher libgeos-dev portaudio19-dev
 
       - name: Setup uv and Python
-        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
         with:
           enable-cache: true
           version: ${{ env.UV_VERSION }}
           python-version: ${{ env.PYTHON_VERSION }}
 
       - name: Install lerobot with all extras
-        run: uv sync --extra all # TODO(Steven): Make flash-attn optional
+        run: uv sync --locked --extra all # TODO(Steven): Make flash-attn optional
 
       - name: Login to Hugging Face
         if: env.HF_USER_TOKEN != ''
@@ -136,21 +137,21 @@ jobs:
           sudo apt-get update
           sudo apt-get install git-lfs
           git lfs install
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           lfs: true
           persist-credentials: false
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3
         with:
           cache-binary: false
       - name: Login to Docker Hub
-        uses: docker/login-action@v3 # zizmor: ignore[unpinned-uses]
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9  # v3
         with:
           username: ${{ secrets.DOCKERHUB_LEROBOT_USERNAME }}
           password: ${{ secrets.DOCKERHUB_LEROBOT_PASSWORD }}
       - name: Build and push Docker image
-        uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6
         with:
           context: .
           file: ./docker/Dockerfile.internal

.github/workflows/latest_deps_tests.yml

@@ -12,38 +12,81 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# This workflow handles full testing with unboud dependencies versions.
-name: Unbound Dependency Tests
+# This workflow tests the project against the latest upstream dependencies
+# (within pyproject.toml constraints) and opens a PR to update uv.lock
+# if the tests pass and the lockfile has changed.
+name: Latest Dependency Tests
 
 on:
   # Allows running this workflow manually from the Actions tab
   workflow_dispatch:
 
-  # Run on the 1st and 15th of every month at 09:00 UTC
-  # schedule:
-  #  - cron: '0 2 1,15 * *'
-
-permissions:
-  contents: read
+  # Runs at 03:00 UTC
+  schedule:
+    - cron: "0 3 * * *"
 
 # Sets up the environment variables
 env:
   UV_VERSION: "0.8.0"
   PYTHON_VERSION: "3.12"
-  DOCKER_IMAGE_NAME: huggingface/lerobot-gpu:unbound
+  DOCKER_IMAGE_NAME: huggingface/lerobot-gpu:latest-deps
 
-# Ensures that only the latest action is built, canceling older runs.
+# Ensures that only the latest run is active, canceling older runs.
 concurrency:
-  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  group: ${{ github.workflow }}
   cancel-in-progress: true
 
 jobs:
 
-  # This job runs the E2E tests + pytest with all unbound extras
-  full-tests:
-    name: Full Unbound Tests
+  # This job upgrades the lockfile and checks if dependencies have changed
+  upgrade-lock:
+    name: Upgrade Lockfile
     runs-on: ubuntu-latest
     if: github.repository == 'huggingface/lerobot'
+    permissions:
+      contents: read
+    outputs:
+      changed: ${{ steps.diff.outputs.changed }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+
+      - name: Setup uv and Python
+        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
+        with:
+          version: ${{ env.UV_VERSION }}
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Upgrade uv.lock
+        run: uv lock --upgrade
+
+      - name: Check for changes
+        id: diff
+        run: |
+          if git diff --quiet uv.lock; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+            echo "uv.lock is up to date — no dependency changes."
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+            echo "uv.lock has changed — running tests."
+          fi
+
+      - name: Upload updated lockfile
+        if: steps.diff.outputs.changed == 'true'
+        uses: actions/upload-artifact@v4 # zizmor: ignore[unpinned-uses]
+        with:
+          name: uv-lock
+          path: uv.lock
+
+  # This job runs the full test suite with the upgraded dependencies
+  cpu-tests:
+    name: CPU Tests (Latest Deps)
+    needs: [upgrade-lock]
+    if: needs.upgrade-lock.outputs.changed == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
     env:
       MUJOCO_GL: egl
       HF_HOME: /mnt/cache/.cache/huggingface
@@ -55,6 +98,11 @@ jobs:
           lfs: true
           persist-credentials: false
 
+      - name: Download updated lockfile
+        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
+        with:
+          name: uv-lock
+
       # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
       # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
@@ -73,34 +121,32 @@ jobs:
           version: ${{ env.UV_VERSION }}
           python-version: ${{ env.PYTHON_VERSION }}
 
-      - name: Unbound dependencies
-        run: |
-          sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml
-          echo "Dependencies unbound:" && cat pyproject.toml
-
       - name: Install lerobot with all extras
-        run: uv sync --extra all # TODO(Steven): Make flash-attn optional
+        run: uv sync --locked --extra all # TODO(Steven): Make flash-attn optional
+
       - name: Login to Hugging Face
         if: env.HF_USER_TOKEN != ''
         run: |
           uv run hf auth login --token "$HF_USER_TOKEN" --add-to-git-credential
           uv run hf auth whoami
+
       - name: Run pytest (all extras)
-        run: uv run pytest tests -vv
+        run: uv run pytest tests -vv --maxfail=10
 
       - name: Run end-to-end tests
         run: uv run make test-end-to-end
 
-  # This job builds a GPU enabled image for testing
+  # This job builds a GPU-enabled Docker image with the upgraded dependencies
   build-and-push-docker:
     name: Build and Push Docker
+    needs: [upgrade-lock]
+    if: needs.upgrade-lock.outputs.changed == 'true'
+    permissions:
+      contents: read
     runs-on:
       group: aws-general-8-plus
-    if: github.repository == 'huggingface/lerobot'
     outputs:
       image_tag: ${{ env.DOCKER_IMAGE_NAME }}
-    env:
-      GITHUB_REF: ${{ github.ref }}
     steps:
       - name: Install Git LFS
         run: |
@@ -111,6 +157,12 @@ jobs:
         with:
           lfs: true
           persist-credentials: false
+
+      - name: Download updated lockfile
+        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
+        with:
+          name: uv-lock
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3 # zizmor: ignore[unpinned-uses]
         with:
@@ -127,14 +179,13 @@ jobs:
           file: ./docker/Dockerfile.internal
           push: true
           tags: ${{ env.DOCKER_IMAGE_NAME }}
-          build-args: |
-            UNBOUND_DEPS=true
 
-  # This job runs pytest with all unbound extras in a GPU enabled host
-  # It runs everytime a test image is created
+  # This job runs pytest with all extras on a GPU-enabled host
   gpu-tests:
-    name: GPU Unbound Tests
+    name: GPU Tests (Latest Deps)
     needs: [build-and-push-docker]
+    permissions:
+      contents: read
     runs-on:
       group: aws-g6-4xlarge-plus
     env:
@@ -159,17 +210,69 @@ jobs:
         run: |
           hf auth login --token "$HF_USER_TOKEN" --add-to-git-credential
           hf auth whoami
+      - name: Fix ptxas permissions
+        run: chmod +x /lerobot/.venv/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas
       - name: Run pytest on GPU
-        run: pytest tests -vv
+        run: pytest tests -vv --maxfail=10
       - name: Run end-to-end tests
         run: make test-end-to-end
 
-  # This job deletes the test image recently created
-  # It runs everytime after the gpu-tests have finished
-  delete-unbound-image:
-    name: Delete Unbound Image
+  # This job creates or updates a PR with the upgraded lockfile
+  open-pr:
+    name: Open PR
+    needs: [cpu-tests, gpu-tests, upgrade-lock]
+    if: success() && needs.upgrade-lock.outputs.changed == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+    env:
+      GH_TOKEN: ${{ secrets.UPDATE_LOCK_TOKEN }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+
+      - name: Download updated lockfile
+        uses: actions/download-artifact@v4 # zizmor: ignore[unpinned-uses]
+        with:
+          name: uv-lock
+
+      - name: Create or update PR
+        run: |
+          set -euo pipefail
+          BRANCH="auto/update-uv-lock"
+
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git"
+
+          git checkout -B "$BRANCH"
+          git add uv.lock
+          git commit -m "chore(dependencies): update uv.lock"
+          git push --force origin "$BRANCH"
+
+          # Create PR only if one doesn't already exist for this branch
+          EXISTING_PR=$(gh pr list --head "$BRANCH" --state open --json number --jq '.[0].number')
+          if [ -z "$EXISTING_PR" ]; then
+            gh pr create \
+              --title "chore(dependencies): update uv.lock" \
+              --body "Automated update of \`uv.lock\` after successful latest dependency tests (CPU + GPU).
+
+          This PR upgrades all dependencies to their latest versions within the ranges specified in \`pyproject.toml\`." \
+              --head "$BRANCH" \
+              --base main
+          else
+            echo "PR #$EXISTING_PR already exists, branch has been updated."
+          fi
+
+  # This job deletes the temporary Docker image after tests complete
+  cleanup-docker:
+    name: Cleanup Docker Image
     needs: [gpu-tests, build-and-push-docker]
     if: always() && needs.build-and-push-docker.result == 'success'
+    permissions:
+      contents: read
     runs-on: ubuntu-latest
     steps:
       - name: Get Docker Hub Token and Delete Image
@@ -180,8 +283,7 @@ jobs:
           IMAGE_FULL: ${{ needs.build-and-push-docker.outputs.image_tag }}
         run: |
           IMAGE_NAME=$(echo "$IMAGE_FULL" | cut -d':' -f1)
-          IMAGE_TAG=$(echo "$IMAGE_FULL" | cut -d':' -f2)
-
+          IMAGE_TAG=$(echo "$IMAGE_FULL" | cut -d':' -f2-)
           echo "Attempting to delete image: $IMAGE_NAME:$IMAGE_TAG"
 
           TOKEN=$(curl -s -H "Content-Type: application/json" \

.github/workflows/quality.yml

@@ -43,16 +43,16 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
 
       - name: Set up Python
-        uses: actions/setup-python@v6
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6
         with:
           python-version: '3.12'
 
       - name: Run pre-commit hooks
-        uses: pre-commit/action@v3.0.1 # zizmor: ignore[unpinned-uses]
+        uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd  # v3.0.1
         with:
           extra_args: --all-files --show-diff-on-failure --color=always

.github/workflows/release.yml

@@ -38,12 +38,12 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
 
       - name: Set up Python
-        uses: actions/setup-python@v6
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6
         with:
           python-version: '3.12'
 
@@ -104,7 +104,7 @@ jobs:
       - name: Publish to TestPyPI for pre-releases
         # True for tags like 'v0.2.0-rc1'
         if: startsWith(github.ref, 'refs/tags/v') && contains(github.ref, '-')
-        uses: pypa/gh-action-pypi-publish@v1.13.0 # zizmor: ignore[unpinned-uses, use-trusted-publishing]
+        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e  # v1.13.0
         with:
           repository-url: https://test.pypi.org/legacy/
           verbose: true
@@ -112,7 +112,7 @@ jobs:
 
       - name: Publish to PyPI
         if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-')
-        uses: pypa/gh-action-pypi-publish@v1.13.0 # zizmor: ignore[unpinned-uses, use-trusted-publishing]
+        uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e  # v1.13.0
         with:
           verbose: true
           print-hash: true
@@ -127,7 +127,7 @@ jobs:
     env:
       MUJOCO_GL: egl
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           lfs: true
           persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
           git curl libglib2.0-0 libegl1-mesa-dev ffmpeg libusb-1.0-0-dev \
           speech-dispatcher libgeos-dev portaudio19-dev
       - name: Setup uv and Python
-        uses: astral-sh/setup-uv@v6 # zizmor: ignore[unpinned-uses]
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e  # v6
         with:
           enable-cache: true # zizmor: ignore[cache-poisoning]
           version: ${{ env.UV_VERSION }}

.github/workflows/security.yml

@@ -43,12 +43,12 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@v6 # zizmor: ignore[unpinned-uses]
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           fetch-depth: 0
           persist-credentials: false
 
       - name: Secret Scanning
-        uses: trufflesecurity/trufflehog@v3.90.0  # zizmor: ignore[unpinned-uses]
+        uses: trufflesecurity/trufflehog@eafb8c5f6a06175141c27f17bcc17941853d0047  # v3.90.0
         with:
           extra_args: --only-verified

.gitignore

@@ -25,7 +25,6 @@ node_modules/
 
 # Lock files
 poetry.lock
-uv.lock
 Pipfile.lock
 
 ### Build & Distribution ###
@@ -173,5 +172,7 @@ outputs/
 
 # Dev folders
 .cache/*
+*.stl
+*.urdf
 *.xml
 *.part

README.md

@@ -4,7 +4,8 @@
 
 <div align="center">
 
-[![Tests](https://github.com/huggingface/lerobot/actions/workflows/nightly.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/nightly.yml?query=branch%3Amain)
+[![Tests](https://github.com/huggingface/lerobot/actions/workflows/latest_deps_tests.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/latest_deps_tests.yml?query=branch%3Amain)
+[![Tests](https://github.com/huggingface/lerobot/actions/workflows/docker_publish.yml/badge.svg?branch=main)](https://github.com/huggingface/lerobot/actions/workflows/docker_publish.yml?query=branch%3Amain)
 [![Python versions](https://img.shields.io/pypi/pyversions/lerobot)](https://www.python.org/downloads/)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/huggingface/lerobot/blob/main/LICENSE)
 [![Status](https://img.shields.io/pypi/status/lerobot)](https://pypi.org/project/lerobot/)

docker/Dockerfile.internal

@@ -73,17 +73,10 @@ ENV HOME=/home/user_lerobot \
 RUN uv venv --python python${PYTHON_VERSION}
 
 # Install Python dependencies for caching
-COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml README.md MANIFEST.in ./
+COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml uv.lock README.md MANIFEST.in ./
 COPY --chown=user_lerobot:user_lerobot src/ src/
 
-ARG UNBOUND_DEPS=false
-
-RUN if [ "$UNBOUND_DEPS" = "true" ]; then \
-    sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml; \
-    echo "Dependencies unbound:" && cat pyproject.toml; \
-    fi
-
-RUN uv pip install --no-cache ".[all]"
+RUN uv sync --locked --extra all --no-cache
 
 RUN chmod +x /lerobot/.venv/lib/python${PYTHON_VERSION}/site-packages/triton/backends/nvidia/bin/ptxas
 

docker/Dockerfile.user

@@ -61,17 +61,10 @@ ENV HOME=/home/user_lerobot \
 RUN uv venv
 
 # Install Python dependencies for caching
-COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml README.md MANIFEST.in ./
+COPY --chown=user_lerobot:user_lerobot setup.py pyproject.toml uv.lock README.md MANIFEST.in ./
 COPY --chown=user_lerobot:user_lerobot src/ src/
 
-ARG UNBOUND_DEPS=false
-
-RUN if [ "$UNBOUND_DEPS" = "true" ]; then \
-    sed -i 's/,[[:space:]]*<[0-9\.]*//g' pyproject.toml; \
-    echo "Dependencies unbound:" && cat pyproject.toml; \
-    fi
-
-RUN uv pip install --no-cache ".[all]"
+RUN uv sync --locked --extra all --no-cache
 
 # Copy the rest of the application code
 # Make sure to have the git-LFS files for testing

docker/README.md

@@ -0,0 +1,77 @@
+# Docker
+
+This directory contains Dockerfiles for running LeRobot in containerized environments. Both images are **built nightly from `main`** and published to Docker Hub with the full environment pre-baked — no dependency setup required.
+
+## Pre-built Images
+
+```bash
+# CPU-only image (based on Dockerfile.user)
+docker pull huggingface/lerobot-cpu:latest
+
+# GPU image with CUDA support (based on Dockerfile.internal)
+docker pull huggingface/lerobot-gpu:latest
+```
+
+## Quick Start
+
+The fastest way to start training is to pull the GPU image and run `lerobot-train` directly. This is the same environment used for all of our CI, so it is a well-tested, batteries-included setup.
+
+```bash
+docker run -it --rm --gpus all --shm-size 16gb huggingface/lerobot-gpu:latest
+
+# inside the container:
+lerobot-train --policy.type=act --dataset.repo_id=lerobot/aloha_sim_transfer_cube_human
+```
+
+## Dockerfiles
+
+### `Dockerfile.user` (CPU)
+
+A lightweight image based on `python:3.12-slim`. Includes all Python dependencies and system libraries but does not include CUDA — there is no GPU support. Useful for exploring the codebase, running scripts, or working with robots, but not practical for training.
+
+### `Dockerfile.internal` (GPU)
+
+A CUDA-enabled image based on `nvidia/cuda`. This is the image for training — mostly used for internal interactions with the GPU cluster.
+
+## Usage
+
+### Running a pre-built image
+
+```bash
+# CPU
+docker run -it --rm huggingface/lerobot-cpu:latest
+
+# GPU
+docker run -it --rm --gpus all --shm-size 16gb huggingface/lerobot-gpu:latest
+```
+
+### Building locally
+
+From the repo root:
+
+```bash
+# CPU
+docker build -f docker/Dockerfile.user -t lerobot-user .
+docker run -it --rm lerobot-user
+
+# GPU
+docker build -f docker/Dockerfile.internal -t lerobot-internal .
+docker run -it --rm --gpus all --shm-size 16gb lerobot-internal
+```
+
+### Multi-GPU training
+
+To select specific GPUs, set `CUDA_VISIBLE_DEVICES` when launching the container:
+
+```bash
+# Use 4 GPUs
+docker run -it --rm --gpus all --shm-size 16gb \
+  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
+  huggingface/lerobot-gpu:latest
+```
+
+### USB device access (e.g. robots, cameras)
+
+```bash
+docker run -it --device=/dev/ -v /dev/:/dev/ --rm huggingface/lerobot-cpu:latest
+```

docs/source/_toctree.yml

@@ -17,12 +17,12 @@
     title: Train RL in Simulation
   - local: multi_gpu_training
     title: Multi GPU training
+  - local: hil_data_collection
+    title: Human In the Loop Data Collection
   - local: peft_training
     title: Training with PEFT (e.g., LoRA)
   - local: rename_map
     title: Using Rename Map and Empty Cameras
-  - local: umi_pi0_relative_ee
-    title: UMI Data with pi0 Relative EE Actions
   title: "Tutorials"
 - sections:
   - local: lerobot-dataset-v3
@@ -71,13 +71,17 @@
     title: Environments from the Hub
   - local: envhub_leisaac
     title: Control & Train Robots in Sim (LeIsaac)
+  title: "Simulation"
+- sections:
+  - local: adding_benchmarks
+    title: Adding a New Benchmark
+  - local: libero
+    title: LIBERO
+  - local: metaworld
+    title: Meta-World
   - local: envhub_isaaclab_arena
     title: NVIDIA IsaacLab Arena Environments
-  - local: libero
-    title: Using Libero
-  - local: metaworld
-    title: Using MetaWorld
-  title: "Simulation"
+  title: "Benchmarks"
 - sections:
   - local: introduction_processors
     title: Introduction to Robot Processors

docs/source/action_representations.mdx

@@ -202,22 +202,11 @@ Here is how the different processors compose. Each arrow is a processor step, an
                     └─────────────────────────────────────────┘
 
                     ┌─────────────────────────────────────────┐
-   State Derivation │   Action column  ────→  State + Action  │
-                    │   DeriveStateFromActionStep (pre only)  │
-                    │   (UMI-style: state from action chunk)  │
-                    └─────────────────────────────────────────┘
-
-                    ┌─────────────────────────────────────────┐
-   Action Repr.     │   Absolute  ←────→  Relative            │
+   Representation   │   Absolute  ←────→  Relative            │
                     │   RelativeActionsProcessorStep (pre)    │
                     │   AbsoluteActionsProcessorStep (post)   │
                     └─────────────────────────────────────────┘
 
-                    ┌─────────────────────────────────────────┐
-   State Repr.      │   Absolute  ────→  Relative             │
-                    │   RelativeStateProcessorStep (pre only) │
-                    └─────────────────────────────────────────┘
-
                     ┌─────────────────────────────────────────┐
    Normalization    │   Raw  ←────→  Normalized               │
                     │   NormalizerProcessorStep (pre)         │
@@ -227,10 +216,6 @@ Here is how the different processors compose. Each arrow is a processor step, an
 
 A typical training preprocessor might chain: `raw absolute joint actions → relative → normalize`. A typical inference postprocessor: `unnormalize → absolute → (optionally IK to joints)`.
 
-With UMI-style relative proprioception (`use_relative_state=True`), the preprocessor also converts observation.state to offsets from the current timestep via `RelativeStateProcessorStep` before normalization. This is a pre-processing-only step (state is an input, not an output).
-
-With `derive_state_from_action=True`, the preprocessor first runs `DeriveStateFromActionStep` to extract a 2-step state from the extended action chunk. This enables full UMI-style training without a separate `observation.state` column. See the [UMI pi0 guide](umi_pi0_relative_ee) for details.
-
 ## References
 
 - [Universal Manipulation Interface (UMI)](https://arxiv.org/abs/2402.10329) - Chi et al., 2024. Defines the relative trajectory action representation and compares it with absolute and delta actions.

docs/source/adding_benchmarks.mdx

@@ -0,0 +1,320 @@
+# Adding a New Benchmark
+
+This guide walks you through adding a new simulation benchmark to LeRobot. Follow the steps in order and use the existing benchmarks as templates.
+
+A benchmark in LeRobot is a set of [Gymnasium](https://gymnasium.farama.org/) environments that wrap a third-party simulator (like LIBERO or Meta-World) behind a standard `gym.Env` interface. The `lerobot-eval` CLI then runs evaluation uniformly across all benchmarks.
+
+## Existing benchmarks at a glance
+
+Before diving in, here is what is already integrated:
+
+| Benchmark      | Env file            | Config class       | Tasks               | Action dim   | Processor                    |
+| -------------- | ------------------- | ------------------ | ------------------- | ------------ | ---------------------------- |
+| LIBERO         | `envs/libero.py`    | `LiberoEnv`        | 130 across 5 suites | 7            | `LiberoProcessorStep`        |
+| Meta-World     | `envs/metaworld.py` | `MetaworldEnv`     | 50 (MT50)           | 4            | None                         |
+| IsaacLab Arena | Hub-hosted          | `IsaaclabArenaEnv` | Configurable        | Configurable | `IsaaclabArenaProcessorStep` |
+
+Use `src/lerobot/envs/libero.py` and `src/lerobot/envs/metaworld.py` as reference implementations.
+
+## How it all fits together
+
+### Data flow
+
+During evaluation, data moves through four stages:
+
+```
+1. gym.Env  ──→  raw observations (numpy dicts)
+
+2. Preprocessing  ──→  standard LeRobot keys + task description
+   (preprocess_observation, add_envs_task in envs/utils.py)
+
+3. Processors  ──→  env-specific then policy-specific transforms
+   (env_preprocessor, policy_preprocessor)
+
+4. Policy  ──→  select_action()  ──→  action tensor
+   then reverse: policy_postprocessor → env_postprocessor → numpy action → env.step()
+```
+
+Most benchmarks only need to care about stage 1 (producing observations in the right format) and optionally stage 3 (if env-specific transforms are needed).
+
+### Environment structure
+
+`make_env()` returns a nested dict of vectorized environments:
+
+```python
+dict[str, dict[int, gym.vector.VectorEnv]]
+#    ^suite       ^task_id
+```
+
+A single-task env (e.g. PushT) looks like `{"pusht": {0: vec_env}}`.
+A multi-task benchmark (e.g. LIBERO) looks like `{"libero_spatial": {0: vec0, 1: vec1, ...}, ...}`.
+
+### How evaluation runs
+
+All benchmarks are evaluated the same way by `lerobot-eval`:
+
+1. `make_env()` builds the nested `{suite: {task_id: VectorEnv}}` dict.
+2. `eval_policy_all()` iterates over every suite and task.
+3. For each task, it runs `n_episodes` rollouts via `rollout()`.
+4. Results are aggregated hierarchically: episode, task, suite, overall.
+5. Metrics include `pc_success` (success rate), `avg_sum_reward`, and `avg_max_reward`.
+
+The critical piece: your env must return `info["is_success"]` on every `step()` call. This is how the eval loop knows whether a task was completed.
+
+## What your environment must provide
+
+LeRobot does not enforce a strict observation schema. Instead it relies on a set of conventions that all benchmarks follow.
+
+### Env attributes
+
+Your `gym.Env` must set these attributes:
+
+| Attribute            | Type  | Why                                                  |
+| -------------------- | ----- | ---------------------------------------------------- |
+| `_max_episode_steps` | `int` | `rollout()` uses this to cap episode length          |
+| `task_description`   | `str` | Passed to VLA policies as a language instruction     |
+| `task`               | `str` | Fallback identifier if `task_description` is not set |
+
+### Success reporting
+
+Your `step()` and `reset()` must include `"is_success"` in the `info` dict:
+
+```python
+info = {"is_success": True}   # or False
+return observation, reward, terminated, truncated, info
+```
+
+### Observations
+
+The simplest approach is to map your simulator's outputs to the standard keys that `preprocess_observation()` already understands. Do this inside your `gym.Env` (e.g. in a `_format_raw_obs()` helper):
+
+| Your env should output    | LeRobot maps it to         | What it is                            |
+| ------------------------- | -------------------------- | ------------------------------------- |
+| `"pixels"` (single array) | `observation.image`        | Single camera image, HWC uint8        |
+| `"pixels"` (dict)         | `observation.images.<cam>` | Multiple cameras, each HWC uint8      |
+| `"agent_pos"`             | `observation.state`        | Proprioceptive state vector           |
+| `"environment_state"`     | `observation.env_state`    | Full environment state (e.g. PushT)   |
+| `"robot_state"`           | `observation.robot_state`  | Nested robot state dict (e.g. LIBERO) |
+
+If your simulator uses different key names, you have two options:
+
+1. **Recommended:** Rename them to the standard keys inside your `gym.Env` wrapper.
+2. **Alternative:** Write an env processor to transform observations after `preprocess_observation()` runs (see step 4 below).
+
+### Actions
+
+Actions are continuous numpy arrays in a `gym.spaces.Box`. The dimensionality depends on your benchmark (7 for LIBERO, 4 for Meta-World, etc.). Policies adapt to different action dimensions through their `input_features` / `output_features` config.
+
+### Feature declaration
+
+Each `EnvConfig` subclass declares two dicts that tell the policy what to expect:
+
+- `features` — maps feature names to `PolicyFeature(type, shape)` (e.g. action dim, image shape).
+- `features_map` — maps raw observation keys to LeRobot convention keys (e.g. `"agent_pos"` to `"observation.state"`).
+
+## Step by step
+
+<Tip>
+  At minimum, you need two files: a **gym.Env wrapper** and an **EnvConfig
+  subclass** with a `create_envs()` override. Everything else is optional or
+  documentation. No changes to `factory.py` are needed.
+</Tip>
+
+### Checklist
+
+| File                                     | Required | Why                                                          |
+| ---------------------------------------- | -------- | ------------------------------------------------------------ |
+| `src/lerobot/envs/<benchmark>.py`        | Yes      | Wraps the simulator as a standard gym.Env                    |
+| `src/lerobot/envs/configs.py`            | Yes      | Registers your benchmark and its `create_envs()` for the CLI |
+| `src/lerobot/processor/env_processor.py` | Optional | Custom observation/action transforms                         |
+| `src/lerobot/envs/utils.py`              | Optional | Only if you need new raw observation keys                    |
+| `pyproject.toml`                         | Yes      | Declares benchmark-specific dependencies                     |
+| `docs/source/<benchmark>.mdx`            | Yes      | User-facing documentation page                               |
+| `docs/source/_toctree.yml`               | Yes      | Adds your page to the docs sidebar                           |
+
+### 1. The gym.Env wrapper (`src/lerobot/envs/<benchmark>.py`)
+
+Create a `gym.Env` subclass that wraps the third-party simulator:
+
+```python
+class MyBenchmarkEnv(gym.Env):
+    metadata = {"render_modes": ["rgb_array"], "render_fps": <fps>}
+
+    def __init__(self, task_suite, task_id, ...):
+        super().__init__()
+        self.task = <task_name_string>
+        self.task_description = <natural_language_instruction>
+        self._max_episode_steps = <max_steps>
+        self.observation_space = spaces.Dict({...})
+        self.action_space = spaces.Box(low=..., high=..., shape=(...,), dtype=np.float32)
+
+    def reset(self, seed=None, **kwargs):
+        ...  # return (observation, info) — info must contain {"is_success": False}
+
+    def step(self, action: np.ndarray):
+        ...  # return (obs, reward, terminated, truncated, info) — info must contain {"is_success": <bool>}
+
+    def render(self):
+        ...  # return RGB image as numpy array
+
+    def close(self):
+        ...
+```
+
+Also provide a factory function that returns the nested dict structure:
+
+```python
+def create_mybenchmark_envs(
+    task: str,
+    n_envs: int,
+    gym_kwargs: dict | None = None,
+    env_cls: type | None = None,
+) -> dict[str, dict[int, Any]]:
+    """Create {suite_name: {task_id: VectorEnv}} for MyBenchmark."""
+    ...
+```
+
+See `create_libero_envs()` (multi-suite, multi-task) and `create_metaworld_envs()` (difficulty-grouped tasks) for reference.
+
+### 2. The config (`src/lerobot/envs/configs.py`)
+
+Register a config dataclass so users can select your benchmark with `--env.type=<name>`. Each config owns its environment creation and processor logic via two methods:
+
+- **`create_envs(n_envs, use_async_envs)`** — Returns `{suite: {task_id: VectorEnv}}`. The base class default uses `gym.make()` for single-task envs. Multi-task benchmarks override this.
+- **`get_env_processors()`** — Returns `(preprocessor, postprocessor)`. The base class default returns identity (no-op) pipelines. Override if your benchmark needs observation/action transforms.
+
+```python
+@EnvConfig.register_subclass("<benchmark_name>")
+@dataclass
+class MyBenchmarkEnvConfig(EnvConfig):
+    task: str = "<default_task>"
+    fps: int = <fps>
+    obs_type: str = "pixels_agent_pos"
+
+    features: dict[str, PolicyFeature] = field(default_factory=lambda: {
+        ACTION: PolicyFeature(type=FeatureType.ACTION, shape=(<action_dim>,)),
+    })
+    features_map: dict[str, str] = field(default_factory=lambda: {
+        ACTION: ACTION,
+        "agent_pos": OBS_STATE,
+        "pixels": OBS_IMAGE,
+    })
+
+    def __post_init__(self):
+        ...  # populate features based on obs_type
+
+    @property
+    def gym_kwargs(self) -> dict:
+        return {"obs_type": self.obs_type, "render_mode": self.render_mode}
+
+    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+        """Override for multi-task benchmarks or custom env creation."""
+        from lerobot.envs.<benchmark> import create_<benchmark>_envs
+        return create_<benchmark>_envs(task=self.task, n_envs=n_envs, ...)
+
+    def get_env_processors(self):
+        """Override if your benchmark needs observation/action transforms."""
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+        from lerobot.processor.env_processor import MyBenchmarkProcessorStep
+        return (
+            PolicyProcessorPipeline(steps=[MyBenchmarkProcessorStep()]),
+            PolicyProcessorPipeline(steps=[]),
+        )
+```
+
+Key points:
+
+- The `register_subclass` name is what users pass on the CLI (`--env.type=<name>`).
+- `features` tells the policy what the environment produces.
+- `features_map` maps raw observation keys to LeRobot convention keys.
+- **No changes to `factory.py` needed** — the factory delegates to `cfg.create_envs()` and `cfg.get_env_processors()` automatically.
+
+### 3. Env processor (optional — `src/lerobot/processor/env_processor.py`)
+
+Only needed if your benchmark requires observation transforms beyond what `preprocess_observation()` handles (e.g. image flipping, coordinate conversion). Define the processor step here and return it from `get_env_processors()` in your config (see step 2):
+
+```python
+@dataclass
+@ProcessorStepRegistry.register(name="<benchmark>_processor")
+class MyBenchmarkProcessorStep(ObservationProcessorStep):
+    def _process_observation(self, observation):
+        processed = observation.copy()
+        # your transforms here
+        return processed
+
+    def transform_features(self, features):
+        return features  # update if shapes change
+
+    def observation(self, observation):
+        return self._process_observation(observation)
+```
+
+See `LiberoProcessorStep` for a full example (image rotation, quaternion-to-axis-angle conversion).
+
+### 4. Dependencies (`pyproject.toml`)
+
+Add a new optional-dependency group:
+
+```toml
+mybenchmark = ["my-benchmark-pkg==1.2.3", "lerobot[scipy-dep]"]
+```
+
+Pinning rules:
+
+- **Always pin** benchmark packages to exact versions for reproducibility (e.g. `metaworld==3.0.0`).
+- **Add platform markers** when needed (e.g. `; sys_platform == 'linux'`).
+- **Pin fragile transitive deps** if known (e.g. `gymnasium==1.1.0` for Meta-World).
+- **Document constraints** in your benchmark doc page.
+
+Users install with:
+
+```bash
+pip install -e ".[mybenchmark]"
+```
+
+### 5. Documentation (`docs/source/<benchmark>.mdx`)
+
+Write a user-facing page following the template in the next section. See `docs/source/libero.mdx` and `docs/source/metaworld.mdx` for full examples.
+
+### 6. Table of contents (`docs/source/_toctree.yml`)
+
+Add your benchmark to the "Benchmarks" section:
+
+```yaml
+- sections:
+    - local: libero
+      title: LIBERO
+    - local: metaworld
+      title: Meta-World
+    - local: envhub_isaaclab_arena
+      title: NVIDIA IsaacLab Arena Environments
+    - local: <your_benchmark>
+      title: <Your Benchmark Name>
+  title: "Benchmarks"
+```
+
+## Verifying your integration
+
+After completing the steps above, confirm that everything works:
+
+1. **Install** — `pip install -e ".[mybenchmark]"` and verify the dependency group installs cleanly.
+2. **Smoke test env creation** — call `make_env()` with your config in Python, check that the returned dict has the expected `{suite: {task_id: VectorEnv}}` shape, and that `reset()` returns observations with the right keys.
+3. **Run a full eval** — `lerobot-eval --env.type=<name> --env.task=<task> --eval.n_episodes=1 --eval.batch_size=1 --policy.path=<any_compatible_policy>` to exercise the full pipeline end-to-end.
+4. **Check success detection** — verify that `info["is_success"]` flips to `True` when the task is actually completed. This is what the eval loop uses to compute success rates.
+
+## Writing a benchmark doc page
+
+Each benchmark `.mdx` page should include:
+
+- **Title and description** — 1-2 paragraphs on what the benchmark tests and why it matters.
+- **Links** — paper, GitHub repo, project website (if available).
+- **Overview image or GIF.**
+- **Available tasks** — table of task suites with counts and brief descriptions.
+- **Installation** — `pip install -e ".[<benchmark>]"` plus any extra steps (env vars, system packages).
+- **Evaluation** — recommended `lerobot-eval` command with `n_episodes` and `batch_size` for reproducible results. Include single-task and multi-task examples if applicable.
+- **Policy inputs and outputs** — observation keys with shapes, action space description.
+- **Recommended evaluation episodes** — how many episodes per task is standard.
+- **Training** — example `lerobot-train` command.
+- **Reproducing published results** — link to pretrained model, eval command, results table (if available).
+
+See `docs/source/libero.mdx` and `docs/source/metaworld.mdx` for complete examples.

docs/source/env_processor.mdx

@@ -90,11 +90,17 @@ The same policy can work with different environment processors, and the same env
 
 ```python
 # Use SmolVLA policy with LIBERO environment
-libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(libero_cfg)
+# Use SmolVLA policy with LIBERO environment
+libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(
+    env_cfg=libero_cfg,
+    policy_cfg=smolvla_cfg,
+)
 smolvla_preprocessor, smolvla_postprocessor = make_pre_post_processors(smolvla_cfg)
-
 # Or use ACT policy with the same LIBERO environment
-libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(libero_cfg)
+libero_preprocessor, libero_postprocessor = make_env_pre_post_processors(
+    env_cfg=libero_cfg,
+    policy_cfg=act_cfg,
+)
 act_preprocessor, act_postprocessor = make_pre_post_processors(act_cfg)
 ```
 
@@ -151,7 +157,7 @@ observation = {
 
 ### Factory Function
 
-The `make_env_pre_post_processors` function follows the same pattern as `make_pre_post_processors` for policies:
+The `make_env_pre_post_processors` function delegates to `env_cfg.get_env_processors()`:
 
 ```python
 from lerobot.envs.factory import make_env_pre_post_processors
@@ -159,47 +165,31 @@ from lerobot.envs.configs import LiberoEnv, PushtEnv
 
 # For LIBERO: Returns LiberoProcessorStep in preprocessor
 libero_cfg = LiberoEnv(task="libero_spatial", camera_name=["agentview"])
-env_preprocessor, env_postprocessor = make_env_pre_post_processors(libero_cfg)
+env_preprocessor, env_postprocessor = make_env_pre_post_processors(libero_cfg, policy_cfg)
 
 # For other environments: Returns identity processors (no-op)
 pusht_cfg = PushtEnv()
-env_preprocessor, env_postprocessor = make_env_pre_post_processors(pusht_cfg)
+env_preprocessor, env_postprocessor = make_env_pre_post_processors(pusht_cfg, policy_cfg)
 ```
 
-### Implementation in `envs/factory.py`
+### How It Works
+
+Each `EnvConfig` subclass can override `get_env_processors()` to return benchmark-specific
+processor pipelines. The base class returns identity (no-op) processors by default.
 
 ```python
-def make_env_pre_post_processors(
-    env_cfg: EnvConfig,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-]:
-    """
-    Create preprocessor and postprocessor pipelines for environment observations.
-
-    Args:
-        env_cfg: The configuration of the environment.
-
-    Returns:
-        A tuple containing:
-            - preprocessor: Pipeline that processes environment observations
-            - postprocessor: Pipeline that processes environment outputs
-    """
-    # For LIBERO environments, add the LiberoProcessorStep to preprocessor
-    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
-        preprocessor = PolicyProcessorPipeline(steps=[LiberoProcessorStep()])
-    else:
-        # For all other environments, return an identity preprocessor
-        preprocessor = PolicyProcessorPipeline(steps=[])
-
-    # Postprocessor is currently identity for all environments
-    # Future: Could add environment-specific action transformations
-    postprocessor = PolicyProcessorPipeline(steps=[])
-
-    return preprocessor, postprocessor
+# In your EnvConfig subclass:
+def get_env_processors(self):
+    from lerobot.processor.pipeline import PolicyProcessorPipeline
+    return (
+        PolicyProcessorPipeline(steps=[MyProcessorStep()]),
+        PolicyProcessorPipeline(steps=[]),
+    )
 ```
 
+The factory function `make_env_pre_post_processors` simply delegates to this method,
+with a special case for `XVLAConfig` policies which override the env processors entirely.
+
 ### Integration in Evaluation
 
 In `lerobot_eval.py`, the environment processors are created once and used throughout:
@@ -219,7 +209,10 @@ def eval_main(cfg: EvalPipelineConfig):
     )
 
     # Create environment processors (NEW!)
-    env_preprocessor, env_postprocessor = make_env_pre_post_processors(env_cfg=cfg.env)
+    env_preprocessor, env_postprocessor = make_env_pre_post_processors(
+        env_cfg=cfg.env,
+        policy_cfg=cfg.policy,
+    )
 
     # Run evaluation with both processor types
     eval_policy_all(
@@ -323,21 +316,22 @@ class MyEnvProcessorStep(ObservationProcessorStep):
         return processed
 ```
 
-### 2. Update the Factory
+### 2. Update Your `EnvConfig` Subclass
 
 ```python
-# In src/lerobot/envs/factory.py
+# In src/lerobot/envs/configs.py
+@EnvConfig.register_subclass("myenv")
+@dataclass
+class MyEnvConfig(EnvConfig):
+    # ... task/features/gym kwargs ...
 
-def make_env_pre_post_processors(env_cfg: EnvConfig):
-    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
-        preprocessor = PolicyProcessorPipeline(steps=[LiberoProcessorStep()])
-    elif isinstance(env_cfg, MyEnvConfig) or "myenv" in env_cfg.type:
-        preprocessor = PolicyProcessorPipeline(steps=[MyEnvProcessorStep()])
-    else:
-        preprocessor = PolicyProcessorPipeline(steps=[])
+    def get_env_processors(self):
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
 
-    postprocessor = PolicyProcessorPipeline(steps=[])
-    return preprocessor, postprocessor
+        return (
+            PolicyProcessorPipeline(steps=[MyEnvProcessorStep()]),
+            PolicyProcessorPipeline(steps=[]),
+        )
 ```
 
 ### 3. Use in Evaluation

docs/source/groot.mdx

@@ -131,4 +131,4 @@ lerobot-record \
 
 ## License
 
-This model follows the **Apache 2.0 License**, consistent with the original [GR00T repository](https://github.com/NVIDIA/Isaac-GR00T).
+This model follows NVIDIA's proprietary license, consistent with the original [GR00T repository](https://github.com/NVIDIA/Isaac-GR00T). Future versions (starting from N1.7) will follow **Apache 2.0 License**.

docs/source/hil_data_collection.mdx

@@ -0,0 +1,269 @@
+# Human-In-the-Loop Data Collection
+
+Human-In-the-Loop (HIL) data collection lets you improve a trained policy by deploying it on a real robot while a human operator monitors and intervenes when needed. The intervention data (recovery movements and corrections) is recorded alongside autonomous segments, producing a richer training dataset that teaches the policy how to handle failures.
+
+---
+
+## Why Human-In-the-Loop?
+
+Standard behavioral cloning trains policies on successful demonstrations only. During deployment, small errors can compound and push the robot into states never seen during training (distribution shift). HIL data collection addresses this by:
+
+- Running the trained policy on the real robot
+- Having a human intervene when the robot is about to fail
+- Recording the human's recovery and correction as training data
+- Fine-tuning the policy on the combined dataset
+
+This produces a policy that not only knows how to perform the task, but also how to recover when things go wrong.
+
+---
+
+## How It Works
+
+During a HIL session, the human operator follows this loop within each episode:
+
+1. **Watch** the policy run autonomously
+2. **Pause** when failure is imminent, the robot holds its position
+3. **Take control** and teleoperate the robot back to a good state (recovery), then correct the behavior
+4. **Return control to the policy**, the policy resumes autonomous execution
+5. Repeat steps 2–4 as many times as needed during the episode
+6. **End the episode** when the task is complete, save and move on to the next rollout
+
+Both autonomous and human-controlled segments are recorded. The policy and human can alternate control multiple times within a single episode, and the episode continues from the current state after each handoff (no reset required just because intervention happened). This captures autonomous execution, recovery, and correction in one continuous trajectory. After collection, the combined dataset (original demonstrations + HIL data) is used to fine-tune the policy.
+
+This process can be repeated iteratively: deploy, collect, fine-tune, repeat. Each round targets the current policy's failure modes.
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Policy v0 (trained on demos)                                           │
+│       ↓                                                                 │
+│  HIL Collection (target current failure modes) → Fine-tune → Policy v1  │
+│       ↓                                                                 │
+│  HIL Collection (target new failure modes) → Fine-tune → Policy v2      │
+│       ↓                                                                 │
+│  ... (repeat until satisfactory performance)                            │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Hardware Requirements
+
+### Teleoperator Requirements
+
+The `examples/hil` HIL scripts require **teleoperators with active motors** that can:
+
+- Enable/disable torque programmatically
+- Move to target positions (to mirror the robot state when pausing)
+
+**Compatible teleoperators in the current `examples/hil` scripts:**
+
+- `openarm_mini` - OpenArm Mini
+- `so_leader` - SO100 / SO101 leader arm
+
+> [!IMPORTANT]
+> The provided `examples/hil` commands default to `bi_openarm_follower` + `openarm_mini`.
+> `so_follower` + `so_leader` configs are also registered and can be used via CLI flags.
+
+---
+
+## Script
+
+A single script handles both synchronous and RTC-based inference. Toggle RTC with `--rtc.enabled=true`:
+
+| Mode                     | Flag                 | Models                |
+| ------------------------ | -------------------- | --------------------- |
+| Standard (default)       | _(no flag needed)_   | ACT, Diffusion Policy |
+| Real-Time Chunking (RTC) | `--rtc.enabled=true` | Pi0, Pi0.5, SmolVLA   |
+
+---
+
+## Step-by-Step Guide
+
+### Step 1: Pre-train a Base Policy
+
+First, train a policy on your demonstration dataset:
+
+```bash
+python src/lerobot/scripts/lerobot_train.py \
+    --dataset.repo_id=your-username/demo-dataset \
+    --policy.type=pi0 \
+    --output_dir=outputs/pretrain \
+    --batch_size=32 \
+    --steps=50000
+```
+
+### Step 2: Collect HIL Data
+
+**Standard inference (ACT, Diffusion Policy):**
+
+```bash
+python examples/hil/hil_data_collection.py \
+    --robot.type=bi_openarm_follower \
+    --robot.left_arm_config.port=can1 \
+    --robot.left_arm_config.side=left \
+    --robot.right_arm_config.port=can0 \
+    --robot.right_arm_config.side=right \
+    --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \
+    --teleop.type=openarm_mini \
+    --teleop.port_left=/dev/ttyACM0 \
+    --teleop.port_right=/dev/ttyACM1 \
+    --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --dataset.repo_id=your-username/hil-dataset \
+    --dataset.single_task="Fold the T-shirt properly" \
+    --dataset.fps=30 \
+    --dataset.episode_time_s=1000 \
+    --dataset.num_episodes=50 \
+    --interpolation_multiplier=2
+```
+
+**With RTC for large models (Pi0, Pi0.5, SmolVLA):**
+
+For models with high inference latency, enable RTC for smooth execution:
+
+```bash
+python examples/hil/hil_data_collection.py \
+    --rtc.enabled=true \
+    --rtc.execution_horizon=20 \
+    --rtc.max_guidance_weight=5.0 \
+    --rtc.prefix_attention_schedule=LINEAR \
+    --robot.type=bi_openarm_follower \
+    --robot.left_arm_config.port=can1 \
+    --robot.left_arm_config.side=left \
+    --robot.right_arm_config.port=can0 \
+    --robot.right_arm_config.side=right \
+    --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}}' \
+    --teleop.type=openarm_mini \
+    --teleop.port_left=/dev/ttyACM0 \
+    --teleop.port_right=/dev/ttyACM1 \
+    --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --dataset.repo_id=your-username/hil-rtc-dataset \
+    --dataset.single_task="Fold the T-shirt properly" \
+    --dataset.fps=30 \
+    --dataset.episode_time_s=1000 \
+    --dataset.num_episodes=50 \
+    --interpolation_multiplier=3
+```
+
+**Controls (Conceptual):**
+
+The interaction model is:
+
+- **Pause input**: pause autonomous policy execution
+- **Takeover input**: transfer control to the human operator and record intervention data
+- **Return-to-policy input**: hand control back to the policy and continue the same episode
+- **Episode control inputs**: save/re-record/stop/reset as needed
+
+Exact key/pedal bindings can differ across scripts and hardware integrations. Use each script's printed controls as the source of truth for the concrete mapping on your setup.
+
+**The HIL Protocol:**
+
+1. Watch the policy run autonomously (teleop is idle/free)
+2. When you see imminent failure, trigger the **pause input**
+   - Policy stops
+   - Teleoperator moves to match robot position (torque enabled)
+   - No frames recorded during pause
+3. Trigger the **takeover input** to take control
+   - Teleoperator torque disabled, free to move
+   - **Recovery**: Teleoperate the robot back to a good state
+   - **Correction**: Correct the behavior
+   - All movements are recorded
+4. Trigger the **return-to-policy input**
+   - Policy resumes autonomous execution from the current state
+   - You can intervene again at any time (repeat steps 2–4)
+5. End and save the episode when the task is complete (or episode time limit is reached)
+6. **Reset**: Teleop moves to robot position, you can move the robot to the starting position
+7. Start the next episode
+
+**Foot Pedal Setup (Linux):**
+
+If using a USB foot pedal (PCsensor FootSwitch), ensure access:
+
+```bash
+sudo setfacl -m u:$USER:rw /dev/input/by-id/usb-PCsensor_FootSwitch-event-kbd
+```
+
+### Step 3: Fine-tune the Policy
+
+Fine-tune on the **combined** dataset (`demo-dataset` + `hil-dataset` merged together):
+
+```bash
+python src/lerobot/scripts/lerobot_train.py \
+    --dataset.repo_id=your-username/hil-dataset \
+    --policy.type=pi0 \
+    --policy.pretrained_path=outputs/pretrain/checkpoints/last/pretrained_model \
+    --output_dir=outputs/hil_finetune \
+    --steps=20000
+```
+
+Then deploy the fine-tuned policy and repeat from Step 2 to target its remaining failure modes.
+
+---
+
+## Tips for Effective HIL Collection
+
+### When to Intervene
+
+Intervene when you see:
+
+- Robot about to make an irreversible mistake
+- Robot hesitating or showing uncertain behavior
+- Robot deviating from the expected trajectory
+
+### Recovery: Teleoperating Back to a Good State
+
+During recovery, teleoperate the robot back to a state where:
+
+- The robot is in a familiar, in-distribution configuration
+- The current subtask can still be completed
+- The recovery trajectory itself is informative training data
+
+### Quality of Corrections
+
+During correction:
+
+- Provide **confident, clean** trajectories
+- Complete the current subtask fully
+- Don't overcorrect or add unnecessary movements
+
+---
+
+## Related Work
+
+This HIL data collection approach builds on ideas from interactive imitation learning:
+
+- **DAgger** (Ross et al., 2011) introduced the core idea: instead of only training on expert demonstrations, query the expert for corrections on states the _learner_ visits. This breaks the compounding-error cycle of standard behavioral cloning by iteratively collecting on-policy data.
+
+- **HG-DAgger** (Kelly et al., 2019) made this practical for robotics: a human expert monitors the robot and only intervenes when needed, rather than labeling every state. The gating between autonomous and human control is exactly the pause → takeover → return-to-policy loop used in the scripts here.
+
+- **RaC** (Hu et al., 2025) scales this loop to long-horizon tasks by explicitly decomposing interventions into **recovery** (teleoperating back to a good state) and **correction** (demonstrating the right behavior from there). This decomposition is the protocol followed by the HIL scripts in `examples/hil`.
+
+- **π0.6/RECAP** (Physical Intelligence, 2025) applies the same iterative collect-and-finetune loop at scale with VLA models, showing that even large pretrained policies benefit substantially from targeted human corrections on their own failure modes. π0.6 is trained using RECAP.
+
+```bibtex
+@article{ross2011dagger,
+  title={A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning},
+  author={Ross, Stéphane and Gordon, Geoffrey and Bagnell, Drew},
+  journal={Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics},
+  year={2011}
+}
+
+@article{kelly2019hgdagger,
+  title={HG-DAgger: Interactive Imitation Learning with Human Experts},
+  author={Kelly, Michael and Sidrane, Chelsea and Driggs-Campbell, Katherine and Kochenderfer, Mykel J},
+  journal={arXiv preprint arXiv:1810.02890},
+  year={2019}
+}
+
+@article{hu2025rac,
+  title={RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction},
+  author={Hu, Zheyuan and Wu, Robyn and Enock, Naveen and Li, Jasmine and Kadakia, Riya and Erickson, Zackory and Kumar, Aviral},
+  journal={arXiv preprint arXiv:2509.07953},
+  year={2025}
+}
+
+@article{pi2025recap,
+  title={π0.6: a VLA That Learns From Experience},
+  author={Physical Intelligence},
+  year={2025}
+}
+```

docs/source/installation.mdx

@@ -1,6 +1,6 @@
 # Installation
 
-This guide uses `conda` (via miniforge) to manage environments (recommended). If you prefer another environment manager (e.g. `uv`, `venv`), ensure you have Python >=3.12 and `ffmpeg` installed with the `libsvtav1` encoder, then skip ahead to [Environment Setup](#step-2-environment-setup).
+This guide uses `conda` (via miniforge) to manage environments (recommended). If you prefer another environment manager (e.g. `uv`, `venv`), ensure you have Python >=3.12 and support PyTorch >= 2.10, then skip ahead to [Environment Setup](#step-2-environment-setup).
 
 ## Step 1 (`conda` only): Install [`miniforge`](https://conda-forge.org/download/)
 
@@ -20,7 +20,7 @@ Create a virtual environment with Python 3.12:
 conda create -y -n lerobot python=3.12
 ```
 </hfoption>
-<hfoption id="uv">
+<hfoption id="uv (PyTorch >= 2.10 only)">
 ```bash
 uv python install 3.12
 uv venv --python 3.12
@@ -32,48 +32,87 @@ uv venv --python 3.12
 Then activate your virtual environment, you have to do this each time you open a shell to use lerobot:
 
 <!-- prettier-ignore-start -->
+
 <hfoptions id="activate_venv">
-<hfoption id="conda">```bash
+<hfoption id="conda">
+```bash
 conda activate lerobot
-```</hfoption>
-<hfoption id="uv">
-```bash
-# Linux/macOSsource
-source .venv/bin/activate
-# Windows PowerShell
-source .venv\Scripts\Activate.ps1
 ```
-</hfoption>
-</hfoptions>
-<!-- prettier-ignore-end -->
-
-When using `conda`, install `ffmpeg` in your environment:
-
-```bash
-conda install ffmpeg -c conda-forge
-ffmpeg -version  # ffmpeg 8.X is not yet supported !
-```
-
-> [!TIP]
-> This usually installs `ffmpeg 7.X` for your platform compiled with the `libsvtav1` encoder. If `libsvtav1` is not supported (check supported encoders with `ffmpeg -encoders`), you can:
->
-> - _[On any platform]_ Explicitly install `ffmpeg 7.X` using:
->
-> ```bash
-> conda install ffmpeg=7.1.1 -c conda-forge
-> ```
->
-> - _[On Linux only]_ If you want to bring your own ffmpeg: Install [ffmpeg build dependencies](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu#GettheDependencies) and [compile ffmpeg from source with libsvtav1](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu#libsvtav1), and make sure you use the corresponding ffmpeg binary to your install with `which ffmpeg`.
 
 > [!NOTE]
-> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to install `evdev` with the following command:
+> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to also install `evdev`:
 >
 > ```bash
 > conda install evdev -c conda-forge
 > ```
 
+</hfoption>
+<hfoption id="uv (PyTorch >= 2.10 only)">
+```bash
+# Linux/macOS
+source .venv/bin/activate
+# Windows PowerShell
+.venv\Scripts\activate
+```
+
+> [!NOTE]
+> When installing LeRobot inside WSL (Windows Subsystem for Linux), make sure to also install `evdev`:
+>
+> ```bash
+> sudo apt install libevdev-dev
+> uv pip install evdev
+> ```
+
+</hfoption>
+</hfoptions>
+<!-- prettier-ignore-end -->
+
+### Install `ffmpeg` (for video decoding)
+
+LeRobot uses [TorchCodec](https://github.com/meta-pytorch/torchcodec) for video decoding by default, which requires `ffmpeg`.
+
+> [!NOTE]
+> **Platform support:** TorchCodec is **not available** on macOS Intel (x86_64), Linux ARM (aarch64, arm64, armv7l), or Windows with PyTorch < 2.8. On these platforms, LeRobot automatically falls back to `pyav` — so you do not need to install `ffmpeg` and can skip to Step 3.
+
+If your platform supports TorchCodec, install `ffmpeg` using one of the methods below:
+
+<!-- prettier-ignore-start -->
+
+<hfoptions id="install_ffmpeg">
+<hfoption id="conda (any PyTorch version)">
+
+Install `ffmpeg` in your conda environment. This works with **all PyTorch versions** and is **required for PyTorch < 2.10**:
+
+```bash
+conda install ffmpeg -c conda-forge
+```
+
+> [!TIP]
+> This usually installs `ffmpeg 8.X` with the `libsvtav1` encoder. If you run into issues (e.g. `libsvtav1` missing — check with `ffmpeg -encoders` — or a version mismatch with `torchcodec`), you can explicitly install `ffmpeg 7.1.1` using:
+>
+> ```bash
+> conda install ffmpeg=7.1.1 -c conda-forge
+> ```
+
+</hfoption>
+<hfoption id="uv (PyTorch >= 2.10 only)">
+
+Starting with **PyTorch >= 2.10** (TorchCodec ≥ 0.10), TorchCodec can dynamically link to a system-wide `ffmpeg` installation. This is useful when using `uv` or other non-`conda` environment managers:
+
+```bash
+# Ubuntu/Debian
+sudo apt install ffmpeg
+
+# macOS (Apple Silicon)
+brew install ffmpeg
+```
+
 > [!IMPORTANT]
-> If you are using `uv` you will have to install `ffmpeg` system-wide (outside of the virtual environment). You rely on `uv` and `torchcodec` ability to dynamically link to the system `ffmpeg`.
+> System-wide `ffmpeg` is **only supported with PyTorch >= 2.10** (TorchCodec ≥ 0.10). For older PyTorch versions, you **must** use `conda install ffmpeg -c conda-forge` instead.
+
+</hfoption>
+</hfoptions>
+<!-- prettier-ignore-end -->
 
 ## Step 3: Install LeRobot 🤗
 

docs/source/libero.mdx

@@ -1,36 +1,61 @@
 # LIBERO
 
-**LIBERO** is a benchmark designed to study **lifelong robot learning**. The idea is that robots won’t just be pretrained once in a factory, they’ll need to keep learning and adapting with their human users over time. This ongoing adaptation is called **lifelong learning in decision making (LLDM)**, and it’s a key step toward building robots that become truly personalized helpers.
+LIBERO is a benchmark designed to study **lifelong robot learning** — the idea that robots need to keep learning and adapting with their users over time, not just be pretrained once. It provides a set of standardized manipulation tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other's work.
 
-- 📄 [LIBERO paper](https://arxiv.org/abs/2306.03310)
-- 💻 [Original LIBERO repo](https://github.com/Lifelong-Robot-Learning/LIBERO)
-
-To make progress on this challenge, LIBERO provides a set of standardized tasks that focus on **knowledge transfer**: how well a robot can apply what it has already learned to new situations. By evaluating on LIBERO, different algorithms can be compared fairly and researchers can build on each other’s work.
-
-LIBERO includes **five task suites**:
-
-- **LIBERO-Spatial (`libero_spatial`)** – tasks that require reasoning about spatial relations.
-- **LIBERO-Object (`libero_object`)** – tasks centered on manipulating different objects.
-- **LIBERO-Goal (`libero_goal`)** – goal-conditioned tasks where the robot must adapt to changing targets.
-- **LIBERO-90 (`libero_90`)** – 90 short-horizon tasks from the LIBERO-100 collection.
-- **LIBERO-Long (`libero_10`)** – 10 long-horizon tasks from the LIBERO-100 collection.
-
-Together, these suites cover **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios. LIBERO is meant to grow over time, and to serve as a shared benchmark where the community can test and improve lifelong learning algorithms.
+- Paper: [Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://arxiv.org/abs/2306.03310)
+- GitHub: [Lifelong-Robot-Learning/LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)
+- Project website: [libero-project.github.io](https://libero-project.github.io)
 
 ![An overview of the LIBERO benchmark](https://libero-project.github.io/assets/img/libero/fig1.png)
 
-## Evaluating with LIBERO
+## Available tasks
 
-At **LeRobot**, we ported [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) into our framework and used it mainly to **evaluate [SmolVLA](https://huggingface.co/docs/lerobot/en/smolvla)**, our lightweight Vision-Language-Action model.
+LIBERO includes **five task suites** covering **130 tasks**, ranging from simple object manipulations to complex multi-step scenarios:
 
-LIBERO is now part of our **multi-eval supported simulation**, meaning you can benchmark your policies either on a **single suite of tasks** or across **multiple suites at once** with just a flag.
+| Suite          | CLI name         | Tasks | Description                                        |
+| -------------- | ---------------- | ----- | -------------------------------------------------- |
+| LIBERO-Spatial | `libero_spatial` | 10    | Tasks requiring reasoning about spatial relations  |
+| LIBERO-Object  | `libero_object`  | 10    | Tasks centered on manipulating different objects   |
+| LIBERO-Goal    | `libero_goal`    | 10    | Goal-conditioned tasks with changing targets       |
+| LIBERO-90      | `libero_90`      | 90    | Short-horizon tasks from the LIBERO-100 collection |
+| LIBERO-Long    | `libero_10`      | 10    | Long-horizon tasks from the LIBERO-100 collection  |
 
-To Install LIBERO, after following LeRobot official instructions, just do:
-`pip install -e ".[libero]"`
+## Installation
+
+After following the LeRobot installation instructions:
+
+```bash
+pip install -e ".[libero]"
+```
+
+<Tip>
+LIBERO requires Linux (`sys_platform == 'linux'`). LeRobot uses MuJoCo for simulation — set the rendering backend before training or evaluation:
+
+```bash
+export MUJOCO_GL=egl  # for headless servers (HPC, cloud)
+```
+
+</Tip>
+
+## Evaluation
+
+### Default evaluation (recommended)
+
+Evaluate across the four standard suites (10 episodes per task):
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=libero \
+  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10 \
+  --env.max_parallel_tasks=1
+```
 
 ### Single-suite evaluation
 
-Evaluate a policy on one LIBERO suite:
+Evaluate on one LIBERO suite:
 
 ```bash
 lerobot-eval \
@@ -42,15 +67,13 @@ lerobot-eval \
 ```
 
 - `--env.task` picks the suite (`libero_object`, `libero_spatial`, etc.).
-- `--env.task_ids` picks task ids to run (`[0]`, `[1,2,3]`, etc.). Omit this flag (or set it to `null`) to run all tasks in the suite.
+- `--env.task_ids` restricts to specific task indices (`[0]`, `[1,2,3]`, etc.). Omit to run all tasks in the suite.
 - `--eval.batch_size` controls how many environments run in parallel.
-- `--eval.n_episodes` sets how many episodes to run in total.
-
----
+- `--eval.n_episodes` sets how many episodes to run per task.
 
 ### Multi-suite evaluation
 
-Benchmark a policy across multiple suites at once:
+Benchmark a policy across multiple suites at once by passing a comma-separated list:
 
 ```bash
 lerobot-eval \
@@ -61,50 +84,49 @@ lerobot-eval \
   --eval.n_episodes=2
 ```
 
-- Pass a comma-separated list to `--env.task` for multi-suite evaluation.
+### Control mode
 
-### Control Mode
+LIBERO supports two control modes — `relative` (default) and `absolute`. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:
 
-LIBERO now supports two control modes: relative and absolute. This matters because different VLA checkpoints are trained with different mode of action to output hence control parameterizations.
-You can switch them with: `env.control_mode = "relative"` and `env.control_mode = "absolute"`
+```bash
+--env.control_mode=relative   # or "absolute"
+```
 
 ### Policy inputs and outputs
 
-When using LIBERO through LeRobot, policies interact with the environment via **observations** and **actions**:
+**Observations:**
 
-- **Observations**
-  - `observation.state` – proprioceptive features (agent state).
-  - `observation.images.image` – main camera view (`agentview_image`).
-  - `observation.images.image2` – wrist camera view (`robot0_eye_in_hand_image`).
+- `observation.state` — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
+- `observation.images.image` — main camera view (`agentview_image`), HWC uint8
+- `observation.images.image2` — wrist camera view (`robot0_eye_in_hand_image`), HWC uint8
 
-  ⚠️ **Note:** LeRobot enforces the `.images.*` prefix for any multi-modal visual features. Always ensure that your policy config `input_features` use the same naming keys, and that your dataset metadata keys follow this convention during evaluation.
-  If your data contains different keys, you must rename the observations to match what the policy expects, since naming keys are encoded inside the normalization statistics layer.
-  This will be fixed with the upcoming Pipeline PR.
+<Tip warning={true}>
+  LeRobot enforces the `.images.*` prefix for visual features. Ensure your
+  policy config `input_features` use the same naming keys, and that your dataset
+  metadata keys follow this convention. If your data contains different keys,
+  you must rename the observations to match what the policy expects, since
+  naming keys are encoded inside the normalization statistics layer.
+</Tip>
 
-- **Actions**
-  - Continuous control values in a `Box(-1, 1, shape=(7,))` space.
+**Actions:**
 
-We also provide a notebook for quick testing:
-Training with LIBERO
+- Continuous control in `Box(-1, 1, shape=(7,))` — 6D end-effector delta + 1D gripper
 
-## Training with LIBERO
+### Recommended evaluation episodes
 
-When training on LIBERO tasks, make sure your dataset parquet and metadata keys follow the LeRobot convention.
+For reproducible benchmarking, use **10 episodes per task** across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results.
 
-The environment expects:
+## Training
 
-- `observation.state` → 8-dim agent state
-- `observation.images.image` → main camera (`agentview_image`)
-- `observation.images.image2` → wrist camera (`robot0_eye_in_hand_image`)
+### Dataset
 
-⚠️ Cleaning the dataset upfront is **cleaner and more efficient** than remapping keys inside the code.
-To avoid potential mismatches and key errors, we provide a **preprocessed LIBERO dataset** that is fully compatible with the current LeRobot codebase and requires no additional manipulation:
-👉 [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero)
+We provide a preprocessed LIBERO dataset fully compatible with LeRobot:
 
-For reference, here is the **original dataset** published by Physical Intelligence:
-👉 [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
+- [HuggingFaceVLA/libero](https://huggingface.co/datasets/HuggingFaceVLA/libero)
 
----
+For reference, the original dataset published by Physical Intelligence:
+
+- [physical-intelligence/libero](https://huggingface.co/datasets/physical-intelligence/libero)
 
 ### Example training command
 
@@ -121,52 +143,39 @@ lerobot-train \
   --batch_size=4 \
   --eval.batch_size=1 \
   --eval.n_episodes=1 \
-  --eval_freq=1000 \
+  --eval_freq=1000
 ```
 
----
+## Reproducing published results
 
-### Note on rendering
+We reproduce the results of Pi0.5 on the LIBERO benchmark. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
 
-LeRobot uses MuJoCo for simulation. You need to set the rendering backend before training or evaluation:
+The finetuned model: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
 
-- `export MUJOCO_GL=egl` → for headless servers (e.g. HPC, cloud)
-
-## Reproducing π₀.₅ results
-
-We reproduce the results of π₀.₅ on the LIBERO benchmark using the LeRobot implementation. We take the Physical Intelligence LIBERO base model (`pi05_libero`) and finetune for an additional 6k steps in bfloat16, with batch size of 256 on 8 H100 GPUs using the [HuggingFace LIBERO dataset](https://huggingface.co/datasets/HuggingFaceVLA/libero).
-
-The finetuned model can be found here:
-
-- **π₀.₅ LIBERO**: [lerobot/pi05_libero_finetuned](https://huggingface.co/lerobot/pi05_libero_finetuned)
-
-We then evaluate the finetuned model using the LeRobot LIBERO implementation, by running the following command:
+### Evaluation command
 
 ```bash
 lerobot-eval \
-  --output_dir=/logs/ \
+  --output_dir=./eval_logs/ \
   --env.type=libero \
   --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
   --eval.batch_size=1 \
   --eval.n_episodes=10 \
   --policy.path=pi05_libero_finetuned \
   --policy.n_action_steps=10 \
-  --output_dir=./eval_logs/ \
   --env.max_parallel_tasks=1
 ```
 
-**Note:** We set `n_action_steps=10`, similar to the original OpenPI implementation.
+We set `n_action_steps=10`, matching the original OpenPI implementation.
 
 ### Results
 
-We obtain the following results on the LIBERO benchmark:
+| Model               | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average  |
+| ------------------- | -------------- | ------------- | ----------- | --------- | -------- |
+| **Pi0.5 (LeRobot)** | 97.0           | 99.0          | 98.0        | 96.0      | **97.5** |
 
-| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average  |
-| -------- | -------------- | ------------- | ----------- | --------- | -------- |
-| **π₀.₅** | 97.0           | 99.0          | 98.0        | 96.0      | **97.5** |
+These results are consistent with the [original results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
 
-These results are consistent with the original [results](https://github.com/Physical-Intelligence/openpi/tree/main/examples/libero#results) reported by Physical Intelligence:
-
-| Model    | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average   |
-| -------- | -------------- | ------------- | ----------- | --------- | --------- |
-| **π₀.₅** | 98.8           | 98.2          | 98.0        | 92.4      | **96.85** |
+| Model              | LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average   |
+| ------------------ | -------------- | ------------- | ----------- | --------- | --------- |
+| **Pi0.5 (OpenPI)** | 98.8           | 98.2          | 98.0        | 92.4      | **96.85** |

docs/source/metaworld.mdx

@@ -1,32 +1,111 @@
 # Meta-World
 
-Meta-World is a well-designed, open-source simulation benchmark for multi-task and meta reinforcement learning in continuous-control robotic manipulation. It gives researchers a shared, realistic playground to test whether algorithms can _learn many different tasks_ and _generalize quickly to new ones_ — two central challenges for real-world robotics.
+Meta-World is an open-source simulation benchmark for **multi-task and meta reinforcement learning** in continuous-control robotic manipulation. It bundles 50 diverse manipulation tasks using everyday objects and a common tabletop Sawyer arm, providing a standardized playground to test whether algorithms can learn many different tasks and generalize quickly to new ones.
 
-- 📄 [MetaWorld paper](https://arxiv.org/pdf/1910.10897)
-- 💻 [Original MetaWorld repo](https://github.com/Farama-Foundation/Metaworld)
+- Paper: [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://arxiv.org/abs/1910.10897)
+- GitHub: [Farama-Foundation/Metaworld](https://github.com/Farama-Foundation/Metaworld)
+- Project website: [metaworld.farama.org](https://metaworld.farama.org)
 
 ![MetaWorld MT10 demo](https://meta-world.github.io/figures/ml45.gif)
 
-## Why Meta-World matters
+## Available tasks
 
-- **Diverse, realistic tasks.** Meta-World bundles a large suite of simulated manipulation tasks (50 in the MT50 suite) using everyday objects and a common tabletop Sawyer arm. This diversity exposes algorithms to a wide variety of dynamics, contacts and goal specifications while keeping a consistent control and observation structure.
-- **Focus on generalization and multi-task learning.** By evaluating across task distributions that share structure but differ in goals and objects, Meta-World reveals whether an agent truly learns transferable skills rather than overfitting to a narrow task.
-- **Standardized evaluation protocol.** It provides clear evaluation modes and difficulty splits, so different methods can be compared fairly across easy, medium, hard and very-hard regimes.
-- **Empirical insight.** Past evaluations on Meta-World show impressive progress on some fronts, but also highlight that current multi-task and meta-RL methods still struggle with large, diverse task sets. That gap points to important research directions.
+Meta-World provides 50 tasks organized into difficulty groups. In LeRobot, you can evaluate on individual tasks, difficulty groups, or the full MT50 suite:
 
-## What it enables in LeRobot
+| Group      | CLI name             | Tasks | Description                                            |
+| ---------- | -------------------- | ----- | ------------------------------------------------------ |
+| Easy       | `easy`               | 28    | Tasks with simple dynamics and single-step goals       |
+| Medium     | `medium`             | 11    | Tasks requiring multi-step reasoning                   |
+| Hard       | `hard`               | 6     | Tasks with complex contacts and precise manipulation   |
+| Very Hard  | `very_hard`          | 5     | The most challenging tasks in the suite                |
+| MT50 (all) | Comma-separated list | 50    | All 50 tasks — the most challenging multi-task setting |
 
-In LeRobot, you can evaluate any policy or vision-language-action (VLA) model on Meta-World tasks and get a clear success-rate measure. The integration is designed to be straightforward:
+You can also pass individual task names directly (e.g., `assembly-v3`, `dial-turn-v3`).
 
-- We provide a LeRobot-ready dataset for Meta-World (MT50) on the HF Hub: `https://huggingface.co/datasets/lerobot/metaworld_mt50`.
-  - This dataset is formatted for the MT50 evaluation that uses all 50 tasks (the most challenging multi-task setting).
-  - MT50 gives the policy a one-hot task vector and uses fixed object/goal positions for consistency.
+We provide a LeRobot-ready dataset for Meta-World MT50 on the HF Hub: [lerobot/metaworld_mt50](https://huggingface.co/datasets/lerobot/metaworld_mt50). This dataset is formatted for the MT50 evaluation that uses all 50 tasks with fixed object/goal positions and one-hot task vectors for consistency.
 
-- Task descriptions and the exact keys required for evaluation are available in the repo/dataset — use these to ensure your policy outputs the right success signals.
+## Installation
 
-## Quick start, train a SmolVLA policy on Meta-World
+After following the LeRobot installation instructions:
 
-Example command to train a SmolVLA policy on a subset of tasks:
+```bash
+pip install -e ".[metaworld]"
+```
+
+<Tip warning={true}>
+If you encounter an `AssertionError: ['human', 'rgb_array', 'depth_array']` when running Meta-World environments, this is a mismatch between Meta-World and your Gymnasium version. Fix it with:
+
+```bash
+pip install "gymnasium==1.1.0"
+```
+
+</Tip>
+
+## Evaluation
+
+### Default evaluation (recommended)
+
+Evaluate on the medium difficulty split (a good balance of coverage and compute):
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=medium \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+### Single-task evaluation
+
+Evaluate on a specific task:
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=assembly-v3 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+### Multi-task evaluation
+
+Evaluate across multiple tasks or difficulty groups:
+
+```bash
+lerobot-eval \
+  --policy.path="your-policy-id" \
+  --env.type=metaworld \
+  --env.task=assembly-v3,dial-turn-v3,handle-press-side-v3 \
+  --eval.batch_size=1 \
+  --eval.n_episodes=10
+```
+
+- `--env.task` accepts explicit task lists (comma-separated) or difficulty groups (e.g., `easy`, `medium`, `hard`, `very_hard`).
+- `--eval.batch_size` controls how many environments run in parallel.
+- `--eval.n_episodes` sets how many episodes to run per task.
+
+### Policy inputs and outputs
+
+**Observations:**
+
+- `observation.image` — single camera view (`corner2`), 480x480 HWC uint8
+- `observation.state` — 4-dim proprioceptive state (end-effector position + gripper)
+
+**Actions:**
+
+- Continuous control in `Box(-1, 1, shape=(4,))` — 3D end-effector delta + 1D gripper
+
+### Recommended evaluation episodes
+
+For reproducible benchmarking, use **10 episodes per task**. For the full MT50 suite this gives 500 total episodes. If you care about generalization, run on the full MT50 — it is intentionally challenging and reveals strengths/weaknesses better than a few narrow tasks.
+
+## Training
+
+### Example training command
+
+Train a SmolVLA policy on a subset of Meta-World tasks:
 
 ```bash
 lerobot-train \
@@ -44,37 +123,8 @@ lerobot-train \
   --eval_freq=1000
 ```
 
-Notes:
-
-- `--env.task` accepts explicit task lists (comma separated) or difficulty groups (e.g., `env.task="hard"`).
-- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget.
-- **Gymnasium Assertion Error**: if you encounter an error like
-  `AssertionError: ['human', 'rgb_array', 'depth_array']` when running MetaWorld environments, this comes from a mismatch between MetaWorld and your Gymnasium version.
-  We recommend using:
-
-```bash
-  pip install "gymnasium==1.1.0"
-```
-
-to ensure proper compatibility.
-
-## Quick start — evaluate a trained policy
-
-To evaluate a trained policy on the Meta-World medium difficulty split:
-
-```bash
-lerobot-eval \
-  --policy.path="your-policy-id" \
-  --env.type=metaworld \
-  --env.task=medium \
-  --eval.batch_size=1 \
-  --eval.n_episodes=2
-```
-
-This will run episodes and return per-task success rates using the standard Meta-World evaluation keys.
-
 ## Practical tips
 
-- If you care about generalization, run on the full MT50 suite — it’s intentionally challenging and reveals strengths/weaknesses better than a few narrow tasks.
-- Use the one-hot task conditioning for multi-task training (MT10 / MT50 conventions) so policies have explicit task context.
+- Use the one-hot task conditioning for multi-task training (MT10/MT50 conventions) so policies have explicit task context.
 - Inspect the dataset task descriptions and the `info["is_success"]` keys when writing post-processing or logging so your success metrics line up with the benchmark.
+- Adjust `batch_size`, `steps`, and `eval_freq` to match your compute budget.

docs/source/multi_task_dit.mdx

@@ -331,6 +331,54 @@ lerobot-train \
   --wandb.project=multitask_dit
 ```
 
+## Libero Results
+
+```
+python -m lerobot.scripts.lerobot_train \
+  --dataset.repo_id=HuggingFaceVLA/libero \
+  --policy.type=multi_task_dit \
+  --policy.push_to_hub=false \
+  --output_dir="./outputs/multitask_dit_libero" \
+  --job_name="multitask-dit-libero" \
+  --wandb.enable=true \
+  --wandb.project=multitask_dit_libero \
+  --dataset.image_transforms.enable=true \
+  --dataset.image_transforms.max_num_transforms=4 \
+  --dataset.image_transforms.tfs='{"brightness":{"type":"ColorJitter","kwargs":{"brightness":[0.75,1.25]}},"contrast":{"type":"ColorJitter","kwargs":{"contrast":[0.6,1.4]}},"saturation":{"type":"ColorJitter","kwargs":{"saturation":[0.8,1.2]}},"hue":{"type":"ColorJitter","kwargs":{"hue":[-0.05,0.05]}},"sharpness":{"type":"SharpnessJitter","kwargs":{"sharpness":[0.6,1.4]}},"rotation":{"type":"RandomRotation","kwargs":{"degrees":[-5,5]}},"translation":{"type":"RandomAffine","kwargs":{"degrees":0,"translate":[0.1,0.1]}}}' \
+  --dataset.video_backend=torchcodec \
+  --policy.use_amp=true \
+  --policy.horizon=48 \
+  --policy.n_obs_steps=2 \
+  --policy.use_rope=true \
+  --policy.use_positional_encoding=false \
+  --policy.hidden_dim=768 \
+  --policy.num_layers=8 \
+  --policy.num_heads=12 \
+  --policy.dropout=0.1 \
+  --policy.timestep_embed_dim=256 \
+  --policy.objective=diffusion \
+  --policy.optimizer_lr=3e-4 \
+  --policy.optimizer_weight_decay=0 \
+  --policy.scheduler_warmup_steps=0 \
+  --policy.vision_encoder_name=openai/clip-vit-base-patch16 \
+  --policy.image_resize_shape=[256,256] \
+  --policy.image_crop_is_random=true \
+  --policy.text_encoder_name=openai/clip-vit-base-patch16 \
+  --policy.vision_encoder_lr_multiplier=0.1 \
+  --policy.device=cuda \
+  --num_workers=8 \
+  --save_freq=4000 \
+  --log_freq=100 \
+  --steps=100000 \
+  --batch_size=320
+```
+
+Results:
+
+| LIBERO Spatial | LIBERO Object | LIBERO Goal | LIBERO 10 | Average |
+| -------------- | ------------- | ----------- | --------- | ------- |
+| 87.0           | 98.2          | 93.8        | 83.2      | 90.6    |
+
 ## References
 
 For more details on the technical implementation and architecture, see:

docs/source/policy_pi05_README.md

@@ -0,0 +1,91 @@
+# π₀.₅ (pi05)
+
+This repository contains the Hugging Face port of **π₀.₅**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
+It is designed as a **Vision-Language-Action model with open-world generalization**.
+
+---
+
+## Model Overview
+
+| Feature              | π₀                                                     | π₀.₅                                      |
+| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
+| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
+| AdaRMS               | Not used                                               | Used in action expert                     |
+| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
+| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
+| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
+
+---
+
+## Relative Actions
+
+π₀.₅ supports training with **relative actions**, where the model learns relative offsets
+from the current robot state instead of absolute joint positions. This mirrors the
+relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
+
+### How it works
+
+1. **During preprocessing**, absolute actions are converted to relative offsets:
+   `relative = action - state` (for selected joints).
+2. The relative actions are normalized using statistics computed from the relative distribution.
+3. **During postprocessing**, predicted relative actions are converted back to absolute:
+   `absolute = relative + state`.
+
+Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
+
+### Configuration
+
+| Parameter                 | Type        | Default       | Description                                                      |
+| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
+| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
+| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
+| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
+
+### Training example
+
+```bash
+python -m lerobot.scripts.lerobot_train \
+  --policy.type=pi05 \
+  --dataset.repo_id=your_org/your_dataset \
+  --policy.use_relative_actions=true \
+  --policy.relative_exclude_joints='["gripper"]'
+```
+
+When `use_relative_actions=true`, the training script automatically:
+
+- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
+- Replaces the standard action stats with relative stats for normalization
+- Broadcasts these stats across all ranks in distributed training
+
+---
+
+## Citation
+
+If you use this work, please cite both **OpenPI** and the π₀.₅ paper:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{intelligence2025pi05visionlanguageactionmodelopenworld,
+  title        = {π₀.₅: a Vision-Language-Action Model with Open-World Generalization},
+  author       = {Physical Intelligence and Kevin Black and Noah Brown and James Darpinian and Karan Dhabalia and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Manuel Y. Galliker and Dibya Ghosh and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Devin LeBlanc and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Allen Z. Ren and Lucy Xiaoyang Shi and Laura Smith and Jost Tobias Springenberg and Kyle Stachowicz and James Tanner and Quan Vuong and Homer Walke and Anna Walling and Haohuan Wang and Lili Yu and Ury Zhilinsky},
+  year         = {2025},
+  eprint       = {2504.16054},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.LG},
+  url          = {https://arxiv.org/abs/2504.16054},
+}
+```
+
+---
+
+## License
+
+This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).

docs/source/policy_pi0_README.md

@@ -0,0 +1,108 @@
+# π₀ (pi0)
+
+This repository contains the Hugging Face port of **π₀**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
+It is designed as a **Vision-Language-Action model for general robot control**.
+
+---
+
+## Model Overview
+
+| Feature              | π₀                                                     | π₀.₅                                      |
+| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
+| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
+| AdaRMS               | Not used                                               | Used in action expert                     |
+| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
+| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
+| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
+
+---
+
+## Relative Actions
+
+π₀ supports training with **relative actions**, where the model learns relative offsets
+from the current robot state instead of absolute joint positions. This mirrors the
+relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
+
+### How it works
+
+1. **During preprocessing**, absolute actions are converted to relative offsets:
+   `relative = action - state` (for selected joints).
+2. The relative actions are normalized using statistics computed from the relative distribution.
+3. **During postprocessing**, predicted relative actions are converted back to absolute:
+   `absolute = relative + state`.
+
+Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
+
+### Configuration
+
+| Parameter                 | Type        | Default       | Description                                                      |
+| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
+| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
+| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
+| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
+
+### Training example
+
+```bash
+python -m lerobot.scripts.lerobot_train \
+  --policy.type=pi0 \
+  --dataset.repo_id=your_org/your_dataset \
+  --policy.use_relative_actions=true \
+  --policy.relative_exclude_joints='["gripper"]'
+```
+
+When `use_relative_actions=true`, the training script automatically:
+
+- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
+- Replaces the standard action stats with relative stats for normalization
+- Broadcasts these stats across all ranks in distributed training
+
+### Recomputing stats for an existing dataset
+
+If you want to precompute relative action stats offline, use `recompute_stats` from
+`lerobot.datasets.dataset_tools`:
+
+```python
+from lerobot.datasets.lerobot_dataset import LeRobotDataset
+from lerobot.datasets.dataset_tools import recompute_stats
+
+dataset = LeRobotDataset("your_org/your_dataset")
+dataset = recompute_stats(
+    dataset,
+    relative_action=True,
+    relative_exclude_joints=["gripper"],
+)
+```
+
+---
+
+## Citation
+
+If you use this work, please cite both **OpenPI** and the π₀ paper:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{black2024pi0visionlanguageactionflowmodel,
+  title        = {π₀: A Vision-Language-Action Flow Model for General Robot Control},
+  author       = {Kevin Black and Noah Brown and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Lucy Xiaoyang Shi and James Tanner and Quan Vuong and Anna Walling and Haohuan Wang and Ury Zhilinsky},
+  year         = {2024},
+  eprint       = {2410.24164},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.LG},
+  url          = {https://arxiv.org/abs/2410.24164},
+}
+```
+
+---
+
+## License
+
+This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).

docs/source/policy_rtc_README.md

@@ -0,0 +1,38 @@
+# Real-Time Chunking (RTC)
+
+This module contains the LeRobot implementation of **Real-Time Chunking (RTC)**, an inference-time technique for flow-matching based policies.
+
+**Note**: RTC is not a policy itself, but rather an inference enhancement that works with flow-matching based policies including [π₀](../pi0/), [π₀.₅](../pi05/), and [SmolVLA](../smolvla/).
+
+---
+
+## Citation
+
+If you use Real-Time Chunking in your work, please cite:
+
+```bibtex
+@misc{openpi2024,
+  author       = {Physical Intelligence Lab},
+  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
+  year         = {2024},
+  publisher    = {GitHub},
+  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
+  license      = {Apache-2.0}
+}
+
+@misc{black2025realtimeexecutionactionchunking,
+      title={Real-Time Execution of Action Chunking Flow Policies},
+      author={Kevin Black and Manuel Y. Galliker and Sergey Levine},
+      year={2025},
+      eprint={2506.07339},
+      archivePrefix={arXiv},
+      primaryClass={cs.RO},
+      url={https://arxiv.org/abs/2506.07339},
+}
+```
+
+---
+
+## License
+
+This implementation follows the **Apache 2.0 License**, consistent with the LeRobot project.

docs/source/policy_sarm_README.md

@@ -0,0 +1,14 @@
+## Paper
+
+https://arxiv.org/abs/2509.25358
+
+## Citation
+
+```bibtex
+@article{chen2025sarm,
+  title={SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation},
+  author={Chen, Qianzhong and Yu, Justin and Schwager, Mac and Abbeel, Pieter and Shentu, Yide and Wu, Philipp},
+  journal={arXiv preprint arXiv:2509.25358},
+  year={2025}
+}
+```

docs/source/umi_pi0_relative_ee.mdx

@@ -1,227 +0,0 @@
-# UMI Data with pi0 Relative EE Actions
-
-This guide explains how to train a pi0 policy with UMI-style relative end-effector (EE) actions and deploy it on a real OpenArm robot.
-
-**What we will do:**
-
-1. Prepare the dataset (EE pose + gripper in the action column).
-2. Recompute statistics for relative actions.
-3. Train pi0 with `derive_state_from_action=true`.
-4. Evaluate the trained policy on a real robot.
-
-## Background
-
-[UMI (Universal Manipulation Interface)](https://umi-gripper.github.io) collects manipulation data with hand-held grippers, recovering 6-DoF EE poses via SLAM. The key insight from UMI (Chi et al., 2024) is that the action space must include **both EE trajectory and gripper width**, and actions should be expressed as **relative trajectories** (offsets from the current pose).
-
-### Dataset layout
-
-The dataset should have this structure:
-
-| Feature                   | Shape     | Content                                                  |
-| ------------------------- | --------- | -------------------------------------------------------- |
-| `observation.images.cam0` | `[3,H,W]` | Wrist camera image                                       |
-| `action`                  | `[8]`     | `[x, y, z, ax, ay, az, proximal, distal]` (EE + gripper) |
-
-No separate `observation.pose` or `observation.joints` columns are needed — the model derives its proprioception state directly from the action column (`derive_state_from_action=true`).
-
-### Why relative actions?
-
-With relative actions, each action in a chunk is an **offset from the current state** rather than an absolute target:
-
-```
-relative_action[i] = absolute_action[t + i] − state[t]
-```
-
-UMI ablations show this is critical: absolute actions achieve only 25% success vs 100% for relative trajectory on the cup arrangement task. Compared to delta actions (each step relative to the previous), relative trajectory avoids error accumulation. See the [Action Representations](action_representations) guide for details.
-
-### `derive_state_from_action`
-
-When `derive_state_from_action=true`, pi0 derives `observation.state` from the action column during training — no separate state column needed. Under the hood:
-
-- `action_delta_indices` extends to `[-1, 0, 1, ..., chunk_size-1]` (one extra leading timestep).
-- `DeriveStateFromActionStep` extracts `[action[t-1], action[t]]` as a 2-step state and strips the extra timestep from the action chunk.
-- `RelativeActionsProcessorStep` converts actions to offsets from `state[t]`.
-- `RelativeStateProcessorStep` converts the 2-step state to relative proprioception (velocity + zeros) and flattens.
-
-This implies `use_relative_state=true` and `state_obs_steps=2`.
-
-During **inference**, `DeriveStateFromActionStep` is a no-op — state comes from the robot via forward kinematics. `RelativeStateProcessorStep` buffers the previous state and applies the same conversion automatically.
-
-## Step 1: Recompute Stats
-
-After preparing the dataset with EE pose in the action column, recompute statistics with `derive_state_from_action=true`. This computes relative action and state stats so the normalizer sees offset distributions:
-
-```bash
-lerobot-edit-dataset \
-    --repo-id=glannuzel/grabette-dataset \
-    --operation=recompute_stats \
-    --operation.relative_action=true \
-    --operation.relative_exclude_joints='["proximal", "distal"]' \
-    --operation.derive_state_from_action=true \
-    --operation.chunk_size=30 \
-    --push_to_hub=true
-```
-
-| Flag                            | Purpose                                                                         |
-| ------------------------------- | ------------------------------------------------------------------------------- |
-| `relative_action=true`          | Compute stats on `action − state` (relative actions)                            |
-| `relative_exclude_joints`       | Keep gripper dims absolute (they don't benefit from relative encoding)          |
-| `derive_state_from_action=true` | Derive state from action column (implies `relative_state`, `state_obs_steps=2`) |
-| `chunk_size=30`                 | Must match training chunk size                                                  |
-
-## Step 2: Train
-
-```bash
-#!/bin/bash
-set -euo pipefail
-
-export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:${LD_LIBRARY_PATH:-}
-
-DATASET="glannuzel/grabette-dataset"
-NUM_PROCESSES=8
-
-echo "=== Training pi0 on $DATASET (UMI relative EE, ${NUM_PROCESSES} GPUs) ==="
-accelerate launch --multi_gpu --num_processes=$NUM_PROCESSES \
-    -m lerobot.scripts.lerobot_train \
-    --dataset.repo_id="$DATASET" \
-    --dataset.video_backend=pyav \
-    --policy.type=pi0 \
-    --policy.pretrained_path=lerobot/pi0_base \
-    --policy.repo_id=pepijn/grabette-umi-pi0 \
-    --policy.chunk_size=30 \
-    --policy.n_action_steps=30 \
-    --policy.derive_state_from_action=true \
-    --use_relative_actions=true \
-    --policy.relative_exclude_joints='["proximal", "distal"]' \
-    --batch_size=32 \
-    --steps=5000 \
-    --policy.scheduler_decay_steps=5000 \
-    --policy.dtype=bfloat16 \
-    --policy.compile_model=false \
-    --policy.gradient_checkpointing=true \
-    --policy.device=cuda \
-    --output_dir=/fsx/pepijn/outputs/grabette-umi \
-    --job_name=grabette-umi-v2 \
-    --wandb.enable=true \
-    --wandb.disable_artifact=true \
-    --wandb.project=grabette-umi \
-    --log_freq=100 \
-    --save_freq=5000
-```
-
-Key flags:
-
-| Flag                            | Purpose                                                                |
-| ------------------------------- | ---------------------------------------------------------------------- |
-| `derive_state_from_action=true` | Derive proprioception from action column (full UMI mode)               |
-| `use_relative_actions=true`     | Actions are offsets from current state                                 |
-| `relative_exclude_joints`       | `["proximal", "distal"]` — gripper stays absolute, EE pose is relative |
-| `chunk_size=30`                 | Action horizon: 30 steps (~0.65s at 46 FPS)                            |
-| `n_action_steps=30`             | Execute full chunk before replanning                                   |
-
-Note: `derive_state_from_action=true` automatically implies `use_relative_state=true` and `state_obs_steps=2`. No `rename_map` is needed since there are no separate observation columns to rename.
-
-## Step 3: Evaluate
-
-The evaluation script in `examples/umi_pi0_relative_ee/evaluate.py` runs inference on a real OpenArm robot:
-
-```bash
-python examples/umi_pi0_relative_ee/evaluate.py
-```
-
-Edit `HF_MODEL_ID`, camera index, and robot configuration at the top of the file.
-
-### How inference works
-
-At inference, the training dataset has no `observation.state` — it was derived from actions. The evaluate script provides `observation.state` from the robot via forward kinematics:
-
-1. **Robot → FK** — Arm joint positions → EE pose `[x,y,z,ax,ay,az]`, gripper → `[proximal, distal]`. Combined into `observation.state` (8D).
-2. **Preprocessor** (loaded from checkpoint) — `DeriveStateFromActionStep` is a no-op. `RelativeStateProcessorStep` buffers previous state, stacks `[prev, current]`, subtracts current → velocity info. `RelativeActionsProcessorStep` caches state. `NormalizerProcessorStep` normalizes.
-3. **pi0 inference** — Predicts normalized relative action chunk (30 steps).
-4. **Postprocessor** — `UnnormalizerProcessorStep` unnormalizes, `AbsoluteActionsProcessorStep` adds cached state → absolute EE targets.
-5. **IK → Robot** — Absolute `[x,y,z,ax,ay,az]` → arm joint targets with full 6-DOF IK (orientation weight = 1.0). `[proximal, distal]` → direct gripper position commands.
-
-### Latency compensation
-
-Set `LATENCY_SKIP_STEPS` to skip the first few predicted action steps, compensating for system latency:
-
-```python
-LATENCY_SKIP_STEPS = 7  # ceil(total_latency_ms / (1000 / FPS))
-```
-
-At 46 FPS (~22ms/step) with ~150ms total latency: `ceil(150/22) ≈ 7`. Start with 0 for a safe first test.
-
-## Replay Viewer
-
-Visualize any dataset episode in a browser-based 3D viewer before running on hardware. The viewer shows the EE trajectory overlaid on the OpenArm URDF model.
-
-### Quick start
-
-```bash
-python examples/umi_pi0_relative_ee/replay.py
-```
-
-### Options
-
-| Flag        | Default                      | Description                          |
-| ----------- | ---------------------------- | ------------------------------------ |
-| `--repo-id` | `glannuzel/grabette-dataset` | HuggingFace dataset repo to load     |
-| `--episode` | `0`                          | Episode index to replay              |
-| `--port`    | `8765`                       | HTTP server port                     |
-| `--force`   | off                          | Re-extract trajectory even if cached |
-
-### Viewer controls
-
-The panel in the top-left corner shows live EE coordinates and gripper state. Transport controls:
-
-- **Play / Pause** — toggle automatic playback.
-- **Step buttons** (◀ ▶) — advance or rewind one frame.
-- **Reset** (⟳) — jump to frame 0.
-- **Scrubber** — drag to seek.
-- **Speed selector** — 0.25× to 4× playback speed.
-
-### Color legend
-
-| Color              | Meaning                                       |
-| ------------------ | --------------------------------------------- |
-| Red sphere         | Current EE position                           |
-| Yellow trail       | Past trajectory                               |
-| Dark trail         | Future trajectory                             |
-| Orange ring + axes | URDF `ee_target` frame (zero-joint reference) |
-
-## How the Pieces Fit Together
-
-```
-Training (derive_state_from_action=true):
-  DataLoader loads action: [B, 31, 8]  (chunk_size=30 + 1 leading)
-      → DeriveStateFromActionStep
-          state  = action[:, :2, :]     → [B, 2, 8]
-          action = action[:, 1:, :]     → [B, 30, 8]
-      → RelativeActionsProcessorStep    (action -= state[:, -1, :])
-      → RelativeStateProcessorStep      (state offsets from current, flatten → [B, 16])
-      → NormalizerProcessorStep         → pi0 model
-
-Inference:
-  arm joints → FK → observation.state [8D: x,y,z,ax,ay,az,prox,dist]
-                        ↓
-                DeriveStateFromActionStep (no-op)
-                        ↓
-                RelativeActionsProcessorStep (caches state)
-                        ↓
-                RelativeStateProcessorStep (buffers prev, stacks, subtracts, flattens)
-                        ↓
-                NormalizerProcessorStep → pi0 model → relative action chunk [30, 8]
-                        ↓
-                UnnormalizerProcessorStep
-                        ↓
-                AbsoluteActionsProcessorStep (+ cached state → absolute EE)
-                        ↓
-                IK → joint targets → robot
-```
-
-## References
-
-- [UMI: Universal Manipulation Interface](https://umi-gripper.github.io) — Chi et al., 2024. Defines relative trajectory actions.
-- [Action Representations](action_representations) — LeRobot guide comparing absolute, relative, and delta actions.
-- [pi0 documentation](pi0) — Full pi0 configuration including `use_relative_actions`.
-- [`examples/so100_to_so100_EE/`](https://github.com/huggingface/lerobot/tree/main/examples/so100_to_so100_EE) — EE-space evaluation example this builds on.

examples/dataset/create_progress_videos.py

@@ -0,0 +1,680 @@
+#!/usr/bin/env python
+
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Create MP4 (or GIF) videos with sarm_progress overlay for specified episodes.
+
+Downloads datasets from HuggingFace, seeks directly into the episode segment
+of the source video, draws a progress line on each frame, and writes the result.
+
+Usage:
+    python examples/dataset/create_progress_videos.py \
+        --repo-id lerobot-data-collection/level2_final_quality3 \
+        --episode 1100
+
+    python examples/dataset/create_progress_videos.py \
+        --repo-id lerobot-data-collection/level2_final_quality3 \
+        --episode 1100 \
+        --camera-key observation.images.top \
+        --output-dir ./my_videos \
+        --gif
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import subprocess
+from pathlib import Path
+
+import cv2
+import numpy as np
+import pandas as pd
+from huggingface_hub import snapshot_download
+
+GRAPH_Y_TOP_FRAC = 0.01
+GRAPH_Y_BOT_FRAC = 0.99
+LINE_THICKNESS = 3
+SHADOW_THICKNESS = 6
+REF_ALPHA = 0.45
+FILL_ALPHA = 0.55
+SCORE_FONT_SCALE = 0.8
+TASK_FONT_SCALE = 0.55
+
+
+def download_episode_metadata(repo_id: str, episode: int) -> Path:
+    """Download only the metadata and sarm_progress files for a dataset.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        episode: Episode index (used for logging only; all meta is fetched).
+
+    Returns:
+        Local cache path for the downloaded snapshot.
+    """
+    logging.info("[1/4] Downloading metadata for %s (episode %d) ...", repo_id, episode)
+    local_path = Path(
+        snapshot_download(
+            repo_id=repo_id,
+            repo_type="dataset",
+            allow_patterns=["meta/**", "sarm_progress.parquet"],
+            ignore_patterns=["*.mp4"],
+        )
+    )
+    return local_path
+
+
+def load_episode_meta(local_path: Path, episode: int, camera_key: str | None) -> dict:
+    """Read info.json and episode parquet to resolve fps, video path, and timestamps.
+
+    Args:
+        local_path: Local cache directory containing meta/.
+        episode: Episode index to look up.
+        camera_key: Camera observation key (e.g. "observation.images.base").
+            If None, the first available video key is used.
+
+    Returns:
+        Dict with keys: fps, camera, video_rel, chunk_index, file_index,
+        from_ts, to_ts, task_name.
+    """
+    info = json.loads((local_path / "meta" / "info.json").read_text())
+    fps = info["fps"]
+    features = info["features"]
+
+    video_keys = [k for k, v in features.items() if v.get("dtype") == "video"]
+    if not video_keys:
+        raise RuntimeError("No video keys found in dataset features")
+
+    if camera_key is not None:
+        if camera_key not in video_keys:
+            raise RuntimeError(f"camera_key='{camera_key}' not found. Available: {video_keys}")
+        selected_camera = camera_key
+    else:
+        selected_camera = video_keys[0]
+    logging.info("   fps=%d  camera='%s'  all_cams=%s", fps, selected_camera, video_keys)
+
+    episode_rows = []
+    for parquet_file in sorted((local_path / "meta" / "episodes").glob("**/*.parquet")):
+        episode_rows.append(pd.read_parquet(parquet_file))
+    episode_df = pd.concat(episode_rows, ignore_index=True)
+    row = episode_df[episode_df["episode_index"] == episode]
+    if row.empty:
+        raise RuntimeError(f"Episode {episode} not found in episode metadata")
+    row = row.iloc[0]
+
+    chunk_col = f"videos/{selected_camera}/chunk_index"
+    file_col = f"videos/{selected_camera}/file_index"
+    ts_from_col = f"videos/{selected_camera}/from_timestamp"
+    ts_to_col = f"videos/{selected_camera}/to_timestamp"
+
+    if chunk_col not in row.index:
+        chunk_col = f"{selected_camera}/chunk_index"
+        file_col = f"{selected_camera}/file_index"
+        ts_from_col = f"{selected_camera}/from_timestamp"
+        ts_to_col = f"{selected_camera}/to_timestamp"
+    if chunk_col not in row.index:
+        raise RuntimeError(
+            f"Cannot find video metadata columns for {selected_camera}.\nAvailable: {list(row.index)}"
+        )
+
+    chunk_index = int(row[chunk_col])
+    file_index = int(row[file_col])
+    from_timestamp = float(row[ts_from_col])
+    to_timestamp = float(row[ts_to_col])
+
+    video_template = info.get(
+        "video_path", "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4"
+    )
+    video_rel = video_template.format(
+        video_key=selected_camera,
+        chunk_index=chunk_index,
+        file_index=file_index,
+    )
+
+    task_name = _resolve_task_name(row, local_path)
+
+    return {
+        "fps": fps,
+        "camera": selected_camera,
+        "video_rel": video_rel,
+        "chunk_index": chunk_index,
+        "file_index": file_index,
+        "from_ts": from_timestamp,
+        "to_ts": to_timestamp,
+        "task_name": task_name,
+    }
+
+
+def _resolve_task_name(row: pd.Series, local_path: Path) -> str:
+    """Best-effort extraction of the task name for an episode row.
+
+    Args:
+        row: Single-episode row from the episodes parquet.
+        local_path: Dataset cache root.
+
+    Returns:
+        Task name string, or empty string if unavailable.
+    """
+    try:
+        if "tasks" in row.index and row["tasks"] is not None:
+            tasks_val = row["tasks"]
+            if isinstance(tasks_val, (list, tuple, np.ndarray)) and len(tasks_val) > 0:
+                return str(tasks_val[0])
+            return str(tasks_val).strip("[]'")
+
+        tasks_parquet = local_path / "meta" / "tasks.parquet"
+        if tasks_parquet.exists():
+            tasks_df = pd.read_parquet(tasks_parquet)
+            task_idx = int(row.get("task_index", 0)) if "task_index" in row.index else 0
+            match = tasks_df[tasks_df["task_index"] == task_idx]
+            if not match.empty:
+                return str(match.index[0])
+    except Exception as exc:
+        logging.warning("Could not load task name: %s", exc)
+    return ""
+
+
+def download_video_file(repo_id: str, local_path: Path, video_rel: str) -> Path:
+    """Download the specific video file if not already cached.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        local_path: Local cache directory.
+        video_rel: Relative path to the video file within the dataset.
+
+    Returns:
+        Absolute path to the downloaded video file.
+    """
+    video_path = local_path / video_rel
+    if video_path.exists():
+        logging.info("   Video already cached: %s", video_path)
+        return video_path
+    logging.info("[2/4] Downloading video file %s ...", video_rel)
+    snapshot_download(
+        repo_id=repo_id,
+        repo_type="dataset",
+        local_dir=str(local_path),
+        allow_patterns=[video_rel],
+    )
+    if not video_path.exists():
+        raise RuntimeError(f"Video not found after download: {video_path}")
+    return video_path
+
+
+def load_progress_data(local_path: Path, episode: int) -> np.ndarray | None:
+    """Load sarm_progress values for an episode.
+
+    Args:
+        local_path: Dataset cache root.
+        episode: Episode index.
+
+    Returns:
+        Sorted (N, 2) array of (frame_index, progress), or None if unavailable.
+    """
+    parquet_path = local_path / "sarm_progress.parquet"
+    if not parquet_path.exists():
+        logging.warning("sarm_progress.parquet not found")
+        return None
+    df = pd.read_parquet(parquet_path)
+    logging.info("   sarm_progress.parquet columns: %s", list(df.columns))
+    episode_df = df[df["episode_index"] == episode].copy()
+    if episode_df.empty:
+        logging.warning("No sarm_progress rows for episode %d", episode)
+        return None
+    episode_df = episode_df.sort_values("frame_index")
+
+    if "progress_dense" in episode_df.columns and episode_df["progress_dense"].notna().any():
+        progress_column = "progress_dense"
+    elif "progress_sparse" in episode_df.columns:
+        progress_column = "progress_sparse"
+    else:
+        progress_columns = [c for c in episode_df.columns if "progress" in c.lower()]
+        if not progress_columns:
+            return None
+        progress_column = progress_columns[0]
+
+    logging.info("   Using progress column: '%s'", progress_column)
+    return episode_df[["frame_index", progress_column]].rename(columns={progress_column: "progress"}).values
+
+
+def _precompute_pixel_coords(
+    progress_data: np.ndarray,
+    num_frames: int,
+    frame_width: int,
+    frame_height: int,
+) -> np.ndarray:
+    """Map progress samples to pixel coordinates for overlay drawing.
+
+    Args:
+        progress_data: (N, 2) array of (frame_index, progress).
+        num_frames: Total number of video frames.
+        frame_width: Video width in pixels.
+        frame_height: Video height in pixels.
+
+    Returns:
+        (N, 2) array of (x, y) pixel coordinates.
+    """
+    frame_indices = progress_data[:, 0].astype(float)
+    progress_values = np.clip(progress_data[:, 1].astype(float), 0.0, 1.0)
+
+    y_top = int(frame_height * GRAPH_Y_TOP_FRAC)
+    y_bot = int(frame_height * GRAPH_Y_BOT_FRAC)
+    graph_height = y_bot - y_top
+
+    x_coords = (frame_indices / (num_frames - 1) * (frame_width - 1)).astype(int)
+    y_coords = (y_bot - progress_values * graph_height).astype(int)
+
+    return np.stack([x_coords, y_coords], axis=1)
+
+
+def _progress_color(normalized_position: float) -> tuple[int, int, int]:
+    """Interpolate BGR color from red to green based on position in [0, 1].
+
+    Args:
+        normalized_position: Value in [0, 1] indicating how far along the episode.
+
+    Returns:
+        BGR color tuple.
+    """
+    red = int(255 * (1.0 - normalized_position))
+    green = int(255 * normalized_position)
+    return (0, green, red)
+
+
+def _prerender_fill_polygon(
+    pixel_coords: np.ndarray,
+    frame_width: int,
+    frame_height: int,
+) -> np.ndarray:
+    """Pre-render the grey fill polygon under the progress curve as a BGRA image.
+
+    Args:
+        pixel_coords: (N, 2) array of (x, y) pixel coordinates.
+        frame_width: Video width in pixels.
+        frame_height: Video height in pixels.
+
+    Returns:
+        BGRA image array of shape (frame_height, frame_width, 4).
+    """
+    y_bot = int(frame_height * GRAPH_Y_BOT_FRAC)
+    fill_image = np.zeros((frame_height, frame_width, 4), dtype=np.uint8)
+    polygon = np.concatenate(
+        [
+            pixel_coords,
+            [[pixel_coords[-1][0], y_bot], [pixel_coords[0][0], y_bot]],
+        ],
+        axis=0,
+    ).astype(np.int32)
+    cv2.fillPoly(fill_image, [polygon], color=(128, 128, 128, int(255 * FILL_ALPHA)))
+    return fill_image
+
+
+def _alpha_composite_region(base: np.ndarray, overlay_bgra: np.ndarray, x_limit: int) -> None:
+    """Blend BGRA overlay onto BGR base in-place, up to x_limit columns.
+
+    Args:
+        base: BGR frame to draw on (modified in-place).
+        overlay_bgra: BGRA overlay image.
+        x_limit: Only blend columns [0, x_limit).
+    """
+    if x_limit <= 0:
+        return
+    region_base = base[:, :x_limit]
+    region_overlay = overlay_bgra[:, :x_limit]
+    alpha = region_overlay[:, :, 3:4].astype(np.float32) / 255.0
+    region_base[:] = np.clip(
+        region_overlay[:, :, :3].astype(np.float32) * alpha + region_base.astype(np.float32) * (1.0 - alpha),
+        0,
+        255,
+    ).astype(np.uint8)
+
+
+def _draw_text_outlined(
+    frame: np.ndarray,
+    text: str,
+    position: tuple[int, int],
+    font_scale: float,
+    thickness: int = 1,
+) -> None:
+    """Draw white text with a dark outline for readability on any background.
+
+    Args:
+        frame: BGR image to draw on (modified in-place).
+        text: String to render.
+        position: (x, y) bottom-left corner of the text.
+        font_scale: OpenCV font scale.
+        thickness: Text stroke thickness.
+    """
+    font = cv2.FONT_HERSHEY_SIMPLEX
+    cv2.putText(frame, text, position, font, font_scale, (0, 0, 0), thickness + 2, cv2.LINE_AA)
+    cv2.putText(frame, text, position, font, font_scale, (255, 255, 255), thickness, cv2.LINE_AA)
+
+
+def composite_progress_video(
+    video_path: Path,
+    from_timestamp: float,
+    to_timestamp: float,
+    progress_data: np.ndarray,
+    output_path: Path,
+    fps: float,
+    task_name: str = "",
+) -> Path:
+    """Read episode frames by seeking into the source video, draw progress overlay, write output.
+
+    Uses cv2.CAP_PROP_POS_MSEC to seek directly into the source video,
+    eliminating the need for an intermediate clip file.
+
+    Args:
+        video_path: Path to the full source video file.
+        from_timestamp: Start timestamp of the episode in seconds.
+        to_timestamp: End timestamp of the episode in seconds.
+        progress_data: (N, 2) array of (frame_index, progress).
+        output_path: Path to write the output MP4.
+        fps: Frames per second for the output video.
+        task_name: Optional task name to display at the top of the video.
+
+    Returns:
+        Path to the written output file (MP4).
+    """
+    capture = cv2.VideoCapture(str(video_path))
+    try:
+        capture.set(cv2.CAP_PROP_POS_MSEC, from_timestamp * 1000)
+
+        frame_width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
+        frame_height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
+        duration_seconds = to_timestamp - from_timestamp
+        num_frames = int(round(duration_seconds * fps))
+
+        logging.info(
+            "   Video: %dx%d, %d frames @ %.1f fps (%.2fs)",
+            frame_width,
+            frame_height,
+            num_frames,
+            fps,
+            duration_seconds,
+        )
+
+        pixel_coords = _precompute_pixel_coords(progress_data, num_frames, frame_width, frame_height)
+        y_ref = int(frame_height * GRAPH_Y_TOP_FRAC)
+
+        fill_image = _prerender_fill_polygon(pixel_coords, frame_width, frame_height)
+
+        ref_line_image = np.zeros((frame_height, frame_width, 4), dtype=np.uint8)
+        cv2.line(
+            ref_line_image,
+            (0, y_ref),
+            (frame_width - 1, y_ref),
+            (200, 200, 200, int(255 * REF_ALPHA)),
+            1,
+            cv2.LINE_AA,
+        )
+
+        frame_indices = progress_data[:, 0].astype(int)
+        progress_values = progress_data[:, 1].astype(float)
+
+        logging.info("[3/4] Compositing %d frames ...", num_frames)
+        fourcc = cv2.VideoWriter_fourcc(*"mp4v")
+        writer = cv2.VideoWriter(str(output_path), fourcc, fps, (frame_width, frame_height))
+
+        for frame_idx in range(num_frames):
+            ret, frame = capture.read()
+            if not ret:
+                break
+
+            drawn_count = int(np.searchsorted(frame_indices, frame_idx, side="right"))
+            x_current = (
+                int(pixel_coords[min(drawn_count, len(pixel_coords)) - 1][0]) + 1 if drawn_count > 0 else 0
+            )
+
+            _alpha_composite_region(frame, ref_line_image, frame_width)
+            _alpha_composite_region(frame, fill_image, x_current)
+
+            if drawn_count >= 2:
+                time_position = (drawn_count - 1) / max(len(progress_values) - 1, 1)
+                line_color = _progress_color(time_position)
+                points = pixel_coords[:drawn_count].reshape(-1, 1, 2).astype(np.int32)
+                cv2.polylines(
+                    frame,
+                    [points],
+                    isClosed=False,
+                    color=(255, 255, 255),
+                    thickness=SHADOW_THICKNESS,
+                    lineType=cv2.LINE_AA,
+                )
+                cv2.polylines(
+                    frame,
+                    [points],
+                    isClosed=False,
+                    color=line_color,
+                    thickness=LINE_THICKNESS,
+                    lineType=cv2.LINE_AA,
+                )
+
+            if drawn_count > 0:
+                score = float(progress_values[min(drawn_count, len(progress_values)) - 1])
+                score_text = f"{score:.2f}"
+                (text_width, _), _ = cv2.getTextSize(
+                    score_text, cv2.FONT_HERSHEY_SIMPLEX, SCORE_FONT_SCALE, 2
+                )
+                score_x = frame_width - text_width - 12
+                score_y = frame_height - 12
+                time_position = (drawn_count - 1) / max(len(progress_values) - 1, 1)
+                score_color = _progress_color(time_position)
+                cv2.putText(
+                    frame,
+                    score_text,
+                    (score_x, score_y),
+                    cv2.FONT_HERSHEY_SIMPLEX,
+                    SCORE_FONT_SCALE,
+                    (0, 0, 0),
+                    4,
+                    cv2.LINE_AA,
+                )
+                cv2.putText(
+                    frame,
+                    score_text,
+                    (score_x, score_y),
+                    cv2.FONT_HERSHEY_SIMPLEX,
+                    SCORE_FONT_SCALE,
+                    score_color,
+                    2,
+                    cv2.LINE_AA,
+                )
+
+            if task_name:
+                (text_width, _), _ = cv2.getTextSize(task_name, cv2.FONT_HERSHEY_SIMPLEX, TASK_FONT_SCALE, 1)
+                task_x = max((frame_width - text_width) // 2, 4)
+                _draw_text_outlined(frame, task_name, (task_x, 22), TASK_FONT_SCALE)
+
+            writer.write(frame)
+            if frame_idx % 100 == 0:
+                logging.info("   Frame %d/%d ...", frame_idx, num_frames)
+
+        writer.release()
+    finally:
+        capture.release()
+
+    logging.info("   MP4 written: %s", output_path)
+    return output_path
+
+
+def convert_mp4_to_gif(mp4_path: Path) -> Path:
+    """Convert an MP4 to an optimized GIF using ffmpeg palette generation.
+
+    Args:
+        mp4_path: Path to the source MP4 file.
+
+    Returns:
+        Path to the generated GIF file.
+    """
+    capture = cv2.VideoCapture(str(mp4_path))
+    frame_width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
+    capture.release()
+
+    gif_path = mp4_path.with_suffix(".gif")
+    palette_path = mp4_path.parent / "_palette.png"
+
+    logging.info("[4/4] Converting to GIF ...")
+    result_palette = subprocess.run(  # nosec B607
+        [
+            "ffmpeg",
+            "-y",
+            "-i",
+            str(mp4_path),
+            "-vf",
+            f"fps=10,scale={frame_width}:-1:flags=lanczos,palettegen=max_colors=128:stats_mode=diff",
+            "-update",
+            "1",
+            str(palette_path),
+        ],
+        capture_output=True,
+        text=True,
+    )
+    if result_palette.returncode != 0:
+        logging.warning("palettegen failed:\n%s", result_palette.stderr[-500:])
+
+    result_gif = subprocess.run(  # nosec B607
+        [
+            "ffmpeg",
+            "-y",
+            "-i",
+            str(mp4_path),
+            "-i",
+            str(palette_path),
+            "-filter_complex",
+            f"fps=10,scale={frame_width}:-1:flags=lanczos[v];[v][1:v]paletteuse=dither=bayer:bayer_scale=3",
+            str(gif_path),
+        ],
+        capture_output=True,
+        text=True,
+    )
+    if result_gif.returncode != 0:
+        logging.warning("GIF encode failed:\n%s", result_gif.stderr[-500:])
+
+    palette_path.unlink(missing_ok=True)
+    logging.info("   GIF written: %s", gif_path)
+    return gif_path
+
+
+def process_dataset(
+    repo_id: str,
+    episode: int,
+    camera_key: str | None,
+    output_dir: Path,
+    create_gif: bool = False,
+) -> Path | None:
+    """Full pipeline: download, extract metadata, composite progress, write output.
+
+    Args:
+        repo_id: HuggingFace dataset repository ID.
+        episode: Episode index.
+        camera_key: Camera key to use, or None for auto-selection.
+        output_dir: Directory to write output files.
+        create_gif: If True, also generate a GIF from the MP4.
+
+    Returns:
+        Path to the final output file, or None on failure.
+    """
+    safe_name = repo_id.replace("/", "_")
+    logging.info("Processing: %s  |  episode %d", repo_id, episode)
+
+    local_path = download_episode_metadata(repo_id, episode)
+    logging.info("   Local cache: %s", local_path)
+
+    episode_meta = load_episode_meta(local_path, episode, camera_key)
+    logging.info("   Episode meta: %s", episode_meta)
+
+    video_path = download_video_file(repo_id, local_path, episode_meta["video_rel"])
+
+    progress_data = load_progress_data(local_path, episode)
+    if progress_data is None:
+        logging.error("Could not load sarm_progress data. Skipping overlay.")
+        return None
+
+    logging.info("   Progress frames: %d", len(progress_data))
+
+    output_path = output_dir / f"{safe_name}_ep{episode}_progress.mp4"
+    final_path = composite_progress_video(
+        video_path=video_path,
+        from_timestamp=episode_meta["from_ts"],
+        to_timestamp=episode_meta["to_ts"],
+        progress_data=progress_data,
+        output_path=output_path,
+        fps=episode_meta["fps"],
+        task_name=episode_meta.get("task_name", ""),
+    )
+
+    if create_gif:
+        final_path = convert_mp4_to_gif(final_path)
+
+    logging.info("Done: %s", final_path)
+    return final_path
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Create MP4/GIF videos with sarm_progress overlay for dataset episodes."
+    )
+    parser.add_argument(
+        "--repo-id",
+        type=str,
+        required=True,
+        help="HuggingFace dataset repository ID (e.g. 'lerobot-data-collection/level2_final_quality3').",
+    )
+    parser.add_argument(
+        "--episode",
+        type=int,
+        required=True,
+        help="Episode index to visualize.",
+    )
+    parser.add_argument(
+        "--camera-key",
+        type=str,
+        default=None,
+        help="Camera observation key (e.g. 'observation.images.base'). Auto-selects first camera if omitted.",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=Path("progress_videos"),
+        help="Directory to write output files (default: ./progress_videos).",
+    )
+    parser.add_argument(
+        "--gif",
+        action="store_true",
+        help="Also generate a GIF from the MP4 output.",
+    )
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+
+    result = process_dataset(
+        repo_id=args.repo_id,
+        episode=args.episode,
+        camera_key=args.camera_key,
+        output_dir=args.output_dir,
+        create_gif=args.gif,
+    )
+
+    if result:
+        logging.info("Output: %s", result)
+
+
+if __name__ == "__main__":
+    main()

examples/hil/hil_utils.py

@@ -0,0 +1,228 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Shared utilities for Human-in-the-Loop data collection scripts."""
+
+import logging
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from lerobot.processor import (
+    IdentityProcessorStep,
+    RobotAction,
+    RobotObservation,
+    RobotProcessorPipeline,
+)
+from lerobot.processor.converters import (
+    observation_to_transition,
+    robot_action_observation_to_transition,
+    transition_to_observation,
+    transition_to_robot_action,
+)
+from lerobot.robots import Robot
+from lerobot.teleoperators import Teleoperator
+from lerobot.utils.control_utils import is_headless
+from lerobot.utils.robot_utils import precise_sleep
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class HILDatasetConfig:
+    repo_id: str
+    single_task: str
+    root: str | Path | None = None
+    fps: int = 30
+    episode_time_s: float = 120
+    num_episodes: int = 50
+    video: bool = True
+    push_to_hub: bool = True
+    private: bool = False
+    tags: list[str] | None = None
+    num_image_writer_processes: int = 0
+    num_image_writer_threads_per_camera: int = 4
+    video_encoding_batch_size: int = 1
+    vcodec: str = "auto"
+    streaming_encoding: bool = True
+    encoder_queue_maxsize: int = 30
+    encoder_threads: int | None = None
+    rename_map: dict[str, str] = field(default_factory=dict)
+
+
+def teleop_has_motor_control(teleop: Teleoperator) -> bool:
+    """Check if teleoperator has motor control capabilities."""
+    return all(hasattr(teleop, attr) for attr in ("enable_torque", "disable_torque", "write_goal_positions"))
+
+
+def teleop_disable_torque(teleop: Teleoperator) -> None:
+    """Disable teleop torque if supported."""
+    if hasattr(teleop, "disable_torque"):
+        teleop.disable_torque()
+
+
+def teleop_enable_torque(teleop: Teleoperator) -> None:
+    """Enable teleop torque if supported."""
+    if hasattr(teleop, "enable_torque"):
+        teleop.enable_torque()
+
+
+def teleop_smooth_move_to(teleop: Teleoperator, target_pos: dict, duration_s: float = 2.0, fps: int = 50):
+    """Smoothly move teleop to target position if motor control is available."""
+    if not teleop_has_motor_control(teleop):
+        logger.warning("Teleop does not support motor control - cannot mirror robot position")
+        return
+
+    teleop_enable_torque(teleop)
+    current = teleop.get_action()
+    steps = max(int(duration_s * fps), 1)
+
+    for step in range(steps + 1):
+        t = step / steps
+        interp = {}
+        for k in current:
+            if k in target_pos:
+                interp[k] = current[k] * (1 - t) + target_pos[k] * t
+            else:
+                interp[k] = current[k]
+        teleop.write_goal_positions(interp)
+        time.sleep(1 / fps)
+
+
+def init_keyboard_listener():
+    """Initialize keyboard listener with HIL controls."""
+    events = {
+        "exit_early": False,
+        "rerecord_episode": False,
+        "stop_recording": False,
+        "policy_paused": False,
+        "correction_active": False,
+        "resume_policy": False,
+        "in_reset": False,
+        "start_next_episode": False,
+    }
+
+    if is_headless():
+        logger.warning("Headless environment - keyboard controls unavailable")
+        return None, events
+
+    from pynput import keyboard
+
+    def on_press(key):
+        try:
+            if events["in_reset"]:
+                if key in [keyboard.Key.space, keyboard.Key.right]:
+                    logger.info("[HIL] Starting next episode...")
+                    events["start_next_episode"] = True
+                elif hasattr(key, "char") and key.char == "c":
+                    events["start_next_episode"] = True
+                elif key == keyboard.Key.esc:
+                    logger.info("[HIL] ESC - Stop recording, pushing to hub...")
+                    events["stop_recording"] = True
+                    events["start_next_episode"] = True
+            else:
+                if key == keyboard.Key.space:
+                    if not events["policy_paused"] and not events["correction_active"]:
+                        logger.info("[HIL] PAUSED - Press 'c' to take control or 'p' to resume policy")
+                        events["policy_paused"] = True
+                elif hasattr(key, "char") and key.char == "c":
+                    if events["policy_paused"] and not events["correction_active"]:
+                        logger.info("[HIL] Taking control...")
+                        events["start_next_episode"] = True
+                elif hasattr(key, "char") and key.char == "p":
+                    if events["policy_paused"] or events["correction_active"]:
+                        logger.info("[HIL] Resuming policy...")
+                        events["resume_policy"] = True
+                elif key == keyboard.Key.right:
+                    logger.info("[HIL] End episode")
+                    events["exit_early"] = True
+                elif key == keyboard.Key.left:
+                    logger.info("[HIL] Re-record episode")
+                    events["rerecord_episode"] = True
+                    events["exit_early"] = True
+                elif key == keyboard.Key.esc:
+                    logger.info("[HIL] ESC - Stop recording...")
+                    events["stop_recording"] = True
+                    events["exit_early"] = True
+        except Exception as e:
+            logger.info(f"Key error: {e}")
+
+    listener = keyboard.Listener(on_press=on_press)
+    listener.start()
+    return listener, events
+
+
+def make_identity_processors():
+    """Create identity processors for recording."""
+    teleop_proc = RobotProcessorPipeline[tuple[RobotAction, RobotObservation], RobotAction](
+        steps=[IdentityProcessorStep()],
+        to_transition=robot_action_observation_to_transition,
+        to_output=transition_to_robot_action,
+    )
+    obs_proc = RobotProcessorPipeline[RobotObservation, RobotObservation](
+        steps=[IdentityProcessorStep()],
+        to_transition=observation_to_transition,
+        to_output=transition_to_observation,
+    )
+    return teleop_proc, obs_proc
+
+
+def reset_loop(robot: Robot, teleop: Teleoperator, events: dict, fps: int):
+    """Reset period where human repositions environment."""
+    logger.info("[HIL] RESET")
+
+    events["in_reset"] = True
+    events["start_next_episode"] = False
+
+    obs = robot.get_observation()
+    robot_pos = {k: v for k, v in obs.items() if k.endswith(".pos") and k in robot.observation_features}
+    teleop_smooth_move_to(teleop, robot_pos, duration_s=2.0, fps=50)
+
+    logger.info("Press any key to enable teleoperation")
+    while not events["start_next_episode"] and not events["stop_recording"]:
+        precise_sleep(0.05)
+
+    if events["stop_recording"]:
+        return
+
+    events["start_next_episode"] = False
+    teleop_disable_torque(teleop)
+    logger.info("Teleop enabled - press any key to start episode")
+
+    while not events["start_next_episode"] and not events["stop_recording"]:
+        loop_start = time.perf_counter()
+        action = teleop.get_action()
+        robot.send_action(action)
+        precise_sleep(1 / fps - (time.perf_counter() - loop_start))
+
+    events["in_reset"] = False
+    events["start_next_episode"] = False
+    events["exit_early"] = False
+    events["policy_paused"] = False
+    events["correction_active"] = False
+    events["resume_policy"] = False
+
+
+def print_controls(rtc: bool = False):
+    """Print control instructions."""
+    mode = "Human-in-the-Loop Data Collection" + (" (RTC)" if rtc else "")
+    logger.info(
+        "%s\n  Controls:\n"
+        "    SPACE  - Pause policy\n"
+        "    c      - Take control\n"
+        "    p      - Resume policy after pause/correction\n"
+        "    →      - End episode\n"
+        "    ESC    - Stop and push to hub",
+        mode,
+    )

examples/rtc/eval_with_real_robot.py

@@ -69,15 +69,20 @@ Usage:
         --policy.path=lerobot-data-collection/folding_final \
         --robot.type=bi_openarm_follower \
         --robot.cameras='{left_wrist: {type: opencv, index_or_path: "/dev/video4", width: 1280, height: 720, fps: 30}, base: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}, right_wrist: {type: opencv, index_or_path: "/dev/video0", width: 1280, height: 720, fps: 30}}' \
-        --robot.left_arm_config.port=can1 \
+        --robot.left_arm_config.port=can0 \
         --robot.left_arm_config.side=left \
         --robot.left_arm_config.can_interface=socketcan \
-        --robot.right_arm_config.port=can0 \
+        --robot.left_arm_config.disable_torque_on_disconnect=true \
+        --robot.left_arm_config.max_relative_target=8.0 \
+        --robot.right_arm_config.port=can1 \
         --robot.right_arm_config.side=right \
         --robot.right_arm_config.can_interface=socketcan \
+        --robot.right_arm_config.disable_torque_on_disconnect=true \
+        --robot.right_arm_config.max_relative_target=8.0 \
         --task="Fold the T-shirt properly" \
         --fps=30 \
         --duration=2000 \
+        --interpolation_multiplier=3 \
         --rtc.enabled=true \
         --rtc.execution_horizon=20 \
         --rtc.max_guidance_weight=5.0 \
@@ -104,9 +109,7 @@ from lerobot.configs.policies import PreTrainedConfig
 from lerobot.configs.types import RTCAttentionSchedule
 from lerobot.datasets.feature_utils import build_dataset_frame, hw_to_dataset_features
 from lerobot.policies.factory import get_policy_class, make_pre_post_processors
-from lerobot.policies.rtc.action_queue import ActionQueue
-from lerobot.policies.rtc.configuration_rtc import RTCConfig
-from lerobot.policies.rtc.latency_tracker import LatencyTracker
+from lerobot.policies.rtc import ActionInterpolator, ActionQueue, LatencyTracker, RTCConfig
 from lerobot.processor import (
     NormalizerProcessorStep,
     RelativeActionsProcessorStep,
@@ -181,6 +184,7 @@ class RTCDemoConfig(HubMixin):
     # Demo parameters
     duration: float = 30.0  # Duration to run the demo (seconds)
     fps: float = 10.0  # Action execution frequency (Hz)
+    interpolation_multiplier: int = 1  # Control rate multiplier (1=off, 2=2x, 3=3x)
 
     # Compute device
     device: str | None = None  # Device to run on (cuda, cpu, auto)
@@ -461,20 +465,23 @@ def actor_control(
         action_keys = [k for k in robot.action_features() if k.endswith(".pos")]
 
         action_count = 0
-        action_interval = 1.0 / cfg.fps
+        interpolator = ActionInterpolator(multiplier=cfg.interpolation_multiplier)
+        action_interval = interpolator.get_control_interval(cfg.fps)
 
         while not shutdown_event.is_set():
             start_time = time.perf_counter()
 
-            # Try to get an action from the queue with timeout
-            action = action_queue.get()
+            if interpolator.needs_new_action():
+                new_action = action_queue.get()
+                if new_action is not None:
+                    interpolator.add(new_action.cpu())
 
+            action = interpolator.get()
             if action is not None:
                 action = action.cpu()
                 action_dict = {key: action[i].item() for i, key in enumerate(action_keys)}
                 action_processed = robot_action_processor((action_dict, None))
                 robot.send_action(action_processed)
-
                 action_count += 1
 
             dt_s = time.perf_counter() - start_time

examples/umi_pi0_relative_ee/evaluate.py

@@ -1,297 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-Inference script for a pi0 model trained with UMI-style relative EE actions
-on an OpenArm robot (single right arm, one wrist camera).
-
-Training dataset layout:
-  observation.images.cam0  [3, 720, 960]
-  action                   [x, y, z, ax, ay, az, proximal, distal]  (shape 8)
-
-The model uses ``derive_state_from_action=true``, so observation.state is
-derived from the action column during training.  At inference the state must
-be provided by the robot — this script uses FK to compute the current EE
-pose and gripper position, which it exposes as ``observation.state``.
-
-Pipeline:
-  1. Read arm joints from robot → FK → observation.state [x,y,z,ax,ay,az,prox,dist]
-  2. Read camera image → observation.images.cam0
-  3. pi0 preprocessor (loaded from checkpoint):
-     - DeriveStateFromActionStep: no-op at inference (state from robot)
-     - RelativeActionsProcessorStep: caches current state
-     - RelativeStateProcessorStep: buffers prev state, stacks [prev,cur],
-       subtracts current → velocity info, flattens
-     - NormalizerProcessorStep: normalizes
-  4. pi0 predicts relative action chunk (30 steps)
-  5. pi0 postprocessor: unnormalize, add cached state → absolute EE
-  6. IK: absolute EE [x,y,z,ax,ay,az] → arm joint targets
-  7. Gripper [proximal, distal] → gripper motor targets
-  8. Send to robot
-
-Usage:
-    python evaluate.py
-"""
-
-from __future__ import annotations
-
-import numpy as np
-from scipy.spatial.transform import Rotation
-
-from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.model.kinematics import RobotKinematics
-from lerobot.policies.factory import make_pre_post_processors
-from lerobot.policies.pi0.modeling_pi0 import PI0Policy
-from lerobot.processor import RelativeStateProcessorStep
-from lerobot.robots.openarm_follower import OpenArmFollower, OpenArmFollowerConfig
-from lerobot.scripts.lerobot_record import record_loop
-from lerobot.types import RobotAction, RobotObservation
-from lerobot.utils.control_utils import init_keyboard_listener
-from lerobot.utils.utils import log_say
-from lerobot.utils.visualization_utils import init_rerun
-
-# ---------------------------------------------------------------------------
-# Configuration — adapt these to your setup
-# ---------------------------------------------------------------------------
-
-FPS = 46
-EPISODE_TIME_SEC = 60
-TASK_DESCRIPTION = "red cube"
-
-HF_MODEL_ID = "pepijn223/grabette-umi-pi0"
-
-# Latency compensation: skip this many predicted action steps to account for
-# camera + inference + execution latency.  Formula: ceil(total_ms / (1000/FPS)).
-# At 46 FPS (~22ms/step) with ~150ms total latency: ceil(150/22) ≈ 7.
-# Start with 0 for a safe first test, then increase to match measured latency.
-LATENCY_SKIP_STEPS = 0
-
-URDF_PATH = "src/lerobot/robots/openarm_follower/urdf/openarm_bimanual_pybullet.urdf"
-URDF_EE_FRAME = "openarm_right_ee_target"
-
-IK_POSITION_WEIGHT = 1.0
-IK_ORIENTATION_WEIGHT = 1.0
-
-# ---------------------------------------------------------------------------
-# Dataset features for inference
-#
-# The training dataset has only observation.images.cam0 and action.
-# observation.state is derived from action during training
-# (derive_state_from_action=true) but must be supplied by the robot at
-# inference.  We define it here so build_dataset_frame can map FK output
-# to the right feature.
-# ---------------------------------------------------------------------------
-
-DATASET_FEATURES: dict = {
-    "observation.state": {
-        "dtype": "float32",
-        "shape": [8],
-        "names": ["x", "y", "z", "ax", "ay", "az", "proximal", "distal"],
-    },
-    "observation.images.cam0": {
-        "dtype": "video",
-        "shape": [3, 720, 960],
-        "names": ["channels", "height", "width"],
-        "info": {
-            "video.height": 720,
-            "video.width": 960,
-            "video.codec": "h264",
-            "video.pix_fmt": "yuv420p",
-            "video.is_depth_map": False,
-            "video.fps": FPS,
-            "video.channels": 3,
-            "has_audio": False,
-        },
-    },
-    "action": {
-        "dtype": "float32",
-        "shape": [8],
-        "names": ["x", "y", "z", "ax", "ay", "az", "proximal", "distal"],
-    },
-    "timestamp": {"dtype": "float32", "shape": [1], "names": None},
-    "frame_index": {"dtype": "int64", "shape": [1], "names": None},
-    "episode_index": {"dtype": "int64", "shape": [1], "names": None},
-    "index": {"dtype": "int64", "shape": [1], "names": None},
-    "task_index": {"dtype": "int64", "shape": [1], "names": None},
-}
-
-
-# ---------------------------------------------------------------------------
-# FK / IK callables
-# ---------------------------------------------------------------------------
-
-
-class JointsToEE:
-    """FK: raw robot observation → flat dict matching observation.state names.
-
-    Arm joint positions → EE pose [x,y,z,ax,ay,az] via forward kinematics.
-    Gripper motor positions → [proximal, distal].
-    Camera images pass through unchanged.
-    """
-
-    def __init__(self, kinematics: RobotKinematics, arm_motor_names: list[str]):
-        self.kin = kinematics
-        self.arm = arm_motor_names
-
-    def __call__(self, obs: RobotObservation) -> RobotObservation:
-        q = np.array([float(obs[f"{m}.pos"]) for m in self.arm])
-        t = self.kin.forward_kinematics(q)
-        rot = Rotation.from_matrix(t[:3, :3]).as_rotvec()
-
-        out: dict = {
-            "x": float(t[0, 3]),
-            "y": float(t[1, 3]),
-            "z": float(t[2, 3]),
-            "ax": float(rot[0]),
-            "ay": float(rot[1]),
-            "az": float(rot[2]),
-            "proximal": float(obs["proximal.pos"]),
-            "distal": float(obs["distal.pos"]),
-        }
-        for k, v in obs.items():
-            if not k.endswith((".pos", ".vel", ".torque")):
-                out[k] = v
-        return out
-
-
-class EEToJoints:
-    """IK: policy action dict → motor position dict for the robot.
-
-    Reads [x,y,z,ax,ay,az] from the action, runs IK for arm joint targets.
-    Passes [proximal, distal] as direct gripper position commands.
-    """
-
-    def __init__(
-        self,
-        kinematics: RobotKinematics,
-        arm_motor_names: list[str],
-        position_weight: float = 1.0,
-        orientation_weight: float = 1.0,
-    ):
-        self.kin = kinematics
-        self.arm = arm_motor_names
-        self.pw = position_weight
-        self.ow = orientation_weight
-        self.q_curr: np.ndarray | None = None
-
-    def __call__(self, args: tuple[RobotAction, RobotObservation]) -> RobotAction:
-        action, obs = args
-
-        q_raw = np.array([float(obs[f"{m}.pos"]) for m in self.arm])
-        if self.q_curr is None:
-            self.q_curr = q_raw
-
-        t_des = np.eye(4)
-        t_des[:3, :3] = Rotation.from_rotvec([action["ax"], action["ay"], action["az"]]).as_matrix()
-        t_des[:3, 3] = [action["x"], action["y"], action["z"]]
-
-        q_target = self.kin.inverse_kinematics(
-            self.q_curr, t_des, position_weight=self.pw, orientation_weight=self.ow
-        )
-        self.q_curr = q_target
-
-        out: dict = {f"{m}.pos": float(q_target[i]) for i, m in enumerate(self.arm)}
-        out["proximal.pos"] = float(action["proximal"])
-        out["distal.pos"] = float(action["distal"])
-        return out
-
-
-# ---------------------------------------------------------------------------
-# Main
-# ---------------------------------------------------------------------------
-
-
-def main():
-    camera_config = {
-        "cam0": OpenCVCameraConfig(index_or_path=0, width=960, height=720, fps=FPS),
-    }
-    robot_config = OpenArmFollowerConfig(
-        port="can0",
-        id="right_openarm",
-        side="right",
-        cameras=camera_config,
-        max_relative_target=8.0,
-        gripper_port="/dev/ttyUSB0",
-    )
-    robot = OpenArmFollower(robot_config)
-
-    policy = PI0Policy.from_pretrained(HF_MODEL_ID)
-    policy.config.latency_skip_steps = LATENCY_SKIP_STEPS
-
-    arm_motor_names = list(robot.bus.motors.keys())
-
-    kinematics = RobotKinematics(
-        urdf_path=URDF_PATH,
-        target_frame_name=URDF_EE_FRAME,
-        joint_names=arm_motor_names,
-    )
-
-    fk = JointsToEE(kinematics, arm_motor_names)
-    ik = EEToJoints(kinematics, arm_motor_names, IK_POSITION_WEIGHT, IK_ORIENTATION_WEIGHT)
-
-    dataset = LeRobotDataset.create(
-        repo_id="tmp/openarm_eval_scratch",
-        fps=FPS,
-        features=DATASET_FEATURES,
-        robot_type=robot.name,
-        use_videos=True,
-        image_writer_threads=4,
-    )
-
-    preprocessor, postprocessor = make_pre_post_processors(
-        policy_cfg=policy,
-        pretrained_path=HF_MODEL_ID,
-        dataset_stats=dataset.meta.stats,
-        preprocessor_overrides={"device_processor": {"device": str(policy.config.device)}},
-    )
-
-    relative_state_steps = [s for s in preprocessor.steps if isinstance(s, RelativeStateProcessorStep)]
-
-    robot.connect()
-
-    listener, events = init_keyboard_listener()
-    init_rerun(session_name="openarm_umi_pi0_relative_ee_evaluate")
-
-    try:
-        if not robot.is_connected:
-            raise ValueError("Robot is not connected!")
-
-        log_say("Starting policy execution")
-        for step in relative_state_steps:
-            step.reset()
-
-        record_loop(
-            robot=robot,
-            events=events,
-            fps=FPS,
-            policy=policy,
-            preprocessor=preprocessor,
-            postprocessor=postprocessor,
-            dataset=dataset,
-            control_time_s=EPISODE_TIME_SEC,
-            single_task=TASK_DESCRIPTION,
-            display_data=True,
-            robot_action_processor=ik,
-            robot_observation_processor=fk,
-        )
-    finally:
-        robot.disconnect()
-        listener.stop()
-
-
-if __name__ == "__main__":
-    main()

examples/umi_pi0_relative_ee/replay.py

@@ -1,113 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-Replay a dataset episode in EE frame using a browser-based URDF viewer.
-
-Extracts ``observation.pose`` from the dataset, saves a trajectory JSON file,
-then launches a local HTTP server and opens the replay viewer.  The trajectory
-is re-centered so frame 0 starts at the OpenArm ``openarm_right_ee_target``
-EE tip (zero-joint pose).
-
-Usage:
-    python replay.py
-    python replay.py --episode 3 --repo-id myuser/mydata
-"""
-
-from __future__ import annotations
-
-import argparse
-import http.server
-import json
-import os
-import threading
-import webbrowser
-from pathlib import Path
-
-VIEWER_DIR = Path(__file__).resolve().parents[2] / "src/lerobot/robots/openarm_follower/urdf"
-TRAJECTORY_FILENAME = "trajectory_ep0.json"
-
-
-def extract_trajectory(repo_id: str, episode: int, output_path: Path) -> dict:
-    from lerobot.datasets.lerobot_dataset import LeRobotDataset
-
-    dataset = LeRobotDataset(repo_id, episodes=[episode])
-    poses = dataset.select_columns("observation.pose")
-    actions = dataset.select_columns("action")
-
-    frames = []
-    for i in range(dataset.num_frames):
-        p = poses[i]["observation.pose"]
-        a = actions[i]["action"]
-        frames.append(
-            {
-                "x": float(p[0]),
-                "y": float(p[1]),
-                "z": float(p[2]),
-                "ax": float(p[3]),
-                "ay": float(p[4]),
-                "az": float(p[5]),
-                "proximal": float(a[0]),
-                "distal": float(a[1]),
-            }
-        )
-    payload = {"fps": dataset.fps, "num_frames": dataset.num_frames, "frames": frames}
-    with open(output_path, "w") as f:
-        json.dump(payload, f)
-    print(f"Extracted {dataset.num_frames} frames at {dataset.fps} FPS → {output_path}")
-    return payload
-
-
-# ---------------------------------------------------------------------------
-# Viewer mode
-# ---------------------------------------------------------------------------
-
-
-def serve_and_open(directory: Path, port: int = 8765):
-    os.chdir(directory)
-    handler = http.server.SimpleHTTPRequestHandler
-    httpd = http.server.HTTPServer(("", port), handler)
-    url = f"http://localhost:{port}/replay_viewer.html"
-    print(f"Serving at {url}")
-    threading.Thread(target=lambda: webbrowser.open(url), daemon=True).start()
-    try:
-        httpd.serve_forever()
-    except KeyboardInterrupt:
-        print("\nServer stopped.")
-        httpd.server_close()
-
-
-def run_viewer(args):
-    trajectory_path = VIEWER_DIR / TRAJECTORY_FILENAME
-    if not trajectory_path.exists() or args.force:
-        extract_trajectory(args.repo_id, args.episode, trajectory_path)
-    else:
-        print(f"Using cached trajectory at {trajectory_path}  (pass --force to re-extract)")
-    serve_and_open(VIEWER_DIR, args.port)
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Replay a dataset episode in EE frame (URDF viewer)")
-    parser.add_argument("--repo-id", default="glannuzel/grabette-dataset")
-    parser.add_argument("--episode", type=int, default=0)
-    parser.add_argument("--port", type=int, default=8765)
-    parser.add_argument("--force", action="store_true", help="Re-extract trajectory even if cached")
-    args = parser.parse_args()
-    run_viewer(args)
-
-
-if __name__ == "__main__":
-    main()

pyproject.toml

@@ -25,7 +25,7 @@ discord = "https://discord.gg/s3KuuzsPFb"
 
 [project]
 name = "lerobot"
-version = "0.5.1"
+version = "0.5.2"
 description = "🤗 LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch"
 dynamic = ["readme"]
 license = { text = "Apache-2.0" }
@@ -71,9 +71,9 @@ dependencies = [
     "cmake>=3.29.0.1,<4.2.0",
     "packaging>=24.2,<26.0",
 
-    "torch>=2.2.1,<2.11.0",
-    "torchcodec>=0.2.1,<0.11.0; sys_platform != 'win32' and (sys_platform != 'linux' or (platform_machine != 'aarch64' and platform_machine != 'arm64' and platform_machine != 'armv7l')) and (sys_platform != 'darwin' or platform_machine != 'x86_64')",
-    "torchvision>=0.21.0,<0.26.0",
+    "torch>=2.7,<2.11.0",
+    "torchcodec>=0.3.0,<0.11.0; sys_platform != 'win32' and (sys_platform != 'linux' or (platform_machine != 'aarch64' and platform_machine != 'arm64' and platform_machine != 'armv7l')) and (sys_platform != 'darwin' or platform_machine != 'x86_64')", # NOTE: Windows support starts at version 0.7 (needs torch==2.8), ffmpeg>=8 support starts at version 0.8.1 (needs torch==2.9), system-wide ffmpeg support starts at version 0.10 (needs torch==2.10).
+    "torchvision>=0.22.0,<0.26.0",
 
     "einops>=0.8.0,<0.9.0",
     "opencv-python-headless>=4.9.0,<4.14.0",
@@ -306,8 +306,7 @@ default.extend-ignore-identifiers-re = [
     "thw",
     "inpt",
     "ROBOTIS",
-    "OT_VALUE",
-    "metalness",
+    "OT_VALUE"
 ]
 
 # TODO: Uncomment when ready to use

src/lerobot/configs/policies.py

@@ -115,17 +115,6 @@ class PreTrainedConfig(draccus.ChoiceRegistry, HubMixin, abc.ABC):  # type: igno
     def reward_delta_indices(self) -> list | None:  # type: ignore[type-arg]    #TODO: No implementation
         raise NotImplementedError
 
-    @property
-    def state_delta_indices(self) -> list | None:  # type: ignore[type-arg]
-        """Delta indices specifically for observation.state.
-
-        When not None, overrides ``observation_delta_indices`` for the
-        ``observation.state`` key only. Useful for loading state history
-        (e.g. ``[-1, 0]`` for UMI-style relative proprioception) without
-        also loading multiple image timesteps.
-        """
-        return None
-
     @abc.abstractmethod
     def get_optimizer_preset(self) -> OptimizerConfig:
         raise NotImplementedError

src/lerobot/datasets/compute_stats.py

@@ -767,94 +767,3 @@ def compute_relative_action_stats(
     )
 
     return stats
-
-
-def compute_relative_state_stats(
-    hf_dataset,
-    features: dict,
-    state_obs_steps: int = 2,
-    exclude_joints: list[str] | None = None,
-    source_key: str = OBS_STATE,
-) -> dict[str, np.ndarray]:
-    """Compute normalization statistics for observation.state after relative conversion.
-
-    For UMI-style relative proprioception with ``state_obs_steps`` timesteps,
-    each state observation becomes a stack of offsets from the current timestep:
-    ``state[t-k] - state[t]`` for k in ``range(state_obs_steps-1, -1, -1)``.
-
-    The stats are computed over the flattened ``[state_obs_steps * state_dim]``
-    vector that the model actually sees after ``prepare_state`` flattening.
-
-    Args:
-        hf_dataset: The HuggingFace dataset with the source column and
-            "episode_index" columns.
-        features: Dataset feature metadata.
-        state_obs_steps: Number of observation timesteps (must be >= 2).
-        exclude_joints: State dimension names to keep absolute.
-        source_key: Column to read data from. Defaults to "observation.state".
-            When ``derive_state_from_action=True``, pass ``ACTION`` to read
-            from the action column instead.
-
-    Returns:
-        Statistics dict with keys "mean", "std", "min", "max", "q01", …, "q99".
-    """
-    from lerobot.processor.relative_action_processor import RelativeStateProcessorStep
-
-    if exclude_joints is None:
-        exclude_joints = []
-
-    state_dim = features[source_key]["shape"][0]
-    state_names = features.get(source_key, {}).get("names")
-    mask_step = RelativeStateProcessorStep(
-        enabled=True,
-        exclude_joints=exclude_joints,
-        state_names=state_names,
-    )
-    relative_mask = np.array(mask_step._build_mask(state_dim), dtype=np.float32)
-
-    logging.info(f"Loading data from '{source_key}' for relative state stats...")
-    all_states = np.array(hf_dataset[source_key], dtype=np.float32)
-    episode_indices = np.array(hf_dataset["episode_index"])
-
-    # Build all valid windows of length state_obs_steps within each episode
-    n = len(all_states)
-    if n < state_obs_steps:
-        raise ValueError(f"Dataset has {n} frames but state_obs_steps={state_obs_steps}")
-
-    max_start = n - state_obs_steps
-    starts = np.arange(max_start + 1)
-    valid = episode_indices[starts] == episode_indices[starts + state_obs_steps - 1]
-    valid_starts = starts[valid]
-
-    if len(valid_starts) == 0:
-        raise RuntimeError("No valid state windows found within single episodes")
-
-    offsets = np.arange(state_obs_steps)
-    mask_dim = len(relative_mask)
-
-    running_stats = RunningQuantileStats()
-
-    batch_size = 50_000
-    for i in range(0, len(valid_starts), batch_size):
-        batch_starts = valid_starts[i : i + batch_size]
-        frame_idx = batch_starts[:, None] + offsets[None, :]  # [N, state_obs_steps]
-        windows = all_states[frame_idx].copy()  # [N, state_obs_steps, state_dim]
-
-        # Subtract current (last) timestep from all timesteps for masked dims
-        current = windows[:, -1:, :]  # [N, 1, state_dim]
-        windows[:, :, :mask_dim] -= current[:, :, :mask_dim] * relative_mask[None, None, :]
-
-        # Flatten to [N, state_obs_steps * state_dim] (same as prepare_state)
-        flattened = windows.reshape(len(batch_starts), -1)
-        running_stats.update(flattened)
-
-    stats = running_stats.get_statistics()
-
-    excluded_dims = int(mask_dim - relative_mask.sum())
-    logging.info(
-        f"Relative state stats ({len(valid_starts)} windows, obs_steps={state_obs_steps}): "
-        f"relative_dims={int(relative_mask.sum())}/{mask_dim} (excluded={excluded_dims}), "
-        f"mean={np.abs(stats['mean']).mean():.4f}, std={stats['std'].mean():.4f}"
-    )
-
-    return stats

src/lerobot/datasets/dataset_tools.py

@@ -41,7 +41,6 @@ from lerobot.datasets.compute_stats import (
     aggregate_stats,
     compute_episode_stats,
     compute_relative_action_stats,
-    compute_relative_state_stats,
 )
 from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata
 from lerobot.datasets.io_utils import (
@@ -1545,10 +1544,6 @@ def recompute_stats(
     relative_exclude_joints: list[str] | None = None,
     chunk_size: int = 50,
     num_workers: int = 0,
-    relative_state: bool = False,
-    relative_exclude_state_joints: list[str] | None = None,
-    state_obs_steps: int = 2,
-    derive_state_from_action: bool = False,
 ) -> LeRobotDataset:
     """Recompute stats.json from scratch by iterating all episodes.
 
@@ -1566,22 +1561,10 @@ def recompute_stats(
             ``policy.chunk_size``. Only used when ``relative_action=True``.
         num_workers: Number of parallel threads for relative action stats computation.
             Values ≤1 mean single-threaded. Only used when ``relative_action=True``.
-        relative_state: If True, compute observation.state stats in relative space
-            (multi-timestep offsets from current). This matches the normalization
-            the model sees during training with ``use_relative_state=True``.
-        relative_exclude_state_joints: State dim names to exclude from relative conversion.
-        state_obs_steps: Number of observation timesteps for relative state stats.
-            Should match ``policy.state_obs_steps``. Only used when ``relative_state=True``.
-        derive_state_from_action: If True, compute relative state stats from the
-            action column instead of observation.state. Implies ``relative_state=True``
-            and ``state_obs_steps=2``.
 
     Returns:
         The same dataset with updated stats.
     """
-    if derive_state_from_action:
-        relative_state = True
-        state_obs_steps = 2
     features = dataset.meta.features
     meta_keys = {"index", "episode_index", "task_index", "frame_index", "timestamp"}
     numeric_features = {
@@ -1613,20 +1596,6 @@ def recompute_stats(
         )
         features_to_compute.pop(ACTION, None)
 
-    # When relative_state is enabled, compute state stats over the flattened
-    # multi-timestep relative representation (matching what the model sees).
-    relative_state_stats = None
-    if relative_state and (OBS_STATE in features or derive_state_from_action):
-        source_key = ACTION if derive_state_from_action else OBS_STATE
-        relative_state_stats = compute_relative_state_stats(
-            hf_dataset=dataset.hf_dataset,
-            features=features,
-            state_obs_steps=state_obs_steps,
-            exclude_joints=relative_exclude_state_joints,
-            source_key=source_key,
-        )
-        features_to_compute.pop(OBS_STATE, None)
-
     logging.info(f"Recomputing stats for features: {list(features_to_compute.keys())}")
 
     data_dir = dataset.root / DATA_DIR
@@ -1663,9 +1632,6 @@ def recompute_stats(
     if relative_action_stats is not None:
         new_stats[ACTION] = relative_action_stats
 
-    if relative_state_stats is not None:
-        new_stats[OBS_STATE] = relative_state_stats
-
     # Merge: keep existing stats for features we didn't recompute
     if dataset.meta.stats:
         for key, value in dataset.meta.stats.items():

src/lerobot/datasets/factory.py

@@ -25,7 +25,7 @@ from lerobot.datasets.lerobot_dataset import LeRobotDataset
 from lerobot.datasets.multi_dataset import MultiLeRobotDataset
 from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
 from lerobot.datasets.transforms import ImageTransforms
-from lerobot.utils.constants import ACTION, OBS_PREFIX, OBS_STATE, REWARD
+from lerobot.utils.constants import ACTION, OBS_PREFIX, REWARD
 
 IMAGENET_STATS = {
     "mean": [[[0.485]], [[0.456]], [[0.406]]],  # (c,1,1)
@@ -52,15 +52,12 @@ def resolve_delta_timestamps(
             returns `None` if the resulting dict is empty.
     """
     delta_timestamps = {}
-    state_delta = getattr(cfg, "state_delta_indices", None)
     for key in ds_meta.features:
         if key == REWARD and cfg.reward_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.reward_delta_indices]
         if key == ACTION and cfg.action_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.action_delta_indices]
-        if key == OBS_STATE and state_delta is not None:
-            delta_timestamps[key] = [i / ds_meta.fps for i in state_delta]
-        elif key.startswith(OBS_PREFIX) and cfg.observation_delta_indices is not None:
+        if key.startswith(OBS_PREFIX) and cfg.observation_delta_indices is not None:
             delta_timestamps[key] = [i / ds_meta.fps for i in cfg.observation_delta_indices]
 
     if len(delta_timestamps) == 0:

src/lerobot/datasets/lerobot_dataset.py

@@ -151,9 +151,11 @@ class LeRobotDataset(torch.utils.data.Dataset):
                 ``$HF_LEROBOT_HOME/hub``.
             episodes (list[int] | None, optional): If specified, this will only load episodes specified by
                 their episode_index in this list. Defaults to None.
-            image_transforms (Callable | None, optional): You can pass standard v2 image transforms from
-                torchvision.transforms.v2 here which will be applied to visual modalities (whether they come
-                from videos or images). Defaults to None.
+            image_transforms (Callable | None, optional):
+                Transform applied to visual modalities inside `__getitem__` after image decoding / tensor
+                conversion. This works for both image-backed and video-backed observations and can later be
+                updated with `set_image_transforms()` or cleared with `clear_image_transforms()`.
+                Defaults to None.
             delta_timestamps (dict[list[float]] | None, optional): _description_. Defaults to None.
             tolerance_s (float, optional): Tolerance in seconds used to ensure data timestamps are actually in
                 sync with the fps value. It is used at the init of the dataset to make sure that each
@@ -192,7 +194,8 @@ class LeRobotDataset(torch.utils.data.Dataset):
         super().__init__()
         self.repo_id = repo_id
         self._requested_root = Path(root) if root else None
-        self.image_transforms = image_transforms
+        self.reader = None
+        self.set_image_transforms(image_transforms)
         self.delta_timestamps = delta_timestamps
         self.episodes = episodes
         self.tolerance_s = tolerance_s
@@ -475,6 +478,18 @@ class LeRobotDataset(torch.utils.data.Dataset):
             f"}})"
         )
 
+    def set_image_transforms(self, image_transforms: Callable | None) -> None:
+        """Replace the transform applied to visual observations."""
+        if image_transforms is not None and not callable(image_transforms):
+            raise TypeError("image_transforms must be callable or None.")
+        self.image_transforms = image_transforms
+        if self.reader is not None:
+            self.reader._image_transforms = image_transforms
+
+    def clear_image_transforms(self) -> None:
+        """Remove the transform applied to visual observations."""
+        self.set_image_transforms(None)
+
     # ── Hub methods (stay on facade) ──────────────────────────────────
 
     def push_to_hub(

src/lerobot/datasets/multi_dataset.py

@@ -89,12 +89,24 @@ class MultiLeRobotDataset(torch.utils.data.Dataset):
                 )
                 self.disabled_features.update(extra_keys)
 
-        self.image_transforms = image_transforms
         self.delta_timestamps = delta_timestamps
         # TODO(rcadene, aliberts): We should not perform this aggregation for datasets
         # with multiple robots of different ranges. Instead we should have one normalization
         # per robot.
         self.stats = aggregate_stats([dataset.meta.stats for dataset in self._datasets])
+        self.set_image_transforms(image_transforms)
+
+    def set_image_transforms(self, image_transforms: Callable | None) -> None:
+        """Replace the transform for this dataset and its children."""
+        if image_transforms is not None and not callable(image_transforms):
+            raise TypeError("image_transforms must be callable or None.")
+        self.image_transforms = image_transforms
+        for dataset in getattr(self, "_datasets", []):
+            dataset.set_image_transforms(self.image_transforms)
+
+    def clear_image_transforms(self) -> None:
+        """Remove the transform from this dataset and its children."""
+        self.set_image_transforms(None)
 
     @property
     def repo_id_to_index(self):

src/lerobot/envs/configs.py

@@ -12,11 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import annotations
+
 import abc
+import importlib
 from dataclasses import dataclass, field, fields
 from typing import Any
 
 import draccus
+import gymnasium as gym
+from gymnasium.envs.registration import registry as gym_registry
 
 from lerobot.configs.types import FeatureType, PolicyFeature
 from lerobot.robots import RobotConfig
@@ -67,6 +72,49 @@ class EnvConfig(draccus.ChoiceRegistry, abc.ABC):
     def gym_kwargs(self) -> dict:
         raise NotImplementedError()
 
+    def create_envs(
+        self,
+        n_envs: int,
+        use_async_envs: bool = False,
+    ) -> dict[str, dict[int, gym.vector.VectorEnv]]:
+        """Create {suite: {task_id: VectorEnv}}.
+
+        Default: single-task env via gym.make(). Multi-task benchmarks override.
+        """
+        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+
+        if self.gym_id not in gym_registry:
+            print(f"gym id '{self.gym_id}' not found, attempting to import '{self.package_name}'...")
+            try:
+                importlib.import_module(self.package_name)
+            except ModuleNotFoundError as e:
+                raise ModuleNotFoundError(
+                    f"Package '{self.package_name}' required for env '{self.type}' not found. "
+                    f"Please install it or check PYTHONPATH."
+                ) from e
+
+            if self.gym_id not in gym_registry:
+                raise gym.error.NameNotFound(
+                    f"Environment '{self.gym_id}' not registered even after importing '{self.package_name}'."
+                )
+
+        def _make_one():
+            return gym.make(self.gym_id, disable_env_checker=self.disable_env_checker, **self.gym_kwargs)
+
+        try:
+            from gymnasium.vector import AutoresetMode
+
+            vec = env_cls([_make_one for _ in range(n_envs)], autoreset_mode=AutoresetMode.SAME_STEP)
+        except ImportError:
+            vec = env_cls([_make_one for _ in range(n_envs)])
+        return {self.type: {0: vec}}
+
+    def get_env_processors(self):
+        """Return (preprocessor, postprocessor) for this env. Default: identity."""
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        return PolicyProcessorPipeline(steps=[]), PolicyProcessorPipeline(steps=[])
+
 
 @dataclass
 class HubEnvConfig(EnvConfig):
@@ -338,6 +386,12 @@ class LiberoEnv(EnvConfig):
         else:
             raise ValueError(f"Unsupported obs_type: {self.obs_type}")
 
+        if self.camera_name_mapping is not None:
+            mapped_agentview = self.camera_name_mapping.get("agentview_image", "image")
+            mapped_eye_in_hand = self.camera_name_mapping.get("robot0_eye_in_hand_image", "image2")
+            self.features_map[LIBERO_KEY_PIXELS_AGENTVIEW] = f"{OBS_IMAGES}.{mapped_agentview}"
+            self.features_map[LIBERO_KEY_PIXELS_EYE_IN_HAND] = f"{OBS_IMAGES}.{mapped_eye_in_hand}"
+
     @property
     def gym_kwargs(self) -> dict:
         kwargs: dict[str, Any] = {"obs_type": self.obs_type, "render_mode": self.render_mode}
@@ -345,6 +399,33 @@ class LiberoEnv(EnvConfig):
             kwargs["task_ids"] = self.task_ids
         return kwargs
 
+    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+        from lerobot.envs.libero import create_libero_envs
+
+        if self.task is None:
+            raise ValueError("LiberoEnv requires a task to be specified")
+        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+        return create_libero_envs(
+            task=self.task,
+            n_envs=n_envs,
+            camera_name=self.camera_name,
+            init_states=self.init_states,
+            gym_kwargs=self.gym_kwargs,
+            env_cls=env_cls,
+            control_mode=self.control_mode,
+            episode_length=self.episode_length,
+            camera_name_mapping=self.camera_name_mapping,
+        )
+
+    def get_env_processors(self):
+        from lerobot.processor.env_processor import LiberoProcessorStep
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        return (
+            PolicyProcessorPipeline(steps=[LiberoProcessorStep()]),
+            PolicyProcessorPipeline(steps=[]),
+        )
+
 
 @EnvConfig.register_subclass("metaworld")
 @dataclass
@@ -387,6 +468,19 @@ class MetaworldEnv(EnvConfig):
             "render_mode": self.render_mode,
         }
 
+    def create_envs(self, n_envs: int, use_async_envs: bool = False):
+        from lerobot.envs.metaworld import create_metaworld_envs
+
+        if self.task is None:
+            raise ValueError("MetaWorld requires a task to be specified")
+        env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
+        return create_metaworld_envs(
+            task=self.task,
+            n_envs=n_envs,
+            gym_kwargs=self.gym_kwargs,
+            env_cls=env_cls,
+        )
+
 
 @EnvConfig.register_subclass("isaaclab_arena")
 @dataclass
@@ -454,3 +548,18 @@ class IsaaclabArenaEnv(HubEnvConfig):
     @property
     def gym_kwargs(self) -> dict:
         return {}
+
+    def get_env_processors(self):
+        from lerobot.processor.env_processor import IsaaclabArenaProcessorStep
+        from lerobot.processor.pipeline import PolicyProcessorPipeline
+
+        state_keys = tuple(k.strip() for k in (self.state_keys or "").split(",") if k.strip())
+        camera_keys = tuple(k.strip() for k in (self.camera_keys or "").split(",") if k.strip())
+        if not state_keys and not camera_keys:
+            raise ValueError("At least one of state_keys or camera_keys must be specified.")
+        return (
+            PolicyProcessorPipeline(
+                steps=[IsaaclabArenaProcessorStep(state_keys=state_keys, camera_keys=camera_keys)]
+            ),
+            PolicyProcessorPipeline(steps=[]),
+        )

src/lerobot/envs/factory.py

@@ -13,90 +13,46 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import importlib
+from __future__ import annotations
+
 from typing import Any
 
 import gymnasium as gym
-from gymnasium.envs.registration import registry as gym_registry
 
-from lerobot.configs.policies import PreTrainedConfig
-from lerobot.envs.configs import AlohaEnv, EnvConfig, HubEnvConfig, IsaaclabArenaEnv, LiberoEnv, PushtEnv
+from lerobot.envs.configs import EnvConfig, HubEnvConfig
 from lerobot.envs.utils import _call_make_env, _download_hub_file, _import_hub_module, _normalize_hub_result
-from lerobot.policies.xvla.configuration_xvla import XVLAConfig
-from lerobot.processor import ProcessorStep
-from lerobot.processor.env_processor import IsaaclabArenaProcessorStep, LiberoProcessorStep
-from lerobot.processor.pipeline import PolicyProcessorPipeline
 
 
 def make_env_config(env_type: str, **kwargs) -> EnvConfig:
-    if env_type == "aloha":
-        return AlohaEnv(**kwargs)
-    elif env_type == "pusht":
-        return PushtEnv(**kwargs)
-    elif env_type == "libero":
-        return LiberoEnv(**kwargs)
-    else:
-        raise ValueError(f"Policy type '{env_type}' is not available.")
+    try:
+        cls = EnvConfig.get_choice_class(env_type)
+    except KeyError as err:
+        raise ValueError(
+            f"Environment type '{env_type}' is not registered. "
+            f"Available: {list(EnvConfig.get_known_choices().keys())}"
+        ) from err
+    return cls(**kwargs)
 
 
 def make_env_pre_post_processors(
     env_cfg: EnvConfig,
-    policy_cfg: PreTrainedConfig,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-]:
+    policy_cfg: Any,
+) -> tuple[Any, Any]:
     """
     Create preprocessor and postprocessor pipelines for environment observations.
 
-    This function creates processor pipelines that transform raw environment
-    observations and actions. By default, it returns identity processors that do nothing.
-    For specific environments like LIBERO, it adds environment-specific processing steps.
-
-    Args:
-        env_cfg: The configuration of the environment.
-
-    Returns:
-        A tuple containing:
-            - preprocessor: Pipeline that processes environment observations
-            - postprocessor: Pipeline that processes environment outputs (currently identity)
+    Returns a tuple of (preprocessor, postprocessor). By default, delegates to
+    ``env_cfg.get_env_processors()``.  The XVLAConfig policy-specific override
+    stays here because it depends on the *policy* config, not the env config.
     """
-    # Preprocessor and Postprocessor steps are Identity for most environments
-    preprocessor_steps: list[ProcessorStep] = []
-    postprocessor_steps: list[ProcessorStep] = []
+    from lerobot.policies.xvla.configuration_xvla import XVLAConfig
+
     if isinstance(policy_cfg, XVLAConfig):
         from lerobot.policies.xvla.processor_xvla import make_xvla_libero_pre_post_processors
 
         return make_xvla_libero_pre_post_processors()
 
-    # For LIBERO environments, add the LiberoProcessorStep to preprocessor
-    if isinstance(env_cfg, LiberoEnv) or "libero" in env_cfg.type:
-        preprocessor_steps.append(LiberoProcessorStep())
-
-    # For Isaaclab Arena environments, add the IsaaclabArenaProcessorStep
-    if isinstance(env_cfg, IsaaclabArenaEnv) or "isaaclab_arena" in env_cfg.type:
-        # Parse comma-separated keys (handle None for state-based policies)
-        if env_cfg.state_keys:
-            state_keys = tuple(k.strip() for k in env_cfg.state_keys.split(",") if k.strip())
-        else:
-            state_keys = ()
-        if env_cfg.camera_keys:
-            camera_keys = tuple(k.strip() for k in env_cfg.camera_keys.split(",") if k.strip())
-        else:
-            camera_keys = ()
-        if not state_keys and not camera_keys:
-            raise ValueError("At least one of state_keys or camera_keys must be specified.")
-        preprocessor_steps.append(
-            IsaaclabArenaProcessorStep(
-                state_keys=state_keys,
-                camera_keys=camera_keys,
-            )
-        )
-
-    preprocessor = PolicyProcessorPipeline(steps=preprocessor_steps)
-    postprocessor = PolicyProcessorPipeline(steps=postprocessor_steps)
-
-    return preprocessor, postprocessor
+    return env_cfg.get_env_processors()
 
 
 def make_env(
@@ -163,57 +119,4 @@ def make_env(
     if n_envs < 1:
         raise ValueError("`n_envs` must be at least 1")
 
-    env_cls = gym.vector.AsyncVectorEnv if use_async_envs else gym.vector.SyncVectorEnv
-
-    if "libero" in cfg.type:
-        from lerobot.envs.libero import create_libero_envs
-
-        if cfg.task is None:
-            raise ValueError("LiberoEnv requires a task to be specified")
-
-        return create_libero_envs(
-            task=cfg.task,
-            n_envs=n_envs,
-            camera_name=cfg.camera_name,
-            init_states=cfg.init_states,
-            gym_kwargs=cfg.gym_kwargs,
-            env_cls=env_cls,
-            control_mode=cfg.control_mode,
-            episode_length=cfg.episode_length,
-        )
-    elif "metaworld" in cfg.type:
-        from lerobot.envs.metaworld import create_metaworld_envs
-
-        if cfg.task is None:
-            raise ValueError("MetaWorld requires a task to be specified")
-
-        return create_metaworld_envs(
-            task=cfg.task,
-            n_envs=n_envs,
-            gym_kwargs=cfg.gym_kwargs,
-            env_cls=env_cls,
-        )
-
-    if cfg.gym_id not in gym_registry:
-        print(f"gym id '{cfg.gym_id}' not found, attempting to import '{cfg.package_name}'...")
-        try:
-            importlib.import_module(cfg.package_name)
-        except ModuleNotFoundError as e:
-            raise ModuleNotFoundError(
-                f"Package '{cfg.package_name}' required for env '{cfg.type}' not found. "
-                f"Please install it or check PYTHONPATH."
-            ) from e
-
-        if cfg.gym_id not in gym_registry:
-            raise gym.error.NameNotFound(
-                f"Environment '{cfg.gym_id}' not registered even after importing '{cfg.package_name}'."
-            )
-
-    def _make_one():
-        return gym.make(cfg.gym_id, disable_env_checker=cfg.disable_env_checker, **(cfg.gym_kwargs or {}))
-
-    vec = env_cls([_make_one for _ in range(n_envs)], autoreset_mode=gym.vector.AutoresetMode.SAME_STEP)
-
-    # normalize to {suite: {task_id: vec_env}} for consistency
-    suite_name = cfg.type  # e.g., "pusht", "aloha"
-    return {suite_name: {0: vec}}
+    return cfg.create_envs(n_envs=n_envs, use_async_envs=use_async_envs)

src/lerobot/envs/libero.py

@@ -223,7 +223,8 @@ class LiberoEnv(gym.Env):
 
     def render(self):
         raw_obs = self._env.env._get_observations()
-        image = self._format_raw_obs(raw_obs)["pixels"]["image"]
+        pixels = self._format_raw_obs(raw_obs)["pixels"]
+        image = next(iter(pixels.values()))
         image = image[::-1, ::-1]  # flip both H and W for visualization
         return image
 
@@ -339,12 +340,6 @@ class LiberoEnv(gym.Env):
         )
         observation = self._format_raw_obs(raw_obs)
         if terminated:
-            info["final_info"] = {
-                "task": self.task,
-                "task_id": self.task_id,
-                "done": bool(done),
-                "is_success": bool(is_success),
-            }
             self.reset()
         truncated = False
         return observation, reward, terminated, truncated, info
@@ -364,6 +359,7 @@ def _make_env_fns(
     init_states: bool,
     gym_kwargs: Mapping[str, Any],
     control_mode: str,
+    camera_name_mapping: dict[str, str] | None = None,
 ) -> list[Callable[[], LiberoEnv]]:
     """Build n_envs factory callables for a single (suite, task_id)."""
 
@@ -379,6 +375,7 @@ def _make_env_fns(
             episode_index=episode_index,
             n_envs=n_envs,
             control_mode=control_mode,
+            camera_name_mapping=camera_name_mapping,
             **local_kwargs,
         )
 
@@ -400,6 +397,7 @@ def create_libero_envs(
     env_cls: Callable[[Sequence[Callable[[], Any]]], Any] | None = None,
     control_mode: str = "relative",
     episode_length: int | None = None,
+    camera_name_mapping: dict[str, str] | None = None,
 ) -> dict[str, dict[int, Any]]:
     """
     Create vectorized LIBERO environments with a consistent return shape.
@@ -449,6 +447,7 @@ def create_libero_envs(
                 init_states=init_states,
                 gym_kwargs=gym_kwargs,
                 control_mode=control_mode,
+                camera_name_mapping=camera_name_mapping,
             )
             out[suite_name][tid] = env_cls(fns)
             print(f"Built vec env | suite={suite_name} | task_id={tid} | n_envs={n_envs}")

src/lerobot/policies/multi_task_dit/README.md

@@ -1,37 +0,0 @@
-# Multitask DiT Policy
-
-## Citation
-
-If you use this work, please cite the following works:
-
-```bibtex
-@misc{jones2025multitaskditpolicy,
-  author = {Bryson Jones},
-  title = {Dissecting and Open-Sourcing Multitask Diffusion Transformer Policy},
-  year = {2025},
-  url = {https://brysonkjones.substack.com/p/dissecting-and-open-sourcing-multitask-diffusion-transformer-policy},
-  note = {Blog post}
-}
-```
-
-```bibtex
-@misc{trilbmteam2025carefulexaminationlargebehaviormodels,
-  author       = {TRI LBM Team},
-  title        = {A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation},
-  year         = {2025},
-  eprint       = {arXiv:2507.05331},
-  archivePrefix = {arXiv},
-  primaryClass = {cs.RO},
-  url          = {https://arxiv.org/abs/2507.05331}
-}
-```
-
-```bibtex
-@misc{bostondynamics2025largebehaviormodelsatlas,
-  author       = {Boston Dynamics and TRI Research Team},
-  title        = {Large Behavior Models and Atlas Find New Footing},
-  year         = {2025},
-  url          = {https://bostondynamics.com/blog/large-behavior-models-atlas-find-new-footing/},
-  note         = {Blog post}
-}
-```

src/lerobot/policies/multi_task_dit/README.md

@@ -0,0 +1 @@
+../../../../docs/source/policy_multi_task_dit_README.md

src/lerobot/policies/pi0/README.md

@@ -1,108 +0,0 @@
-# π₀ (pi0)
-
-This repository contains the Hugging Face port of **π₀**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
-It is designed as a **Vision-Language-Action model for general robot control**.
-
----
-
-## Model Overview
-
-| Feature              | π₀                                                     | π₀.₅                                      |
-| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
-| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
-| AdaRMS               | Not used                                               | Used in action expert                     |
-| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
-| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
-| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
-
----
-
-## Relative Actions
-
-π₀ supports training with **relative actions**, where the model learns relative offsets
-from the current robot state instead of absolute joint positions. This mirrors the
-relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
-
-### How it works
-
-1. **During preprocessing**, absolute actions are converted to relative offsets:
-   `relative = action - state` (for selected joints).
-2. The relative actions are normalized using statistics computed from the relative distribution.
-3. **During postprocessing**, predicted relative actions are converted back to absolute:
-   `absolute = relative + state`.
-
-Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
-
-### Configuration
-
-| Parameter                 | Type        | Default       | Description                                                      |
-| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
-| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
-| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
-| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
-
-### Training example
-
-```bash
-python -m lerobot.scripts.lerobot_train \
-  --policy.type=pi0 \
-  --dataset.repo_id=your_org/your_dataset \
-  --policy.use_relative_actions=true \
-  --policy.relative_exclude_joints='["gripper"]'
-```
-
-When `use_relative_actions=true`, the training script automatically:
-
-- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
-- Replaces the standard action stats with relative stats for normalization
-- Broadcasts these stats across all ranks in distributed training
-
-### Recomputing stats for an existing dataset
-
-If you want to precompute relative action stats offline, use `recompute_stats` from
-`lerobot.datasets.dataset_tools`:
-
-```python
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.datasets.dataset_tools import recompute_stats
-
-dataset = LeRobotDataset("your_org/your_dataset")
-dataset = recompute_stats(
-    dataset,
-    relative_action=True,
-    relative_exclude_joints=["gripper"],
-)
-```
-
----
-
-## Citation
-
-If you use this work, please cite both **OpenPI** and the π₀ paper:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{black2024pi0visionlanguageactionflowmodel,
-  title        = {π₀: A Vision-Language-Action Flow Model for General Robot Control},
-  author       = {Kevin Black and Noah Brown and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Lucy Xiaoyang Shi and James Tanner and Quan Vuong and Anna Walling and Haohuan Wang and Ury Zhilinsky},
-  year         = {2024},
-  eprint       = {2410.24164},
-  archivePrefix= {arXiv},
-  primaryClass = {cs.LG},
-  url          = {https://arxiv.org/abs/2410.24164},
-}
-```
-
----
-
-## License
-
-This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).

src/lerobot/policies/pi0/README.md

@@ -0,0 +1 @@
+../../../../docs/source/policy_pi0_README.md

src/lerobot/policies/pi0/configuration_pi0.py

@@ -57,28 +57,6 @@ class PI0Config(PreTrainedConfig):
     # Populated at runtime from dataset metadata by make_policy.
     action_feature_names: list[str] | None = None
 
-    # Relative state (UMI-style relative proprioception): converts multi-timestep
-    # observation.state to offsets from the current timestep, providing velocity info.
-    # Requires state_obs_steps >= 2. The flattened multi-timestep state is padded to
-    # max_state_dim, so ensure state_obs_steps * state_dim <= max_state_dim.
-    use_relative_state: bool = False
-    state_obs_steps: int = 1
-    relative_exclude_state_joints: list[str] = field(default_factory=list)
-    # Populated at runtime from dataset metadata by make_policy.
-    state_feature_names: list[str] | None = None
-
-    # Derive observation.state from the action column (UMI-style).
-    # When True, action_delta_indices loads one extra leading timestep [-1, 0, ..., chunk_size-1],
-    # DeriveStateFromActionStep extracts [action[t-1], action[t]] as a 2-step state,
-    # and strips the extra timestep from the action chunk.
-    # Implies use_relative_state=True and state_obs_steps=2.
-    derive_state_from_action: bool = False
-
-    # Latency compensation: skip this many steps from the start of each predicted
-    # action chunk during inference. E.g. at 10Hz with ~200ms total latency,
-    # latency_skip_steps=2 compensates for the delay.
-    latency_skip_steps: int = 0
-
     # Real-Time Chunking (RTC) configuration
     rtc_config: RTCConfig | None = None
 
@@ -128,10 +106,6 @@ class PI0Config(PreTrainedConfig):
     def __post_init__(self):
         super().__post_init__()
 
-        if self.derive_state_from_action:
-            self.use_relative_state = True
-            self.state_obs_steps = 2
-
         # Validate configuration
         if self.n_action_steps > self.chunk_size:
             raise ValueError(
@@ -147,13 +121,6 @@ class PI0Config(PreTrainedConfig):
         if self.dtype not in ["bfloat16", "float32"]:
             raise ValueError(f"Invalid dtype: {self.dtype}")
 
-        if self.use_relative_state and self.state_obs_steps < 2:
-            raise ValueError(
-                "use_relative_state requires state_obs_steps >= 2 "
-                f"(got {self.state_obs_steps}). Set state_obs_steps=2 for "
-                "UMI-style relative proprioception."
-            )
-
     def validate_features(self) -> None:
         """Validate and set up input/output features."""
         for i in range(self.empty_cameras):
@@ -199,16 +166,8 @@ class PI0Config(PreTrainedConfig):
     def observation_delta_indices(self) -> None:
         return None
 
-    @property
-    def state_delta_indices(self) -> list[int] | None:
-        if self.state_obs_steps >= 2:
-            return list(range(-(self.state_obs_steps - 1), 1))
-        return None
-
     @property
     def action_delta_indices(self) -> list:
-        if self.derive_state_from_action:
-            return [-1] + list(range(self.chunk_size))
         return list(range(self.chunk_size))
 
     @property

src/lerobot/policies/pi0/modeling_pi0.py

@@ -1230,11 +1230,8 @@ class PI0Policy(PreTrainedPolicy):
         return images, img_masks
 
     def prepare_state(self, batch):
-        """Flatten multi-timestep state and pad to max_state_dim."""
-        state = batch[OBS_STATE]
-        if state.ndim == 3:
-            state = state.flatten(start_dim=1)
-        state = pad_vector(state, self.config.max_state_dim)
+        """Pad state"""
+        state = pad_vector(batch[OBS_STATE], self.config.max_state_dim)
         return state
 
     def prepare_action(self, batch):
@@ -1253,8 +1250,7 @@ class PI0Policy(PreTrainedPolicy):
 
         # Action queue logic for n_action_steps > 1
         if len(self._action_queue) == 0:
-            skip = self.config.latency_skip_steps
-            actions = self.predict_action_chunk(batch)[:, skip : skip + self.config.n_action_steps]
+            actions = self.predict_action_chunk(batch)[:, : self.config.n_action_steps]
             # Transpose to get shape (n_action_steps, batch_size, action_dim)
             self._action_queue.extend(actions.transpose(0, 1))
 

src/lerobot/policies/pi0/processor_pi0.py

@@ -24,7 +24,6 @@ from lerobot.processor import (
     AbsoluteActionsProcessorStep,
     AddBatchDimensionProcessorStep,
     ComplementaryDataProcessorStep,
-    DeriveStateFromActionStep,
     DeviceProcessorStep,
     NormalizerProcessorStep,
     PolicyAction,
@@ -32,7 +31,6 @@ from lerobot.processor import (
     ProcessorStep,
     ProcessorStepRegistry,
     RelativeActionsProcessorStep,
-    RelativeStateProcessorStep,
     RenameObservationsProcessorStep,
     TokenizerProcessorStep,
     UnnormalizerProcessorStep,
@@ -130,25 +128,13 @@ def make_pi0_pre_post_processors(
         A tuple containing the configured pre-processor and post-processor pipelines.
     """
 
-    derive_state_step = DeriveStateFromActionStep(
-        enabled=getattr(config, "derive_state_from_action", False),
-    )
-
     relative_step = RelativeActionsProcessorStep(
         enabled=config.use_relative_actions,
         exclude_joints=getattr(config, "relative_exclude_joints", []),
         action_names=getattr(config, "action_feature_names", None),
     )
 
-    relative_state_step = RelativeStateProcessorStep(
-        enabled=getattr(config, "use_relative_state", False),
-        exclude_joints=getattr(config, "relative_exclude_state_joints", []),
-        state_names=getattr(config, "state_feature_names", None),
-    )
-
-    # Order: DeriveStateFromAction extracts state from the extended action chunk,
-    # then relative_action uses current state[t] for subtraction,
-    # then relative_state converts the multi-timestep state to offsets.
+    # OpenPI order: raw → relative → normalize → model → unnormalize → absolute
     input_steps: list[ProcessorStep] = [
         RenameObservationsProcessorStep(rename_map={}),  # To mimic the same processor as pretrained one
         AddBatchDimensionProcessorStep(),
@@ -160,9 +146,7 @@ def make_pi0_pre_post_processors(
             padding="max_length",
         ),
         DeviceProcessorStep(device=config.device),
-        derive_state_step,
         relative_step,
-        relative_state_step,
         NormalizerProcessorStep(
             features={**config.input_features, **config.output_features},
             norm_map=config.normalization_mapping,

src/lerobot/policies/pi05/README.md

@@ -1,91 +0,0 @@
-# π₀.₅ (pi05)
-
-This repository contains the Hugging Face port of **π₀.₅**, adapted from [OpenPI](https://github.com/Physical-Intelligence/openpi) by the Physical Intelligence.
-It is designed as a **Vision-Language-Action model with open-world generalization**.
-
----
-
-## Model Overview
-
-| Feature              | π₀                                                     | π₀.₅                                      |
-| -------------------- | ------------------------------------------------------ | ----------------------------------------- |
-| Time Conditioning    | Concatenates time with actions via `action_time_mlp_*` | Uses `time_mlp_*` for AdaRMS conditioning |
-| AdaRMS               | Not used                                               | Used in action expert                     |
-| Tokenizer Length     | 48 tokens                                              | 200 tokens                                |
-| Discrete State Input | False (Uses `state_proj` layer)                        | True                                      |
-| Parameter Count      | Higher (includes state embedding)                      | Lower (no state embedding)                |
-
----
-
-## Relative Actions
-
-π₀.₅ supports training with **relative actions**, where the model learns relative offsets
-from the current robot state instead of absolute joint positions. This mirrors the
-relative-action transform in OpenPI (`DeltaActions`) and can improve performance.
-
-### How it works
-
-1. **During preprocessing**, absolute actions are converted to relative offsets:
-   `relative = action - state` (for selected joints).
-2. The relative actions are normalized using statistics computed from the relative distribution.
-3. **During postprocessing**, predicted relative actions are converted back to absolute:
-   `absolute = relative + state`.
-
-Joints listed in `relative_exclude_joints` (e.g., gripper) are kept absolute.
-
-### Configuration
-
-| Parameter                 | Type        | Default       | Description                                                      |
-| ------------------------- | ----------- | ------------- | ---------------------------------------------------------------- |
-| `use_relative_actions`    | `bool`      | `False`       | Enable relative-action training                                  |
-| `relative_exclude_joints` | `list[str]` | `["gripper"]` | Joint names to keep absolute (matched by substring)              |
-| `action_feature_names`    | `list[str]` | `None`        | Auto-populated from dataset metadata at runtime by `make_policy` |
-
-### Training example
-
-```bash
-python -m lerobot.scripts.lerobot_train \
-  --policy.type=pi05 \
-  --dataset.repo_id=your_org/your_dataset \
-  --policy.use_relative_actions=true \
-  --policy.relative_exclude_joints='["gripper"]'
-```
-
-When `use_relative_actions=true`, the training script automatically:
-
-- Computes relative action statistics from the dataset (sampled chunk-level relative actions)
-- Replaces the standard action stats with relative stats for normalization
-- Broadcasts these stats across all ranks in distributed training
-
----
-
-## Citation
-
-If you use this work, please cite both **OpenPI** and the π₀.₅ paper:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{intelligence2025pi05visionlanguageactionmodelopenworld,
-  title        = {π₀.₅: a Vision-Language-Action Model with Open-World Generalization},
-  author       = {Physical Intelligence and Kevin Black and Noah Brown and James Darpinian and Karan Dhabalia and Danny Driess and Adnan Esmail and Michael Equi and Chelsea Finn and Niccolo Fusai and Manuel Y. Galliker and Dibya Ghosh and Lachy Groom and Karol Hausman and Brian Ichter and Szymon Jakubczak and Tim Jones and Liyiming Ke and Devin LeBlanc and Sergey Levine and Adrian Li-Bell and Mohith Mothukuri and Suraj Nair and Karl Pertsch and Allen Z. Ren and Lucy Xiaoyang Shi and Laura Smith and Jost Tobias Springenberg and Kyle Stachowicz and James Tanner and Quan Vuong and Homer Walke and Anna Walling and Haohuan Wang and Lili Yu and Ury Zhilinsky},
-  year         = {2025},
-  eprint       = {2504.16054},
-  archivePrefix= {arXiv},
-  primaryClass = {cs.LG},
-  url          = {https://arxiv.org/abs/2504.16054},
-}
-```
-
----
-
-## License
-
-This port follows the **Apache 2.0 License**, consistent with the original [OpenPI repository](https://github.com/Physical-Intelligence/openpi).

src/lerobot/policies/pi05/README.md

@@ -0,0 +1 @@
+../../../../docs/source/policy_pi05_README.md

src/lerobot/policies/pi0_fast/processor_pi0_fast.py

@@ -136,7 +136,7 @@ def make_pi0_fast_pre_post_processors(
     # Pi0Fast order: relative → normalize → tokenize → model → unnormalize → absolute
     # This matches pi0/pi0.5: RelativeActionsProcessorStep runs first on raw absolute actions,
     # caching the raw state. NormalizerProcessorStep then normalizes the raw relative actions,
-    # so the normalizer (and action tokenizer) sees delta values, relative stats are required.
+    # so the normalizer (and action tokenizer) sees delta values — relative stats are required.
     # NOTE: RelativeActionsProcessorStep only modifies the action in the transition; it reads
     # state from the observation but does not change it. NormalizerProcessorStep still runs
     # before Pi0FastPrepareStateAndLanguageTokenizerProcessorStep, so the state tokenizer

src/lerobot/policies/rtc/README.md

@@ -1,38 +0,0 @@
-# Real-Time Chunking (RTC)
-
-This module contains the LeRobot implementation of **Real-Time Chunking (RTC)**, an inference-time technique for flow-matching based policies.
-
-**Note**: RTC is not a policy itself, but rather an inference enhancement that works with flow-matching based policies including [π₀](../pi0/), [π₀.₅](../pi05/), and [SmolVLA](../smolvla/).
-
----
-
-## Citation
-
-If you use Real-Time Chunking in your work, please cite:
-
-```bibtex
-@misc{openpi2024,
-  author       = {Physical Intelligence Lab},
-  title        = {OpenPI: PyTorch Implementation of π0 and π0.5 Policies},
-  year         = {2024},
-  publisher    = {GitHub},
-  howpublished = {\url{https://github.com/Physical-Intelligence/openpi}},
-  license      = {Apache-2.0}
-}
-
-@misc{black2025realtimeexecutionactionchunking,
-      title={Real-Time Execution of Action Chunking Flow Policies},
-      author={Kevin Black and Manuel Y. Galliker and Sergey Levine},
-      year={2025},
-      eprint={2506.07339},
-      archivePrefix={arXiv},
-      primaryClass={cs.RO},
-      url={https://arxiv.org/abs/2506.07339},
-}
-```
-
----
-
-## License
-
-This implementation follows the **Apache 2.0 License**, consistent with the LeRobot project.

src/lerobot/policies/rtc/README.md

@@ -0,0 +1 @@
+../../../../docs/source/policy_rtc_README.md

src/lerobot/policies/rtc/__init__.py

@@ -0,0 +1,29 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Real-Time Chunking (RTC) utilities for action-chunking policies."""
+
+from lerobot.policies.rtc.action_interpolator import ActionInterpolator
+from lerobot.policies.rtc.action_queue import ActionQueue
+from lerobot.policies.rtc.configuration_rtc import RTCConfig
+from lerobot.policies.rtc.latency_tracker import LatencyTracker
+from lerobot.policies.rtc.modeling_rtc import RTCProcessor
+
+__all__ = [
+    "ActionInterpolator",
+    "ActionQueue",
+    "LatencyTracker",
+    "RTCConfig",
+    "RTCProcessor",
+]

src/lerobot/policies/rtc/action_interpolator.py

@@ -0,0 +1,116 @@
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Action interpolation for smoother robot control.
+
+Provides configurable Nx control rate by interpolating between consecutive actions.
+Useful with RTC and action-chunking policies to reduce jerkiness.
+"""
+
+from torch import Tensor
+
+
+class ActionInterpolator:
+    """Interpolates between consecutive actions for smoother control.
+
+    When enabled with multiplier N, produces N actions per policy action
+    by linearly interpolating between the previous and current action.
+
+    Example with multiplier=3:
+        prev_action -> [1/3 interpolated, 2/3 interpolated, current_action]
+
+    This effectively multiplies the control rate for smoother motion.
+
+    Usage:
+        interpolator = ActionInterpolator(multiplier=2)  # 2x control rate
+
+        # In control loop:
+        if interpolator.needs_new_action():
+            new_action = queue.get()
+            if new_action:
+                interpolator.add(new_action.cpu())
+
+        action = interpolator.get()
+        if action:
+            robot.send_action(action)
+    """
+
+    def __init__(self, multiplier: int = 1):
+        """Initialize the interpolator.
+
+        Args:
+            multiplier: Control rate multiplier (1 = no interpolation, 2 = 2x, 3 = 3x, etc.)
+        """
+        if multiplier < 1:
+            raise ValueError(f"multiplier must be >= 1, got {multiplier}")
+        self.multiplier = multiplier
+        self._prev: Tensor | None = None
+        self._buffer: list[Tensor] = []
+        self._idx = 0
+
+    @property
+    def enabled(self) -> bool:
+        """Whether interpolation is active (multiplier > 1)."""
+        return self.multiplier > 1
+
+    def reset(self):
+        """Reset interpolation state (call between episodes)."""
+        self._prev = None
+        self._buffer = []
+        self._idx = 0
+
+    def needs_new_action(self) -> bool:
+        """Check if a new action is needed from the queue."""
+        return self._idx >= len(self._buffer)
+
+    def add(self, action: Tensor) -> None:
+        """Add a new action and compute interpolated sequence.
+
+        Args:
+            action: New action tensor from policy/queue (already on CPU).
+        """
+        if self.multiplier > 1 and self._prev is not None:
+            self._buffer = []
+            for i in range(1, self.multiplier + 1):
+                t = i / self.multiplier
+                interp = self._prev + t * (action - self._prev)
+                self._buffer.append(interp)
+        else:
+            # First step: no previous action yet, so run at base FPS without interpolation.
+            self._buffer = [action.clone()]
+        self._prev = action.clone()
+        self._idx = 0
+
+    def get(self) -> Tensor | None:
+        """Get the next interpolated action.
+
+        Returns:
+            Next action tensor, or None if buffer is exhausted.
+        """
+        if self._idx >= len(self._buffer):
+            return None
+        action = self._buffer[self._idx]
+        self._idx += 1
+        return action
+
+    def get_control_interval(self, fps: float) -> float:
+        """Get the control interval based on interpolation multiplier.
+
+        Args:
+            fps: Base frames per second.
+
+        Returns:
+            Control interval in seconds (divided by multiplier).
+        """
+        return 1.0 / (fps * self.multiplier)

src/lerobot/policies/rtc/action_queue.py

@@ -79,6 +79,13 @@ class ActionQueue:
             self.last_index += 1
             return action.clone()
 
+    def clear(self) -> None:
+        """Clear queued actions and reset consumption index."""
+        with self.lock:
+            self.queue = None
+            self.original_queue = None
+            self.last_index = 0
+
     def qsize(self) -> int:
         """Get the number of remaining actions in the queue.
 
@@ -123,14 +130,26 @@ class ActionQueue:
         with self.lock:
             if self.original_queue is None:
                 return None
-            return self.original_queue[self.last_index :]
+            return self.original_queue[self.last_index :].clone()
+
+    def get_processed_left_over(self) -> Tensor | None:
+        """Get leftover processed actions (the actions currently executed by the robot).
+
+        Returns:
+            Tensor | None: Remaining processed actions (remaining_steps, action_dim),
+                or None if no processed queue exists.
+        """
+        with self.lock:
+            if self.queue is None:
+                return None
+            return self.queue[self.last_index :].clone()
 
     def merge(
         self,
         original_actions: Tensor,
         processed_actions: Tensor,
         real_delay: int,
-        action_index_before_inference: int | None = 0,
+        action_index_before_inference: int | None = None,
     ):
         """Merge new actions into the queue.
 
@@ -145,10 +164,10 @@ class ActionQueue:
             action_index_before_inference: Index before inference started, for validation.
         """
         with self.lock:
-            self._check_delays(real_delay, action_index_before_inference)
+            delay = self._check_and_resolve_delays(real_delay, action_index_before_inference)
 
             if self.cfg.enabled:
-                self._replace_actions_queue(original_actions, processed_actions, real_delay)
+                self._replace_actions_queue(original_actions, processed_actions, delay)
                 return
 
             self._append_actions_queue(original_actions, processed_actions)
@@ -164,12 +183,13 @@ class ActionQueue:
             processed_actions: Post-processed actions for robot.
             real_delay: Number of time steps to skip due to inference delay.
         """
-        self.original_queue = original_actions[real_delay:].clone()
-        self.queue = processed_actions[real_delay:].clone()
+        clamped_delay = max(0, min(real_delay, len(original_actions), len(processed_actions)))
+        self.original_queue = original_actions[clamped_delay:].clone()
+        self.queue = processed_actions[clamped_delay:].clone()
 
         logger.debug(f"original_actions shape: {self.original_queue.shape}")
         logger.debug(f"processed_actions shape: {self.queue.shape}")
-        logger.debug(f"real_delay: {real_delay}")
+        logger.debug(f"real_delay: {real_delay}, clamped_delay: {clamped_delay}")
 
         self.last_index = 0
 
@@ -196,7 +216,9 @@ class ActionQueue:
 
         self.last_index = 0
 
-    def _check_delays(self, real_delay: int, action_index_before_inference: int | None = None):
+    def _check_and_resolve_delays(
+        self, real_delay: int, action_index_before_inference: int | None = None
+    ) -> int:
         """Validate that computed delays match expectations.
 
         Compares the delay computed from inference latency with the actual
@@ -205,15 +227,20 @@ class ActionQueue:
         Args:
             real_delay: Delay computed from inference latency.
             action_index_before_inference: Action index when inference started.
-        """
-        if action_index_before_inference is None:
-            return
 
-        indexes_diff = self.last_index - action_index_before_inference
-        if indexes_diff != real_delay:
-            # Let's check that action index difference (real delay calculated based on action queue)
-            # is the same as delay calculated based on inference latency
-            logger.warning(
-                f"[ACTION_QUEUE] Indexes diff is not equal to real delay. "
-                f"Indexes diff: {indexes_diff}, real delay: {real_delay}"
-            )
+        Returns:
+            int: Delay to use.
+        """
+        effective_delay = max(0, real_delay)
+
+        if action_index_before_inference is not None:
+            indexes_diff = max(0, self.last_index - action_index_before_inference)
+            if indexes_diff != real_delay:
+                logger.warning(
+                    "Indexes diff is not equal to real delay. indexes_diff=%d, real_delay=%d",
+                    indexes_diff,
+                    real_delay,
+                )
+                return real_delay
+
+        return effective_delay

src/lerobot/policies/sarm/README.md

@@ -1,14 +0,0 @@
-## Paper
-
-https://arxiv.org/abs/2509.25358
-
-## Citation
-
-```bibtex
-@article{chen2025sarm,
-  title={SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation},
-  author={Chen, Qianzhong and Yu, Justin and Schwager, Mac and Abbeel, Pieter and Shentu, Yide and Wu, Philipp},
-  journal={arXiv preprint arXiv:2509.25358},
-  year={2025}
-}
-```

src/lerobot/policies/sarm/README.md

@@ -0,0 +1 @@
+../../../../docs/source/policy_sarm_README.md

src/lerobot/processor/__init__.py

@@ -77,12 +77,9 @@ from .policy_robot_bridge import (
 )
 from .relative_action_processor import (
     AbsoluteActionsProcessorStep,
-    DeriveStateFromActionStep,
     RelativeActionsProcessorStep,
-    RelativeStateProcessorStep,
     to_absolute_actions,
     to_relative_actions,
-    to_relative_state,
 )
 from .rename_processor import RenameObservationsProcessorStep
 from .tokenizer_processor import ActionTokenizerProcessorStep, TokenizerProcessorStep
@@ -110,9 +107,7 @@ __all__ = [
     "make_default_robot_action_processor",
     "make_default_robot_observation_processor",
     "AbsoluteActionsProcessorStep",
-    "DeriveStateFromActionStep",
     "RelativeActionsProcessorStep",
-    "RelativeStateProcessorStep",
     "MapDeltaActionToRobotActionStep",
     "MapTensorToDeltaActionDictStep",
     "NormalizerProcessorStep",
@@ -144,7 +139,6 @@ __all__ = [
     "TruncatedProcessorStep",
     "to_absolute_actions",
     "to_relative_actions",
-    "to_relative_state",
     "UnnormalizerProcessorStep",
     "VanillaObservationProcessorStep",
 ]

src/lerobot/processor/relative_action_processor.py

@@ -30,13 +30,10 @@ from .pipeline import ProcessorStep, ProcessorStepRegistry
 __all__ = [
     "MapDeltaActionToRobotActionStep",
     "MapTensorToDeltaActionDictStep",
-    "DeriveStateFromActionStep",
     "RelativeActionsProcessorStep",
     "AbsoluteActionsProcessorStep",
-    "RelativeStateProcessorStep",
     "to_relative_actions",
     "to_absolute_actions",
-    "to_relative_state",
 ]
 
 
@@ -84,41 +81,6 @@ def to_absolute_actions(actions: Tensor, state: Tensor, mask: Sequence[bool]) ->
     return actions
 
 
-@ProcessorStepRegistry.register("derive_state_from_action_processor")
-@dataclass
-class DeriveStateFromActionStep(ProcessorStep):
-    """Derives 2-step observation.state from the action chunk (UMI-style).
-
-    Expects action with one extra leading timestep: [B, chunk_size+1, D]
-    from action_delta_indices = [-1, 0, 1, ..., chunk_size-1].
-    Extracts [action[t-1], action[t]] as state and strips the extra timestep.
-    No-op during inference (state comes from robot).
-    """
-
-    enabled: bool = False
-
-    def __call__(self, transition: EnvTransition) -> EnvTransition:
-        if not self.enabled:
-            return transition
-        action = transition.get(TransitionKey.ACTION)
-        if action is None or action.ndim < 3:
-            return transition
-        new_transition = transition.copy()
-        new_obs = dict(new_transition.get(TransitionKey.OBSERVATION, {}))
-        new_obs[OBS_STATE] = action[..., :2, :]
-        new_transition[TransitionKey.ACTION] = action[..., 1:, :]
-        new_transition[TransitionKey.OBSERVATION] = new_obs
-        return new_transition
-
-    def get_config(self) -> dict[str, Any]:
-        return {"enabled": self.enabled}
-
-    def transform_features(
-        self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
-    ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        return features
-
-
 @ProcessorStepRegistry.register("delta_actions_processor")
 @dataclass
 class RelativeActionsProcessorStep(ProcessorStep):
@@ -162,14 +124,7 @@ class RelativeActionsProcessorStep(ProcessorStep):
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
         observation = transition.get(TransitionKey.OBSERVATION, {})
-        raw_state = observation.get(OBS_STATE) if observation else None
-
-        # When state_delta_indices loads multi-timestep state [B, n_obs, D],
-        # use only the current (last) timestep for relative action conversion.
-        if raw_state is not None:
-            state = raw_state[..., -1, :] if raw_state.ndim >= 3 else raw_state
-        else:
-            state = None
+        state = observation.get(OBS_STATE) if observation else None
 
         # Always cache state for the paired AbsoluteActionsProcessorStep
         if state is not None:
@@ -200,120 +155,6 @@ class RelativeActionsProcessorStep(ProcessorStep):
         return features
 
 
-def to_relative_state(state: Tensor, mask: Sequence[bool]) -> Tensor:
-    """Convert multi-timestep absolute state to relative (offset from current timestep).
-
-    Each timestep becomes: ``state[..., t, :] - state[..., -1, :]`` for masked dims.
-    The last (current) timestep becomes zeros for masked dims.
-
-    Args:
-        state: (..., n_obs, state_dim) — last timestep is the reference (current).
-        mask: Which dims to convert. Can be shorter than state_dim.
-    """
-    mask_t = torch.tensor(mask, dtype=state.dtype, device=state.device)
-    dims = mask_t.shape[0]
-    current = state[..., -1:, :]  # (..., 1, state_dim)
-    state = state.clone()
-    state[..., :dims] -= current[..., :dims] * mask_t
-    return state
-
-
-@ProcessorStepRegistry.register("relative_state_processor")
-@dataclass
-class RelativeStateProcessorStep(ProcessorStep):
-    """Converts observation.state to relative (offset from current timestep).
-
-    UMI-style relative proprioception: each state timestep is expressed as
-    an offset from the current EE pose, providing velocity information.
-
-    During training (multi-timestep input from ``state_delta_indices``):
-        ``state[..., t, :] -= state[..., -1, :]`` — subtract current from all.
-
-    During inference (single timestep): buffers the previous state and stacks
-    ``[previous, current]`` before applying the relative conversion, producing
-    the same ``[n_obs, D]`` shape the model expects.
-
-    Attributes:
-        enabled: Whether to apply the relative conversion.
-        exclude_joints: Joint/dim names to keep absolute.
-        state_names: State dimension names from dataset metadata.
-    """
-
-    enabled: bool = False
-    exclude_joints: list[str] = field(default_factory=list)
-    state_names: list[str] | None = None
-    _previous_state: torch.Tensor | None = field(default=None, init=False, repr=False)
-
-    def _build_mask(self, state_dim: int) -> list[bool]:
-        if not self.exclude_joints or self.state_names is None:
-            return [True] * state_dim
-
-        exclude_tokens = [str(name).lower() for name in self.exclude_joints if name]
-        if not exclude_tokens:
-            return [True] * state_dim
-
-        mask = []
-        for name in self.state_names[:state_dim]:
-            state_name = str(name).lower()
-            is_excluded = any(token == state_name or token in state_name for token in exclude_tokens)
-            mask.append(not is_excluded)
-
-        if len(mask) < state_dim:
-            mask.extend([True] * (state_dim - len(mask)))
-
-        return mask
-
-    def __call__(self, transition: EnvTransition) -> EnvTransition:
-        if not self.enabled:
-            return transition
-
-        observation = transition.get(TransitionKey.OBSERVATION, {})
-        state = observation.get(OBS_STATE) if observation else None
-
-        if state is None:
-            return transition
-
-        new_transition = transition.copy()
-        new_obs = dict(new_transition.get(TransitionKey.OBSERVATION, {}))
-        mask = self._build_mask(state.shape[-1])
-
-        if state.ndim >= 3:
-            # [B, n_obs, D] — multi-timestep (training with state_delta_indices)
-            relative = to_relative_state(state, mask)
-            new_obs[OBS_STATE] = relative.flatten(start_dim=-2)  # [B, n_obs*D]
-        elif state.ndim == 2:
-            # [B, D] — single timestep (inference): buffer previous and stack
-            current = state
-            if self._previous_state is None:
-                self._previous_state = current.clone()
-            prev = self._previous_state
-            if prev.device != current.device or prev.dtype != current.dtype:
-                prev = prev.to(device=current.device, dtype=current.dtype)
-            stacked = torch.stack([prev, current], dim=-2)  # [B, 2, D]
-            relative = to_relative_state(stacked, mask)
-            new_obs[OBS_STATE] = relative.flatten(start_dim=-2)  # [B, 2*D]
-            self._previous_state = current.clone()
-
-        new_transition[TransitionKey.OBSERVATION] = new_obs
-        return new_transition
-
-    def reset(self) -> None:
-        """Reset the state buffer. Call at episode boundaries during inference."""
-        self._previous_state = None
-
-    def get_config(self) -> dict[str, Any]:
-        return {
-            "enabled": self.enabled,
-            "exclude_joints": self.exclude_joints,
-            "state_names": self.state_names,
-        }
-
-    def transform_features(
-        self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
-    ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        return features
-
-
 @ProcessorStepRegistry.register("absolute_actions_processor")
 @dataclass
 class AbsoluteActionsProcessorStep(ProcessorStep):

src/lerobot/robots/bi_openarm_follower/bi_openarm_follower.py

@@ -62,8 +62,6 @@ class BiOpenArmFollower(Robot):
             can_bitrate=config.left_arm_config.can_bitrate,
             can_data_bitrate=config.left_arm_config.can_data_bitrate,
             motor_config=config.left_arm_config.motor_config,
-            gripper_port=config.left_arm_config.gripper_port,
-            gripper_motor_ids=config.left_arm_config.gripper_motor_ids,
             position_kd=config.left_arm_config.position_kd,
             position_kp=config.left_arm_config.position_kp,
             joint_limits=config.left_arm_config.joint_limits,
@@ -82,8 +80,6 @@ class BiOpenArmFollower(Robot):
             can_bitrate=config.right_arm_config.can_bitrate,
             can_data_bitrate=config.right_arm_config.can_data_bitrate,
             motor_config=config.right_arm_config.motor_config,
-            gripper_port=config.right_arm_config.gripper_port,
-            gripper_motor_ids=config.right_arm_config.gripper_motor_ids,
             position_kd=config.right_arm_config.position_kd,
             position_kp=config.right_arm_config.position_kp,
             joint_limits=config.right_arm_config.joint_limits,
@@ -100,9 +96,11 @@ class BiOpenArmFollower(Robot):
         left_arm_motors_ft = self.left_arm._motors_ft
         right_arm_motors_ft = self.right_arm._motors_ft
 
+        # Right first, then left — matches the teleoperator (OpenArmMini) ordering
+        # and the dataset feature names recorded during data collection.
         return {
-            **{f"left_{k}": v for k, v in left_arm_motors_ft.items()},
             **{f"right_{k}": v for k, v in right_arm_motors_ft.items()},
+            **{f"left_{k}": v for k, v in left_arm_motors_ft.items()},
         }
 
     @property
@@ -154,14 +152,16 @@ class BiOpenArmFollower(Robot):
         left_cam_keys = set(self.left_arm.cameras.keys())
         right_cam_keys = set(self.right_arm.cameras.keys())
 
-        left_obs = self.left_arm.get_observation()
-        for key, value in left_obs.items():
-            obs_dict[key if key in left_cam_keys else f"left_{key}"] = value
-
+        # Right first, then left — matches the teleoperator (OpenArmMini) ordering
+        # and the dataset feature names recorded during data collection.
         right_obs = self.right_arm.get_observation()
         for key, value in right_obs.items():
             obs_dict[key if key in right_cam_keys else f"right_{key}"] = value
 
+        left_obs = self.left_arm.get_observation()
+        for key, value in left_obs.items():
+            obs_dict[key if key in left_cam_keys else f"left_{key}"] = value
+
         return obs_dict
 
     @check_if_not_connected
@@ -187,7 +187,7 @@ class BiOpenArmFollower(Robot):
         prefixed_sent_action_left = {f"left_{key}": value for key, value in sent_action_left.items()}
         prefixed_sent_action_right = {f"right_{key}": value for key, value in sent_action_right.items()}
 
-        return {**prefixed_sent_action_left, **prefixed_sent_action_right}
+        return {**prefixed_sent_action_right, **prefixed_sent_action_left}
 
     @check_if_not_connected
     def disconnect(self):

src/lerobot/robots/bi_openarm_follower/config_bi_openarm_follower.py

@@ -23,10 +23,12 @@ from ..config import RobotConfig
 
 
 @RobotConfig.register_subclass("bi_openarm_follower")
-@dataclass
+@dataclass(kw_only=True)
 class BiOpenArmFollowerConfig(RobotConfig):
     """Configuration class for Bi OpenArm Follower robots."""
 
+    id: str | None = "bi_openarm_follower"
+
     left_arm_config: OpenArmFollowerConfigBase
     right_arm_config: OpenArmFollowerConfigBase
 

src/lerobot/robots/openarm_follower/config_openarm_follower.py

@@ -28,8 +28,7 @@ LEFT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
     "joint_5": (-85.0, 85.0),
     "joint_6": (-40.0, 40.0),
     "joint_7": (-80.0, 80.0),
-    "proximal": (0.0, 100.0),
-    "distal": (0.0, 100.0),
+    "gripper": (-65.0, 0.0),
 }
 
 RIGHT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
@@ -40,8 +39,7 @@ RIGHT_DEFAULT_JOINTS_LIMITS: dict[str, tuple[float, float]] = {
     "joint_5": (-85.0, 85.0),
     "joint_6": (-40.0, 40.0),
     "joint_7": (-80.0, 80.0),
-    "proximal": (0.0, 100.0),
-    "distal": (0.0, 100.0),
+    "gripper": (-65.0, 0.0),
 }
 
 
@@ -75,8 +73,13 @@ class OpenArmFollowerConfigBase:
     # Camera configurations
     cameras: dict[str, CameraConfig] = field(default_factory=dict)
 
-    # Arm motor configuration (7 DOF, Damiao on CAN bus)
+    # Motor configuration for OpenArms (7 DOF per arm)
     # Maps motor names to (send_can_id, recv_can_id, motor_type)
+    # Based on: https://docs.openarm.dev/software/setup/configure-test
+    # OpenArms uses 4 types of motors:
+    # - DM8009 (DM-J8009P-2EC) for shoulders (high torque)
+    # - DM4340P and DM4340 for shoulder rotation and elbow
+    # - DM4310 (DM-J4310-2EC V1.1) for wrist and gripper
     motor_config: dict[str, tuple[int, int, str]] = field(
         default_factory=lambda: {
             "joint_1": (0x01, 0x11, "dm8009"),  # J1 - Shoulder pan (DM8009)
@@ -86,18 +89,19 @@ class OpenArmFollowerConfigBase:
             "joint_5": (0x05, 0x15, "dm4310"),  # J5 - Wrist roll (DM4310)
             "joint_6": (0x06, 0x16, "dm4310"),  # J6 - Wrist pitch (DM4310)
             "joint_7": (0x07, 0x17, "dm4310"),  # J7 - Wrist rotation (DM4310)
+            "gripper": (0x08, 0x18, "dm4310"),  # J8 - Gripper (DM4310)
         }
     )
 
-    # UMI-style gripper (Feetech STS3215 on serial bus)
-    gripper_port: str = "/dev/ttyUSB0"
-    gripper_motor_ids: dict[str, int] = field(default_factory=lambda: {"proximal": 1, "distal": 2})
+    # MIT control parameters for position control (used in send_action)
+    # List of 8 values: [joint_1, joint_2, joint_3, joint_4, joint_5, joint_6, joint_7, gripper]
+    position_kp: list[float] = field(
+        default_factory=lambda: [240.0, 240.0, 240.0, 240.0, 24.0, 31.0, 25.0, 25.0]
+    )
+    position_kd: list[float] = field(default_factory=lambda: [5.0, 5.0, 3.0, 5.0, 0.3, 0.3, 0.3, 0.3])
 
-    # MIT control parameters for the 7 arm joints
-    position_kp: list[float] = field(default_factory=lambda: [240.0, 240.0, 240.0, 240.0, 24.0, 31.0, 25.0])
-    position_kd: list[float] = field(default_factory=lambda: [5.0, 5.0, 3.0, 5.0, 0.3, 0.3, 0.3])
-
-    # Joint limits. Can be overridden via CLI or by setting config.side to 'left' or 'right'.
+    # Values for joint limits. Can be overridden via CLI (for custom values) or by setting config.side to either 'left' or 'right'.
+    # If config.side is left set to None and no CLI values are passed, the default joint limit values are small for safety.
     joint_limits: dict[str, tuple[float, float]] = field(
         default_factory=lambda: {
             "joint_1": (-5.0, 5.0),
@@ -107,8 +111,7 @@ class OpenArmFollowerConfigBase:
             "joint_5": (-5.0, 5.0),
             "joint_6": (-5.0, 5.0),
             "joint_7": (-5.0, 5.0),
-            "proximal": (0.0, 100.0),
-            "distal": (0.0, 100.0),
+            "gripper": (-5.0, 0.0),
         }
     )
 

src/lerobot/robots/openarm_follower/openarm_follower.py

@@ -22,7 +22,6 @@ from typing import Any
 from lerobot.cameras.utils import make_cameras_from_configs
 from lerobot.motors import Motor, MotorCalibration, MotorNormMode
 from lerobot.motors.damiao import DamiaoMotorsBus
-from lerobot.motors.feetech import FeetechMotorsBus, OperatingMode
 from lerobot.types import RobotAction, RobotObservation
 from lerobot.utils.decorators import check_if_already_connected, check_if_not_connected
 
@@ -39,7 +38,8 @@ logger = logging.getLogger(__name__)
 
 class OpenArmFollower(Robot):
     """
-    OpenArms Follower Robot: 7 DOF Damiao arm (CAN) + UMI-style Feetech gripper (serial).
+    OpenArms Follower Robot which uses CAN bus communication to control 7 DOF arm with a gripper.
+    The arm uses Damiao motors in MIT control mode.
     """
 
     config_class = OpenArmFollowerConfig
@@ -49,17 +49,19 @@ class OpenArmFollower(Robot):
         super().__init__(config)
         self.config = config
 
-        # Arm motors (Damiao on CAN bus)
-        arm_motors: dict[str, Motor] = {}
+        # Arm motors
+        motors: dict[str, Motor] = {}
         for motor_name, (send_id, recv_id, motor_type_str) in config.motor_config.items():
-            motor = Motor(send_id, motor_type_str, MotorNormMode.DEGREES)
+            motor = Motor(
+                send_id, motor_type_str, MotorNormMode.DEGREES
+            )  # Always use degrees for Damiao motors
             motor.recv_id = recv_id
             motor.motor_type_str = motor_type_str
-            arm_motors[motor_name] = motor
+            motors[motor_name] = motor
 
         self.bus = DamiaoMotorsBus(
             port=self.config.port,
-            motors=arm_motors,
+            motors=motors,
             calibration=self.calibration,
             can_interface=self.config.can_interface,
             use_can_fd=self.config.use_can_fd,
@@ -67,17 +69,6 @@ class OpenArmFollower(Robot):
             data_bitrate=self.config.can_data_bitrate if self.config.use_can_fd else None,
         )
 
-        # Gripper motors (Feetech STS3215 on serial bus)
-        gripper_motors: dict[str, Motor] = {
-            name: Motor(motor_id, "sts3215", MotorNormMode.RANGE_0_100)
-            for name, motor_id in config.gripper_motor_ids.items()
-        }
-        self.gripper_bus = FeetechMotorsBus(
-            port=config.gripper_port,
-            motors=gripper_motors,
-            calibration=self.calibration,
-        )
-
         if config.side is not None:
             if config.side == "left":
                 config.joint_limits = LEFT_DEFAULT_JOINTS_LIMITS
@@ -93,6 +84,7 @@ class OpenArmFollower(Robot):
             )
         logger.info(f"Values used for joint limits: {config.joint_limits}.")
 
+        # Initialize cameras
         self.cameras = make_cameras_from_configs(config.cameras)
 
     @property
@@ -101,10 +93,8 @@ class OpenArmFollower(Robot):
         features: dict[str, type] = {}
         for motor in self.bus.motors:
             features[f"{motor}.pos"] = float
-            features[f"{motor}.vel"] = float
-            features[f"{motor}.torque"] = float
-        for motor in self.gripper_bus.motors:
-            features[f"{motor}.pos"] = float
+            features[f"{motor}.vel"] = float  # Add this
+            features[f"{motor}.torque"] = float  # Add this
         return features
 
     @property
@@ -126,11 +116,8 @@ class OpenArmFollower(Robot):
 
     @property
     def is_connected(self) -> bool:
-        return (
-            self.bus.is_connected
-            and self.gripper_bus.is_connected
-            and all(cam.is_connected for cam in self.cameras.values())
-        )
+        """Check if robot is connected."""
+        return self.bus.is_connected and all(cam.is_connected for cam in self.cameras.values())
 
     @check_if_already_connected
     def connect(self, calibrate: bool = True) -> None:
@@ -140,12 +127,12 @@ class OpenArmFollower(Robot):
         We assume that at connection time, the arms are in a safe rest position,
         and torque can be safely disabled to run calibration if needed.
         """
+
+        # Connect to CAN bus
         logger.info(f"Connecting arm on {self.config.port}...")
         self.bus.connect()
 
-        logger.info(f"Connecting gripper on {self.config.gripper_port}...")
-        self.gripper_bus.connect()
-
+        # Run calibration if needed
         if not self.is_calibrated and calibrate:
             logger.info(
                 "Mismatch between calibration values in the motor and the calibration file or no calibration file found"
@@ -157,7 +144,7 @@ class OpenArmFollower(Robot):
 
         self.configure()
 
-        if self.bus.is_calibrated:
+        if self.is_calibrated:
             self.bus.set_zero_position()
 
         self.bus.enable_torque()
@@ -166,39 +153,47 @@ class OpenArmFollower(Robot):
 
     @property
     def is_calibrated(self) -> bool:
-        return self.bus.is_calibrated and self.gripper_bus.is_calibrated
+        """Check if robot is calibrated."""
+        return self.bus.is_calibrated
 
     def calibrate(self) -> None:
         """
-        Run calibration for both the Damiao arm and Feetech gripper.
+        Run calibration procedure for OpenArms robot.
 
-        Arm calibration: set zero position with arm hanging, ±90° default range.
-        Gripper calibration: SO100-style half-turn homing + range recording.
+        The calibration procedure:
+        1. Disable torque
+        2. Ask user to position arms in hanging position with grippers closed
+        3. Set this as zero position
+        4. Record range of motion for each joint
+        5. Save calibration
         """
         if self.calibration:
+            # Calibration file exists, ask user whether to use it or run new calibration
             user_input = input(
                 f"Press ENTER to use provided calibration file associated with the id {self.id}, or type 'c' and press ENTER to run calibration: "
             )
             if user_input.strip().lower() != "c":
                 logger.info(f"Writing calibration file associated with the id {self.id} to the motors")
                 self.bus.write_calibration(self.calibration)
-                self.gripper_bus.write_calibration(self.calibration)
                 return
 
         logger.info(f"\nRunning calibration for {self}")
-
-        # --- Arm calibration (Damiao) ---
         self.bus.disable_torque()
+
+        # Step 1: Set zero position
         input(
-            "\nCalibration: Set Zero Position\n"
+            "\nCalibration: Set Zero Position)\n"
             "Position the arm in the following configuration:\n"
             "  - Arm hanging straight down\n"
             "  - Gripper closed\n"
             "Press ENTER when ready..."
         )
+
+        # Set current position as zero for all motors
         self.bus.set_zero_position()
         logger.info("Arm zero position set.")
 
+        logger.info("Setting range: -90° to +90° for safety by default for all joints")
         for motor_name, motor in self.bus.motors.items():
             self.calibration[motor_name] = MotorCalibration(
                 id=motor.id,
@@ -207,52 +202,17 @@ class OpenArmFollower(Robot):
                 range_min=-90,
                 range_max=90,
             )
+
         self.bus.write_calibration(self.calibration)
-
-        # --- Gripper calibration (Feetech) ---
-        self.gripper_bus.disable_torque()
-        for motor in self.gripper_bus.motors:
-            self.gripper_bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
-
-        input("Move gripper to the middle of its range of motion and press ENTER....")
-        homing_offsets = self.gripper_bus.set_half_turn_homings()
-
-        gripper_motor_names = list(self.gripper_bus.motors.keys())
-        print(
-            f"Move gripper joints ({', '.join(gripper_motor_names)}) through their "
-            "entire ranges of motion.\nRecording positions. Press ENTER to stop..."
-        )
-        range_mins, range_maxes = self.gripper_bus.record_ranges_of_motion(gripper_motor_names)
-
-        for motor_name, m in self.gripper_bus.motors.items():
-            self.calibration[motor_name] = MotorCalibration(
-                id=m.id,
-                drive_mode=0,
-                homing_offset=homing_offsets[motor_name],
-                range_min=range_mins[motor_name],
-                range_max=range_maxes[motor_name],
-            )
-        self.gripper_bus.write_calibration(self.calibration)
-
         self._save_calibration()
         print(f"Calibration saved to {self.calibration_fpath}")
 
     def configure(self) -> None:
-        """Configure both arm (Damiao) and gripper (Feetech) motors."""
+        """Configure motors with appropriate settings."""
+        # TODO(Steven, Pepijn): Slightly different from what it is happening in the leader
         with self.bus.torque_disabled():
             self.bus.configure_motors()
 
-        with self.gripper_bus.torque_disabled():
-            self.gripper_bus.configure_motors()
-            for motor in self.gripper_bus.motors:
-                self.gripper_bus.write("Operating_Mode", motor, OperatingMode.POSITION.value)
-                self.gripper_bus.write("P_Coefficient", motor, 16)
-                self.gripper_bus.write("I_Coefficient", motor, 0)
-                self.gripper_bus.write("D_Coefficient", motor, 32)
-                self.gripper_bus.write("Max_Torque_Limit", motor, 500)
-                self.gripper_bus.write("Protection_Current", motor, 250)
-                self.gripper_bus.write("Overload_Torque", motor, 25)
-
     def setup_motors(self) -> None:
         raise NotImplementedError(
             "Motor ID configuration is typically done via manufacturer tools for CAN motors."
@@ -260,23 +220,25 @@ class OpenArmFollower(Robot):
 
     @check_if_not_connected
     def get_observation(self) -> RobotObservation:
-        """Read all motor states from arm (CAN) and gripper (serial), plus cameras."""
+        """
+        Get current observation from robot including position, velocity, and torque.
+
+        Reads all motor states (pos/vel/torque) in one CAN refresh cycle
+        instead of 3 separate reads.
+        """
         start = time.perf_counter()
+
         obs_dict: dict[str, Any] = {}
 
-        # Arm motors (Damiao) — pos/vel/torque in one CAN refresh cycle
         states = self.bus.sync_read_all_states()
+
         for motor in self.bus.motors:
             state = states.get(motor, {})
             obs_dict[f"{motor}.pos"] = state.get("position", 0.0)
             obs_dict[f"{motor}.vel"] = state.get("velocity", 0.0)
             obs_dict[f"{motor}.torque"] = state.get("torque", 0.0)
 
-        # Gripper motors (Feetech) — position only
-        gripper_positions = self.gripper_bus.sync_read("Present_Position")
-        for motor, val in gripper_positions.items():
-            obs_dict[f"{motor}.pos"] = val
-
+        # Capture images from cameras
         for cam_key, cam in self.cameras.items():
             start = time.perf_counter()
             obs_dict[cam_key] = cam.read_latest()
@@ -296,76 +258,86 @@ class OpenArmFollower(Robot):
         custom_kd: dict[str, float] | None = None,
     ) -> RobotAction:
         """
-        Send action command to robot. Arm joints go to Damiao CAN bus,
-        gripper joints go to Feetech serial bus.
+        Send action command to robot.
+
+        The action magnitude may be clipped based on safety limits.
 
         Args:
-            action: Dictionary with motor positions (e.g., "joint_1.pos", "proximal.pos")
-            custom_kp: Optional custom kp gains per arm motor
-            custom_kd: Optional custom kd gains per arm motor
+            action: Dictionary with motor positions (e.g., "joint_1.pos", "joint_2.pos")
+            custom_kp: Optional custom kp gains per motor (e.g., {"joint_1": 120.0, "joint_2": 150.0})
+            custom_kd: Optional custom kd gains per motor (e.g., {"joint_1": 1.5, "joint_2": 2.0})
 
         Returns:
             The action actually sent (potentially clipped)
         """
+
         goal_pos = {key.removesuffix(".pos"): val for key, val in action.items() if key.endswith(".pos")}
 
-        # Apply joint limit clipping
+        # Apply joint limit clipping to arm
         for motor_name, position in goal_pos.items():
             if motor_name in self.config.joint_limits:
                 min_limit, max_limit = self.config.joint_limits[motor_name]
                 clipped_position = max(min_limit, min(max_limit, position))
                 if clipped_position != position:
-                    logger.debug(f"Clipped {motor_name} from {position:.2f} to {clipped_position:.2f}")
+                    logger.debug(f"Clipped {motor_name} from {position:.2f}° to {clipped_position:.2f}°")
                 goal_pos[motor_name] = clipped_position
 
-        # Split into arm and gripper actions
-        arm_motors = set(self.bus.motors.keys())
-        gripper_motors = set(self.gripper_bus.motors.keys())
-        arm_goal = {k: v for k, v in goal_pos.items() if k in arm_motors}
-        gripper_goal = {k: v for k, v in goal_pos.items() if k in gripper_motors}
-
-        # Cap arm goal position when too far away from present position
-        if self.config.max_relative_target is not None and arm_goal:
+        # Cap goal position when too far away from present position.
+        # /!\ Slower fps expected due to reading from the follower.
+        if self.config.max_relative_target is not None:
             present_pos = self.bus.sync_read("Present_Position")
-            goal_present_pos = {key: (g_pos, present_pos[key]) for key, g_pos in arm_goal.items()}
-            arm_goal = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target)
+            goal_present_pos = {key: (g_pos, present_pos[key]) for key, g_pos in goal_pos.items()}
+            goal_pos = ensure_safe_goal_position(goal_present_pos, self.config.max_relative_target)
 
-        # Arm: batch MIT control (Damiao)
-        if arm_goal:
-            arm_motor_names = list(self.bus.motors.keys())
-            commands = {}
-            for motor_name, position_degrees in arm_goal.items():
-                idx = arm_motor_names.index(motor_name) if motor_name in arm_motor_names else 0
-                if custom_kp is not None and motor_name in custom_kp:
-                    kp = custom_kp[motor_name]
-                else:
-                    kp = (
-                        self.config.position_kp[idx]
-                        if isinstance(self.config.position_kp, list)
-                        else self.config.position_kp
-                    )
-                if custom_kd is not None and motor_name in custom_kd:
-                    kd = custom_kd[motor_name]
-                else:
-                    kd = (
-                        self.config.position_kd[idx]
-                        if isinstance(self.config.position_kd, list)
-                        else self.config.position_kd
-                    )
-                commands[motor_name] = (kp, kd, position_degrees, 0.0, 0.0)
-            self.bus._mit_control_batch(commands)
+        # TODO(Steven, Pepijn): Refactor writing
+        # Motor name to index mapping for gains
+        motor_index = {
+            "joint_1": 0,
+            "joint_2": 1,
+            "joint_3": 2,
+            "joint_4": 3,
+            "joint_5": 4,
+            "joint_6": 5,
+            "joint_7": 6,
+            "gripper": 7,
+        }
 
-        # Gripper: position control (Feetech)
-        if gripper_goal:
-            self.gripper_bus.sync_write("Goal_Position", gripper_goal)
+        # Use batch MIT control for arm (sends all commands, then collects responses)
+        commands = {}
+        for motor_name, position_degrees in goal_pos.items():
+            idx = motor_index.get(motor_name, 0)
+            # Use custom gains if provided, otherwise use config defaults
+            if custom_kp is not None and motor_name in custom_kp:
+                kp = custom_kp[motor_name]
+            else:
+                kp = (
+                    self.config.position_kp[idx]
+                    if isinstance(self.config.position_kp, list)
+                    else self.config.position_kp
+                )
+            if custom_kd is not None and motor_name in custom_kd:
+                kd = custom_kd[motor_name]
+            else:
+                kd = (
+                    self.config.position_kd[idx]
+                    if isinstance(self.config.position_kd, list)
+                    else self.config.position_kd
+                )
+            commands[motor_name] = (kp, kd, position_degrees, 0.0, 0.0)
+
+        self.bus._mit_control_batch(commands)
 
-        goal_pos.update(arm_goal)
         return {f"{motor}.pos": val for motor, val in goal_pos.items()}
 
     @check_if_not_connected
     def disconnect(self):
+        """Disconnect from robot."""
+
+        # Disconnect CAN bus
         self.bus.disconnect(self.config.disable_torque_on_disconnect)
-        self.gripper_bus.disconnect(self.config.disable_torque_on_disconnect)
+
+        # Disconnect cameras
         for cam in self.cameras.values():
             cam.disconnect()
+
         logger.info(f"{self} disconnected.")

src/lerobot/robots/openarm_follower/urdf/openarm_bimanual_pybullet.urdf

@@ -1,630 +0,0 @@
-<?xml version='1.0' encoding='utf-8'?>
-<robot name="openarm">
-  <link name="world" />
-  <joint name="openarm_body_world_joint" type="fixed">
-    <parent link="world" />
-    <child link="openarm_body_link0" />
-    <origin rpy="0 0 0" xyz="0 0 0" />
-  </joint>
-  <link name="openarm_body_link0">
-    <visual name="openarm_body_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/body/v10/visual/body_link0.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_body_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/body/v10/collision/body_link0_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <mass value="13.89" />
-      <inertia ixx="1.653" ixy="0.0" ixz="0.0" iyy="1.653" iyz="0.0" izz="0.051" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_openarm_body_link0_joint" type="fixed">
-    <parent link="openarm_body_link0" />
-    <child link="openarm_left_link0" />
-    <origin rpy="-1.5708 0 0" xyz="0.0 0.031 0.698" />
-  </joint>
-  <link name="openarm_left_link0">
-    <visual name="openarm_left_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link0.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link0_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0009483362816297526 -0.0001580207020448382 0.03076860287587199" />
-      <mass value="1.1432284943239561" />
-      <inertia ixx="0.001128" ixy="-4e-06" ixz="-3.3e-05" iyy="0.000962" iyz="-7e-06" izz="0.00147" />
-    </inertial>
-  </link>
-  <link name="openarm_left_link1">
-    <visual name="openarm_left_link1_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link1.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link1_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link1_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0011467657911800769 -3.319987657026362e-05 0.05395284380736254" />
-      <mass value="1.1416684646202298" />
-      <inertia ixx="0.001567" ixy="-1e-06" ixz="-2.9e-05" iyy="0.001273" iyz="1e-06" izz="0.001016" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint1" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 0.0 0.0625" />
-    <parent link="openarm_left_link0" />
-    <child link="openarm_left_link1" />
-    <axis xyz="0 0 1" />
-    <limit effort="40" lower="-3.490659" upper="1.3962629999999998" velocity="16.754666" />
-  </joint>
-  <link name="openarm_left_link2">
-    <visual name="openarm_left_link2_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link2.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link2_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link2_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.00839629182351943 2.0145102027597523e-08 0.03256649300522363" />
-      <mass value="0.2775092746011571" />
-      <inertia ixx="0.000359" ixy="1e-06" ixz="-0.000109" iyy="0.000376" iyz="1e-06" izz="0.000232" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint2" type="revolute">
-    <origin rpy="-1.57079632679 0 0" xyz="-0.0301 0.0 0.06" />
-    <parent link="openarm_left_link1" />
-    <child link="openarm_left_link2" />
-    <axis xyz="-1 0 0" />
-    <limit effort="40" lower="-3.3161253267948965" upper="0.17453267320510335" velocity="16.754666" />
-  </joint>
-  <link name="openarm_left_link3">
-    <visual name="openarm_left_link3_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link3.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link3_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link3_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.002104752099628911 -0.0005549085042607548 0.09047470545721961" />
-      <mass value="1.073863338202347" />
-      <inertia ixx="0.004372" ixy="1e-06" ixz="1.1e-05" iyy="0.004319" iyz="-3.6e-05" izz="0.000661" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint3" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0301 0.0 0.06625" />
-    <parent link="openarm_left_link2" />
-    <child link="openarm_left_link3" />
-    <axis xyz="0 0 1" />
-    <limit effort="27" lower="-1.570796" upper="1.570796" velocity="5.445426" />
-  </joint>
-  <link name="openarm_left_link4">
-    <visual name="openarm_left_link4_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link4.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link4_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link4_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0029006831074562967 -0.03030575826634669 0.06339637422196209" />
-      <mass value="0.6348534566833373" />
-      <inertia ixx="0.000623" ixy="-1e-06" ixz="-1.9e-05" iyy="0.000511" iyz="3.8e-05" izz="0.000334" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint4" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0 0.0315 0.15375" />
-    <parent link="openarm_left_link3" />
-    <child link="openarm_left_link4" />
-    <axis xyz="0 1 0" />
-    <limit effort="27" lower="0.0" upper="2.443461" velocity="5.445426" />
-  </joint>
-  <link name="openarm_left_link5">
-    <visual name="openarm_left_link5_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link5.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link5_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link5_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.003049665024221911 -0.0008866902457326625 0.043079803024980934" />
-      <mass value="0.6156588026168502" />
-      <inertia ixx="0.000423" ixy="-8e-06" ixz="6e-06" iyy="0.000445" iyz="-6e-06" izz="0.000324" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint5" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 -0.0315 0.0955" />
-    <parent link="openarm_left_link4" />
-    <child link="openarm_left_link5" />
-    <axis xyz="0 0 1" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_link6">
-    <visual name="openarm_left_link6_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link6.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link6_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link6_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.037136587005447405 -0.00033230528343419053 -9.498374522309838e-05" />
-      <mass value="0.475202773187987" />
-      <inertia ixx="0.000143" ixy="1e-06" ixz="1e-06" iyy="0.000157" iyz="1e-06" izz="0.000159" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint6" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0375 0.0 0.1205" />
-    <parent link="openarm_left_link5" />
-    <child link="openarm_left_link6" />
-    <axis xyz="1 0 0" />
-    <limit effort="7" lower="-0.785398" upper="0.785398" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_link7">
-    <visual name="openarm_left_link7_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link7.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_link7_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link7_symp.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="6.875510271106056e-05 -0.01266175250761268 0.06951945409987448" />
-      <mass value="0.4659771327380578" />
-      <inertia ixx="0.000639" ixy="1e-06" ixz="1e-06" iyy="0.000497" iyz="8.9e-05" izz="0.000342" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_joint7" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0375 0.0 0.0" />
-    <parent link="openarm_left_link6" />
-    <child link="openarm_left_link7" />
-    <axis xyz="0 -1 0" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <joint name="openarm_right_openarm_body_link0_joint" type="fixed">
-    <parent link="openarm_body_link0" />
-    <child link="openarm_right_link0" />
-    <origin rpy="1.5708 0 0" xyz="0.0 -0.031 0.698" />
-  </joint>
-  <link name="openarm_right_link0">
-    <visual name="openarm_right_link0_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link0.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link0_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 0.0" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link0_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0009483362816297526 0.0001580207020448382 0.03076860287587199" />
-      <mass value="1.1432284943239561" />
-      <inertia ixx="0.001128" ixy="-4e-06" ixz="-3.3e-05" iyy="0.000962" iyz="-7e-06" izz="0.00147" />
-    </inertial>
-  </link>
-  <link name="openarm_right_link1">
-    <visual name="openarm_right_link1_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link1.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link1_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 0.0 -0.0625" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link1_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.0011467657911800769 3.319987657026362e-05 0.05395284380736254" />
-      <mass value="1.1416684646202298" />
-      <inertia ixx="0.001567" ixy="-1e-06" ixz="-2.9e-05" iyy="0.001273" iyz="1e-06" izz="0.001016" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint1" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 0.0 0.0625" />
-    <parent link="openarm_right_link0" />
-    <child link="openarm_right_link1" />
-    <axis xyz="0 0 1" />
-    <limit effort="40" lower="-1.396263" upper="3.490659" velocity="16.754666" />
-  </joint>
-  <link name="openarm_right_link2">
-    <visual name="openarm_right_link2_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link2.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link2_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0301 0.0 -0.1225" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link2_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="0.00839629182351943 -2.0145102027597523e-08 0.03256649300522363" />
-      <mass value="0.2775092746011571" />
-      <inertia ixx="0.000359" ixy="1e-06" ixz="-0.000109" iyy="0.000376" iyz="1e-06" izz="0.000232" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint2" type="revolute">
-    <origin rpy="1.57079632679 0 0" xyz="-0.0301 0.0 0.06" />
-    <parent link="openarm_right_link1" />
-    <child link="openarm_right_link2" />
-    <axis xyz="-1 0 0" />
-    <limit effort="40" lower="-0.17453267320510335" upper="3.3161253267948965" velocity="16.754666" />
-  </joint>
-  <link name="openarm_right_link3">
-    <visual name="openarm_right_link3_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link3.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link3_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.18875" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link3_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.002104752099628911 0.0005549085042607548 0.09047470545721961" />
-      <mass value="1.073863338202347" />
-      <inertia ixx="0.004372" ixy="1e-06" ixz="1.1e-05" iyy="0.004319" iyz="-3.6e-05" izz="0.000661" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint3" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0301 0.0 0.06625" />
-    <parent link="openarm_right_link2" />
-    <child link="openarm_right_link3" />
-    <axis xyz="0 0 1" />
-    <limit effort="27" lower="-1.570796" upper="1.570796" velocity="5.445426" />
-  </joint>
-  <link name="openarm_right_link4">
-    <visual name="openarm_right_link4_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link4.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link4_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0315 -0.3425" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link4_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0029006831074562967 -0.03030575826634669 0.06339637422196209" />
-      <mass value="0.6348534566833373" />
-      <inertia ixx="0.000623" ixy="-1e-06" ixz="-1.9e-05" iyy="0.000511" iyz="3.8e-05" izz="0.000334" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint4" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0 0.0315 0.15375" />
-    <parent link="openarm_right_link3" />
-    <child link="openarm_right_link4" />
-    <axis xyz="0 1 0" />
-    <limit effort="27" lower="0.0" upper="2.443461" velocity="5.445426" />
-  </joint>
-  <link name="openarm_right_link5">
-    <visual name="openarm_right_link5_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link5.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link5_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0 -0.0 -0.438" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link5_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.003049665024221911 0.0008866902457326625 0.043079803024980934" />
-      <mass value="0.6156588026168502" />
-      <inertia ixx="0.000423" ixy="-8e-06" ixz="6e-06" iyy="0.000445" iyz="-6e-06" izz="0.000324" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint5" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0 -0.0315 0.0955" />
-    <parent link="openarm_right_link4" />
-    <child link="openarm_right_link5" />
-    <axis xyz="0 0 1" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_right_link6">
-    <visual name="openarm_right_link6_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link6.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link6_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="-0.0375 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link6_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="-0.037136587005447405 0.00033230528343419053 -9.498374522309838e-05" />
-      <mass value="0.475202773187987" />
-      <inertia ixx="0.000143" ixy="1e-06" ixz="1e-06" iyy="0.000157" iyz="1e-06" izz="0.000159" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint6" type="revolute">
-    <origin rpy="0 0 0" xyz="0.0375 0.0 0.1205" />
-    <parent link="openarm_right_link5" />
-    <child link="openarm_right_link6" />
-    <axis xyz="1 0 0" />
-    <limit effort="7" lower="-0.785398" upper="0.785398" velocity="20.943946" />
-  </joint>
-  <link name="openarm_right_link7">
-    <visual name="openarm_right_link7_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/visual/link7.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_link7_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.0 -0.5585" />
-      <geometry>
-        <mesh filename="./meshes/arm/v10/collision/link7_symp.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0.0 0.0 0.0" xyz="6.875510271106056e-05 0.01266175250761268 0.06951945409987448" />
-      <mass value="0.4659771327380578" />
-      <inertia ixx="0.000639" ixy="1e-06" ixz="1e-06" iyy="0.000497" iyz="8.9e-05" izz="0.000342" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_joint7" type="revolute">
-    <origin rpy="0 0 0" xyz="-0.0375 0.0 0.0" />
-    <parent link="openarm_right_link6" />
-    <child link="openarm_right_link7" />
-    <axis xyz="0 1 0" />
-    <limit effort="7" lower="-1.570796" upper="1.570796" velocity="20.943946" />
-  </joint>
-  <link name="openarm_left_hand">
-    <visual name="openarm_left_hand_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/hand.dae" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_hand_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/hand.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0 0.002 0.03" />
-      <mass value="0.35" />
-      <inertia ixx="0.0002473" ixy="1e-06" ixz="1e-06" iyy="1.763e-05" iyz="1e-06" izz="0.0002521" />
-    </inertial>
-  </link>
-  <joint name="left_openarm_hand_joint" type="fixed">
-    <parent link="openarm_left_link7" />
-    <child link="openarm_left_hand" />
-    <origin rpy="0 0 0" xyz="0 -0.0 0.1001" />
-  </joint>
-  <link name="openarm_left_hand_tcp">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_hand_tcp_joint" type="fixed">
-    <origin rpy="0 0 0" xyz="0 -0.0 0.08" />
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_hand_tcp" />
-  </joint>
-  <link name="openarm_left_left_finger">
-    <visual name="openarm_left_left_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_left_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <link name="openarm_left_right_finger">
-    <visual name="openarm_left_right_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_left_right_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 -0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <joint name="openarm_left_finger_joint1" type="prismatic">
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_right_finger" />
-    <origin rpy="0 0 0" xyz="0 -0.006 0.015" />
-    <axis xyz="0 -1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-  </joint>
-  <joint name="openarm_left_finger_joint2" type="prismatic">
-    <parent link="openarm_left_hand" />
-    <child link="openarm_left_left_finger" />
-    <origin rpy="0 0 0" xyz="0 0.006 0.015" />
-    <axis xyz="0 1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-    <mimic joint="openarm_left_finger_joint1" />
-  </joint>
-  <link name="openarm_right_hand">
-    <visual name="openarm_right_hand_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/hand.dae" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_hand_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.0 -0.6585" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/hand.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0 0.002 0.03" />
-      <mass value="0.35" />
-      <inertia ixx="0.0002473" ixy="1e-06" ixz="1e-06" iyy="1.763e-05" iyz="1e-06" izz="0.0002521" />
-    </inertial>
-  </link>
-  <link name="openarm_right_ee_target">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_ee_target_joint" type="fixed">
-    <parent link="openarm_right_link7" />
-    <child link="openarm_right_ee_target" />
-    <origin rpy="0 0 0" xyz="0 0.0 0.07" />
-  </joint>
-  <joint name="right_openarm_hand_joint" type="fixed">
-    <parent link="openarm_right_link7" />
-    <child link="openarm_right_hand" />
-    <origin rpy="0 0 0" xyz="0 -0.0 0.1001" />
-  </joint>
-  <link name="openarm_right_hand_tcp">
-    <inertial>
-      <origin xyz="0 0 0" rpy="0 0 0" />
-      <mass value="0.001" />
-      <inertia ixx="0.000001" ixy="0.0" ixz="0.0" iyy="0.000001" iyz="0.0" izz="0.000001" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_hand_tcp_joint" type="fixed">
-    <origin rpy="0 0 0" xyz="0 -0.0 0.08" />
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_hand_tcp" />
-  </joint>
-  <link name="openarm_right_left_finger">
-    <visual name="openarm_right_left_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_left_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 -0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <link name="openarm_right_right_finger">
-    <visual name="openarm_right_right_finger_visual">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/visual/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </visual>
-    <collision name="openarm_right_right_finger_collision">
-      <origin rpy="0.0 0.0 0.0" xyz="0.0 0.05 -0.673001" />
-      <geometry>
-        <mesh filename="./meshes/ee/openarm_hand/collision/finger.stl" scale="0.001 -0.001 0.001" />
-      </geometry>
-    </collision>
-    <inertial>
-      <origin rpy="0 0 0" xyz="0.0064528 -0.01702 0.0219685" />
-      <mass value="0.03602545343277134" />
-      <inertia ixx="2.3749999999999997e-06" ixy="1e-06" ixz="1e-06" iyy="2.3749999999999997e-06" iyz="1e-06" izz="7.5e-07" />
-    </inertial>
-  </link>
-  <joint name="openarm_right_finger_joint1" type="prismatic">
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_right_finger" />
-    <origin rpy="0 0 0" xyz="0 -0.006 0.015" />
-    <axis xyz="0 -1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-  </joint>
-  <joint name="openarm_right_finger_joint2" type="prismatic">
-    <parent link="openarm_right_hand" />
-    <child link="openarm_right_left_finger" />
-    <origin rpy="0 0 0" xyz="0 0.006 0.015" />
-    <axis xyz="0 1 0" />
-    <limit effort="333" lower="0.0" upper="0.044" velocity="10.0" />
-    <mimic joint="openarm_right_finger_joint1" />
-  </joint>
-</robot>

src/lerobot/robots/openarm_follower/urdf/replay_viewer.html

@@ -1,408 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8" />
-<title>Dataset Replay — EE Frame Viewer</title>
-<style>
-  * { margin: 0; padding: 0; box-sizing: border-box; }
-  body { background: #0d1117; overflow: hidden; font-family: 'JetBrains Mono', monospace; color: #c9d1d9; }
-  canvas { display: block; }
-
-  #panel {
-    position: absolute; top: 14px; left: 14px;
-    background: rgba(13,17,23,0.92); border: 1px solid #30363d;
-    border-radius: 10px; padding: 16px 20px; z-index: 10;
-    width: 340px; backdrop-filter: blur(8px);
-  }
-  #panel h2 { font-size: 14px; color: #58a6ff; margin-bottom: 10px; letter-spacing: 0.5px; }
-
-  .row { display: flex; align-items: center; gap: 8px; margin: 6px 0; font-size: 12px; }
-  .row label { width: 70px; color: #8b949e; flex-shrink: 0; }
-  .row .val { color: #f0f6fc; font-variant-numeric: tabular-nums; }
-
-  #transport {
-    margin-top: 12px; display: flex; align-items: center; gap: 8px;
-  }
-  #transport button {
-    background: #21262d; color: #c9d1d9; border: 1px solid #30363d;
-    padding: 6px 14px; border-radius: 6px; cursor: pointer;
-    font-family: inherit; font-size: 12px; transition: background 0.15s;
-  }
-  #transport button:hover { background: #30363d; }
-  #transport button.active { background: #1f6feb; border-color: #1f6feb; color: #fff; }
-
-  #scrubber {
-    width: 100%; margin-top: 8px;
-    -webkit-appearance: none; appearance: none;
-    height: 6px; border-radius: 3px; background: #21262d; outline: none;
-  }
-  #scrubber::-webkit-slider-thumb {
-    -webkit-appearance: none; width: 14px; height: 14px;
-    border-radius: 50%; background: #58a6ff; cursor: pointer;
-  }
-
-  #speed-ctrl { margin-top: 6px; }
-  #speed-ctrl select {
-    background: #21262d; color: #c9d1d9; border: 1px solid #30363d;
-    padding: 4px 8px; border-radius: 4px; font-family: inherit; font-size: 11px;
-  }
-
-  #frame-counter {
-    font-size: 11px; color: #8b949e; margin-top: 6px;
-    font-variant-numeric: tabular-nums;
-  }
-
-  .legend { display: flex; align-items: center; gap: 6px; margin: 3px 0; font-size: 11px; }
-  .dot { width: 10px; height: 10px; border-radius: 50%; display: inline-block; }
-</style>
-<link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet">
-</head>
-<body>
-
-<div id="panel">
-  <h2>DATASET REPLAY — EE FRAME</h2>
-  <div style="font-size:11px;color:#8b949e;margin-bottom:8px;">glannuzel/grabette-dataset · episode 0</div>
-
-  <div class="legend"><span class="dot" style="background:#ff6b6b"></span> EE target (dataset)</div>
-  <div class="legend"><span class="dot" style="background:#ffd43b"></span> Trajectory (past)</div>
-  <div class="legend"><span class="dot" style="background:#30363d"></span> Trajectory (future)</div>
-
-  <div class="row"><label>x</label><span class="val" id="v-x">—</span></div>
-  <div class="row"><label>y</label><span class="val" id="v-y">—</span></div>
-  <div class="row"><label>z</label><span class="val" id="v-z">—</span></div>
-  <div class="row"><label>ax</label><span class="val" id="v-ax">—</span></div>
-  <div class="row"><label>ay</label><span class="val" id="v-ay">—</span></div>
-  <div class="row"><label>az</label><span class="val" id="v-az">—</span></div>
-  <div class="row"><label>gripper</label><span class="val" id="v-grip">—</span></div>
-
-  <div id="transport">
-    <button id="btn-play" onclick="togglePlay()">▶ Play</button>
-    <button onclick="stepFrame(-1)">◀</button>
-    <button onclick="stepFrame(1)">▶</button>
-    <button onclick="resetPlay()">⟳</button>
-  </div>
-  <input type="range" id="scrubber" min="0" max="1" value="0" step="1" />
-  <div id="speed-ctrl">
-    <label style="font-size:11px;color:#8b949e;">Speed:</label>
-    <select id="speed-select" onchange="setSpeed(this.value)">
-      <option value="0.25">0.25×</option>
-      <option value="0.5">0.5×</option>
-      <option value="1" selected>1×</option>
-      <option value="2">2×</option>
-      <option value="4">4×</option>
-    </select>
-  </div>
-  <div id="frame-counter">Frame 0 / 0 · 0.00s</div>
-</div>
-
-<script type="importmap">
-{
-  "imports": {
-    "three": "https://cdn.jsdelivr.net/npm/three@0.169.0/build/three.module.js",
-    "three/examples/jsm/": "https://cdn.jsdelivr.net/npm/three@0.169.0/examples/jsm/"
-  }
-}
-</script>
-
-<script type="module">
-import * as THREE from 'three';
-import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
-import { STLLoader } from 'three/examples/jsm/loaders/STLLoader.js';
-
-let trajectory = null;
-let currentFrame = 0;
-let playing = false;
-let speed = 1.0;
-let lastTime = 0;
-let accumulator = 0;
-
-// Anchor: EE tip world position at zero-joint pose (in Y-up Three.js space)
-const eeAnchor = new THREE.Vector3();
-// Z-up → Y-up rotation (same as robotGroup): -90° around X
-const zUpToYUp = new THREE.Quaternion().setFromAxisAngle(new THREE.Vector3(1, 0, 0), -Math.PI / 2);
-
-const scene = new THREE.Scene();
-scene.background = new THREE.Color(0x0d1117);
-
-const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.01, 100);
-
-const renderer = new THREE.WebGLRenderer({ antialias: true });
-renderer.setSize(window.innerWidth, window.innerHeight);
-renderer.setPixelRatio(window.devicePixelRatio);
-renderer.shadowMap.enabled = true;
-document.body.appendChild(renderer.domElement);
-
-const controls = new OrbitControls(camera, renderer.domElement);
-controls.enableDamping = true;
-controls.dampingFactor = 0.08;
-
-scene.add(new THREE.AmbientLight(0xffffff, 0.8));
-const dirLight = new THREE.DirectionalLight(0xffffff, 1.4);
-dirLight.position.set(2, 4, 3);
-scene.add(dirLight);
-scene.add(new THREE.DirectionalLight(0x8899cc, 0.6).translateX(-2).translateY(1).translateZ(-3));
-scene.add(new THREE.DirectionalLight(0xffffff, 0.5).translateY(-1).translateZ(2));
-
-const grid = new THREE.GridHelper(2, 20, 0x21262d, 0x161b22);
-scene.add(grid);
-scene.add(new THREE.AxesHelper(0.15));
-
-// EE marker
-const eeMarker = new THREE.Mesh(
-  new THREE.SphereGeometry(0.012, 20, 20),
-  new THREE.MeshStandardMaterial({ color: 0xff6b6b, emissive: 0xff6b6b, emissiveIntensity: 0.7 })
-);
-scene.add(eeMarker);
-eeMarker.add(new THREE.AxesHelper(0.06));
-
-// Trajectory lines
-const MAX_POINTS = 2000;
-const pastGeo = new THREE.BufferGeometry();
-pastGeo.setAttribute('position', new THREE.Float32BufferAttribute(new Float32Array(MAX_POINTS * 3), 3));
-const pastLine = new THREE.Line(pastGeo, new THREE.LineBasicMaterial({ color: 0xffd43b, linewidth: 2 }));
-scene.add(pastLine);
-
-const futureGeo = new THREE.BufferGeometry();
-futureGeo.setAttribute('position', new THREE.Float32BufferAttribute(new Float32Array(MAX_POINTS * 3), 3));
-const futureLine = new THREE.Line(futureGeo, new THREE.LineBasicMaterial({ color: 0x30363d, linewidth: 1 }));
-scene.add(futureLine);
-
-// URDF
-const stlLoader = new STLLoader();
-const robotGroup = new THREE.Group();
-// URDF is Z-up; Three.js is Y-up → rotate -90° around X
-robotGroup.rotation.x = -Math.PI / 2;
-scene.add(robotGroup);
-let urdfLinks = {};
-
-function rotvecToQuat(ax, ay, az) {
-  const angle = Math.sqrt(ax * ax + ay * ay + az * az);
-  if (angle < 1e-8) return new THREE.Quaternion();
-  return new THREE.Quaternion().setFromAxisAngle(
-    new THREE.Vector3(ax / angle, ay / angle, az / angle), angle
-  );
-}
-
-async function loadURDF() {
-  const resp = await fetch('./openarm_bimanual_pybullet.urdf');
-  const text = await resp.text();
-  const xml = new DOMParser().parseFromString(text, 'text/xml');
-
-  const links = {};
-
-  for (const linkEl of xml.querySelectorAll('link')) {
-    const name = linkEl.getAttribute('name');
-    const group = new THREE.Group();
-    group.name = name;
-
-    const visual = linkEl.querySelector('visual');
-    if (visual) {
-      const meshEl = visual.querySelector('mesh');
-      const originEl = visual.querySelector('origin');
-      if (meshEl) {
-        const filename = meshEl.getAttribute('filename');
-        const scaleStr = meshEl.getAttribute('scale');
-        const sc = scaleStr ? scaleStr.split(' ').map(Number) : [1, 1, 1];
-        let xyz = [0, 0, 0];
-        if (originEl && originEl.getAttribute('xyz'))
-          xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-        if (filename.endsWith('.stl')) {
-          try {
-            const geo = await new Promise((res, rej) =>
-              stlLoader.load(filename, res, undefined, rej));
-            const mesh = new THREE.Mesh(geo, new THREE.MeshStandardMaterial({
-              color: 0x8899bb, metalness: 0.3, roughness: 0.5,
-            }));
-            mesh.scale.set(sc[0], sc[1], sc[2]);
-            mesh.position.set(xyz[0], xyz[1], xyz[2]);
-            group.add(mesh);
-          } catch (e) { /* skip missing mesh */ }
-        }
-      }
-    }
-    links[name] = group;
-  }
-
-  const rootLinks = new Set(Object.keys(links));
-
-  for (const jointEl of xml.querySelectorAll('joint')) {
-    const parentName = jointEl.querySelector('parent').getAttribute('link');
-    const childName = jointEl.querySelector('child').getAttribute('link');
-    rootLinks.delete(childName);
-
-    const originEl = jointEl.querySelector('origin');
-    let xyz = [0, 0, 0], rpy = [0, 0, 0];
-    if (originEl) {
-      if (originEl.getAttribute('xyz')) xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-      if (originEl.getAttribute('rpy')) rpy = originEl.getAttribute('rpy').split(' ').map(Number);
-    }
-
-    const parent = links[parentName];
-    const child = links[childName];
-    if (!parent || !child) continue;
-
-    child.position.set(xyz[0], xyz[1], xyz[2]);
-    if (rpy[0] || rpy[1] || rpy[2])
-      child.rotation.set(rpy[0], rpy[1], rpy[2], 'XYZ');
-    parent.add(child);
-  }
-
-  for (const n of rootLinks)
-    if (links[n]) robotGroup.add(links[n]);
-
-  // EE target marker on the URDF
-  const eeTargetLink = links['openarm_right_ee_target'];
-  if (eeTargetLink) {
-    eeTargetLink.add(new THREE.Mesh(
-      new THREE.TorusGeometry(0.02, 0.002, 8, 32),
-      new THREE.MeshStandardMaterial({ color: 0xffaa00, emissive: 0xffaa00, emissiveIntensity: 0.5 })
-    ));
-    eeTargetLink.add(new THREE.AxesHelper(0.05));
-  }
-
-  urdfLinks = links;
-}
-
-async function loadTrajectory() {
-  const resp = await fetch('./trajectory_ep0.json');
-  trajectory = await resp.json();
-  document.getElementById('scrubber').max = trajectory.num_frames - 1;
-  document.getElementById('scrubber').value = 0;
-}
-
-function computeOffset() {
-  if (!trajectory || !urdfLinks['openarm_right_ee_target']) return;
-
-  robotGroup.updateMatrixWorld(true);
-  const eeLink = urdfLinks['openarm_right_ee_target'];
-  eeLink.getWorldPosition(eeAnchor);
-
-  controls.target.copy(eeAnchor);
-  camera.position.set(eeAnchor.x + 0.8, eeAnchor.y + 0.3, eeAnchor.z + 0.0);
-  controls.update();
-
-  updateFrame(0);
-}
-
-function mapFramePos(f) {
-  const f0 = trajectory.frames[0];
-  const delta = new THREE.Vector3(f.x - f0.x, f.y - f0.y, f.z - f0.z);
-  delta.applyQuaternion(zUpToYUp);
-  return delta.add(eeAnchor);
-}
-
-function updateFrame(idx) {
-  if (!trajectory) return;
-  currentFrame = Math.max(0, Math.min(idx, trajectory.num_frames - 1));
-
-  const f = trajectory.frames[currentFrame];
-  const pos = mapFramePos(f);
-
-  eeMarker.position.copy(pos);
-  // Orientation: rotate the dataset axis-angle into Y-up space
-  const q = rotvecToQuat(f.ax, f.ay, f.az);
-  eeMarker.quaternion.copy(zUpToYUp).multiply(q);
-
-  // Past trajectory
-  const pastArr = pastGeo.attributes.position.array;
-  let pi = 0;
-  for (let i = 0; i <= currentFrame && i < MAX_POINTS; i++) {
-    const p = mapFramePos(trajectory.frames[i]);
-    pastArr[pi++] = p.x; pastArr[pi++] = p.y; pastArr[pi++] = p.z;
-  }
-  pastGeo.setDrawRange(0, Math.min(currentFrame + 1, MAX_POINTS));
-  pastGeo.attributes.position.needsUpdate = true;
-
-  // Future trajectory
-  const futArr = futureGeo.attributes.position.array;
-  let fi = 0;
-  for (let i = currentFrame; i < trajectory.num_frames && (i - currentFrame) < MAX_POINTS; i++) {
-    const p = mapFramePos(trajectory.frames[i]);
-    futArr[fi++] = p.x; futArr[fi++] = p.y; futArr[fi++] = p.z;
-  }
-  futureGeo.setDrawRange(0, Math.min(trajectory.num_frames - currentFrame, MAX_POINTS));
-  futureGeo.attributes.position.needsUpdate = true;
-
-  // UI
-  document.getElementById('v-x').textContent = pos.x.toFixed(4);
-  document.getElementById('v-y').textContent = pos.y.toFixed(4);
-  document.getElementById('v-z').textContent = pos.z.toFixed(4);
-  document.getElementById('v-ax').textContent = f.ax.toFixed(4);
-  document.getElementById('v-ay').textContent = f.ay.toFixed(4);
-  document.getElementById('v-az').textContent = f.az.toFixed(4);
-  document.getElementById('v-grip').textContent =
-    `p=${f.proximal.toFixed(2)} d=${f.distal.toFixed(2)}`;
-
-  document.getElementById('scrubber').value = currentFrame;
-  const timeS = (currentFrame / trajectory.fps).toFixed(2);
-  document.getElementById('frame-counter').textContent =
-    `Frame ${currentFrame} / ${trajectory.num_frames - 1} · ${timeS}s`;
-}
-
-// Playback controls
-window.togglePlay = function() {
-  playing = !playing;
-  const btn = document.getElementById('btn-play');
-  btn.textContent = playing ? '⏸ Pause' : '▶ Play';
-  btn.classList.toggle('active', playing);
-  if (playing) { lastTime = performance.now(); accumulator = 0; }
-};
-
-window.stepFrame = function(delta) {
-  playing = false;
-  document.getElementById('btn-play').textContent = '▶ Play';
-  document.getElementById('btn-play').classList.remove('active');
-  updateFrame(currentFrame + delta);
-};
-
-window.resetPlay = function() {
-  playing = false;
-  document.getElementById('btn-play').textContent = '▶ Play';
-  document.getElementById('btn-play').classList.remove('active');
-  updateFrame(0);
-};
-
-window.setSpeed = function(v) { speed = parseFloat(v); };
-
-document.getElementById('scrubber').addEventListener('input', (e) => {
-  updateFrame(parseInt(e.target.value));
-});
-
-window.addEventListener('resize', () => {
-  camera.aspect = window.innerWidth / window.innerHeight;
-  camera.updateProjectionMatrix();
-  renderer.setSize(window.innerWidth, window.innerHeight);
-});
-
-function animate(now) {
-  requestAnimationFrame(animate);
-  controls.update();
-
-  if (playing && trajectory) {
-    const dt = (now - lastTime) / 1000;
-    lastTime = now;
-    accumulator += dt * speed;
-    const frameDuration = 1.0 / trajectory.fps;
-    while (accumulator >= frameDuration) {
-      accumulator -= frameDuration;
-      if (currentFrame < trajectory.num_frames - 1) {
-        updateFrame(currentFrame + 1);
-      } else {
-        playing = false;
-        document.getElementById('btn-play').textContent = '▶ Play';
-        document.getElementById('btn-play').classList.remove('active');
-        break;
-      }
-    }
-  }
-
-  renderer.render(scene, camera);
-}
-requestAnimationFrame(animate);
-
-Promise.all([loadURDF(), loadTrajectory()])
-  .then(() => computeOffset())
-  .catch(err => console.error(err));
-</script>
-</body>
-</html>

src/lerobot/robots/openarm_follower/urdf/viewer.html

@@ -1,311 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8" />
-<title>OpenArm URDF Viewer</title>
-<style>
-  * { margin: 0; padding: 0; box-sizing: border-box; }
-  body { background: #1a1a2e; overflow: hidden; font-family: 'IBM Plex Mono', monospace; }
-  canvas { display: block; }
-  #info {
-    position: absolute; top: 16px; left: 16px;
-    color: #e0e0e0; font-size: 13px; line-height: 1.6;
-    background: rgba(0,0,0,0.7); padding: 14px 18px; border-radius: 8px;
-    border: 1px solid #333; max-width: 340px; z-index: 10;
-  }
-  #info h2 { font-size: 15px; color: #fff; margin-bottom: 8px; }
-  .legend { display: flex; align-items: center; gap: 8px; margin: 4px 0; }
-  .dot { width: 12px; height: 12px; border-radius: 50%; display: inline-block; flex-shrink: 0; }
-  .dot-red { background: #ff4444; }
-  .dot-green { background: #44ff44; }
-  .dot-blue { background: #4488ff; }
-  #frame-select { margin-top: 10px; }
-  #frame-select button {
-    background: #333; color: #e0e0e0; border: 1px solid #555;
-    padding: 6px 10px; margin: 2px; border-radius: 4px; cursor: pointer;
-    font-family: inherit; font-size: 12px;
-  }
-  #frame-select button:hover { background: #555; }
-  #frame-select button.active { background: #4488ff; color: #fff; border-color: #4488ff; }
-  #status { margin-top: 8px; font-size: 11px; color: #888; }
-</style>
-<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;600&display=swap" rel="stylesheet">
-</head>
-<body>
-
-<div id="info">
-  <h2>OpenArm Right Arm — EE Frame Options</h2>
-  <div class="legend"><span class="dot dot-red"></span> openarm_right_link7 (wrist output)</div>
-  <div class="legend"><span class="dot" style="background:#ffaa00"></span> openarm_right_ee_target (+5cm)</div>
-  <div class="legend"><span class="dot dot-green"></span> openarm_right_hand (+10cm)</div>
-  <div class="legend"><span class="dot dot-blue"></span> openarm_right_hand_tcp (+18cm)</div>
-  <div id="frame-select">
-    <button onclick="focusFrame('link7')" class="active">link7</button>
-    <button onclick="focusFrame('ee_target')">ee_target</button>
-    <button onclick="focusFrame('hand')">hand</button>
-    <button onclick="focusFrame('tcp')">hand_tcp</button>
-  </div>
-  <div id="status">Loading URDF...</div>
-  <p style="margin-top:8px;font-size:11px;color:#888;">Drag to orbit · Scroll to zoom · Right-drag to pan</p>
-</div>
-
-<script type="importmap">
-{
-  "imports": {
-    "three": "https://cdn.jsdelivr.net/npm/three@0.169.0/build/three.module.js",
-    "three/examples/jsm/": "https://cdn.jsdelivr.net/npm/three@0.169.0/examples/jsm/"
-  }
-}
-</script>
-
-<script type="module">
-import * as THREE from 'three';
-import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
-import { STLLoader } from 'three/examples/jsm/loaders/STLLoader.js';
-
-const statusEl = document.getElementById('status');
-
-const scene = new THREE.Scene();
-scene.background = new THREE.Color(0x1a1a2e);
-
-const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.01, 100);
-camera.position.set(0.8, 1.0, 1.8);
-
-const renderer = new THREE.WebGLRenderer({ antialias: true });
-renderer.setSize(window.innerWidth, window.innerHeight);
-renderer.setPixelRatio(window.devicePixelRatio);
-renderer.shadowMap.enabled = true;
-document.body.appendChild(renderer.domElement);
-
-const controls = new OrbitControls(camera, renderer.domElement);
-controls.target.set(0, 0, 0.9);
-controls.enableDamping = true;
-controls.dampingFactor = 0.08;
-controls.update();
-
-// Lighting
-scene.add(new THREE.AmbientLight(0xffffff, 0.6));
-const dirLight = new THREE.DirectionalLight(0xffffff, 1.2);
-dirLight.position.set(3, 5, 4);
-scene.add(dirLight);
-scene.add(new THREE.DirectionalLight(0x8888ff, 0.4).translateX(-2).translateY(1).translateZ(-3));
-
-// Ground grid
-scene.add(new THREE.GridHelper(4, 40, 0x333355, 0x222244));
-scene.add(new THREE.AxesHelper(0.3));
-
-// Parse URDF manually — build the kinematic tree and load STL meshes
-const stlLoader = new STLLoader();
-const robotGroup = new THREE.Group();
-scene.add(robotGroup);
-
-async function loadURDF() {
-  const resp = await fetch('./openarm_bimanual_pybullet.urdf');
-  const text = await resp.text();
-  const parser = new DOMParser();
-  const xml = parser.parseFromString(text, 'text/xml');
-
-  // Parse links and joints
-  const links = {};
-  const joints = [];
-
-  for (const linkEl of xml.querySelectorAll('link')) {
-    const name = linkEl.getAttribute('name');
-    const group = new THREE.Group();
-    group.name = name;
-
-    // Try to load visual mesh
-    const visual = linkEl.querySelector('visual');
-    if (visual) {
-      const meshEl = visual.querySelector('mesh');
-      const originEl = visual.querySelector('origin');
-      if (meshEl) {
-        const filename = meshEl.getAttribute('filename');
-        const scaleStr = meshEl.getAttribute('scale');
-        const scale = scaleStr ? scaleStr.split(' ').map(Number) : [1, 1, 1];
-
-        let xyz = [0, 0, 0];
-        if (originEl && originEl.getAttribute('xyz')) {
-          xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-        }
-
-        if (filename.endsWith('.stl')) {
-          try {
-            const geo = await new Promise((resolve, reject) => {
-              stlLoader.load(filename, resolve, undefined, reject);
-            });
-            const mat = new THREE.MeshStandardMaterial({
-              color: 0x6688aa,
-              metalness: 0.3,
-              roughness: 0.6,
-              transparent: true,
-              opacity: 0.7,
-            });
-            const mesh = new THREE.Mesh(geo, mat);
-            mesh.scale.set(scale[0], scale[1], scale[2]);
-            mesh.position.set(xyz[0], xyz[1], xyz[2]);
-            group.add(mesh);
-          } catch (e) {
-            // Mesh file not found, skip
-          }
-        }
-      }
-    }
-
-    links[name] = group;
-  }
-
-  // Parse joints and build hierarchy
-  for (const jointEl of xml.querySelectorAll('joint')) {
-    const name = jointEl.getAttribute('name');
-    const type = jointEl.getAttribute('type');
-    const parentName = jointEl.querySelector('parent').getAttribute('link');
-    const childName = jointEl.querySelector('child').getAttribute('link');
-    const originEl = jointEl.querySelector('origin');
-
-    let xyz = [0, 0, 0];
-    let rpy = [0, 0, 0];
-    if (originEl) {
-      if (originEl.getAttribute('xyz')) xyz = originEl.getAttribute('xyz').split(' ').map(Number);
-      if (originEl.getAttribute('rpy')) rpy = originEl.getAttribute('rpy').split(' ').map(Number);
-    }
-
-    joints.push({ name, type, parentName, childName, xyz, rpy });
-  }
-
-  // Build tree
-  const rootLinks = new Set(Object.keys(links));
-  for (const j of joints) {
-    rootLinks.delete(j.childName);
-  }
-
-  for (const j of joints) {
-    const parent = links[j.parentName];
-    const child = links[j.childName];
-    if (parent && child) {
-      child.position.set(j.xyz[0], j.xyz[1], j.xyz[2]);
-      if (j.rpy[0] || j.rpy[1] || j.rpy[2]) {
-        child.rotation.set(j.rpy[0], j.rpy[1], j.rpy[2], 'XYZ');
-      }
-      parent.add(child);
-    }
-  }
-
-  // Add root links to scene
-  for (const name of rootLinks) {
-    if (links[name]) robotGroup.add(links[name]);
-  }
-
-  // Place markers at the three EE frame candidates
-  const targets = {
-    link7: 'openarm_right_link7',
-    ee_target: 'openarm_right_ee_target',
-    hand: 'openarm_right_hand',
-    tcp: 'openarm_right_hand_tcp',
-  };
-  const colors = { link7: 0xff4444, ee_target: 0xffaa00, hand: 0x44ff44, tcp: 0x4488ff };
-  const labels = { link7: 'link7', ee_target: 'ee_target', hand: 'hand', tcp: 'hand_tcp' };
-
-  for (const [key, linkName] of Object.entries(targets)) {
-    const link = links[linkName];
-    if (!link) continue;
-
-    // Marker sphere
-    const sphere = new THREE.Mesh(
-      new THREE.SphereGeometry(0.018, 16, 16),
-      new THREE.MeshStandardMaterial({ color: colors[key], emissive: colors[key], emissiveIntensity: 0.6 })
-    );
-    link.add(sphere);
-
-    // Ring around sphere for visibility
-    const ring = new THREE.Mesh(
-      new THREE.TorusGeometry(0.03, 0.003, 8, 32),
-      new THREE.MeshStandardMaterial({ color: colors[key], emissive: colors[key], emissiveIntensity: 0.4 })
-    );
-    link.add(ring);
-
-    // Axes helper
-    link.add(new THREE.AxesHelper(0.08));
-
-    // Sprite label
-    const canvas = document.createElement('canvas');
-    canvas.width = 512; canvas.height = 80;
-    const ctx = canvas.getContext('2d');
-    ctx.font = 'bold 36px IBM Plex Mono, monospace';
-    ctx.fillStyle = '#' + colors[key].toString(16).padStart(6, '0');
-    ctx.fillText(labels[key], 4, 50);
-    const tex = new THREE.CanvasTexture(canvas);
-    const sprite = new THREE.Sprite(new THREE.SpriteMaterial({ map: tex, depthTest: false }));
-    sprite.scale.set(0.3, 0.05, 1);
-    sprite.position.set(0.06, 0.0, 0.03);
-    link.add(sprite);
-  }
-
-  // Dashed lines between markers (in world space)
-  robotGroup.updateMatrixWorld(true);
-  const positions = {};
-  for (const [key, linkName] of Object.entries(targets)) {
-    const link = links[linkName];
-    if (link) {
-      const wp = new THREE.Vector3();
-      link.getWorldPosition(wp);
-      positions[key] = wp;
-    }
-  }
-
-  function addDashedLine(from, to) {
-    const geo = new THREE.BufferGeometry().setFromPoints([from, to]);
-    const mat = new THREE.LineDashedMaterial({ color: 0xaaaaaa, dashSize: 0.012, gapSize: 0.008 });
-    const line = new THREE.Line(geo, mat);
-    line.computeLineDistances();
-    scene.add(line);
-  }
-  if (positions.link7 && positions.hand) addDashedLine(positions.link7, positions.hand);
-  if (positions.hand && positions.tcp) addDashedLine(positions.hand, positions.tcp);
-
-  // Store for focus buttons
-  window._framePositions = positions;
-  window._links = links;
-  window._targets = targets;
-
-  // Focus on the hand area
-  if (positions.hand) {
-    controls.target.copy(positions.hand);
-    camera.position.set(positions.hand.x + 0.5, positions.hand.y + 0.4, positions.hand.z + 0.5);
-    controls.update();
-  }
-
-  const meshCount = robotGroup.children.length;
-  statusEl.textContent = `Loaded. Right arm chain visible with ${Object.keys(links).length} links.`;
-}
-
-window.focusFrame = function(key) {
-  const pos = window._framePositions?.[key];
-  if (!pos) return;
-  controls.target.copy(pos);
-  camera.position.set(pos.x + 0.35, pos.y + 0.25, pos.z + 0.35);
-  controls.update();
-  document.querySelectorAll('#frame-select button').forEach(b => b.classList.remove('active'));
-  event.target.classList.add('active');
-};
-
-window.addEventListener('resize', () => {
-  camera.aspect = window.innerWidth / window.innerHeight;
-  camera.updateProjectionMatrix();
-  renderer.setSize(window.innerWidth, window.innerHeight);
-});
-
-function animate() {
-  requestAnimationFrame(animate);
-  controls.update();
-  renderer.render(scene, camera);
-}
-animate();
-
-loadURDF().catch(err => {
-  statusEl.textContent = `Error: ${err.message}`;
-  console.error(err);
-});
-</script>
-</body>
-</html>

src/lerobot/robots/so_follower/robot_kinematic_processor.py

@@ -255,16 +255,19 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
     """
     Computes desired joint positions from a target end-effector pose using inverse kinematics (IK).
 
+    This step translates a Cartesian command (position and orientation of the end-effector) into
+    the corresponding joint-space commands for each motor.
+
     Attributes:
         kinematics: The robot's kinematic model for inverse kinematics.
-        motor_names: Arm joint names for IK computation.
-        gripper_names: Gripper joint name(s). ee.gripper_pos is written to all of them.
+        motor_names: A list of motor names for which to compute joint positions.
+        q_curr: Internal state storing the last joint positions, used as an initial guess for the IK solver.
         initial_guess_current_joints: If True, use the robot's current joint state as the IK guess.
+            If False, use the solution from the previous step.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
     q_curr: np.ndarray | None = field(default=None, init=False, repr=False)
     initial_guess_current_joints: bool = True
 
@@ -275,73 +278,63 @@ class InverseKinematicsEEToJoints(RobotActionProcessorStep):
         wx = action.pop("ee.wx")
         wy = action.pop("ee.wy")
         wz = action.pop("ee.wz")
+        gripper_pos = action.pop("ee.gripper_pos")
 
-        ee_keys = [x, y, z, wx, wy, wz]
-
-        if self.gripper_names:
-            gripper_pos = action.pop("ee.gripper_pos")
-            ee_keys.append(gripper_pos)
-        if None in ee_keys:
-            raise ValueError("Missing required end-effector pose components in action")
+        if None in (x, y, z, wx, wy, wz, gripper_pos):
+            raise ValueError(
+                "Missing required end-effector pose components: ee.x, ee.y, ee.z, ee.wx, ee.wy, ee.wz, ee.gripper_pos must all be present in action"
+            )
 
         observation = self.transition.get(TransitionKey.OBSERVATION).copy()
         if observation is None:
-            raise ValueError("Joints observation is required for computing robot kinematics")
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
         q_raw = np.array(
-            [
-                float(v)
-                for k, v in observation.items()
-                if isinstance(k, str) and k.endswith(".pos") and k.removesuffix(".pos") in self.motor_names
-            ],
+            [float(v) for k, v in observation.items() if isinstance(k, str) and k.endswith(".pos")],
             dtype=float,
         )
+        if q_raw is None:
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
-        if self.initial_guess_current_joints:
+        if self.initial_guess_current_joints:  # Use current joints as initial guess
             self.q_curr = q_raw
-        else:
+        else:  # Use previous ik solution as initial guess
             if self.q_curr is None:
                 self.q_curr = q_raw
 
+        # Build desired 4x4 transform from pos + rotvec (twist)
         t_des = np.eye(4, dtype=float)
         t_des[:3, :3] = Rotation.from_rotvec([wx, wy, wz]).as_matrix()
         t_des[:3, 3] = [x, y, z]
 
+        # Compute inverse kinematics
         q_target = self.kinematics.inverse_kinematics(self.q_curr, t_des)
         self.q_curr = q_target
 
+        # TODO: This is sentitive to order of motor_names = q_target mapping
         for i, name in enumerate(self.motor_names):
-            action[f"{name}.pos"] = float(q_target[i])
+            if name != "gripper":
+                action[f"{name}.pos"] = float(q_target[i])
+            else:
+                action["gripper.pos"] = float(gripper_pos)
 
-        if self.gripper_names:
-            for gname in self.gripper_names:
-                action[f"{gname}.pos"] = float(gripper_pos)
-
-        # When gripper_names is empty, gripper keys (e.g. proximal.pos, distal.pos)
-        # are already in the action dict as absolute positions — left untouched.
         return action
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        ee_feats = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            ee_feats.append("gripper_pos")
-        for feat in ee_feats:
+        for feat in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION].pop(f"ee.{feat}", None)
 
         for name in self.motor_names:
             features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
                 type=FeatureType.ACTION, shape=(1,)
             )
-        for name in self.gripper_names:
-            features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
-                type=FeatureType.ACTION, shape=(1,)
-            )
 
         return features
 
     def reset(self):
+        """Resets the initial guess for the IK solver."""
         self.q_curr = None
 
 
@@ -409,39 +402,24 @@ class GripperVelocityToJoint(RobotActionProcessorStep):
 
 
 def compute_forward_kinematics_joints_to_ee(
-    joints: dict[str, Any],
-    kinematics: RobotKinematics,
-    motor_names: list[str],
-    gripper_names: list[str] | None = None,
+    joints: dict[str, Any], kinematics: RobotKinematics, motor_names: list[str]
 ) -> dict[str, Any]:
-    if gripper_names is None:
-        gripper_names = ["gripper"]
-
     motor_joint_values = [joints[f"{n}.pos"] for n in motor_names]
 
     q = np.array(motor_joint_values, dtype=float)
     t = kinematics.forward_kinematics(q)
     pos = t[:3, 3]
     tw = Rotation.from_matrix(t[:3, :3]).as_rotvec()
-
+    gripper_pos = joints["gripper.pos"]
     for n in motor_names:
         joints.pop(f"{n}.pos")
-
     joints["ee.x"] = float(pos[0])
     joints["ee.y"] = float(pos[1])
     joints["ee.z"] = float(pos[2])
     joints["ee.wx"] = float(tw[0])
     joints["ee.wy"] = float(tw[1])
     joints["ee.wz"] = float(tw[2])
-
-    # When gripper_names is non-empty, fold them into ee.gripper_pos (e.g. SO100).
-    # When empty, gripper joints pass through as-is (absolute position control).
-    if gripper_names:
-        gripper_pos = joints[f"{gripper_names[0]}.pos"]
-        for n in gripper_names:
-            joints.pop(f"{n}.pos", None)
-        joints["ee.gripper_pos"] = float(gripper_pos)
-
+    joints["ee.gripper_pos"] = float(gripper_pos)
     return joints
 
 
@@ -451,33 +429,27 @@ class ForwardKinematicsJointsToEEObservation(ObservationProcessorStep):
     """
     Computes the end-effector pose from joint positions using forward kinematics (FK).
 
+    This step is typically used to add the robot's Cartesian pose to the observation space,
+    which can be useful for visualization or as an input to a policy.
+
     Attributes:
         kinematics: The robot's kinematic model.
-        motor_names: Arm joint names used for FK computation.
-        gripper_names: Gripper joint name(s) to fold into ee.gripper_pos.
-            Empty list means gripper joints pass through as absolute positions.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def observation(self, observation: RobotObservation) -> RobotObservation:
-        return compute_forward_kinematics_joints_to_ee(
-            observation, self.kinematics, self.motor_names, self.gripper_names
-        )
+        return compute_forward_kinematics_joints_to_ee(observation, self.kinematics, self.motor_names)
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
+        # We only use the ee pose in the dataset, so we don't need the joint positions
         for n in self.motor_names:
             features[PipelineFeatureType.OBSERVATION].pop(f"{n}.pos", None)
-        ee_keys = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            for n in self.gripper_names:
-                features[PipelineFeatureType.OBSERVATION].pop(f"{n}.pos", None)
-            ee_keys.append("gripper_pos")
-        for k in ee_keys:
+        # We specify the dataset features of this step that we want to be stored in the dataset
+        for k in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.OBSERVATION][f"ee.{k}"] = PolicyFeature(
                 type=FeatureType.STATE, shape=(1,)
             )
@@ -490,33 +462,27 @@ class ForwardKinematicsJointsToEEAction(RobotActionProcessorStep):
     """
     Computes the end-effector pose from joint positions using forward kinematics (FK).
 
+    This step is typically used to add the robot's Cartesian pose to the observation space,
+    which can be useful for visualization or as an input to a policy.
+
     Attributes:
         kinematics: The robot's kinematic model.
-        motor_names: Arm joint names used for FK computation.
-        gripper_names: Gripper joint name(s) to fold into ee.gripper_pos.
-            Empty list means gripper joints pass through as absolute positions.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def action(self, action: RobotAction) -> RobotAction:
-        return compute_forward_kinematics_joints_to_ee(
-            action, self.kinematics, self.motor_names, self.gripper_names
-        )
+        return compute_forward_kinematics_joints_to_ee(action, self.kinematics, self.motor_names)
 
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
+        # We only use the ee pose in the dataset, so we don't need the joint positions
         for n in self.motor_names:
             features[PipelineFeatureType.ACTION].pop(f"{n}.pos", None)
-        ee_keys = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            for n in self.gripper_names:
-                features[PipelineFeatureType.ACTION].pop(f"{n}.pos", None)
-            ee_keys.append("gripper_pos")
-        for k in ee_keys:
+        # We specify the dataset features of this step that we want to be stored in the dataset
+        for k in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION][f"ee.{k}"] = PolicyFeature(
                 type=FeatureType.STATE, shape=(1,)
             )
@@ -528,14 +494,13 @@ class ForwardKinematicsJointsToEEAction(RobotActionProcessorStep):
 class ForwardKinematicsJointsToEE(ProcessorStep):
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
 
     def __post_init__(self):
         self.joints_to_ee_action_processor = ForwardKinematicsJointsToEEAction(
-            kinematics=self.kinematics, motor_names=self.motor_names, gripper_names=self.gripper_names
+            kinematics=self.kinematics, motor_names=self.motor_names
         )
         self.joints_to_ee_observation_processor = ForwardKinematicsJointsToEEObservation(
-            kinematics=self.kinematics, motor_names=self.motor_names, gripper_names=self.gripper_names
+            kinematics=self.kinematics, motor_names=self.motor_names
         )
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
@@ -559,13 +524,13 @@ class ForwardKinematicsJointsToEE(ProcessorStep):
 @dataclass
 class InverseKinematicsRLStep(ProcessorStep):
     """
-    IK step for the RL pipeline. Same logic as InverseKinematicsEEToJoints but
-    operates on EnvTransition directly and stores the IK solution.
+    Computes desired joint positions from a target end-effector pose using inverse kinematics (IK).
+
+    This is modified from the InverseKinematicsEEToJoints step to be used in the RL pipeline.
     """
 
     kinematics: RobotKinematics
     motor_names: list[str]
-    gripper_names: list[str] = field(default_factory=lambda: ["gripper"])
     q_curr: np.ndarray | None = field(default=None, init=False, repr=False)
     initial_guess_current_joints: bool = True
 
@@ -573,7 +538,7 @@ class InverseKinematicsRLStep(ProcessorStep):
         new_transition = dict(transition)
         action = new_transition.get(TransitionKey.ACTION)
         if action is None:
-            raise ValueError("Action is required for InverseKinematicsRLStep")
+            raise ValueError("Action is required for InverseKinematicsEEToJoints")
         action = dict(action)
 
         x = action.pop("ee.x")
@@ -582,46 +547,45 @@ class InverseKinematicsRLStep(ProcessorStep):
         wx = action.pop("ee.wx")
         wy = action.pop("ee.wy")
         wz = action.pop("ee.wz")
+        gripper_pos = action.pop("ee.gripper_pos")
 
-        ee_keys = [x, y, z, wx, wy, wz]
-        if self.gripper_names:
-            gripper_pos = action.pop("ee.gripper_pos")
-            ee_keys.append(gripper_pos)
-        if None in ee_keys:
-            raise ValueError("Missing required end-effector pose components in action")
+        if None in (x, y, z, wx, wy, wz, gripper_pos):
+            raise ValueError(
+                "Missing required end-effector pose components: ee.x, ee.y, ee.z, ee.wx, ee.wy, ee.wz, ee.gripper_pos must all be present in action"
+            )
 
         observation = new_transition.get(TransitionKey.OBSERVATION).copy()
         if observation is None:
-            raise ValueError("Joints observation is required for computing robot kinematics")
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
         q_raw = np.array(
-            [
-                float(v)
-                for k, v in observation.items()
-                if isinstance(k, str) and k.endswith(".pos") and k.removesuffix(".pos") in self.motor_names
-            ],
+            [float(v) for k, v in observation.items() if isinstance(k, str) and k.endswith(".pos")],
             dtype=float,
         )
+        if q_raw is None:
+            raise ValueError("Joints observation is require for computing robot kinematics")
 
-        if self.initial_guess_current_joints:
+        if self.initial_guess_current_joints:  # Use current joints as initial guess
             self.q_curr = q_raw
-        else:
+        else:  # Use previous ik solution as initial guess
             if self.q_curr is None:
                 self.q_curr = q_raw
 
+        # Build desired 4x4 transform from pos + rotvec (twist)
         t_des = np.eye(4, dtype=float)
         t_des[:3, :3] = Rotation.from_rotvec([wx, wy, wz]).as_matrix()
         t_des[:3, 3] = [x, y, z]
 
+        # Compute inverse kinematics
         q_target = self.kinematics.inverse_kinematics(self.q_curr, t_des)
         self.q_curr = q_target
 
+        # TODO: This is sentitive to order of motor_names = q_target mapping
         for i, name in enumerate(self.motor_names):
-            action[f"{name}.pos"] = float(q_target[i])
-
-        if self.gripper_names:
-            for gname in self.gripper_names:
-                action[f"{gname}.pos"] = float(gripper_pos)
+            if name != "gripper":
+                action[f"{name}.pos"] = float(q_target[i])
+            else:
+                action["gripper.pos"] = float(gripper_pos)
 
         new_transition[TransitionKey.ACTION] = action
         complementary_data = new_transition.get(TransitionKey.COMPLEMENTARY_DATA, {})
@@ -632,22 +596,16 @@ class InverseKinematicsRLStep(ProcessorStep):
     def transform_features(
         self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
     ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
-        ee_feats = ["x", "y", "z", "wx", "wy", "wz"]
-        if self.gripper_names:
-            ee_feats.append("gripper_pos")
-        for feat in ee_feats:
+        for feat in ["x", "y", "z", "wx", "wy", "wz", "gripper_pos"]:
             features[PipelineFeatureType.ACTION].pop(f"ee.{feat}", None)
 
         for name in self.motor_names:
             features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
                 type=FeatureType.ACTION, shape=(1,)
             )
-        for name in self.gripper_names:
-            features[PipelineFeatureType.ACTION][f"{name}.pos"] = PolicyFeature(
-                type=FeatureType.ACTION, shape=(1,)
-            )
 
         return features
 
     def reset(self):
+        """Resets the initial guess for the IK solver."""
         self.q_curr = None

src/lerobot/scripts/lerobot_edit_dataset.py

@@ -254,10 +254,6 @@ class RecomputeStatsConfig(OperationConfig):
     relative_exclude_joints: list[str] | None = None
     chunk_size: int = 50
     num_workers: int = 0
-    relative_state: bool = False
-    relative_exclude_state_joints: list[str] | None = None
-    state_obs_steps: int = 2
-    derive_state_from_action: bool = False
 
 
 @OperationConfig.register_subclass("info")
@@ -567,14 +563,6 @@ def handle_recompute_stats(cfg: EditDatasetConfig) -> None:
             f"Relative action stats enabled (chunk_size={cfg.operation.chunk_size}, "
             f"exclude_joints={cfg.operation.relative_exclude_joints})"
         )
-    if cfg.operation.relative_state:
-        logging.info(
-            f"Relative state stats enabled (state_obs_steps={cfg.operation.state_obs_steps}, "
-            f"exclude_state_joints={cfg.operation.relative_exclude_state_joints})"
-        )
-
-    if cfg.operation.derive_state_from_action:
-        logging.info("Derive state from action enabled (implies relative_state=True, state_obs_steps=2)")
 
     recompute_stats(
         dataset,
@@ -583,10 +571,6 @@ def handle_recompute_stats(cfg: EditDatasetConfig) -> None:
         relative_exclude_joints=cfg.operation.relative_exclude_joints,
         chunk_size=cfg.operation.chunk_size,
         num_workers=cfg.operation.num_workers,
-        relative_state=cfg.operation.relative_state,
-        relative_exclude_state_joints=cfg.operation.relative_exclude_state_joints,
-        state_obs_steps=cfg.operation.state_obs_steps,
-        derive_state_from_action=cfg.operation.derive_state_from_action,
     )
 
     logging.info(f"Stats written to {dataset.root}")

src/lerobot/scripts/lerobot_eval.py

@@ -201,6 +201,11 @@ def rollout(
                     "You're likely using an older version of gymnasium (< 1.0). Please upgrade."
                 )
             successes = final_info["is_success"].tolist()
+        elif "is_success" in info:
+            is_success = info["is_success"]
+            successes = (
+                is_success.tolist() if hasattr(is_success, "tolist") else [bool(is_success)] * env.num_envs
+            )
         else:
             successes = [False] * env.num_envs
 

src/lerobot/scripts/lerobot_record.py

@@ -74,6 +74,8 @@ from pathlib import Path
 from pprint import pformat
 from typing import Any
 
+import torch
+
 from lerobot.cameras import (  # noqa: F401
     CameraConfig,  # noqa: F401
 )
@@ -90,6 +92,7 @@ from lerobot.datasets.pipeline_features import aggregate_pipeline_dataset_featur
 from lerobot.datasets.video_utils import VideoEncodingManager
 from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
+from lerobot.policies.rtc import ActionInterpolator
 from lerobot.policies.utils import make_robot_action
 from lerobot.processor import (
     PolicyAction,
@@ -226,6 +229,9 @@ class RecordConfig:
     play_sounds: bool = True
     # Resume recording on an existing dataset.
     resume: bool = False
+    # Action interpolation multiplier for smoother policy control (1=off, 2=2x, 3=3x)
+    # Only applies when using a policy (not teleop)
+    interpolation_multiplier: int = 1
 
     def __post_init__(self):
         # HACK: We parse again the cli args here to get the pretrained path if there was one.
@@ -298,6 +304,7 @@ def record_loop(
     control_time_s: int | None = None,
     single_task: str | None = None,
     display_data: bool = False,
+    interpolator: ActionInterpolator | None = None,
     display_compressed_images: bool = False,
 ):
     if dataset is not None and dataset.fps != fps:
@@ -334,6 +341,16 @@ def record_loop(
         preprocessor.reset()
         postprocessor.reset()
 
+    # Reset interpolator if provided
+    if interpolator is not None:
+        interpolator.reset()
+
+    # Calculate control interval based on interpolation
+    use_interpolation = interpolator is not None and interpolator.enabled and policy is not None
+    control_interval = interpolator.get_control_interval(fps) if interpolator else 1 / fps
+    # Pre-compute action key order outside the hot loop — it won't change mid-episode.
+    action_keys = sorted(robot.action_features) if use_interpolation else []
+
     no_action_count = 0
     timestamp = 0
     start_episode_t = time.perf_counter()
@@ -353,28 +370,68 @@ def record_loop(
         if policy is not None or dataset is not None:
             observation_frame = build_dataset_frame(dataset.features, obs_processed, prefix=OBS_STR)
 
+        # Track whether this iteration should be recorded to the dataset.
+        # Interpolated-only iterations send actions to the robot but don't record frames,
+        # keeping the dataset at the original fps while the robot moves at the higher rate.
+        is_record_frame = True
+
         # Get action from either policy or teleop
         if policy is not None and preprocessor is not None and postprocessor is not None:
-            action_values = predict_action(
-                observation=observation_frame,
-                policy=policy,
-                device=get_safe_torch_device(policy.config.device),
-                preprocessor=preprocessor,
-                postprocessor=postprocessor,
-                use_amp=policy.config.use_amp,
-                task=single_task,
-                robot_type=robot.robot_type,
-            )
+            # With interpolation: only call policy when interpolator needs new action
+            if use_interpolation:
+                ran_inference = False
 
-            act_processed_policy: RobotAction = make_robot_action(action_values, dataset.features)
+                if interpolator.needs_new_action():
+                    action_values = predict_action(
+                        observation=observation_frame,
+                        policy=policy,
+                        device=get_safe_torch_device(policy.config.device),
+                        preprocessor=preprocessor,
+                        postprocessor=postprocessor,
+                        use_amp=policy.config.use_amp,
+                        task=single_task,
+                        robot_type=robot.robot_type,
+                    )
+                    act_processed_policy = make_robot_action(action_values, dataset.features)
+                    robot_action_to_send = robot_action_processor((act_processed_policy, obs))
+
+                    action_tensor = torch.tensor([robot_action_to_send[k] for k in action_keys])
+                    interpolator.add(action_tensor)
+                    ran_inference = True
+
+                interp_action = interpolator.get()
+                if interp_action is not None:
+                    robot_action_to_send = {k: interp_action[i].item() for i, k in enumerate(action_keys)}
+                    action_values = robot_action_to_send
+                else:
+                    continue
+
+                is_record_frame = ran_inference
+            else:
+                action_values = predict_action(
+                    observation=observation_frame,
+                    policy=policy,
+                    device=get_safe_torch_device(policy.config.device),
+                    preprocessor=preprocessor,
+                    postprocessor=postprocessor,
+                    use_amp=policy.config.use_amp,
+                    task=single_task,
+                    robot_type=robot.robot_type,
+                )
+                act_processed_policy: RobotAction = make_robot_action(action_values, dataset.features)
+                # Applies a pipeline to the action, default is IdentityProcessor
+                robot_action_to_send = robot_action_processor((act_processed_policy, obs))
+                action_values = robot_action_to_send
 
         elif policy is None and isinstance(teleop, Teleoperator):
+            act = teleop.get_action()
             if robot.name == "unitree_g1":
                 teleop.send_feedback(obs)
-            act = teleop.get_action()
 
             # Applies a pipeline to the raw teleop action, default is IdentityProcessor
             act_processed_teleop = teleop_action_processor((act, obs))
+            action_values = act_processed_teleop
+            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
 
         elif policy is None and isinstance(teleop, list):
             arm_action = teleop_arm.get_action()
@@ -383,6 +440,8 @@ def record_loop(
             base_action = robot._from_keyboard_to_base_action(keyboard_action)
             act = {**arm_action, **base_action} if len(base_action) > 0 else arm_action
             act_processed_teleop = teleop_action_processor((act, obs))
+            action_values = act_processed_teleop
+            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
         else:
             no_action_count += 1
             if no_action_count == 1 or no_action_count % 10 == 0:
@@ -393,22 +452,14 @@ def record_loop(
                 )
             continue
 
-        # Applies a pipeline to the action, default is IdentityProcessor
-        if policy is not None and act_processed_policy is not None:
-            action_values = act_processed_policy
-            robot_action_to_send = robot_action_processor((act_processed_policy, obs))
-        else:
-            action_values = act_processed_teleop
-            robot_action_to_send = robot_action_processor((act_processed_teleop, obs))
-
         # Send action to robot
         # Action can eventually be clipped using `max_relative_target`,
         # so action actually sent is saved in the dataset. action = postprocessor.process(action)
         # TODO(steven, pepijn, adil): we should use a pipeline step to clip the action, so the sent action is the action that we input to the robot.
         _sent_action = robot.send_action(robot_action_to_send)
 
-        # Write to dataset
-        if dataset is not None:
+        # Write to dataset (only on real policy frames, not interpolated-only iterations)
+        if dataset is not None and is_record_frame:
             action_frame = build_dataset_frame(dataset.features, action_values, prefix=ACTION)
             frame = {**observation_frame, **action_frame, "task": single_task}
             dataset.add_frame(frame)
@@ -420,7 +471,7 @@ def record_loop(
 
         dt_s = time.perf_counter() - start_loop_t
 
-        sleep_time_s: float = 1 / fps - dt_s
+        sleep_time_s: float = control_interval - dt_s
         if sleep_time_s < 0:
             logging.warning(
                 f"Record loop is running slower ({1 / dt_s:.1f} Hz) than the target FPS ({fps} Hz). Dataset frames might be dropped and robot control might be unstable. Common causes are: 1) Camera FPS not keeping up 2) Policy inference taking too long 3) CPU starvation"
@@ -506,6 +557,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
         policy = None if cfg.policy is None else make_policy(cfg.policy, ds_meta=dataset.meta)
         preprocessor = None
         postprocessor = None
+        interpolator = None
         if cfg.policy is not None:
             preprocessor, postprocessor = make_pre_post_processors(
                 policy_cfg=cfg.policy,
@@ -516,6 +568,10 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
                     "rename_observations_processor": {"rename_map": cfg.dataset.rename_map},
                 },
             )
+            # Create interpolator for smoother policy control
+            if cfg.interpolation_multiplier > 1:
+                interpolator = ActionInterpolator(multiplier=cfg.interpolation_multiplier)
+                logging.info(f"Action interpolation enabled: {cfg.interpolation_multiplier}x control rate")
 
         robot.connect()
         if teleop is not None:
@@ -547,6 +603,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
                     control_time_s=cfg.dataset.episode_time_s,
                     single_task=cfg.dataset.single_task,
                     display_data=cfg.display_data,
+                    interpolator=interpolator,
                     display_compressed_images=display_compressed_images,
                 )
 

src/lerobot/teleoperators/openarm_mini/openarm_mini.py

@@ -32,9 +32,15 @@ from .config_openarm_mini import OpenArmMiniConfig
 logger = logging.getLogger(__name__)
 
 # Motors whose direction is inverted during readout
-RIGHT_MOTORS_TO_FLIP = ["joint_1", "joint_2", "joint_3", "joint_4", "joint_5"]
+RIGHT_MOTORS_TO_FLIP = ["joint_1", "joint_2", "joint_3", "joint_4", "joint_5", "joint_7"]
 LEFT_MOTORS_TO_FLIP = ["joint_1", "joint_3", "joint_4", "joint_5", "joint_6", "joint_7"]
 
+# Leader joint 6 maps to follower joint 7 and vice versa
+JOINT_REMAP = {"joint_6": "joint_7", "joint_7": "joint_6"}
+JOINT_REMAP_REVERSE = {"joint_7": "joint_6", "joint_6": "joint_7"}
+
+GRIPPER_TELEOP_TO_DEGREES = -0.65
+
 
 class OpenArmMini(Teleoperator):
     """
@@ -95,6 +101,8 @@ class OpenArmMini(Teleoperator):
 
     @property
     def action_features(self) -> dict[str, type]:
+        # Right first, then left — matches the robot (BiOpenArmFollower) ordering
+        # and the dataset feature names recorded during data collection.
         features: dict[str, type] = {}
         for motor in self.bus_right.motors:
             features[f"right_{motor}.pos"] = float
@@ -276,16 +284,70 @@ class OpenArmMini(Teleoperator):
         right_positions = self.bus_right.sync_read("Present_Position")
         left_positions = self.bus_left.sync_read("Present_Position")
 
+        # Right first, then left — matches the robot (BiOpenArmFollower) ordering
+        # and the dataset feature names recorded during data collection.
+        # Joint 6↔7 remap: leader joint_6 → follower joint_7 and vice versa.
         action: dict[str, Any] = {}
         for motor, val in right_positions.items():
-            action[f"right_{motor}.pos"] = -val if motor in RIGHT_MOTORS_TO_FLIP else val
+            target = JOINT_REMAP.get(motor, motor)
+            if motor == "gripper":
+                # Convert gripper from teleop 0-100 to openarms degrees: 0→0°, 100→-65°
+                action[f"right_{target}.pos"] = val * GRIPPER_TELEOP_TO_DEGREES
+            else:
+                action[f"right_{target}.pos"] = -val if motor in RIGHT_MOTORS_TO_FLIP else val
         for motor, val in left_positions.items():
-            action[f"left_{motor}.pos"] = -val if motor in LEFT_MOTORS_TO_FLIP else val
+            target = JOINT_REMAP.get(motor, motor)
+            if motor == "gripper":
+                action[f"left_{target}.pos"] = val * GRIPPER_TELEOP_TO_DEGREES
+            else:
+                action[f"left_{target}.pos"] = -val if motor in LEFT_MOTORS_TO_FLIP else val
 
         dt_ms = (time.perf_counter() - start) * 1e3
         logger.debug(f"{self} read action: {dt_ms:.1f}ms")
         return action
 
+    def enable_torque(self) -> None:
+        """Enable torque on both arms for position control."""
+        self.bus_right.enable_torque()
+        self.bus_left.enable_torque()
+
+    def disable_torque(self) -> None:
+        """Disable torque on both arms for free movement."""
+        self.bus_right.disable_torque()
+        self.bus_left.disable_torque()
+
+    def write_goal_positions(self, positions: dict[str, float]) -> None:
+        """Write goal positions to motors (inverse of get_action flip/gripper/remap logic)."""
+        right_goals: dict[str, float] = {}
+        left_goals: dict[str, float] = {}
+
+        for key, val in positions.items():
+            if not key.endswith(".pos"):
+                continue
+            motor_name = key.removesuffix(".pos")
+            if motor_name.startswith("right_"):
+                base = motor_name.removeprefix("right_")
+                # Reverse remap: follower joint_7 → leader joint_6 and vice versa
+                target = JOINT_REMAP_REVERSE.get(base, base)
+                if base == "gripper":
+                    # Convert robot degrees to teleop 0-100: 0°→0, -65°→100
+                    right_goals[target] = val / GRIPPER_TELEOP_TO_DEGREES
+                else:
+                    # Un-flip using the ORIGINAL motor name (target = leader motor)
+                    right_goals[target] = -val if target in RIGHT_MOTORS_TO_FLIP else val
+            elif motor_name.startswith("left_"):
+                base = motor_name.removeprefix("left_")
+                target = JOINT_REMAP_REVERSE.get(base, base)
+                if base == "gripper":
+                    left_goals[target] = val / GRIPPER_TELEOP_TO_DEGREES
+                else:
+                    left_goals[target] = -val if target in LEFT_MOTORS_TO_FLIP else val
+
+        if right_goals:
+            self.bus_right.sync_write("Goal_Position", right_goals)
+        if left_goals:
+            self.bus_left.sync_write("Goal_Position", left_goals)
+
     def send_feedback(self, feedback: dict[str, float]) -> None:
         raise NotImplementedError("Feedback is not yet implemented for OpenArm Mini.")