Merge branch 'fix/storage-ci-runners' into fix/add-xvla-ci-main

Added detailed instructions for implementing a custom optimizer and modifying parameter retrieval for X-VLA finetuning.

Signed-off-by: Jinliang Zheng <54488861+2toinf@users.noreply.github.com>

.github/workflows/fast_tests.yml

@@ -68,8 +68,6 @@ jobs:
           persist-credentials: false
           lfs: true
 
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 
@@ -87,8 +85,14 @@ jobs:
           version: ${{ env.UV_VERSION }}
           python-version: ${{ env.PYTHON_VERSION }}
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Install lerobot with test extras
         run: uv sync --extra "test"
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run pytest
         run: uv run pytest tests -vv --maxfail=10

.github/workflows/full_tests.yml

@@ -66,8 +66,6 @@ jobs:
           lfs: true
           persist-credentials: false
 
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 
@@ -87,12 +85,21 @@ jobs:
       - name: Install lerobot with all extras
         run: uv sync --all-extras --no-extra groot # TODO(Steven): Make flash-attn optional
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run pytest (all extras)
         run: uv run pytest tests -vv --maxfail=10
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run end-to-end tests
         run: uv run make test-end-to-end
 
+      - name: Check disk usage
+        run: df -h
+
   # This job builds a GPU enabled image for testing
   # It runs everytime a PR is approved or a push to main
   # TODO(Steven): For now we skip this job for community PRs

.github/workflows/unbound_deps_tests.yml

@@ -52,9 +52,6 @@ jobs:
         with:
           lfs: true
           persist-credentials: false
-
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 

docs/source/_toctree.yml

@@ -9,8 +9,6 @@
     title: Imitation Learning for Robots
   - local: cameras
     title: Cameras
-  - local: bring_your_own_policies
-    title: Bring Your Own Policies
   - local: integrate_hardware
     title: Bring Your Own Hardware
   - local: hilserl
@@ -83,19 +81,11 @@
     title: Hope Jr
   - local: reachy2
     title: Reachy 2
-  - local: unitree_g1
-    title: Unitree G1
-  - local: earthrover_mini_plus
-    title: Earth Rover Mini
   title: "Robots"
 - sections:
   - local: phone_teleop
     title: Phone
   title: "Teleoperators"
-- sections:
-  - local: torch_accelerators
-    title: PyTorch accelerators
-  title: "Supported Hardware"
 - sections:
   - local: notebooks
     title: Notebooks

docs/source/async.mdx

@@ -278,7 +278,7 @@ We found the default values of `actions_per_chunk` and `chunk_size_threshold` to
 2. **Adjust your `fps` based on inference latency.** While the server generates a new action chunk, the client is not idle and is stepping through its current action queue. If the two processes happen at fundamentally different speeds, the client might end up with an empty queue. As such, you should reduce your fps if you consistently run out of actions in queue.
 3. **Adjust `chunk_size_threshold`**.
    - Values closer to `0.0` result in almost sequential behavior. Values closer to `1.0` → send observation every step (more bandwidth, relies on good world-model).
-   - We found values around 0.5-0.6 to work well. If you want to tweak this, spin up a `RobotClient` setting the `--debug_visualize_queue_size` to `True`. This will plot the action queue size evolution at runtime, and you can use it to find the value of `chunk_size_threshold` that works best for your setup.
+   - We found values around 0.5-0.6 to work well. If you want to tweak this, spin up a `RobotClient` setting the `--debug-visualize-queue-size` to `True`. This will plot the action queue size evolution at runtime, and you can use it to find the value of `chunk_size_threshold` that works best for your setup.
 
 <p align="center">
   <img
@@ -289,7 +289,7 @@ We found the default values of `actions_per_chunk` and `chunk_size_threshold` to
 <p align="center">
   <i>
     The action queue size is plotted at runtime when the
-    `--debug_visualize_queue_size` flag is passed, for various levels of
+    `--debug-visualize-queue-size` flag is passed, for various levels of
     `chunk_size_threshold` (`g` in the SmolVLA paper).
   </i>
 </p>

docs/source/bring_your_own_policies.mdx

@@ -1,175 +0,0 @@
-# Bring Your Own Policies
-
-This tutorial explains how to integrate your own custom policy implementations into the LeRobot ecosystem, allowing you to leverage all LeRobot tools for training, evaluation, and deployment while using your own algorithms.
-
-## Step 1: Create a Policy Package
-
-Your custom policy should be organized as an installable Python package following LeRobot's plugin conventions.
-
-### Package Structure
-
-Create a package with the prefix `lerobot_policy_` (IMPORTANT!) followed by your policy name:
-
-```bash
-lerobot_policy_my_custom_policy/
-├── pyproject.toml
-└── src/
-    └── lerobot_policy_my_custom_policy/
-        ├── __init__.py
-        ├── configuration_my_custom_policy.py
-        ├── modeling_my_custom_policy.py
-        └── processor_my_custom_policy.py
-```
-
-### Package Configuration
-
-Set up your `pyproject.toml`:
-
-```toml
-[project]
-name = "lerobot_policy_my_custom_policy"
-version = "0.1.0"
-dependencies = [
-    # your policy-specific dependencies
-]
-requires-python = ">= 3.11"
-
-[build-system]
-build-backend = # your-build-backend
-requires = # your-build-system
-```
-
-## Step 2: Define the Policy Configuration
-
-Create a configuration class that inherits from `PreTrainedConfig` and registers your policy type:
-
-```python
-# configuration_my_custom_policy.py
-from dataclasses import dataclass, field
-from lerobot.configs.policies import PreTrainedConfig
-from lerobot.configs.types import NormalizationMode
-
-@PreTrainedConfig.register_subclass("my_custom_policy")
-@dataclass
-class MyCustomPolicyConfig(PreTrainedConfig):
-    """Configuration class for MyCustomPolicy.
-
-    Args:
-        n_obs_steps: Number of observation steps to use as input
-        horizon: Action prediction horizon
-        n_action_steps: Number of action steps to execute
-        hidden_dim: Hidden dimension for the policy network
-        # Add your policy-specific parameters here
-    """
-    # ...PreTrainedConfig fields...
-    pass
-
-    def __post_init__(self):
-        super().__post_init__()
-        # Add any validation logic here
-
-    def validate_features(self) -> None:
-        """Validate input/output feature compatibility."""
-        # Implement validation logic for your policy's requirements
-        pass
-```
-
-## Step 3: Implement the Policy Class
-
-Create your policy implementation by inheriting from LeRobot's base `PreTrainedPolicy` class:
-
-```python
-# modeling_my_custom_policy.py
-import torch
-import torch.nn as nn
-from typing import Dict, Any
-
-from lerobot.policies.pretrained import PreTrainedPolicy
-from .configuration_my_custom_policy import MyCustomPolicyConfig
-
-class MyCustomPolicy(PreTrainedPolicy):
-    config_class = MyCustomPolicyConfig
-    name = "my_custom_policy"
-
-    def __init__(self, config: MyCustomPolicyConfig, dataset_stats: Dict[str, Any] = None):
-        super().__init__(config, dataset_stats)
-        ...
-```
-
-## Step 4: Add Data Processors
-
-Create processor functions:
-
-```python
-# processor_my_custom_policy.py
-from typing import Dict, Any
-import torch
-
-
-def make_my_custom_policy_pre_post_processors(
-    config,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[PolicyAction, PolicyAction],
-]:
-    """Create preprocessing and postprocessing functions for your policy."""
-    pass  # Define your preprocessing and postprocessing logic here
-
-```
-
-## Step 5: Package Initialization
-
-Expose your classes in the package's `__init__.py`:
-
-```python
-# __init__.py
-"""Custom policy package for LeRobot."""
-
-try:
-    import lerobot  # noqa: F401
-except ImportError:
-    raise ImportError(
-        "lerobot is not installed. Please install lerobot to use this policy package."
-    )
-
-from .configuration_my_custom_policy import MyCustomPolicyConfig
-from .modeling_my_custom_policy import MyCustomPolicy
-from .processor_my_custom_policy import make_my_custom_policy_pre_post_processors
-
-__all__ = [
-    "MyCustomPolicyConfig",
-    "MyCustomPolicy",
-    "make_my_custom_policy_pre_post_processors",
-]
-```
-
-## Step 6: Installation and Usage
-
-### Install Your Policy Package
-
-```bash
-cd lerobot_policy_my_custom_policy
-pip install -e .
-
-# Or install from PyPI if published
-pip install lerobot_policy_my_custom_policy
-```
-
-### Use Your Policy
-
-Once installed, your policy automatically integrates with LeRobot's training and evaluation tools:
-
-```bash
-lerobot-train \
-    --policy.type my_custom_policy \
-    --env.type pusht \
-    --steps 200000
-```
-
-## Examples and Community Contributions
-
-Check out these example policy implementations:
-
-- [DiTFlow Policy](https://github.com/danielsanjosepro/lerobot_policy_ditflow) - Diffusion Transformer policy with flow-matching objective. Try it out in this example: [DiTFlow Example](https://github.com/danielsanjosepro/test_lerobot_policy_ditflow)
-
-Share your policy implementations with the community! 🤗

docs/source/earthrover_mini_plus.mdx

@@ -1,206 +0,0 @@
-# EarthRover Mini Plus
-
-The EarthRover Mini Plus is a fully open source mobile robot that connects through the cloud using the Frodobots SDK. This lets you control the robot and record datasets for training AI models.
-
-## What You Need
-
-### Hardware
-
-- EarthRover Mini robot
-- Computer with Python 3.10 or newer
-- Internet connection
-
-### Setting Up the Frodobots SDK
-
-The robot needs the [Frodobots SDK](https://github.com/Frodobots/earth-rovers-sdk) running on your computer. Here's how:
-
-1. Download and install the SDK:
-
-```bash
-git clone https://github.com/Frodobots/earth-rovers-sdk.git
-cd earth-rovers-sdk
-pip install -r requirements.txt
-```
-
-2. Start the SDK:
-
-```bash
-hypercorn main:app --reload
-```
-
-3. Open your web browser and go to `http://localhost:8000`, then click "Join"
-
-The SDK gives you:
-
-- Live video from front and rear cameras
-
-> [!IMPORTANT]
-> The SDK must be running before you can use the robot.
-
-## Install LeRobot
-
-Follow our [Installation Guide](./installation) to install LeRobot.
-
-In addition to the base installation, install the EarthRover Mini dependencies:
-
-```bash
-pip install -e .
-```
-
-## How It Works
-
-The robot uses the internet to communicate:
-
-- **Movement commands**: Sent through the SDK
-- **Camera video**: Received from the SDK
-- **Robot info**: Battery, location, speed from the SDK
-
-You don't need to plug anything in - it all works through the SDK.
-
-## Calibration
-
-No calibration needed! The robot is ready to use as soon as the SDK is running.
-
-## Controlling the Robot
-
-You control the robot using your keyboard - just like playing a video game with WASD keys.
-
-### Keyboard Controls
-
-| Key | Action                           |
-| --- | -------------------------------- |
-| W   | Move forward                     |
-| S   | Move backward                    |
-| A   | Turn left (with forward motion)  |
-| D   | Turn right (with forward motion) |
-| Q   | Rotate left in place             |
-| E   | Rotate right in place            |
-| X   | Stop all movement                |
-| +/= | Increase speed                   |
-| -   | Decrease speed                   |
-| ESC | Disconnect                       |
-
-### Speed Settings
-
-You can adjust how fast the robot moves:
-
-- **Forward/backward speed**: Default is full speed (1.0)
-- **Turning speed**: Default is full speed (1.0)
-- **Speed changes**: Use +/- keys to adjust by 0.1 each time
-
-### Try It Out
-
-Test driving the robot before recording data:
-
-```python
-from lerobot.robots.earthrover_mini_plus import EarthRoverMiniPlus, EarthRoverMiniPlusConfig
-from lerobot.teleoperators.keyboard import KeyboardRoverTeleop, KeyboardRoverTeleopConfig
-
-# Initialize robot
-robot_config = EarthRoverMiniPlusConfig()
-robot = EarthRoverMiniPlus(robot_config)
-
-# Initialize teleoperator
-teleop_config = KeyboardRoverTeleopConfig(
-    linear_speed=1.0,
-    angular_speed=1.0,
-    speed_increment=0.1
-)
-teleop = KeyboardRoverTeleop(teleop_config)
-
-# Connect
-robot.connect()
-teleop.connect()
-
-# Teleoperate (use keyboard controls)
-try:
-    while True:
-        action = teleop.get_action()
-        robot.send_action(action)
-except KeyboardInterrupt:
-    pass
-finally:
-    robot.disconnect()
-    teleop.disconnect()
-```
-
-> [!TIP]
-> If you're using a Mac, you might need to give Terminal permission to access your keyboard for teleoperation. Go to System Preferences > Security & Privacy > Input Monitoring and check the box for Terminal.
-
-## Recording Data
-
-Once you can drive the robot well, you can start recording data to train AI models. The system records:
-
-- **What you do**: How you move the robot (forward, backward, turning)
-- **What the robot sees**:
-  - Videos from both cameras
-  - Robot speed and direction
-  - Battery level and location
-  - GPS position and signal
-  - Other sensor data
-- **When it happened**: Timestamps for everything
-
-### Setting Up Hugging Face
-
-We use Hugging Face to store your data online. First, log in with your token from [Hugging Face settings](https://huggingface.co/settings/tokens):
-
-```bash
-huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
-```
-
-Store your Hugging Face username:
-
-```bash
-HF_USER=$(huggingface-cli whoami | head -n 1)
-echo $HF_USER
-```
-
-### Start Recording
-
-Use the standard recording command:
-
-```bash
-python src/lerobot/scripts/lerobot_record.py \
-    --robot.type=earthrover_mini_plus \
-    --teleop.type=keyboard_rover \
-    --dataset.repo_id=your_username/dataset_name \
-    --dataset.num_episodes=2 \
-    --dataset.fps=10 \
-    --dataset.single_task="Navigate around obstacles" \
-    --display_data=true
-```
-
-Replace `your_username/dataset_name` with your Hugging Face username and a name for your dataset.
-
-### What Gets Saved
-
-Your dataset includes:
-
-**Your Actions (2 things)**:
-
-- How much you moved forward/backward
-- How much you turned left/right
-
-**Robot Observations (12 things)**:
-
-- Front camera video
-- Rear camera video
-- Current speed
-- Battery level
-- Which way the robot is facing
-- GPS location (latitude, longitude, signal strength)
-- Network signal strength
-- Vibration level
-- Lamp status (on/off)
-
-### Where Your Data Goes
-
-On your computer: `~/.cache/huggingface/lerobot/{repo-id}`
-
-After recording, your data automatically uploads to your Hugging Face page:
-
-```bash
-echo https://huggingface.co/datasets/${HF_USER}/earthrover-navigation
-```
-
-Your dataset will be tagged with `LeRobot` for community discovery.

docs/source/il_robots.mdx

@@ -428,7 +428,7 @@ Your robot should replicate movements similar to those you recorded. For example
 
 ## Train a policy
 
-To train a policy to control your robot, use the [`lerobot-train`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/lerobot_train.py) script. A few arguments are required. Here is an example command:
+To train a policy to control your robot, use the [`lerobot-train`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/train.py) script. A few arguments are required. Here is an example command:
 
 ```bash
 lerobot-train \
@@ -485,7 +485,7 @@ huggingface-cli upload ${HF_USER}/act_so101_test${CKPT} \
 
 ## Run inference and evaluate your policy
 
-You can use the `record` script from [`lerobot-record`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/lerobot_record.py) with a policy checkpoint as input, to run inference and evaluate your policy. For instance, run this command or API example to run inference and record 10 evaluation episodes:
+You can use the `record` script from [`lerobot/record.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/record.py) with a policy checkpoint as input, to run inference and evaluate your policy. For instance, run this command or API example to run inference and record 10 evaluation episodes:
 
 <hfoptions id="eval">
 <hfoption id="Command">

docs/source/installation.mdx

@@ -90,7 +90,7 @@ If you encounter build errors, you may need to install additional dependencies:
 To install these for linux run:
 
 ```bash
-sudo apt-get install cmake build-essential python3-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev
+sudo apt-get install cmake build-essential python-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev pkg-config
 ```
 
 For other systems, see: [Compiling PyAV](https://pyav.org/docs/develop/overview/installation.html#bring-your-own-ffmpeg)

docs/source/so101.mdx

@@ -30,6 +30,131 @@ The follower arm uses 6x STS3215 motors with 1/345 gearing. The leader, however,
 | Wrist Roll          |   5   |  1 / 147   |
 | Gripper             |   6   |  1 / 147   |
 
+### Clean Parts
+
+Remove all support material from the 3D-printed parts. The easiest way to do this is using a small screwdriver to get underneath the support material.
+
+It is advisable to install one 3-pin cable in the motor after placing them before continuing assembly.
+
+### Joint 1
+
+- Place the first motor into the base.
+- Fasten the motor with 4 M2x6mm screws (smallest screws). Two from the top and two from the bottom.
+- Slide over the first motor holder and fasten it using two M2x6mm screws (one on each side).
+- Install both motor horns, securing the top horn with a M3x6mm screw.
+- Attach the shoulder part.
+- Tighten the shoulder part with 4 M3x6mm screws on top and 4 M3x6mm screws on the bottom
+- Add the shoulder motor holder.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint1_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 2
+
+- Slide the second motor in from the top.
+- Fasten the second motor with 4 M2x6mm screws.
+- Attach both motor horns to motor 2, again use the M3x6mm horn screw.
+- Attach the upper arm with 4 M3x6mm screws on each side.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint2_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 3
+
+- Insert motor 3 and fasten using 4 M2x6mm screws
+- Attach both motor horns to motor 3 and secure one again with a M3x6mm horn screw.
+- Connect the forearm to motor 3 using 4 M3x6mm screws on each side.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint3_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 4
+
+- Slide over motor holder 4.
+- Slide in motor 4.
+- Fasten motor 4 with 4 M2x6mm screws and attach its motor horns, use a M3x6mm horn screw.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint4_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 5
+
+- Insert motor 5 into the wrist holder and secure it with 2 M2x6mm front screws.
+- Install only one motor horn on the wrist motor and secure it with a M3x6mm horn screw.
+- Secure the wrist to motor 4 using 4 M3x6mm screws on both sides.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint5_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Gripper / Handle
+
+<hfoptions id="assembly">
+<hfoption id="Follower">
+
+- Attach the gripper to motor 5, attach it to the motor horn on the wrist using 4 M3x6mm screws.
+- Insert the gripper motor and secure it with 2 M2x6mm screws on each side.
+- Attach the motor horns and again use a M3x6mm horn screw.
+- Install the gripper claw and secure it with 4 M3x6mm screws on both sides.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Gripper_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+</hfoption>
+<hfoption id="Leader">
+
+- Mount the leader holder onto the wrist and secure it with 4 M3x6mm screws.
+- Attach the handle to motor 5 using 1 M2x6mm screw.
+- Insert the gripper motor, secure it with 2 M2x6mm screws on each side, attach a motor horn using a M3x6mm horn screw.
+- Attach the follower trigger with 4 M3x6mm screws.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Leader_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+</hfoption>
+</hfoptions>
+
 ## Configure the motors
 
 ### 1. Find the USB ports associated with each arm
@@ -215,131 +340,6 @@ leader.setup_motors()
 </hfoption>
 </hfoptions>
 
-### Clean Parts
-
-Remove all support material from the 3D-printed parts. The easiest way to do this is using a small screwdriver to get underneath the support material.
-
-It is advisable to install one 3-pin cable in the motor after placing them before continuing assembly.
-
-### Joint 1
-
-- Place the first motor into the base.
-- Fasten the motor with 4 M2x6mm screws (smallest screws). Two from the top and two from the bottom.
-- Slide over the first motor holder and fasten it using two M2x6mm screws (one on each side).
-- Install both motor horns, securing the top horn with a M3x6mm screw.
-- Attach the shoulder part.
-- Tighten the shoulder part with 4 M3x6mm screws on top and 4 M3x6mm screws on the bottom
-- Add the shoulder motor holder.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint1_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 2
-
-- Slide the second motor in from the top.
-- Fasten the second motor with 4 M2x6mm screws.
-- Attach both motor horns to motor 2, again use the M3x6mm horn screw.
-- Attach the upper arm with 4 M3x6mm screws on each side.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint2_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 3
-
-- Insert motor 3 and fasten using 4 M2x6mm screws
-- Attach both motor horns to motor 3 and secure one again with a M3x6mm horn screw.
-- Connect the forearm to motor 3 using 4 M3x6mm screws on each side.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint3_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 4
-
-- Slide over motor holder 4.
-- Slide in motor 4.
-- Fasten motor 4 with 4 M2x6mm screws and attach its motor horns, use a M3x6mm horn screw.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint4_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 5
-
-- Insert motor 5 into the wrist holder and secure it with 2 M2x6mm front screws.
-- Install only one motor horn on the wrist motor and secure it with a M3x6mm horn screw.
-- Secure the wrist to motor 4 using 4 M3x6mm screws on both sides.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint5_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Gripper / Handle
-
-<hfoptions id="assembly">
-<hfoption id="Follower">
-
-- Attach the gripper to motor 5, attach it to the motor horn on the wrist using 4 M3x6mm screws.
-- Insert the gripper motor and secure it with 2 M2x6mm screws on each side.
-- Attach the motor horns and again use a M3x6mm horn screw.
-- Install the gripper claw and secure it with 4 M3x6mm screws on both sides.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Gripper_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-</hfoption>
-<hfoption id="Leader">
-
-- Mount the leader holder onto the wrist and secure it with 4 M3x6mm screws.
-- Attach the handle to motor 5 using 1 M2x6mm screw.
-- Insert the gripper motor, secure it with 2 M2x6mm screws on each side, attach a motor horn using a M3x6mm horn screw.
-- Attach the follower trigger with 4 M3x6mm screws.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Leader_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-</hfoption>
-</hfoptions>
-
 ## Calibrate
 
 Next, you'll need to calibrate your robot to ensure that the leader and follower arms have the same position values when they are in the same physical position.

docs/source/torch_accelerators.mdx

@@ -1,42 +0,0 @@
-# PyTorch accelerators
-
-LeRobot supports multiple hardware acceleration options for both training and inference.
-
-These options include:
-
-- **CPU**: CPU executes all computations, no dedicated accelerator is used
-- **CUDA**: acceleration with NVIDIA & AMD GPUs
-- **MPS**: acceleration with Apple Silicon GPUs
-- **XPU**: acceleration with Intel integrated and discrete GPUs
-
-## Getting Started
-
-To use particular accelerator, a suitable version of PyTorch should be installed.
-
-For CPU, CUDA, and MPS backends follow instructions provided on [PyTorch installation page](https://pytorch.org/get-started/locally).
-For XPU backend, follow instructions from [PyTorch documentation](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
-
-### Verifying the installation
-
-After installation, accelerator availability can be verified by running
-
-```python
-import torch
-print(torch.<backend_name>.is_available())  # <backend_name> is cuda, mps, or xpu
-```
-
-## How to run training or evaluation
-
-To select the desired accelerator, use the `--policy.device` flag when running `lerobot-train` or `lerobot-eval`. For example, to use MPS on Apple Silicon, run:
-
-```bash
-lerobot-train
-    --policy.device=mps ...
-```
-
-```bash
-lerobot-eval \
-    --policy.device=mps ...
-```
-
-However, in most cases, presence of an accelerator is detected automatically and `policy.device` parameter can be omitted from CLI commands.

docs/source/unitree_g1.mdx

@@ -1,203 +0,0 @@
-# Unitree G1 Robot Setup and Control
-
-This guide covers the complete setup process for the Unitree G1 humanoid, from initial connection to running gr00t_wbc locomotion.
-
-## About the Unitree G1
-
-We offer support for both 29 and 23 DOF G1. In this first PR we introduce:
-
-- **`unitree g1` robot class, handling low level communication with the humanoid**
-- **ZMQ socket bridge** for remote communication over WiFi, allowing one to deploy policies remotely instead of over ethernet or directly on the Orin
-- **GR00T locomotion policy** for bipedal walking and balance
-
----
-
-## Part 1: Connect to Robot over Ethernet
-
-### Step 1: Configure Your Computer's Ethernet Interface
-
-Set a static IP on the same subnet as the robot:
-
-```bash
-# Replace 'enp131s0' with your ethernet interface name (check with `ip a`)
-sudo ip addr flush dev enp131s0
-sudo ip addr add 192.168.123.200/24 dev enp131s0
-sudo ip link set enp131s0 up
-```
-
-**Note**: The robot's Ethernet IP is fixed at `192.168.123.164`. Your computer must use `192.168.123.x` where x ≠ 164.
-
-### Step 2: SSH into the Robot
-
-```bash
-ssh unitree@192.168.123.164
-# Password: 123
-```
-
-You should now be connected to the robot's onboard computer.
-
----
-
-## Part 2: Enable WiFi on the Robot
-
-Once connected via Ethernet, follow these steps to enable WiFi:
-
-### Step 1: Enable WiFi Hardware
-
-```bash
-# Unblock WiFi radio
-sudo rfkill unblock wifi
-sudo rfkill unblock all
-
-# Bring up WiFi interface
-sudo ip link set wlan0 up
-
-# Enable NetworkManager control
-sudo nmcli radio wifi on
-sudo nmcli device set wlan0 managed yes
-sudo systemctl restart NetworkManager
-```
-
-### Step 2: Enable Internet Forwarding
-
-**On your laptop:**
-
-```bash
-# Enable IP forwarding
-sudo sysctl -w net.ipv4.ip_forward=1
-
-# Set up NAT (replace wlp132s0f0 with your WiFi interface)
-sudo iptables -t nat -A POSTROUTING -o wlp132s0f0 -s 192.168.123.0/24 -j MASQUERADE
-sudo iptables -A FORWARD -i wlp132s0f0 -o enp131s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-sudo iptables -A FORWARD -i enp131s0 -o wlp132s0f0 -j ACCEPT
-```
-
-**On the robot:**
-
-```bash
-# Add laptop as default gateway
-sudo ip route del default 2>/dev/null || true
-sudo ip route add default via 192.168.123.200 dev eth0
-echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
-
-# Test connection
-ping -c 3 8.8.8.8
-```
-
-### Step 3: Connect to WiFi Network
-
-```bash
-# List available networks
-nmcli device wifi list
-
-# Connect to your WiFi (example)
-sudo nmcli connection add type wifi ifname wlan0 con-name "YourNetwork" ssid "YourNetwork"
-sudo nmcli connection modify "YourNetwork" wifi-sec.key-mgmt wpa-psk
-sudo nmcli connection modify "YourNetwork" wifi-sec.psk "YourPassword"
-sudo nmcli connection modify "YourNetwork" connection.autoconnect yes
-sudo nmcli connection up "YourNetwork"
-
-# Check WiFi IP address
-ip a show wlan0
-```
-
-### Step 4: SSH Over WiFi
-
-Once connected to WiFi, note the robot's IP address and disconnect the Ethernet cable. You can now SSH over WiFi:
-
-```bash
-ssh unitree@<YOUR_ROBOT_IP>
-# Password: 123
-```
-
-Replace `<YOUR_ROBOT_IP>` with your robot's actual WiFi IP address (e.g., `172.18.129.215`).
-
----
-
-## Part 3: Robot Server Setup
-
-### Step 1: Install LeRobot on the Orin
-
-SSH into the robot and install LeRobot:
-
-```bash
-ssh unitree@<YOUR_ROBOT_IP>
-
-conda create -y -n lerobot python=3.10
-conda activate lerobot
-git clone https://github.com/huggingface/lerobot.git
-cd lerobot
-pip install -e '.[unitree_g1]'
-git clone https://github.com/unitreerobotics/unitree_sdk2_python.git
-cd unitree_sdk2_python  && pip install -e .
-```
-
-**Note**: The Unitree SDK requires CycloneDDS v0.10.2 to be installed. See the [Unitree SDK documentation](https://github.com/unitreerobotics/unitree_sdk2_python) for details.
-
-### Step 2: Run the Robot Server
-
-On the robot:
-
-```bash
-python src/lerobot/robots/unitree_g1/run_g1_server.py
-```
-
-**Important**: Keep this terminal running. The server must be active for remote control.
-
----
-
-## Part 4: Running GR00T Locomotion
-
-With the robot server running, you can now control the robot from your laptop.
-
-### Step 1: Install LeRobot on your machine
-
-```bash
-conda create -y -n lerobot python=3.10
-conda activate lerobot
-git clone https://github.com/huggingface/lerobot.git
-cd lerobot
-pip install -e '.[unitree_g1]'
-git clone https://github.com/unitreerobotics/unitree_sdk2_python.git
-cd unitree_sdk2_python  && pip install -e .
-```
-
-### Step 2: Update Robot IP in Config
-
-Edit the config file to match your robot's WiFi IP:
-
-```python
-# In src/lerobot/robots/unitree_g1/config_unitree_g1.py
-robot_ip: str = "<YOUR_ROBOT_IP>"  # Replace with your robot's WiFi IP.
-```
-
-**Note**: When running directly on the G1 (not remotely), set `robot_ip: str = "127.0.0.1"` instead.
-
-### Step 3: Run the Locomotion Policy
-
-```bash
-# Run GR00T locomotion controller
-python examples/unitree_g1/gr00t_locomotion.py --repo-id "nepyope/GR00T-WholeBodyControl_g1"
-```
-
-### Step 4: Control with Remote
-
-- **Left stick**: Forward/backward and left/right movement
-- **Right stick**: Rotation
-- **R1 button**: Raise waist height
-- **R2 button**: Lower waist height
-
-Press `Ctrl+C` to stop the policy.
-
----
-
-## Additional Resources
-
-- [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python)
-- [GR00T Policy Repository](https://huggingface.co/nepyope/GR00T-WholeBodyControl_g1)
-- [LeRobot Documentation](https://github.com/huggingface/lerobot)
-- [Unitree_IL_Lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot)
-
----
-
-_Last updated: December 2025_

docs/source/xvla.mdx

@@ -10,36 +10,20 @@ Inspired by meta-learning and prompt learning, we ask: **"What if a VLA model co
 
 **X-VLA** is a soft-prompted, flow-matching VLA framework that treats each hardware setup as a "task" and encodes it using a small set of learnable embeddings. These **Soft Prompts** capture embodiment and domain-specific variations, guiding the Transformer from the earliest stages of multimodal fusion. With this mechanism, X-VLA can reconcile diverse robot morphologies, data types, and sensor setups within a single unified architecture.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture.png"
-    alt="XVLA Architecture"
-    style="max-width: 100%; height: auto; width: 800px;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture.png" width="400">
 
 Built from pure Transformer encoders, X-VLA scales naturally with model size and dataset diversity. Across 6 simulation benchmarks and 3 real robots, Soft Prompts consistently outperform existing methods in handling hardware and domain differences. X-VLA-0.9B, trained on 290K episodes spanning seven robotic platforms, learns an embodiment-agnostic generalist policy in Phase I, and adapts efficiently to new robots in Phase II simply by learning a new set of prompts, while keeping the backbone frozen.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture2.png"
-    alt="XVLA Architecture 2"
-    style="width: 32%; max-width: 450px; height: auto;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture2.png" width="400">
 
 With only 1% of parameters tuned (9M), X-VLA-0.9B achieves near-π₀ performance on LIBERO and Simpler-WidowX, despite using **300× fewer trainable parameters**. It also demonstrates strong real-world dexterity with minimal demonstrations, including folding cloths in under two minutes.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-fold.png"
-    alt="XVLA fold visualization"
-    style="width: 95%; max-width: 1100px; height: auto;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-fold.png" width="400">
 
 X-VLA shows that generalist robot intelligence does not require increasingly complex architectures, only the right way to absorb heterogeneity. Soft Prompts offer a simple, scalable mechanism for unifying diverse robotic data, paving the way toward adaptable, cross-embodiment robot foundation models.
 
+---
+
 ## Installation
 
 After installing LeRobot, install the X-VLA dependencies:
@@ -54,6 +38,8 @@ After the new release, you'll be able to do:
 pip install lerobot[xvla]
 ```
 
+---
+
 ## Quick Start
 
 ### Basic Usage
@@ -80,6 +66,8 @@ lerobot-eval \
   --seed=142
 ```
 
+---
+
 ## Available Checkpoints
 
 ### 🎯 Base Model
@@ -92,7 +80,7 @@ A 0.9B parameter instantiation of X-VLA, trained with a carefully designed data
 
 - **Phase II: Domain Adaptation** - Adapted to deployable policies for target domains. A new set of soft prompts is introduced and optimized to encode the hardware configuration of the novel domain, while the pretrained backbone remains frozen.
 
-### Simulation Checkpoints
+### 🎮 Simulation Checkpoints
 
 **[lerobot/xvla-libero](https://huggingface.co/lerobot/xvla-libero)**
 
@@ -116,6 +104,8 @@ Optimized for AgileX robot dexterous manipulation tasks.
 
 Adapted for Google Robot platforms.
 
+---
+
 ## Training X-VLA
 
 ### Recommended Training Configuration
@@ -242,6 +232,8 @@ This ensures the optimizer receives a dict of named parameters, allowing it to c
 Completely matching the official reported performance may require an additional warm-up LR schedule for soft-prompts, which can bring minor improvements.
 We encourage implementing this in your customized training pipeline for optimal results.
 
+---
+
 ## Core Concepts
 
 ### 1. Action Modes
@@ -250,13 +242,13 @@ X-VLA uses an **Action Registry** system to handle different action spaces and e
 
 #### Available Action Modes
 
-| Action Mode      | Action Dim              | Description                                 | Use Case                             |
-| ---------------- | ----------------------- | ------------------------------------------- | ------------------------------------ |
-| `ee6d`           | 20                      | End-effector with xyz, 6D rotation, gripper | Dual-arm setups with spatial control |
-| `joint`          | 14                      | Joint-space with gripper                    | Direct joint control robots          |
-| `agibot_ee6d`    | 20                      | AGI-bot variant with MSE loss               | AGI-bot platforms                    |
-| `so101_bimanual` | 20 (model), 12 (real)   | SO101 bimanual robot                        | Bimanual manipulation tasks          |
-| `auto`           | 20 (model), auto (real) | Auto-detects action dim from dataset        | **Recommended** for new robots       |
+| Action Mode      | Action Dim            | Description                                 | Use Case                             |
+| ---------------- | --------------------- | ------------------------------------------- | ------------------------------------ |
+| `ee6d`           | 20                    | End-effector with xyz, 6D rotation, gripper | Dual-arm setups with spatial control |
+| `joint`          | 14                    | Joint-space with gripper                    | Direct joint control robots          |
+| `agibot_ee6d`    | 20                    | AGI-bot variant with MSE loss               | AGI-bot platforms                    |
+| `franka_joint7`  | 7                     | Franka Panda 7-joint control                | Franka robots without gripper        |
+| `so101_bimanual` | 20 (model), 12 (real) | SO101 bimanual robot                        | Bimanual manipulation tasks          |
 
 #### Why Action Modes Matter
 
@@ -284,27 +276,6 @@ REAL_DIM = 12
 
 See the [action_hub.py](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/action_hub.py) implementation for details.
 
-#### Auto Action Mode (Recommended)
-
-The `auto` action mode is the easiest way to use X-VLA with any robot. It automatically detects your dataset's action dimension and handles padding/trimming:
-
-```bash
-lerobot-train \
-  --policy.path="lerobot/xvla-base" \
-  --policy.action_mode=auto \
-  --policy.max_action_dim=20 \
-  ...
-```
-
-**How it works:**
-
-- Reads `action_feature.shape[-1]` from your dataset (e.g., 7 for Franka)
-- Model outputs `max_action_dim` (default 20) for pretrained compatibility
-- Loss is computed **only on the real dimensions**: `MSE(pred[:,:,:real_dim], target[:,:,:real_dim])`
-- Postprocess trims output back to `real_dim` for robot control
-
-This eliminates the need to create custom action modes for most robots.
-
 ### 2. Domain IDs
 
 Domain IDs are learnable identifiers for different robot configurations and camera setups. They allow X-VLA to distinguish between:
@@ -366,10 +337,9 @@ num_domains: int = 30          # Maximum number of domain IDs
 len_soft_prompts: int = 32     # Length of soft prompt embeddings
 
 # Action space
-action_mode: str = "ee6d"      # Action space type (use "auto" for auto-detection)
+action_mode: str = "ee6d"      # Action space type
 use_proprio: bool = True       # Use proprioceptive state
 max_state_dim: int = 32        # Maximum state dimension
-max_action_dim: int = 20       # Max action dim for padding (used by "auto" mode)
 
 # Vision
 num_image_views: int | None    # Number of camera views
@@ -379,6 +349,8 @@ resize_imgs_with_padding: tuple[int, int] | None  # Target image size with paddi
 num_denoising_steps: int = 10  # Flow matching denoising steps
 ```
 
+---
+
 ## Creating Custom Action Modes
 
 If your robot has a unique action space, you can create a custom action mode:
@@ -440,6 +412,8 @@ lerobot-train \
   ...
 ```
 
+---
+
 ## Advanced Topics
 
 ### Multi-Camera Support
@@ -481,16 +455,7 @@ preprocessor = PolicyProcessorPipeline(
 
 When your dataset has fewer action dimensions than the pretrained model:
 
-**Option 1 (Recommended)**: Use `auto` action mode
-
-```bash
-# Automatically detects your dataset's action dimension
-# Works with any robot without custom code
-policy.action_mode=auto
-policy.max_action_dim=20  # Match pretrained model
-```
-
-**Option 2**: Use a predefined action mode with built-in padding
+**Option 1**: Use padding (automatic in most action modes)
 
 ```python
 # Model expects 20D, dataset has 12D
@@ -511,6 +476,8 @@ class MappedActionSpace(BaseActionSpace):
         ...
 ```
 
+---
+
 ## Troubleshooting
 
 ### Common Issues
@@ -543,6 +510,8 @@ class MappedActionSpace(BaseActionSpace):
   3. Reduce batch size
   4. Freeze more components
 
+---
+
 ## Citation
 
 If you use X-VLA in your research, please cite:
@@ -557,13 +526,17 @@ If you use X-VLA in your research, please cite:
 }
 ```
 
+---
+
 ## Additional Resources
 
-- [X-VLA Paper](https://arxiv.org/pdf/2510.10274)
+- [X-VLA Paper](https://arxiv.org) (coming soon)
 - [LeRobot Documentation](https://github.com/huggingface/lerobot)
-- [Action Registry Implementation](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/action_hub.py)
-- [Processor Implementation](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/processor_xvla.py)
-- [Model Configuration](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/configuration_xvla.py)
+- [Action Registry Implementation](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/action_hub.py)
+- [Processor Implementation](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/processor_xvla.py)
+- [Model Configuration](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/configuration_xvla.py)
+
+---
 
 ## Contributing
 

examples/dataset/PGEN_SUMMARY.md

@@ -1,243 +0,0 @@
-# Synthetic Data Generation Script - Summary
-
-## ✅ What Was Created
-
-### Main Script: `annotate_pgen.py` (717 lines)
-A production-ready script implementing the Hi-Robot synthetic data generation pipeline.
-
-**Key Features:**
-- ✅ Loads LeRobot datasets with skill annotations
-- ✅ Generates synthetic user prompts and robot utterances using Qwen VLM
-- ✅ **Temporal sampling** - generates dialogue every N seconds (default: 1s)
-- ✅ Adds `task_index_high_level` feature to dataset parquets
-- ✅ Saves high-level tasks to `meta/tasks_high_level.parquet`
-- ✅ Exports debug JSONL for quality analysis
-- ✅ Supports both Qwen2-VL and Qwen3-VL models
-- ✅ Multi-view camera support
-- ✅ Episode-aware processing with automatic first-frame sampling
-- ✅ Modular architecture for easy extension
-
-### Supporting Files Created
-
-1. **`run_pgen.sh`** - Convenience script with sensible defaults
-2. **`README_PGEN.md`** - Comprehensive documentation with examples
-3. **`example_pgen_usage.md`** - Practical examples and performance estimates
-4. **`SAMPLING_DIAGRAM.md`** - Visual explanation of temporal sampling strategy
-5. **`PGEN_SUMMARY.md`** - This file
-
-## 🚀 Key Innovation: Temporal Sampling
-
-The script processes **ALL episodes** in the dataset efficiently via `--sample-interval`:
-
-```bash
-# Instead of calling VLM for every frame (expensive):
-# 15,000 frames × VLM call = ~5 hours
-
-# Generate dialogue every 1 second (efficient):
-python annotate_pgen.py --repo-id dataset --model qwen --sample-interval 1.0
-# 15,000 frames processed, only ~500 VLM calls (30x speedup!)
-```
-
-**How it works:**
-- Process ALL frames in ALL episodes (complete coverage)
-- Generate dialogue at sampled timepoints (e.g., every 1 second)
-- Propagate task indices to intermediate frames
-- Always sample first frame of each episode
-- All frames get labeled, but VLM is only called for samples
-- No dummy values or skipped episodes
-
-**Benefits:**
-- 30-100x speedup depending on interval
-- Maintains temporal coherence
-- Reduces cost without losing quality
-- Configurable based on skill duration
-
-## 📊 Efficiency Comparison
-
-For a typical 15,000 frame dataset at 30 fps:
-
-| Method | VLM Calls | Time | Cost |
-|--------|-----------|------|------|
-| Every frame | 15,000 | ~5 hours | $$$$ |
-| Every 0.5s | 1,000 | ~20 min | $$$ |
-| **Every 1s** (default) | **500** | **~10 min** | **$$** |
-| Every 2s | 250 | ~5 min | $ |
-
-## 🎯 Usage
-
-### Quick Test (5s sampling for fast iteration)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 5.0 \
-    --output-dir ./outputs/test_quick
-```
-
-### Production Run (Recommended Settings)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/full_pgen
-```
-
-### High-Quality with Qwen3
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --sample-interval 0.5 \
-    --temperature 0.6 \
-    --output-dir ./outputs/high_quality
-```
-
-## 📦 Output Structure
-
-After running, you'll have:
-
-```
-dataset_root/
-├── meta/
-│   ├── tasks_high_level.parquet      # High-level tasks with prompts/utterances
-│   └── syn_annotations.jsonl         # Debug: full context for each sample
-└── data/
-    └── chunk-000/
-        └── file-000.parquet           # Updated with task_index_high_level
-```
-
-**New feature added to all parquet files:**
-- `task_index_high_level` (int64): Links to tasks_high_level.parquet
-
-## 🔧 All Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--repo-id` / `--data-dir` | - | Dataset source |
-| `--model` | Qwen/Qwen2-VL-7B-Instruct | VLM model |
-| `--device` | cuda | Device to use |
-| `--dtype` | bfloat16 | Model precision |
-| `--temperature` | 0.7 | Sampling temperature |
-| **`--sample-interval`** | **1.0** | **Generate every N seconds (all episodes processed)** |
-| `--num-image-views-per-sample` | 1 | Number of cameras |
-| `--batch-size` | 1 | Batch size (currently unused) |
-| `--output-dir` | None | Output directory |
-| `--push-to-hub` | False | Push to HuggingFace |
-
-## 🎨 Generated Data Format
-
-Each sampled frame produces:
-
-```json
-{
-  "scenario_type": "specific_object",
-  "response_type": "confirmation",
-  "user_prompt": "Can you pick up the pink brick?",
-  "robot_utterance": "Sure, I'll grab the pink lego brick.",
-  "skill": "robot arm picks up pink lego brick",
-  "episode_id": 0,
-  "frame_index": 45,
-  "timestamp": 1.5,
-  "skill_history": ["robot arm moves towards pink lego brick"],
-  "task_description": "pink lego brick into the transparent box"
-}
-```
-
-**Scenario Types:**
-- specific_object, negative_task, situated_correction, implicit_request, constraint_based
-
-**Response Types:**
-- confirmation, clarification, acknowledgment, constraint_acknowledgment
-
-## 🔬 Code Architecture
-
-```python
-# Main components (modular design)
-
-class QwenPgen:
-    """VLM wrapper supporting Qwen2/3"""
-    def call_qwen(images, prompt) -> dict
-
-def construct_prompt(task, history, skill) -> str:
-    """Build contextual prompt with history"""
-
-def annotate_sample(pgen, images, ...) -> dict:
-    """Generate dialogue for one sample"""
-
-def generate_synthetic_data(dataset, pgen, ...) -> tuple:
-    """Process entire dataset with temporal sampling"""
-    # Core sampling logic:
-    # - Track last_sample_timestamp per episode
-    # - Sample if time_elapsed >= sample_interval
-    # - Always sample first frame of episodes
-    # - Propagate task_index to intermediate frames
-
-def main():
-    """CLI entrypoint with argparse"""
-```
-
-## ✨ Next Steps
-
-1. **Quick test with large interval:**
-   ```bash
-   # Fast iteration - samples every 5 seconds
-   python examples/dataset/annotate_pgen.py \
-       --data-dir /path/to/dataset \
-       --model Qwen/Qwen2-VL-7B-Instruct \
-       --sample-interval 5.0 \
-       --output-dir ./outputs/quick_test
-   ```
-
-2. **Verify output quality:**
-   ```bash
-   head outputs/quick_test/meta/syn_annotations.jsonl
-   ```
-
-3. **Production run:**
-   ```bash
-   # Standard 1 second sampling for production
-   bash examples/dataset/run_pgen.sh
-   ```
-
-4. **Use in training:**
-   ```python
-   from lerobot.datasets.lerobot_dataset import LeRobotDataset
-   
-   ds = LeRobotDataset(repo_id="...", root="outputs/pgen_annotations")
-   
-   # Access high-level task for each frame
-   frame = ds[100]
-   task_idx = frame["task_index_high_level"].item()
-   ```
-
-## 📚 Documentation Files
-
-- **`README_PGEN.md`**: Full API reference and troubleshooting
-- **`example_pgen_usage.md`**: Practical examples with performance estimates
-- **`SAMPLING_DIAGRAM.md`**: Visual explanation of temporal sampling
-- **`PGEN_SUMMARY.md`**: This overview document
-
-## 🎯 Success Criteria
-
-✅ Script generates synthetic dialogue using Qwen VLM  
-✅ Adds `task_index_high_level` feature to dataset  
-✅ Saves tasks to `tasks_high_level.parquet`  
-✅ Implements efficient temporal sampling (30-100x speedup)  
-✅ Handles episode boundaries correctly  
-✅ Produces diverse interaction types (scenarios + responses)  
-✅ Maintains temporal coherence within episodes  
-✅ Includes comprehensive documentation and examples  
-✅ Ready for production use on real datasets  
-
-## 💡 Key Takeaway
-
-**The script processes ALL episodes with intelligent sampling:**
-- `--sample-interval` controls how often VLM is called (default: 1.0s)
-- ALL frames in ALL episodes get labeled (complete coverage)
-- Intermediate frames inherit from most recent sample (temporal coherence)
-- Achieves 30-100x speedup while maintaining quality
-- Adjust interval based on use case: 5.0s for testing, 1.0s for production, 0.5s for fine detail
-
-This makes the synthetic data generation **practical, scalable, and complete** for real-world datasets!
-

examples/dataset/README_PGEN.md

@@ -1,243 +0,0 @@
-# Synthetic Data Generation for Hierarchical Robot Policies
-
-This directory contains `annotate_pgen.py`, a script for generating synthetic user prompts and robot utterances for hierarchical policy training using Vision-Language Models (VLMs).
-
-## Overview
-
-The script implements the synthetic data generation pipeline described in the Hi-Robot paper:
-
-1. **Load** a LeRobot dataset with skill annotations (from `annotate.py`)
-2. **Generate** synthetic dialogue using Qwen VLM:
-   - User prompts (ℓ_t): Natural requests that lead to specific skills
-   - Robot utterances (u_t): Acknowledgments and clarifications
-3. **Save** results as a new dataset feature `task_index_high_level`
-
-## Prerequisites
-
-1. First, annotate your dataset with skills using `annotate.py`:
-
-```bash
-python examples/dataset/annotate.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --video-key observation.images.base \
-    --model Qwen/Qwen2-VL-7B-Instruct
-```
-
-This creates `meta/skills.json` with skill segmentation for each episode.
-
-## Usage
-
-### Basic Usage
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/pgen_dataset
-```
-
-**Note**: The script processes **all episodes** in the dataset. It generates dialogue every 1 second (`--sample-interval 1.0`) using temporal sampling. Frames between samples reuse the last generated dialogue. This makes the process efficient while ensuring complete dataset coverage.
-
-### Advanced Options
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --temperature 0.8 \
-    --sample-interval 0.5 \
-    --num-image-views-per-sample 2 \
-    --output-dir ./outputs/pgen_dataset \
-    --push-to-hub
-```
-
-This example uses a more powerful model and samples every 0.5 seconds for finer granularity.
-
-### Fast Testing (larger interval)
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 5.0 \
-    --output-dir ./outputs/pgen_quick_test
-```
-
-Use a larger interval (5.0 seconds) for rapid iteration during development. All episodes are still processed.
-
-### Using Local Dataset
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --output-dir ./outputs/pgen_dataset
-```
-
-## Output Files
-
-The script produces several outputs:
-
-1. **`meta/tasks_high_level.parquet`**: High-level tasks with user prompts and robot utterances
-   - Columns: task_index, user_prompt, robot_utterance, skill, scenario_type, response_type
-
-2. **`meta/syn_annotations.jsonl`**: Debug file with all generated dialogues
-   - One JSON object per line with full context for each frame
-
-3. **Modified dataset**: New dataset with `task_index_high_level` feature added to all parquet files
-
-## Scenario and Response Types
-
-The generator produces diverse interaction types:
-
-### Scenario Types
-- **specific_object**: Direct specification of objects/actions
-- **negative_task**: Instructions about what NOT to do
-- **situated_correction**: Adjustments based on current state
-- **implicit_request**: Implied needs without direct commands
-- **constraint_based**: Specific constraints or preferences
-
-### Response Types
-- **confirmation**: Simple acknowledgment ("OK, I'll do X")
-- **clarification**: Seeking confirmation ("Just to confirm...")
-- **acknowledgment**: Action acknowledgment ("Got it, doing X")
-- **constraint_acknowledgment**: Acknowledging constraints ("Sure, I'll X while Y")
-
-## Example Generated Data
-
-```json
-{
-  "episode_id": 0,
-  "frame_index": 45,
-  "timestamp": 2.5,
-  "skill_current": "robot arm picks up pink lego brick",
-  "skill_history": ["robot arm moves towards pink lego brick"],
-  "task_description": "pink lego brick into the transparent box",
-  "scenario_type": "specific_object",
-  "response_type": "confirmation",
-  "user_prompt": "Can you grab the pink brick?",
-  "robot_utterance": "Sure, I'll pick up the pink lego brick."
-}
-```
-
-## Accessing the Data
-
-After running the script, access the synthetic data in your code:
-
-```python
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-import pandas as pd
-
-# Load modified dataset
-dataset = LeRobotDataset(repo_id="lerobot/svla_so101_pickplace_with_high_level_tasks")
-
-# Access frame with high-level task
-frame = dataset[100]
-high_level_task_idx = frame["task_index_high_level"].item()
-
-# Load high-level tasks
-tasks_df = pd.read_parquet(dataset.root / "meta" / "tasks_high_level.parquet")
-task_info = tasks_df.iloc[high_level_task_idx]
-
-print(f"User prompt: {task_info['user_prompt']}")
-print(f"Robot utterance: {task_info['robot_utterance']}")
-print(f"Skill: {task_info['skill']}")
-```
-
-## Architecture
-
-The script is modular and extensible:
-
-```python
-# Core components
-class QwenPgen:
-    """VLM wrapper for generation"""
-    def call_qwen(images, prompt) -> dict
-    
-def construct_prompt(task, history, skill) -> str
-    """Build prompt for VLM"""
-    
-def annotate_sample(pgen, images, ...) -> dict
-    """Generate dialogue for one sample"""
-    
-def generate_synthetic_data(dataset, pgen, ...) -> tuple
-    """Process entire dataset"""
-```
-
-## Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--repo-id` | - | HuggingFace dataset ID |
-| `--data-dir` | - | Local dataset path |
-| `--model` | Qwen/Qwen2-VL-7B-Instruct | VLM model name |
-| `--device` | cuda | Device (cuda/cpu) |
-| `--dtype` | bfloat16 | Model precision |
-| `--temperature` | 0.7 | Sampling temperature |
-| `--sample-interval` | 1.0 | Generate dialogue every N seconds (all episodes processed) |
-| `--num-image-views-per-sample` | 1 | Number of cameras |
-| `--output-dir` | None | Output directory |
-| `--push-to-hub` | False | Push to HuggingFace Hub |
-
-## Sampling Strategy
-
-The script uses **temporal sampling** to efficiently generate dialogue:
-
-- **Default**: Generate dialogue every 1 second (`--sample-interval 1.0`)
-- **Efficiency**: If a dataset runs at 30fps, this samples ~3% of frames
-- **Propagation**: Frames between samples reuse the last generated task_index
-- **Episode-aware**: Always samples the first frame of each episode
-
-### Example with 30 fps dataset:
-```bash
-# Sample every 1 second (every 30 frames)
---sample-interval 1.0  # ~3,000 generations for a 100 episode dataset (3 sec/episode)
-
-# Sample every 0.5 seconds (every 15 frames)
---sample-interval 0.5  # ~6,000 generations (more granular)
-
-# Sample every 2 seconds (every 60 frames)
---sample-interval 2.0  # ~1,500 generations (more efficient)
-```
-
-### Why sampling works:
-- Skills typically last 1-3 seconds
-- Dialogue doesn't need to change every frame
-- Reduces computational cost by 30-100x
-- Still provides good coverage for training
-
-## Tips
-
-1. **Quick testing**: Use larger `--sample-interval` (e.g., 5.0 or 10.0) for rapid iteration
-2. **Monitor GPU**: VLM inference is memory-intensive
-3. **Check outputs**: Review `syn_annotations.jsonl` for quality
-4. **Adjust temperature**: Higher = more diverse, lower = more consistent
-5. **Multiple views**: Use `--num-image-views-per-sample 2+` for better context
-6. **Tune sampling**: Start with 1.0s, increase for speed (testing), decrease for granularity (production)
-
-## Troubleshooting
-
-### No skills.json found
-Run `annotate.py` first to generate skill annotations.
-
-### Out of memory
-- Reduce batch size to 1
-- Use smaller model (Qwen2-VL-7B instead of Qwen3-VL-30B)
-- Process fewer samples at a time
-
-### Poor quality generations
-- Adjust temperature (try 0.6-0.9)
-- Check that skills.json has good annotations
-- Ensure images are loading correctly
-
-## Citation
-
-Based on the Hi-Robot paper's synthetic data generation approach:
-```
-@article{hirobot2024,
-  title={Hi-Robot: Hierarchical Robot Learning with Vision-Language Models},
-  year={2024}
-}
-```
-

examples/dataset/SAMPLING_DIAGRAM.md

@@ -1,141 +0,0 @@
-# Temporal Sampling Strategy Visualization
-
-## How `--sample-interval` Works
-
-### Example: 30 fps dataset, `--sample-interval 1.0` (1 second)
-
-```
-Timeline (seconds):  0.0      0.5      1.0      1.5      2.0      2.5      3.0
-                     │        │        │        │        │        │        │
-Frames:              0───15───30───45───60───75───90───105──120──135──150
-                     │        │        │        │        │        │        │
-                     ▼                 ▼                 ▼                 ▼
-Sampled:            YES      NO       YES      NO       YES      NO       YES
-                     │                 │                 │                 │
-Task Index:         [0]──────────────>[1]──────────────>[2]──────────────>[3]
-                     │                 │                 │                 │
-VLM Called:         ✓ Gen             ✓ Gen             ✓ Gen             ✓ Gen
-                    dialogue          dialogue          dialogue          dialogue
-                     │                 │                 │                 │
-Frames 0-29    ─────┘                 │                 │                 │
-get task 0                             │                 │                 │
-                                       │                 │                 │
-Frames 30-59  ────────────────────────┘                 │                 │
-get task 1                                               │                 │
-                                                         │                 │
-Frames 60-89  ──────────────────────────────────────────┘                 │
-get task 2                                                                 │
-                                                                           │
-Frames 90-119 ────────────────────────────────────────────────────────────┘
-get task 3
-```
-
-## Comparison: Different Sampling Intervals
-
-### `--sample-interval 2.0` (every 2 seconds)
-```
-Timeline:    0.0      1.0      2.0      3.0      4.0      5.0      6.0
-             │        │        │        │        │        │        │
-Sampled:    YES      NO       YES      NO       YES      NO       YES
-             │                 │                 │                 │
-Tasks:      [0]───────────────>[1]───────────────>[2]───────────────>[3]
-             
-VLM Calls:   4 (fewer calls, faster but less granular)
-```
-
-### `--sample-interval 1.0` (every 1 second) - **DEFAULT**
-```
-Timeline:    0.0   0.5   1.0   1.5   2.0   2.5   3.0   3.5   4.0   4.5   5.0   5.5   6.0
-             │     │     │     │     │     │     │     │     │     │     │     │     │
-Sampled:    YES   NO   YES   NO   YES   NO   YES   NO   YES   NO   YES   NO   YES
-             │           │           │           │           │           │           │
-Tasks:      [0]─────────>[1]─────────>[2]─────────>[3]─────────>[4]─────────>[5]─────>[6]
-             
-VLM Calls:   7 (balanced coverage and speed)
-```
-
-### `--sample-interval 0.5` (every 0.5 seconds)
-```
-Timeline:    0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0
-             │    │    │    │    │    │    │    │    │    │    │    │    │
-Sampled:    YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES
-             │    │    │    │    │    │    │    │    │    │    │    │    │
-Tasks:      [0]─>[1]─>[2]─>[3]─>[4]─>[5]─>[6]─>[7]─>[8]─>[9]─>[10]>[11]>[12]
-             
-VLM Calls:   13 (high granularity, slower but more detailed)
-```
-
-## Episode Boundaries
-
-The script always samples the **first frame** of each episode:
-
-```
-Episode 0                          Episode 1                          Episode 2
-├─────────────────────────────────┤├─────────────────────────────────┤├──────...
-│                                 ││                                 ││
-Frame: 0    30    60    90   120  130   160   190   220  250  260   290  320
-Time:  0.0  1.0   2.0   3.0  4.0  0.0   1.0   2.0   3.0  4.0  0.0   1.0  2.0
-       │    │     │     │    │    │     │     │     │    │    │     │    │
-       ▼    ▼     ▼     ▼    ▼    ▼     ▼     ▼     ▼    ▼    ▼     ▼    ▼
-Sample:YES  YES   YES   YES  YES  YES   YES   YES   YES  YES  YES   YES  YES
-       │    │     │     │    │    │     │     │     │    │    │     │    │
-Task:  0────1─────2─────3────4    5─────6─────7─────8────9    10────11───12
-
-Note: Frames 0, 130, 260 are ALWAYS sampled (episode starts)
-      Even if they're within the sample-interval window
-```
-
-## Real-World Example: svla_so101_pickplace Dataset
-
-Typical stats:
-- **Total episodes**: 50
-- **Avg episode length**: 300 frames (10 seconds at 30 fps)
-- **Total frames**: 15,000
-
-### Without Sampling (every frame)
-```
-Frames processed:    15,000
-VLM calls:           15,000
-Time estimate:       ~5 hours
-Unique tasks:        ~12,000 (lots of duplicates)
-```
-
-### With `--sample-interval 1.0` (every 1 second)
-```
-Frames processed:    15,000 ✓
-VLM calls:           500
-Time estimate:       ~10 minutes
-Unique tasks:        ~450 (meaningful variety)
-Efficiency gain:     30x faster
-```
-
-### With `--sample-interval 2.0` (every 2 seconds)
-```
-Frames processed:    15,000 ✓
-VLM calls:           250
-Time estimate:       ~5 minutes
-Unique tasks:        ~220
-Efficiency gain:     60x faster
-```
-
-## Key Points
-
-1. **All frames get labeled**: Every frame gets a `task_index_high_level`
-2. **Only sampled frames call VLM**: Huge efficiency gain
-3. **Temporal coherence**: Nearby frames share the same task
-4. **Episode-aware**: Always samples episode starts
-5. **Configurable**: Adjust `--sample-interval` based on your needs
-
-## Choosing Your Sampling Interval
-
-| Use Case | Recommended Interval | Why |
-|----------|---------------------|-----|
-| Quick testing | 2.0s | Fastest iteration |
-| Standard training | 1.0s | Good balance |
-| High-quality dataset | 0.5s | Better coverage |
-| Fine-grained control | 0.33s | Very detailed |
-| Dense annotations | 0.1s | Nearly every frame |
-
-**Rule of thumb**: Match your sampling interval to your typical skill duration.
-If skills last 1-3 seconds, sampling every 1 second captures each skill multiple times.
-

examples/dataset/example_pgen_usage.md

@@ -1,143 +0,0 @@
-# Example: Synthetic Data Generation with Sampling
-
-## Quick Start
-
-### 1. Test with 100 frames and 1 second sampling
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --num-samples 100 \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/test_pgen
-```
-
-**Expected behavior** (assuming 30 fps):
-- Total frames: 100
-- Frames sampled: ~4 (every 30 frames = 1 second)
-- Efficiency: 96% fewer VLM calls
-- Output: All 100 frames get `task_index_high_level`, but only 4 unique dialogues generated
-
-### 2. Process full dataset with different sampling rates
-
-#### Conservative (every 2 seconds)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 2.0 \
-    --output-dir ./outputs/pgen_2s
-```
-
-#### Standard (every 1 second) - **RECOMMENDED**
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/pgen_1s
-```
-
-#### Fine-grained (every 0.5 seconds)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 0.5 \
-    --output-dir ./outputs/pgen_0.5s
-```
-
-## Performance Estimates
-
-For a dataset with:
-- 100 episodes
-- 10 seconds per episode (average)
-- 30 fps
-- Total frames: 30,000
-
-| Sampling Interval | Frames Sampled | % Sampled | Speedup | Time Estimate |
-|-------------------|----------------|-----------|---------|---------------|
-| Every frame (0.033s) | 30,000 | 100% | 1x | ~10 hours |
-| 0.5 seconds | 2,000 | 6.7% | 15x | ~40 min |
-| **1.0 seconds** | **1,000** | **3.3%** | **30x** | **~20 min** |
-| 2.0 seconds | 500 | 1.7% | 60x | ~10 min |
-
-*Note: Times are approximate and depend on GPU, model size, and generation speed*
-
-## Understanding the Output
-
-### Console Output Example
-```
-[cyan]Generating synthetic data for 30000 frames...[/cyan]
-[cyan]Sampling interval: 1.0s (fps: 30)[/cyan]
-Generating synthetic dialogue: 100%|████████| 30000/30000 [20:15<00:00, 24.68it/s]
-[green]✓ Sampled 1000 frames out of 30000 (3.3%)[/green]
-[green]✓ Generated 450 unique high-level tasks[/green]
-```
-
-### What happens:
-1. **Frame 0 (t=0.0s)**: Generate dialogue → Task index 0
-2. **Frames 1-29 (t=0.033s-0.967s)**: Reuse task index 0
-3. **Frame 30 (t=1.0s)**: Generate new dialogue → Task index 1
-4. **Frames 31-59 (t=1.033s-1.967s)**: Reuse task index 1
-5. And so on...
-
-### Result:
-- Every frame has a `task_index_high_level`
-- Only sampled frames have unique dialogues generated
-- Intermediate frames inherit from the most recent sample
-- Maintains temporal coherence within episodes
-
-## Checking Your Results
-
-After running, verify the output:
-
-```bash
-# Check the generated tasks
-python -c "
-import pandas as pd
-from pathlib import Path
-
-tasks = pd.read_parquet('outputs/test_pgen/meta/tasks_high_level.parquet')
-print(f'Total unique tasks: {len(tasks)}')
-print(f'Sample tasks:')
-print(tasks[['user_prompt', 'robot_utterance', 'skill']].head())
-"
-
-# Check debug output
-head outputs/test_pgen/meta/syn_annotations.jsonl
-
-# Load and verify dataset
-python -c "
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-
-ds = LeRobotDataset(repo_id='local_with_high_level_tasks', 
-                    root='outputs/test_pgen')
-print(f'Dataset has {len(ds)} frames')
-print(f'Features: {list(ds.features.keys())}')
-assert 'task_index_high_level' in ds.features
-print('✓ task_index_high_level feature added successfully!')
-"
-```
-
-## Common Use Cases
-
-### Development/Testing
-```bash
---sample-interval 2.0  # Fast iteration
---num-samples 500      # Small subset
-```
-
-### Production Training
-```bash
---sample-interval 1.0  # Good coverage
-# Process all samples (no --num-samples)
-```
-
-### High-Quality Dataset
-```bash
---sample-interval 0.5  # Fine-grained
---temperature 0.6      # More consistent
---model Qwen/Qwen3-VL-30B-A3B-Instruct  # Larger model
-```
-

examples/dataset/inference_gemma.py

@@ -1,17 +0,0 @@
-from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
-
-model_id = "google/paligemma-3b-pt-224"
-model = PaliGemmaForConditionalGeneration.from_pretrained(model_id)
-processor = AutoProcessor.from_pretrained(model_id)
-
-breakpoint()
-prefix_output = model.language_model.forward(
-    inputs_embeds=inputs_embeds[0],
-    attention_mask=attention_mask,
-    position_ids=position_ids,
-    adarms_cond=adarms_cond[0] if adarms_cond is not None else None,
-)
-prefix_past_key_values = prefix_output.past_key_values
-# prefix_output to be used for the language head
-# shape: [batch_size, seq_len, hidden_size] with hidden_size = 2048
-prefix_output = prefix_output.last_hidden_state

examples/dataset/inference_pi05.py

@@ -1,58 +0,0 @@
-import torch
-from huggingface_hub import HfApi
-
-import lerobot
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-# import make_pre_post_processors
-from lerobot.policies.factory import make_pre_post_processors
-from lerobot.policies.pi05.configuration_pi05 import PI05Config
-from lerobot.policies.factory import make_policy, make_policy_config
-from lerobot.configs.policies import PreTrainedConfig
-
-cfg = PreTrainedConfig.from_pretrained(
-    pretrained_name_or_path="/fsx/jade_choghari/outputs/pi0_training_new/checkpoints/last/pretrained_model",
-)
-cfg.dtype = "bfloat16"
-
-pre_processor, post_processor = make_pre_post_processors(
-    policy_cfg=cfg,
-    pretrained_path="/fsx/jade_choghari/outputs/pi0_training_new/checkpoints/last/pretrained_model",
-)
-
-
-dataset = LeRobotDataset(repo_id="local", root="/fsx/jade_choghari/outputs/pgen_annotations1")
-# rename map --rename_map='{
-#         "observation.images.side": "observation.images.base_0_rgb",
-#         "observation.images.up": "observation.images.left_wrist_0_rgb"
-#         }'
-rename_map = {
-    "observation.images.side": "observation.images.base_0_rgb",
-    "observation.images.up": "observation.images.left_wrist_0_rgb"
-}
-policy = make_policy(
-    cfg=cfg,
-    ds_meta=dataset.meta,
-    rename_map=rename_map,
-)
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=4,
-        shuffle=True,
-)
-
-batch = next(iter(dataloader))
-
-batch = pre_processor(batch)
-
-# Test training forward pass
-policy.train()
-loss, loss_dict = policy.forward(batch)
-print(f"Training loss: {loss_dict}")
-
-# Test inference
-policy.eval()
-with torch.no_grad():
-    actions = policy.predict_action_chunk(batch)
-    print(f"Predicted actions shape: {actions.shape}")

examples/dataset/load_lerobot_high.py

@@ -1,23 +0,0 @@
-import torch
-from huggingface_hub import HfApi
-
-import lerobot
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-
-dataset = LeRobotDataset(repo_id="local", root="/fsx/jade_choghari/outputs/pgen_annotations1")
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=32,
-        shuffle=True,
-)
-
-batch = next(iter(dataloader))
-print(batch.keys())
-print(batch['task_index_high_level'].shape)
-print(batch['task_index_high_level'])
-print(batch['user_prompt'][0])
-print(batch['robot_utterance'][0])
-print(batch['task'][0])
-breakpoint()

examples/dataset/prompt.txt

@@ -1,334 +0,0 @@
-Generate annotate_pgen.py using Qwen for synthetic data generation
-
-You are writing a Python script called annotate_pgen.py.
-This script generates synthetic user prompts (ℓ_t) and robot utterances (u_t) for Hi Robot–style hierarchical policy training, using Qwen 3vl as the generator model (pgen).
-
-SCRIPT PURPOSE
-
-The script must:
-
-Load Dlabeled which is a LeRobot Dataset that has been annotate using the annotate.py script, which contains:
-
-images: list of image paths at time t
-
-skill_current: the annotated skill label (ℓ̂_t)
-
-skill_history: list of previous skill labels (ℓ̂₀ … ℓ̂_{t−1}), those where annotated, and you can find details on them stored in teh dataset inside the the DATA_PATH/meta/skills.json
-
-you will find something like 
-
-{
-  "coarse_description": "pink lego brick into the transparent box",
-  "skill_to_task_index": {
-    "robot arm picks up pink lego brick": 19,
-    "robot arm approaches transparent box": 3,
-    "robot arm retracts from transparent box": 28,
-    "robot arm moves towards pink lego brick": 12,
-    "robot arm releases red lego brick into box": 26,
-    "robot arm releases red lego brick into transparent box": 27,
-    "robot arm closes gripper to pick up the pink lego brick": 5,
-    "robot arm lifts the pink lego brick": 7,
-    etc..
-  },
-  "episodes": {
-    "0": {
-      "episode_index": 0,
-      "description": "pink lego brick into the transparent box",
-      "skills": [
-        {
-          "name": "robot arm moves towards pink lego brick",
-          "start": 0.0,
-          "end": 1.8
-        },
-        {
-          "name": "robot arm picks up pink lego brick",
-          "start": 1.8,
-          "end": 3.1
-        },
-        {
-          "name": "robot arm moves towards transparent box",
-          "start": 3.1,
-          "end": 5.5
-        },
-        {
-          "name": "robot arm releases pink lego brick into transparent box",
-          "start": 5.5,
-          "end": 7.0
-        },
-        {
-          "name": "robot arm retracts from transparent box",
-          "start": 7.0,
-          "end": 10.1
-        }
-      ]
-    },
-    "1": {
-      "episode_index": 1,
-      "description": "pink lego brick into the transparent box",
-      "skills": [
-        {
-          "name": "robot arm moves towards red lego brick",
-          "start": 0.0,
-          "end": 1.2
-        },
-        {
-          "name": "robot arm picks up red lego brick",
-          "start": 1.2,
-          "end": 2.0
-        },
-        {
-          "name": "robot arm moves towards transparent box",
-          "start": 2.0,
-          "end": 3.8
-        },
-        {
-          "name": "robot arm places red lego brick into transparent box",
-          "start": 3.8,
-          "end": 5.0
-        },
-        {
-          "name": "robot arm moves away from transparent box",
-          "start": 5.0,
-          "end": 8.9
-        }
-      ]
-    },
-
-notice how task_description: is a high-level description (e.g., "make a sandwich") stored in description for each episode
-
-For each sample, call Qwen VLM to generate:
-
-synthetic user prompt ℓ_t
-
-synthetic robot response u_t
-
-Save results to D_syn in Parquet format insdie DATA_PATH/meta/tasks.parquet ; note tasks.parquet already contains the other tasks, so you need to update
-
-Should be modular, clean, easy to extend, with:
-
-a PGEN_PROMPT_TEMPLATE
-
-a construct_prompt() method
-
-a call_qwen() method
-
-a annotate_sample() method
-
-a CLI entrypoint (if __name__ == "__main__":)
-
-📦 INPUT FORMAT (Dlabeled)
-
-The script should expect Dlabeled as a .jsonl file where each line has:
-
-{
-  "episode_id": "ep_001",
-  "t": 37,
-  "images": ["path/to/cam0_t.jpg", "path/to/cam1_t.jpg"],
-  "skill_current": "pick up the KitKat",
-  "skill_history": ["open fridge", "pick up lettuce", "place lettuce"],
-  "task_description": "making a sandwich"
-}
-
-📤 OUTPUT FORMAT (D_syn)
-
-Each line of synthetically generated data should be:
-
-{
-  "episode_id": "ep_001",
-  "t": 37,
-  "images": ["path/to/cam0_t.jpg", "path/to/cam1_t.jpg"],
-  "skill_current": "pick up the KitKat",
-  "skill_history": [...],
-  "user_prompt": "Can you grab me something sweet?",
-  "robot_utterance": "Sure, I can pick up the KitKat.",
-  "task_description": "making a sandwich"
-}
-
-
-Store as syn_annotations.jsonl. for debugging
-
-🧠 pgen MODEL (Qwen) REQUIREMENTS
-
-Use HuggingFace Transformers:
-
-Qwen/Qwen2-VL-7B-Instruct (or any Qwen2-VL Vision-Language model available)
-
-Use the image + text chat interface
-
-Vision inputs should be loaded with PIL
-
-Use a single forward pass that outputs BOTH ℓ_t and u_t in a structured JSON
-
-📝 PROMPT FORMAT FOR pgen
-
-Create a template like:
-
-You are a robot-assistant dialogue generator for hierarchical robot policies.
-
-You will receive:
-- A list of images showing the current robot scene.
-- The high-level task: {task_description}
-- Previous skill steps completed: {skill_history}
-- The next skill to be performed by the robot: {skill_current}
-
-Generate two things in JSON:
-1. "user_prompt": a natural-sounding user request that logically leads to the robot performing the skill "{skill_current}" given the task and history.
-2. "robot_utterance": a natural robot reply acknowledging or clarifying the request.
-
-The responses must be grounded in the visual scene, the task, and the skill history.
-
-Respond ONLY in JSON:
-{
-  "user_prompt": "...",
-  "robot_utterance": "..."
-}
-
-This resposne will have a corresponsing task_index, and the task will be saved in task.parqeut and you must update each dataset parquet in for example /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace/data/chunk-000/
-file-000.parquet to include this new feature called task_index_high_level consider udpatign the metadata in info.json as well
-📌 LOGIC REQUIRED
-construct_prompt(sample)
-
-Loads sample dict
-
-Inserts:
-
-task_description
-
-skill_history
-
-skill_current
-
-Returns a full text prompt string
-
-call_qwen(images, prompt)
-
-Loads images into Qwen-VL multimodal input format
-
-Calls model.generate
-
-Parses JSON output
-
-annotate_sample(sample)
-
-Builds prompt
-
-Calls Qwen
-
-Returns augmented sample with user_prompt + robot_utterance
-
-🚀 CLI Usage
-
-The script should run as:
-
-python annotate_pgen.py \
-  --output-dir PATH \
-  --model Qwen/Qwen2-VL-7B-Instruct \
-  --repo-id lerobot/svla_so101_pickplace \
-  --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-  --batch-size 1
-
-
-Include arguments via argparse.
-
-🔧 OTHER REQUIREMENTS
-
-Use tqdm for progress bars
-
-Log errors gracefully and continue
-
-Support GPU acceleration (device="cuda")
-
-Cache model loading so it's not reloaded every call
-
-Make the prompt deterministic but allow temperature parameter
-
-Add a flag --num-image-views-per-sample
-
-Add automatic JSON parsing with helpful error messages
-
-🎯 FINAL DELIVERABLE
-
-Cursor must now generate:
-A full Python file named annotate_pgen.py implementing the above functionality end-to-end.
-
-It should be production-ready, runnable on real data, cleanly structured, and easy to modify.
-
-
-from the paper:
-Next, we use a large vision-language model (VLM) pgen
-to produce synthetic user prompts and interjections ℓt,
-and corresponding robot utterance ut. Given Dlabeled, we
-prompt pgen with both the visual context I1
-t ,...,In
-t and the
-skill labelˆ
-ℓt (e.g., pick up the lettuce). pgen then imag-
-ines an appropriate interaction that might have led toˆ
-ℓt in a
-real user interaction: it generates possible user prompts ℓt
-(e.g., “Can you add some lettuce for me?”) along with the
-robot’s verbal responses and clarifications ut. We detail the
-A. Synthetic Data Generation
-A.1. Scenario and Response Categorization
-To ensure the quality and diversity of the synthetic data,
-we incorporate structured scenario classification and re-
-sponse categorization into the prompt design for pgen, fol-
-lowing (Stephan et al., 2024). Specifically, we classify
-interactions into different scenario types, such as nega-
-tive task (where the user instructs the robot what not to
-do), situated correction (where the user adjusts an earlier
-command based on the evolving task state), and specific
-constraint (where the user specifies particular constraints,
-such as dietary preferences). In addition, we categorize
-the robot’s responses into types such as simple confirma-
-tions, clarifications, and error handling. These classifica-
-tions guide the generation process to ensure a broad range
-of user-robot interactions.
-A.2. Prompt Construction for Contextual Grounding
-In prompt P, we include a detailed description of the task
-(e.g., bussing a table, making a sandwich, grocery shop-
-ping) and instruct the model to ground responses in visual
-observations and prior context. A key advantage of lever-
-aging large pretrained VLMs is their ability to incorporate
-world knowledge when generating interactions. For in-
-stance, the model can infer dietary constraints when gener-
-ating prompts for sandwich-making, producing user com-
-mands such as “Can you make a sandwich for me? I’m
-lactose intolerant” and an appropriate robot response like
-“Sure, I won’t put cheese on it.” Similarly, it can reason
-over ambiguous or implicit requests, such as inferring that
-“I want something sweet” in a grocery shopping scenario
-should lead to suggestions like chocolate or candy.
-To maintain consistency in multi-step tasks, we condition
-pgen on prior skill labels within an episodeˆ
-ˆ
-ℓ0,...,
-ℓt−1,
-allowing it to generate coherent user commands that
-account for past actions. For instance, if the robot
-has already placed lettuce and tomato on a sandwich,
-the generated user prompt might request additional in-
-gredients that logically follow. This ensures that the
-synthetic interactions reflect realistic task progression
-rather than isolated commands. As such, we leverage
-ˆ
-ˆ
-ˆ
-pgen(ℓt,ut|I1
-t ,...,In
-t ,
-ℓ0,...,
-ℓt−1,
-ℓt,P) to produce a richer,
-more diverse synthetic dataset Dsyn that provides mean-
-ingful supervision for training our high-level policy.
-While in this work we generate a separate Dsyn and train
-a separate high-level policy for each task (e.g., sandwich
-making vs. table cleaning) for clarity and ease of bench-
-marking, the architecture is readily amenable to a unified
-multi-task formulation. In principle, the same hierarchical
-approach could be used to train a single high-level policy
-across a multitude of tasks, facilitating knowledge transfer
-
-
-The result should be a new LeRobotDataset with a new feature called task_index_high_level inside each dataset parquet

examples/dataset/run.sh

@@ -1,10 +0,0 @@
-# python examples/dataset/annotate.py \
-#     --repo-id lerobot/svla_so101_pickplace \
-#     --video-key observation.images.side \
-#     --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-
-python examples/dataset/annotate.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --video-key observation.images.side \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --episodes 3 5 7 44

examples/dataset/run_pgen.sh

@@ -1,42 +0,0 @@
-#!/bin/bash
-
-# Example script to run synthetic data generation with Qwen VLM
-# This generates user prompts and robot utterances for hierarchical policy training
-
-# Configuration
-REPO_ID="lerobot/svla_so101_pickplace"
-MODEL="Qwen/Qwen3-VL-30B-A3B-Instruct"
-# Alternative: MODEL="Qwen/Qwen2-VL-7B-Instruct"
-
-
-OUTPUT_DIR="/fsx/jade_choghari/outputs/pgen_annotations1"
-BATCH_SIZE=32
-TEMPERATURE=0.9
-SAMPLE_INTERVAL=5.0  # Generate dialogue every 1 second (all episodes processed)
-
-# Run synthetic data generation (processes ALL episodes)
-python examples/dataset/annotate_pgen.py \
-    --repo-id "$REPO_ID" \
-    --model "$MODEL" \
-    --output-dir "$OUTPUT_DIR" \
-    --temperature "$TEMPERATURE" \
-    --batch-size "$BATCH_SIZE" \
-    --sample-interval "$SAMPLE_INTERVAL" \
-    --num-image-views-per-sample 1
-
-# For faster testing, increase sample interval:
-# --sample-interval 5.0  # Samples every 5 seconds (much faster)
-
-# To push to hub after generation:
-# Add --push-to-hub flag
-
-# Efficient batch processing: 4 episodes at once
-# python examples/dataset/annotate_pgen.py \
-#     --repo-id "$REPO_ID" \
-#     --model "$MODEL" \
-#     --output-dir "$OUTPUT_DIR" \
-#     --video-mode \
-#     --video-key observation.images.up \
-#     --video-batch-size "$BATCH_SIZE" \
-#     --sample-interval 1.0
-

examples/dataset/subtask_annotation.py

@@ -1,802 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-SARM Subtask Annotation using local GPU (Qwen3-VL).
-
-This script implements the annotation approach from the SARM paper using local GPU inference:
-"SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation"
-Paper: https://arxiv.org/pdf/2509.25358
-
-What it does:
-1. Takes videos from a LeRobot dataset
-2. Uses Qwen3-VL running locally on GPU to identify when subtasks occur
-3. Saves subtask timestamps to the dataset metadata
-4. Optionally pushes the annotated dataset to HuggingFace Hub
-
-SARM trains reward models that predict:
-  - Stage: Which subtask is currently being executed (discrete classification)
-  - Progress: How far along the subtask we are (continuous 0-1)
-
-Supports three annotation modes:
-  1. No annotations (no args): Auto-creates single sparse "task" stage covering full episode.
-     Use with SARM config annotation_mode="single_stage" for simple tasks.
-
-  2. Dense-only (--dense-only --dense-subtasks): Dense annotations from VLM, auto-generated
-     single sparse "task" stage. Use with annotation_mode="dense_only".
-
-  3. Dual mode (--sparse-subtasks + --dense-subtasks): Both sparse and dense annotations
-     from VLM. Use with annotation_mode="dual".
-
-Requirements:
-  - GPU with sufficient VRAM (16GB+ recommended for 30B model)
-  - `pip install transformers, torch, qwen-vl-utils`
-
-Run with:
-```bash
-python examples/dataset_annotation/subtask_annotation.py \
-  --repo-id your-username/your-dataset \
-  --sparse-subtasks "Do ..." \
-  --dense-subtasks "Do task 1, Do task 2, Do task 3" \
-  --video-key observation.images.base \
-  --push-to-hub
-```
-"""
-
-import argparse
-import json
-import multiprocessing as mp
-import re
-import subprocess
-import tempfile
-import textwrap
-import time
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-import cv2
-import pandas as pd
-import torch
-from qwen_vl_utils import process_vision_info
-from rich.console import Console
-from transformers import AutoProcessor, Qwen3VLMoeForConditionalGeneration
-
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.policies.sarm.sarm_utils import (
-    Subtask,
-    SubtaskAnnotation,
-    Timestamp,
-    compute_temporal_proportions,
-)
-
-
-def create_sarm_prompt(subtask_list: list[str]) -> str:
-    subtask_str = "\n".join([f"  - {name}" for name in subtask_list])
-
-    return textwrap.dedent(f"""\
-        # Role
-        You are a Robotics Vision System specializing in temporal action localization for robot manipulation. Your job is to segment a single demonstration video into distinct, non-overlapping atomic actions from a fixed subtask list.
-
-        # Subtask Label Set (Closed Vocabulary)
-        You must strictly identify the video segments using ONLY the following labels. Do not create new labels or modify existing ones:
-
-        [
-        {subtask_str}
-        ]
-
-        The video shows one successful execution of all subtasks in a logical order.
-
-        # Ground-Truth Semantics (Very Important)
-        Use **visual state changes** to define when a subtask starts and ends. Do NOT assume equal durations for the subtasks.
-
-        - A subtask **starts** at the first frame where the robot's motion clearly initiates that subtask.
-        - A subtask **ends** at the first frame where that specific action is visually completed and the manipulated object reaches a temporary, stable configuration.
-
-        If there are short pauses or micro-motions that don't clearly correspond to a new subtask, they belong to the **current** subtask.
-
-        # Hard Constraints & Logic
-        1. **Continuous Coverage (No Gaps):**
-           - The entire video duration from "00:00" to the final timestamp must be covered by subtasks.
-           - There can be no gaps between subtasks.
-           - If there is any idle or ambiguous time between clear actions, extend the *preceding* subtask to cover it.
-
-        2. **Boundary Consistency:**
-           - The `"end"` timestamp of one subtask must be exactly equal to the `"start"` timestamp of the next subtask.
-           - Boundaries must coincide with a real visual state transition, not just a convenient time split.
-
-        3. **Chronological Order, One Occurrence Each:**
-           - This is a single successful demonstration.
-           - Each subtask from the vocabulary appears **exactly once**, in the correct logical order.
-           - **Durations may be very different** between subtasks. Never assume they are similar lengths. Base all boundaries only on the video.
-
-        4. **Reject Uniform Segmentation (Important):**
-           - Do NOT simply divide the video into equal or nearly equal time chunks.
-           - If your boundaries would result in subtasks with similar durations (e.g. all around 5 seconds), treat this as evidence that your segmentation is wrong and refine the boundaries.
-           - Only use nearly equal durations if the video truly shows each subtask taking the same amount of time (this is very rare).
-
-        5. **Timestamps:**
-           - Timestamps must be in `"MM:SS"` format.
-           - The first subtask always starts at `"00:00"`.
-           - The last subtask ends at the final visible frame of the video.
-
-        # Step 1 — Textual Timeline (must do this first)
-        First, write a extensive and detailed textual timeline describing what happens in the video with approximate timestamps.
-        For each subtask, include:
-        - its name
-        - an approximate start and end time,
-        - an description of the visual event at the boundary (e.g. "shirt fully folded to the left", "robot rotates folded shirt 90 degrees").
-
-        Format this as a bullet list.
-
-        # Step 2 — JSON Output (final answer)
-        After the textual timeline, output **only** valid JSON with this structure.
-        The JSON **must** be consistent with the textual timeline above:
-
-        {{
-          "subtasks": [
-            {{
-              "name": "EXACT_NAME_FROM_LIST",
-              "timestamps": {{
-                "start": "MM:SS",
-                "end":   "MM:SS"
-              }}
-            }},
-            {{
-              "name": "EXACT_NAME_FROM_LIST",
-              "timestamps": {{
-                "start": "MM:SS",
-                "end":   "MM:SS"
-              }}
-            }}
-          ]
-        }}
-
-        Do not add any extra keys to the JSON.
-        """)
-
-
-class VideoAnnotator:
-    """Annotates robot manipulation videos using local Qwen3-VL model on GPU"""
-
-    def __init__(
-        self,
-        subtask_list: list[str],
-        model_name: str = "Qwen/Qwen3-VL-30B-A3B-Instruct",
-        device: str = "cuda",
-        torch_dtype: torch.dtype = torch.bfloat16,
-        model: "Qwen3VLMoeForConditionalGeneration | None" = None,
-        processor: "AutoProcessor | None" = None,
-    ):
-        """
-        Initialize the video annotator with local model.
-
-        Args:
-            subtask_list: List of allowed subtask names (for consistency)
-            model_name: Hugging Face model name (default: Qwen/Qwen3-VL-30B-A3B-Instruct)
-            device: Device to use (cuda, cpu)
-            torch_dtype: Data type for model (bfloat16, float16, float32)
-            model: Pre-loaded model instance (optional, to share between annotators)
-            processor: Pre-loaded processor instance (optional, to share between annotators)
-        """
-        self.subtask_list = subtask_list
-        self.prompt = create_sarm_prompt(subtask_list)
-        self.console = Console()
-        self.device = device
-
-        # Use provided model/processor or load new ones
-        if model is not None and processor is not None:
-            self.model = model
-            self.processor = processor
-            self.console.print(f"[green]✓ Using shared model on {device}[/green]")
-        else:
-            self.console.print(f"[cyan]Loading model: {model_name}...[/cyan]")
-
-            self.model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
-                model_name, torch_dtype=torch_dtype, device_map=device, trust_remote_code=True
-            )
-
-            self.processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
-
-            self.console.print(f"[green]✓ Model loaded successfully on {device}[/green]")
-
-    def extract_episode_segment(
-        self, file_path: Path, start_timestamp: float, end_timestamp: float, target_fps: int = 1
-    ) -> Path:
-        """
-        Extract a specific episode segment from concatenated video.
-        Uses minimal compression to preserve quality for local inference.
-
-        Args:
-            file_path: Path to the concatenated video file
-            start_timestamp: Starting timestamp in seconds (within this video file)
-            end_timestamp: Ending timestamp in seconds (within this video file)
-            target_fps: Target FPS (default: 1 for faster processing)
-
-        Returns:
-            Path to extracted video file
-        """
-        # Create temporary file for extracted video
-        tmp_file = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
-        tmp_path = Path(tmp_file.name)
-        tmp_file.close()
-
-        try:
-            # Check if ffmpeg is available
-            subprocess.run(
-                ["ffmpeg", "-version"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True
-            )
-        except (subprocess.CalledProcessError, FileNotFoundError):
-            raise RuntimeError("ffmpeg not found, cannot extract episode segment") from e
-
-        try:
-            # Calculate duration
-            duration = end_timestamp - start_timestamp
-
-            self.console.print(
-                f"[cyan]Extracting episode: {start_timestamp:.1f}s-{end_timestamp:.1f}s ({duration:.1f}s)[/cyan]"
-            )
-
-            # Use ffmpeg to extract segment with minimal quality loss
-            cmd = [
-                "ffmpeg",
-                "-i",
-                str(file_path),
-                "-ss",
-                str(start_timestamp),
-                "-t",
-                str(duration),
-                "-r",
-                str(target_fps),
-                "-c:v",
-                "libx264",
-                "-preset",
-                "ultrafast",
-                "-crf",
-                "23",
-                "-an",
-                "-y",
-                str(tmp_path),
-            ]
-
-            subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True)
-
-            # Verify the output file was created and is not empty
-            if not tmp_path.exists() or tmp_path.stat().st_size == 0:
-                self.console.print("[red]✗ Video extraction failed (0 bytes) - skipping episode[/red]")
-                if tmp_path.exists():
-                    tmp_path.unlink()
-                raise RuntimeError("FFmpeg produced empty video file")
-
-            # Show extraction results
-            file_size_mb = tmp_path.stat().st_size / (1024 * 1024)
-
-            # Fail if file is too small (< 100KB likely means extraction failed)
-            if file_size_mb < 0.1:
-                self.console.print(
-                    f"[red]✗ Extracted video too small ({file_size_mb:.2f}MB) - skipping episode[/red]"
-                )
-                tmp_path.unlink()
-                raise RuntimeError(f"Video extraction produced invalid file ({file_size_mb:.2f}MB)")
-
-            self.console.print(f"[green]✓ Extracted: {file_size_mb:.1f}MB ({target_fps} FPS)[/green]")
-
-            return tmp_path
-
-        except subprocess.CalledProcessError as e:
-            raise RuntimeError(f"ffmpeg failed ({e})") from e
-
-    def annotate(
-        self,
-        file_path: str | Path,
-        fps: int,
-        start_timestamp: float = 0.0,
-        end_timestamp: float | None = None,
-        max_retries: int = 3,
-    ) -> SubtaskAnnotation:
-        """Annotate a video segment using local GPU."""
-        file_path = Path(file_path)
-
-        if end_timestamp is None:
-            cap = cv2.VideoCapture(str(file_path))
-            end_timestamp = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) / (cap.get(cv2.CAP_PROP_FPS) or 1)
-            cap.release()
-
-        duration = end_timestamp - start_timestamp
-        duration_str = f"{int(duration // 60):02d}:{int(duration % 60):02d}"
-
-        extracted_path = self.extract_episode_segment(file_path, start_timestamp, end_timestamp, 1)
-        is_extracted = extracted_path != file_path
-
-        try:
-            messages = [
-                {"role": "system", "content": [{"type": "text", "text": self.prompt}]},
-                {
-                    "role": "user",
-                    "content": [
-                        {"type": "video", "video": str(extracted_path), "fps": 1.0},
-                        {
-                            "type": "text",
-                            "text": f"Video is {duration_str} (~{duration:.1f}s). Follow instructions.",
-                        },
-                    ],
-                },
-            ]
-
-            for attempt in range(max_retries):
-                try:
-                    text = self.processor.apply_chat_template(
-                        messages, tokenize=False, add_generation_prompt=True
-                    )
-                    image_inputs, video_inputs = process_vision_info(messages)
-                    inputs = self.processor(
-                        text=[text],
-                        images=image_inputs,
-                        videos=video_inputs,
-                        padding=True,
-                        return_tensors="pt",
-                    ).to(self.device)
-
-                    with torch.no_grad():
-                        generated_ids = self.model.generate(
-                            **inputs, max_new_tokens=1024, do_sample=True, temperature=0.7
-                        )
-
-                    response = self.processor.batch_decode(
-                        [out[len(inp) :] for inp, out in zip(inputs.input_ids, generated_ids)],
-                        skip_special_tokens=True,
-                    )[0].strip()
-
-                    # Extract JSON
-                    if "```json" in response:
-                        response = response.split("```json")[1].split("```")[0]
-                    elif "```" in response:
-                        response = response.split("```")[1].split("```")[0]
-
-                    try:
-                        return SubtaskAnnotation.model_validate(json.loads(response))
-                    except json.JSONDecodeError:
-                        match = re.search(r"\{.*\}", response, re.DOTALL)
-                        if match:
-                            return SubtaskAnnotation.model_validate(json.loads(match.group()))
-                        raise ValueError("No JSON found")
-                except Exception as e:
-                    if attempt == max_retries - 1:
-                        raise RuntimeError(f"Failed after {max_retries} attempts") from e
-                    time.sleep(1)
-        finally:
-            if is_extracted and extracted_path.exists():
-                extracted_path.unlink()
-
-
-def display_annotation(
-    annotation: SubtaskAnnotation, console: Console, episode_idx: int, fps: int, prefix: str = ""
-):
-    """Display annotation summary."""
-    subtask_summary = ", ".join(
-        f"{s.name}({s.timestamps.start}-{s.timestamps.end})" for s in annotation.subtasks
-    )
-    console.print(
-        f"[green]Episode {episode_idx} {prefix}: {len(annotation.subtasks)} subtasks - {subtask_summary}[/green]"
-    )
-
-
-def timestamp_to_seconds(timestamp: str) -> float:
-    """Convert MM:SS or SS timestamp to seconds"""
-    parts = timestamp.split(":")
-    if len(parts) == 2:
-        return int(parts[0]) * 60 + int(parts[1])
-    else:
-        return int(parts[0])
-
-
-def save_annotations_to_dataset(
-    dataset_path: Path, annotations: dict[int, SubtaskAnnotation], fps: int, prefix: str = "sparse"
-):
-    """Save annotations to LeRobot dataset parquet format."""
-    from lerobot.datasets.utils import DEFAULT_EPISODES_PATH, load_episodes
-
-    episodes_dataset = load_episodes(dataset_path)
-    if not episodes_dataset or len(episodes_dataset) == 0:
-        return
-
-    episodes_df = episodes_dataset.to_pandas()
-    cols = [
-        f"{prefix}_{c}"
-        for c in [
-            "subtask_names",
-            "subtask_start_times",
-            "subtask_end_times",
-            "subtask_start_frames",
-            "subtask_end_frames",
-        ]
-    ]
-    for col in cols:
-        episodes_df[col] = None
-
-    for ep_idx, ann in annotations.items():
-        if ep_idx >= len(episodes_df):
-            continue
-        names, starts, ends, start_frames, end_frames = [], [], [], [], []
-        for s in ann.subtasks:
-            names.append(s.name)
-            st, et = timestamp_to_seconds(s.timestamps.start), timestamp_to_seconds(s.timestamps.end)
-            starts.append(st)
-            ends.append(et)
-            start_frames.append(int(st * fps))
-            end_frames.append(int(et * fps))
-        episodes_df.at[ep_idx, cols[0]] = names
-        episodes_df.at[ep_idx, cols[1]] = starts
-        episodes_df.at[ep_idx, cols[2]] = ends
-        episodes_df.at[ep_idx, cols[3]] = start_frames
-        episodes_df.at[ep_idx, cols[4]] = end_frames
-
-    # Group by file and write
-    for ep_idx in episodes_df.index:
-        key = (
-            episodes_df.loc[ep_idx, "meta/episodes/chunk_index"],
-            episodes_df.loc[ep_idx, "meta/episodes/file_index"],
-        )
-        path = dataset_path / DEFAULT_EPISODES_PATH.format(chunk_index=key[0], file_index=key[1])
-        if path.exists():
-            file_df = pd.read_parquet(path)
-            for col in cols + (
-                [
-                    "subtask_names",
-                    "subtask_start_times",
-                    "subtask_end_times",
-                    "subtask_start_frames",
-                    "subtask_end_frames",
-                ]
-                if prefix == "sparse"
-                else []
-            ):
-                if col not in file_df.columns:
-                    file_df[col] = None
-            if ep_idx in annotations:
-                for col in cols:
-                    file_df.at[ep_idx, col] = episodes_df.loc[ep_idx, col]
-                if prefix == "sparse":  # Legacy columns
-                    for i, legacy in enumerate(
-                        [
-                            "subtask_names",
-                            "subtask_start_times",
-                            "subtask_end_times",
-                            "subtask_start_frames",
-                            "subtask_end_frames",
-                        ]
-                    ):
-                        file_df.at[ep_idx, legacy] = episodes_df.loc[ep_idx, cols[i]]
-            file_df.to_parquet(path, engine="pyarrow", compression="snappy")
-
-
-def generate_auto_sparse_annotations(
-    dataset: LeRobotDataset, episode_indices: list[int], video_key: str
-) -> dict[int, SubtaskAnnotation]:
-    """Auto-generate single 'task' stage annotations for all episodes."""
-    annotations = {}
-    for ep_idx in episode_indices:
-        start = float(dataset.meta.episodes[f"videos/{video_key}/from_timestamp"][ep_idx])
-        end = float(dataset.meta.episodes[f"videos/{video_key}/to_timestamp"][ep_idx])
-        duration = end - start
-        end_str = f"{int(duration // 60):02d}:{int(duration % 60):02d}"
-        annotations[ep_idx] = SubtaskAnnotation(
-            subtasks=[Subtask(name="task", timestamps=Timestamp(start="00:00", end=end_str))]
-        )
-    return annotations
-
-
-def load_annotations_from_dataset(dataset_path: Path, prefix: str = "sparse") -> dict[int, SubtaskAnnotation]:
-    """Load annotations from LeRobot dataset parquet files."""
-    from lerobot.datasets.utils import load_episodes
-
-    episodes_dataset = load_episodes(dataset_path)
-    if not episodes_dataset or len(episodes_dataset) == 0:
-        return {}
-
-    col_names = f"{prefix}_subtask_names"
-    col_start = f"{prefix}_subtask_start_times"
-    col_end = f"{prefix}_subtask_end_times"
-
-    # Fall back to legacy columns for sparse
-    if col_names not in episodes_dataset.column_names:
-        if prefix == "sparse" and "subtask_names" in episodes_dataset.column_names:
-            col_names, col_start, col_end = "subtask_names", "subtask_start_times", "subtask_end_times"
-        else:
-            return {}
-
-    df = episodes_dataset.to_pandas()
-    annotations = {}
-    for ep_idx in df.index:
-        names = df.loc[ep_idx, col_names]
-        if names is None or (isinstance(names, float) and pd.isna(names)):
-            continue
-        starts, ends = df.loc[ep_idx, col_start], df.loc[ep_idx, col_end]
-        annotations[int(ep_idx)] = SubtaskAnnotation(
-            subtasks=[
-                Subtask(
-                    name=n,
-                    timestamps=Timestamp(
-                        start=f"{int(s) // 60:02d}:{int(s) % 60:02d}",
-                        end=f"{int(e) // 60:02d}:{int(e) % 60:02d}",
-                    ),
-                )
-                for n, s, e in zip(names, starts, ends)
-            ]
-        )
-    return annotations
-
-
-def process_single_episode(
-    ep_idx: int,
-    dataset_root: Path,
-    dataset_meta,
-    video_key: str,
-    fps: int,
-    annotator: VideoAnnotator,
-    console: Console,
-) -> tuple[int, SubtaskAnnotation | None, str | None]:
-    """Process a single episode annotation."""
-    try:
-        video_path = dataset_root / dataset_meta.get_video_file_path(ep_idx, video_key)
-        if not video_path.exists():
-            return ep_idx, None, f"Video not found: {video_path}"
-
-        start = float(dataset_meta.episodes[f"videos/{video_key}/from_timestamp"][ep_idx])
-        end = float(dataset_meta.episodes[f"videos/{video_key}/to_timestamp"][ep_idx])
-        return ep_idx, annotator.annotate(video_path, fps, start, end), None
-    except Exception as e:
-        return ep_idx, None, str(e)
-
-
-def worker_process_episodes(
-    worker_id: int,
-    gpu_id: int,
-    episode_indices: list[int],
-    repo_id: str,
-    video_key: str,
-    sparse_subtask_list: list[str],
-    dense_subtask_list: list[str] | None,
-    model_name: str,
-    torch_dtype: torch.dtype,
-) -> tuple[dict, dict | None]:
-    """Worker for parallel processing across GPUs."""
-    device = f"cuda:{gpu_id}"
-    console = Console()
-    dataset = LeRobotDataset(repo_id, download_videos=False)
-
-    sparse_annotator = VideoAnnotator(sparse_subtask_list, model_name, device, torch_dtype)
-    dense_annotator = (
-        VideoAnnotator(
-            dense_subtask_list,
-            model_name,
-            device,
-            torch_dtype,
-            sparse_annotator.model,
-            sparse_annotator.processor,
-        )
-        if dense_subtask_list
-        else None
-    )
-
-    sparse_annotations, dense_annotations = {}, {} if dense_subtask_list else None
-
-    for ep_idx in episode_indices:
-        _, sparse_ann, err = process_single_episode(
-            ep_idx, dataset.root, dataset.meta, video_key, dataset.fps, sparse_annotator, console
-        )
-        if sparse_ann:
-            sparse_annotations[ep_idx] = sparse_ann
-
-        if dense_annotator:
-            _, dense_ann, _ = process_single_episode(
-                ep_idx, dataset.root, dataset.meta, video_key, dataset.fps, dense_annotator, console
-            )
-            if dense_ann:
-                dense_annotations[ep_idx] = dense_ann
-
-    return sparse_annotations, dense_annotations
-
-
-def main():
-    parser = argparse.ArgumentParser(description="SARM-style subtask annotation using local GPU (Qwen3-VL)")
-    parser.add_argument("--repo-id", type=str, required=True, help="HuggingFace dataset repository ID")
-    parser.add_argument(
-        "--sparse-subtasks", type=str, default=None, help="Comma-separated sparse subtask names"
-    )
-    parser.add_argument(
-        "--dense-subtasks", type=str, default=None, help="Comma-separated dense subtask names"
-    )
-    parser.add_argument(
-        "--dense-only", action="store_true", help="Dense-only mode with auto-generated sparse 'task' stage"
-    )
-    parser.add_argument("--episodes", type=int, nargs="+", default=None, help="Episode indices to annotate")
-    parser.add_argument("--model", type=str, default="Qwen/Qwen3-VL-30B-A3B-Instruct", help="VLM model")
-    parser.add_argument("--skip-existing", action="store_true", help="Skip already annotated episodes")
-    parser.add_argument("--video-key", type=str, default=None, help="Video key (default: first available)")
-    parser.add_argument("--push-to-hub", action="store_true", help="Push to HuggingFace Hub")
-    parser.add_argument("--output-repo-id", type=str, default=None, help="Output repo ID for push")
-    parser.add_argument("--device", type=str, default="cuda", help="Device (cuda/cpu)")
-    parser.add_argument("--dtype", type=str, default="bfloat16", choices=["bfloat16", "float16", "float32"])
-    parser.add_argument("--num-workers", type=int, default=1, help="Parallel workers for multi-GPU")
-    parser.add_argument("--gpu-ids", type=int, nargs="+", default=None, help="GPU IDs to use")
-
-    args = parser.parse_args()
-    console = Console()
-
-    # Validate arguments
-    if args.dense_only and not args.dense_subtasks:
-        return console.print("[red]Error: --dense-only requires --dense-subtasks[/red]")
-    if args.dense_subtasks and not args.sparse_subtasks and not args.dense_only:
-        return console.print("[red]Error: --dense-subtasks requires --sparse-subtasks or --dense-only[/red]")
-
-    sparse_subtask_list = (
-        [s.strip() for s in args.sparse_subtasks.split(",")] if args.sparse_subtasks else None
-    )
-    dense_subtask_list = [s.strip() for s in args.dense_subtasks.split(",")] if args.dense_subtasks else None
-    auto_sparse = sparse_subtask_list is None
-    dense_mode = dense_subtask_list is not None
-    torch_dtype = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}[args.dtype]
-
-    console.print(f"[cyan]Loading dataset: {args.repo_id}[/cyan]")
-    dataset = LeRobotDataset(args.repo_id, download_videos=True)
-    fps = dataset.fps
-
-    if not dataset.meta.video_keys:
-        raise ValueError("No video keys found")
-
-    video_key = (
-        args.video_key if args.video_key in (dataset.meta.video_keys or []) else dataset.meta.video_keys[0]
-    )
-    console.print(f"[cyan]Using camera: {video_key}, FPS: {fps}[/cyan]")
-
-    # Determine episodes
-    episode_indices = args.episodes or list(range(dataset.meta.total_episodes))
-
-    existing_annotations = load_annotations_from_dataset(dataset.root, prefix="sparse")
-    if args.skip_existing:
-        episode_indices = [ep for ep in episode_indices if ep not in existing_annotations]
-
-    if not episode_indices:
-        return console.print("[green]All episodes already annotated![/green]")
-    console.print(f"[cyan]Annotating {len(episode_indices)} episodes[/cyan]")
-
-    # GPU setup
-    gpu_ids = args.gpu_ids or list(
-        range(min(args.num_workers, torch.cuda.device_count() if torch.cuda.is_available() else 1))
-    )
-    args.num_workers = len(gpu_ids)
-
-    sparse_annotations = existing_annotations.copy()
-    dense_annotations = {} if dense_mode else None
-
-    # Auto-sparse mode
-    if auto_sparse:
-        sparse_annotations.update(generate_auto_sparse_annotations(dataset, episode_indices, video_key))
-        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-        console.print(f"[green]Auto-generated {len(episode_indices)} sparse 'task' annotations[/green]")
-
-    # VLM annotation (for sparse if not auto, and for dense)
-    need_vlm = (not auto_sparse) or dense_mode
-
-    if need_vlm:
-        if args.num_workers > 1 and not auto_sparse:
-            # Parallel processing
-            console.print(f"[cyan]Parallel processing with {args.num_workers} workers[/cyan]")
-            episodes_per_worker = [[] for _ in range(args.num_workers)]
-            for i, ep_idx in enumerate(episode_indices):
-                episodes_per_worker[i % args.num_workers].append(ep_idx)
-
-            with ProcessPoolExecutor(
-                max_workers=args.num_workers, mp_context=mp.get_context("spawn")
-            ) as executor:
-                futures = [
-                    executor.submit(
-                        worker_process_episodes,
-                        w,
-                        gpu_ids[w],
-                        episodes_per_worker[w],
-                        args.repo_id,
-                        video_key,
-                        sparse_subtask_list,
-                        dense_subtask_list,
-                        args.model,
-                        torch_dtype,
-                    )
-                    for w in range(args.num_workers)
-                    if episodes_per_worker[w]
-                ]
-
-                for future in as_completed(futures):
-                    try:
-                        worker_sparse, worker_dense = future.result()
-                        sparse_annotations.update(worker_sparse)
-                        if dense_mode and worker_dense:
-                            dense_annotations.update(worker_dense)
-                        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-                        if dense_mode:
-                            save_annotations_to_dataset(dataset.root, dense_annotations, fps, prefix="dense")
-                    except Exception as e:
-                        raise RuntimeError(f"Worker failed: {e}") from e
-        else:
-            # Sequential processing
-            sparse_annotator = (
-                VideoAnnotator(sparse_subtask_list, args.model, args.device, torch_dtype)
-                if not auto_sparse and sparse_subtask_list
-                else None
-            )
-            dense_annotator = (
-                VideoAnnotator(
-                    dense_subtask_list,
-                    args.model,
-                    args.device,
-                    torch_dtype,
-                    sparse_annotator.model if sparse_annotator else None,
-                    sparse_annotator.processor if sparse_annotator else None,
-                )
-                if dense_mode
-                else None
-            )
-
-            for i, ep_idx in enumerate(episode_indices):
-                console.print(f"[cyan]Episode {ep_idx} ({i + 1}/{len(episode_indices)})[/cyan]")
-
-                if sparse_annotator:
-                    _, sparse_ann, err = process_single_episode(
-                        ep_idx, dataset.root, dataset.meta, video_key, fps, sparse_annotator, console
-                    )
-                    if sparse_ann:
-                        sparse_annotations[ep_idx] = sparse_ann
-                        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-                    elif err:
-                        console.print(f"[red]Sparse failed: {err}[/red]")
-
-                if dense_annotator:
-                    _, dense_ann, err = process_single_episode(
-                        ep_idx, dataset.root, dataset.meta, video_key, fps, dense_annotator, console
-                    )
-                    if dense_ann:
-                        dense_annotations[ep_idx] = dense_ann
-                        save_annotations_to_dataset(dataset.root, dense_annotations, fps, prefix="dense")
-                    elif err:
-                        console.print(f"[red]Dense failed: {err}[/red]")
-
-    # Save temporal proportions
-    def save_proportions(annotations, prefix, is_auto=False):
-        props: dict[str, float] = {"task": 1.0} if is_auto else compute_temporal_proportions(annotations, fps)
-        path = dataset.root / "meta" / f"temporal_proportions_{prefix}.json"
-        path.parent.mkdir(parents=True, exist_ok=True)
-        with open(path, "w") as f:
-            json.dump(props, f, indent=2)
-        console.print(f"[green]Saved {prefix} temporal proportions[/green]")
-
-    save_proportions(sparse_annotations, "sparse", auto_sparse)
-    if dense_mode and dense_annotations:
-        save_proportions(dense_annotations, "dense")
-
-    console.print(
-        f"\n[bold green]Complete! {len(sparse_annotations)} sparse, {len(dense_annotations or {})} dense annotations[/bold green]"
-    )
-
-    if args.push_to_hub:
-        try:
-            dataset.push_to_hub(push_videos=True)
-            console.print(f"[green]Pushed to {args.output_repo_id or args.repo_id}[/green]")
-        except Exception as e:
-            console.print(f"[red]Push failed: {e}[/red]")
-
-
-if __name__ == "__main__":
-    main()

examples/dataset/test_pgen_quick.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-# Quick test to verify the fix for task_indices length mismatch
-# This should now work correctly even with --num-samples < full dataset length
-
-echo "Testing annotate_pgen.py with --num-samples=100 on full dataset..."
-
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --num-samples 100 \
-    --sample-interval 1.0 \
-    --output-dir /fsx/jade_choghari/outputs/pgen_test_fixed
-
-if [ $? -eq 0 ]; then
-    echo "✓ SUCCESS: Script completed without errors!"
-    echo ""
-    echo "Verifying output..."
-    
-    # Check that all frames have task_index_high_level
-    python -c "
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-import numpy as np
-
-ds = LeRobotDataset(repo_id='local_test', root='/fsx/jade_choghari/outputs/pgen_test_fixed')
-print(f'Dataset has {len(ds)} frames')
-print(f'Features: {list(ds.features.keys())}')
-
-# Check that task_index_high_level exists
-assert 'task_index_high_level' in ds.features, 'task_index_high_level not in features!'
-
-# Sample some frames
-for idx in [0, 50, 99, 100, 500, 1000, 11938]:
-    if idx < len(ds):
-        frame = ds[idx]
-        task_idx = frame['task_index_high_level'].item()
-        print(f'Frame {idx}: task_index_high_level = {task_idx}')
-
-print('✓ All checks passed!')
-"
-else
-    echo "✗ FAILED: Script exited with error code $?"
-fi
-

examples/unitree_g1/gr00t_locomotion.py

@@ -1,347 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""
-Example: GR00T Locomotion with Pre-loaded Policies
-
-This example demonstrates the NEW pattern for loading GR00T policies externally
-and passing them to the robot class.
-"""
-
-import argparse
-import logging
-import threading
-import time
-from collections import deque
-
-import numpy as np
-import onnxruntime as ort
-from huggingface_hub import hf_hub_download
-
-from lerobot.robots.unitree_g1.config_unitree_g1 import UnitreeG1Config
-from lerobot.robots.unitree_g1.unitree_g1 import UnitreeG1
-
-logger = logging.getLogger(__name__)
-
-GROOT_DEFAULT_ANGLES = np.zeros(29, dtype=np.float32)
-GROOT_DEFAULT_ANGLES[[0, 6]] = -0.1  # hip pitch
-GROOT_DEFAULT_ANGLES[[3, 9]] = 0.3  # knee
-GROOT_DEFAULT_ANGLES[[4, 10]] = -0.2  # ankle pitch
-
-MISSING_JOINTS = []
-G1_MODEL = "g1_23"  # or "g1_29"
-if G1_MODEL == "g1_23":
-    MISSING_JOINTS = [12, 14, 20, 21, 27, 28]  # waist yaw/pitch, wrist pitch/yaw
-
-LOCOMOTION_ACTION_SCALE = 0.25
-
-LOCOMOTION_CONTROL_DT = 0.02
-
-ANG_VEL_SCALE: float = 0.25
-DOF_POS_SCALE: float = 1.0
-DOF_VEL_SCALE: float = 0.05
-CMD_SCALE: list = [2.0, 2.0, 0.25]
-
-
-DEFAULT_GROOT_REPO_ID = "nepyope/GR00T-WholeBodyControl_g1"
-
-
-def load_groot_policies(
-    repo_id: str = DEFAULT_GROOT_REPO_ID,
-) -> tuple[ort.InferenceSession, ort.InferenceSession]:
-    """Load GR00T dual-policy system (Balance + Walk) from Hugging Face Hub.
-
-    Args:
-        repo_id: Hugging Face Hub repository ID containing the ONNX policies.
-    """
-    logger.info(f"Loading GR00T dual-policy system from Hugging Face Hub ({repo_id})...")
-
-    # Download ONNX policies from Hugging Face Hub
-    balance_path = hf_hub_download(
-        repo_id=repo_id,
-        filename="GR00T-WholeBodyControl-Balance.onnx",
-    )
-    walk_path = hf_hub_download(
-        repo_id=repo_id,
-        filename="GR00T-WholeBodyControl-Walk.onnx",
-    )
-
-    # Load ONNX policies
-    policy_balance = ort.InferenceSession(balance_path)
-    policy_walk = ort.InferenceSession(walk_path)
-
-    logger.info("GR00T policies loaded successfully")
-
-    return policy_balance, policy_walk
-
-
-class GrootLocomotionController:
-    """
-    Handles GR00T-style locomotion control for the Unitree G1 robot.
-
-    This controller manages:
-    - Dual-policy system (Balance + Walk)
-    - 29-joint observation processing
-    - 15D action output (legs + waist)
-    - Policy inference and motor command generation
-    """
-
-    def __init__(self, policy_balance, policy_walk, robot, config):
-        self.policy_balance = policy_balance
-        self.policy_walk = policy_walk
-        self.robot = robot
-        self.config = config
-
-        self.locomotion_cmd = np.array([0.0, 0.0, 0.0], dtype=np.float32)  # vx, vy, theta_dot
-
-        # GR00T-specific state
-        self.groot_qj_all = np.zeros(29, dtype=np.float32)
-        self.groot_dqj_all = np.zeros(29, dtype=np.float32)
-        self.groot_action = np.zeros(15, dtype=np.float32)
-        self.groot_obs_single = np.zeros(86, dtype=np.float32)
-        self.groot_obs_history = deque(maxlen=6)
-        self.groot_obs_stacked = np.zeros(516, dtype=np.float32)
-        self.groot_height_cmd = 0.74  # Default base height
-        self.groot_orientation_cmd = np.array([0.0, 0.0, 0.0], dtype=np.float32)
-
-        # input to gr00t is 6 frames (6*86D=516)
-        for _ in range(6):
-            self.groot_obs_history.append(np.zeros(86, dtype=np.float32))
-
-        # Thread management
-        self.locomotion_running = False
-        self.locomotion_thread = None
-
-        logger.info("GrootLocomotionController initialized")
-
-    def groot_locomotion_run(self):
-        # get current observation
-        robot_state = self.robot.get_observation()
-
-        if robot_state is None:
-            return
-
-        # get command from remote controller
-        if robot_state.wireless_remote is not None:
-            self.robot.remote_controller.set(robot_state.wireless_remote)
-            if self.robot.remote_controller.button[0]:  # R1 - raise waist
-                self.groot_height_cmd += 0.001
-                self.groot_height_cmd = np.clip(self.groot_height_cmd, 0.50, 1.00)
-            if self.robot.remote_controller.button[4]:  # R2 - lower waist
-                self.groot_height_cmd -= 0.001
-                self.groot_height_cmd = np.clip(self.groot_height_cmd, 0.50, 1.00)
-        else:
-            self.robot.remote_controller.lx = 0.0
-            self.robot.remote_controller.ly = 0.0
-            self.robot.remote_controller.rx = 0.0
-            self.robot.remote_controller.ry = 0.0
-
-        self.locomotion_cmd[0] = self.robot.remote_controller.ly  # forward/backward
-        self.locomotion_cmd[1] = self.robot.remote_controller.lx * -1  # left/right
-        self.locomotion_cmd[2] = self.robot.remote_controller.rx * -1  # rotation rate
-
-        for i in range(29):
-            self.groot_qj_all[i] = robot_state.motor_state[i].q
-            self.groot_dqj_all[i] = robot_state.motor_state[i].dq
-
-        # adapt observation for g1_23dof
-        for idx in MISSING_JOINTS:
-            self.groot_qj_all[idx] = 0.0
-            self.groot_dqj_all[idx] = 0.0
-
-        # Scale joint positions and velocities
-        qj_obs = self.groot_qj_all.copy()
-        dqj_obs = self.groot_dqj_all.copy()
-
-        # express imu data in gravity frame of reference
-        quat = robot_state.imu_state.quaternion
-        ang_vel = np.array(robot_state.imu_state.gyroscope, dtype=np.float32)
-        gravity_orientation = self.robot.get_gravity_orientation(quat)
-
-        # scale joint positions and velocities before policy inference
-        qj_obs = (qj_obs - GROOT_DEFAULT_ANGLES) * DOF_POS_SCALE
-        dqj_obs = dqj_obs * DOF_VEL_SCALE
-        ang_vel_scaled = ang_vel * ANG_VEL_SCALE
-
-        # build single frame observation
-        self.groot_obs_single[:3] = self.locomotion_cmd * np.array(CMD_SCALE)
-        self.groot_obs_single[3] = self.groot_height_cmd
-        self.groot_obs_single[4:7] = self.groot_orientation_cmd
-        self.groot_obs_single[7:10] = ang_vel_scaled
-        self.groot_obs_single[10:13] = gravity_orientation
-        self.groot_obs_single[13:42] = qj_obs
-        self.groot_obs_single[42:71] = dqj_obs
-        self.groot_obs_single[71:86] = self.groot_action  # 15D previous actions
-
-        # Add to history and stack observations (6 frames × 86D = 516D)
-        self.groot_obs_history.append(self.groot_obs_single.copy())
-
-        # Stack all 6 frames into 516D vector
-        for i, obs_frame in enumerate(self.groot_obs_history):
-            start_idx = i * 86
-            end_idx = start_idx + 86
-            self.groot_obs_stacked[start_idx:end_idx] = obs_frame
-
-        # Run policy inference (ONNX) with 516D stacked observation
-
-        cmd_magnitude = np.linalg.norm(self.locomotion_cmd)
-
-        selected_policy = (
-            self.policy_balance if cmd_magnitude < 0.05 else self.policy_walk
-        )  # balance/standing policy for small commands, walking policy for movement commands
-
-        # run policy inference
-        ort_inputs = {selected_policy.get_inputs()[0].name: np.expand_dims(self.groot_obs_stacked, axis=0)}
-        ort_outs = selected_policy.run(None, ort_inputs)
-        self.groot_action = ort_outs[0].squeeze()
-
-        # transform action back to target joint positions
-        target_dof_pos_15 = GROOT_DEFAULT_ANGLES[:15] + self.groot_action * LOCOMOTION_ACTION_SCALE
-
-        # command motors
-        for i in range(15):
-            motor_idx = i
-            self.robot.msg.motor_cmd[motor_idx].q = target_dof_pos_15[i]
-            self.robot.msg.motor_cmd[motor_idx].qd = 0
-            self.robot.msg.motor_cmd[motor_idx].kp = self.robot.kp[motor_idx]
-            self.robot.msg.motor_cmd[motor_idx].kd = self.robot.kd[motor_idx]
-            self.robot.msg.motor_cmd[motor_idx].tau = 0
-
-        # adapt action for g1_23dof
-        for joint_idx in MISSING_JOINTS:
-            self.robot.msg.motor_cmd[joint_idx].q = 0.0
-            self.robot.msg.motor_cmd[joint_idx].qd = 0
-            self.robot.msg.motor_cmd[joint_idx].kp = self.robot.kp[joint_idx]
-            self.robot.msg.motor_cmd[joint_idx].kd = self.robot.kd[joint_idx]
-            self.robot.msg.motor_cmd[joint_idx].tau = 0
-
-        # send action to robot
-        self.robot.send_action(self.robot.msg)
-
-    def _locomotion_thread_loop(self):
-        """Background thread that runs the locomotion policy at specified rate."""
-        logger.info("Locomotion thread started")
-        while self.locomotion_running:
-            start_time = time.time()
-            try:
-                self.groot_locomotion_run()
-            except Exception as e:
-                logger.error(f"Error in locomotion loop: {e}")
-
-            # Sleep to maintain control rate
-            elapsed = time.time() - start_time
-            sleep_time = max(0, LOCOMOTION_CONTROL_DT - elapsed)
-            time.sleep(sleep_time)
-        logger.info("Locomotion thread stopped")
-
-    def start_locomotion_thread(self):
-        if self.locomotion_running:
-            logger.warning("Locomotion thread already running")
-            return
-
-        logger.info("Starting locomotion control thread...")
-        self.locomotion_running = True
-        self.locomotion_thread = threading.Thread(target=self._locomotion_thread_loop, daemon=True)
-        self.locomotion_thread.start()
-
-        logger.info("Locomotion control thread started!")
-
-    def stop_locomotion_thread(self):
-        if not self.locomotion_running:
-            return
-
-        logger.info("Stopping locomotion control thread...")
-        self.locomotion_running = False
-        if self.locomotion_thread:
-            self.locomotion_thread.join(timeout=2.0)
-        logger.info("Locomotion control thread stopped")
-
-    def reset_robot(self):
-        """Move robot legs to default standing position over 2 seconds (arms are not moved)."""
-        total_time = 3.0
-        num_step = int(total_time / self.robot.control_dt)
-
-        # Only control legs, not arms (first 12 joints)
-        default_pos = GROOT_DEFAULT_ANGLES  # First 12 values are leg angles
-        dof_size = len(default_pos)
-
-        # Get current lowstate
-        robot_state = self.robot.get_observation()
-
-        # Record the current leg positions
-        init_dof_pos = np.zeros(dof_size, dtype=np.float32)
-        for i in range(dof_size):
-            init_dof_pos[i] = robot_state.motor_state[i].q
-
-        # Move legs to default pos
-        for i in range(num_step):
-            alpha = i / num_step
-            for motor_idx in range(dof_size):
-                target_pos = default_pos[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].q = (
-                    init_dof_pos[motor_idx] * (1 - alpha) + target_pos * alpha
-                )
-                self.robot.msg.motor_cmd[motor_idx].qd = 0
-                self.robot.msg.motor_cmd[motor_idx].kp = self.robot.kp[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].kd = self.robot.kd[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].tau = 0
-            self.robot.msg.crc = self.robot.crc.Crc(self.robot.msg)
-            self.robot.lowcmd_publisher.Write(self.robot.msg)
-            time.sleep(self.robot.control_dt)
-        logger.info("Reached default position (legs only)")
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="GR00T Locomotion Controller for Unitree G1")
-    parser.add_argument(
-        "--repo-id",
-        type=str,
-        default=DEFAULT_GROOT_REPO_ID,
-        help=f"Hugging Face Hub repo ID for GR00T policies (default: {DEFAULT_GROOT_REPO_ID})",
-    )
-    args = parser.parse_args()
-
-    # load policies
-    policy_balance, policy_walk = load_groot_policies(repo_id=args.repo_id)
-
-    # initialize robot
-    config = UnitreeG1Config()
-    robot = UnitreeG1(config)
-
-    # initialize gr00t locomotion controller
-    groot_controller = GrootLocomotionController(
-        policy_balance=policy_balance,
-        policy_walk=policy_walk,
-        robot=robot,
-        config=config,
-    )
-
-    # reset legs and start locomotion thread
-    try:
-        groot_controller.reset_robot()
-        groot_controller.start_locomotion_thread()
-
-        # log status
-        logger.info("Robot initialized with GR00T locomotion policies")
-        logger.info("Locomotion controller running in background thread")
-        logger.info("Press Ctrl+C to stop")
-
-        # keep robot alive
-        while True:
-            time.sleep(1.0)
-    except KeyboardInterrupt:
-        print("\nStopping locomotion...")
-        groot_controller.stop_locomotion_thread()
-        print("Done!")

pyproject.toml

@@ -107,10 +107,6 @@ dynamixel = ["dynamixel-sdk>=3.7.31,<3.9.0"]
 gamepad = ["lerobot[pygame-dep]", "hidapi>=0.14.0,<0.15.0"]
 hopejr = ["lerobot[feetech]", "lerobot[pygame-dep]"]
 lekiwi = ["lerobot[feetech]", "pyzmq>=26.2.1,<28.0.0"]
-unitree_g1 = [
-    "pyzmq>=26.2.1,<28.0.0",
-    "onnxruntime>=1.16.0"
-]
 reachy2 = ["reachy2_sdk>=1.0.14,<1.1.0"]
 kinematics = ["lerobot[placo-dep]"]
 intelrealsense = [
@@ -362,9 +358,9 @@ ignore_errors = false
 # module = "lerobot.async_inference.*"
 # ignore_errors = false
 
-[[tool.mypy.overrides]]
-module = "lerobot.transport.*"
-ignore_errors = false
+# [[tool.mypy.overrides]]
+# module = "lerobot.transport.*"
+# ignore_errors = false
 
 # [[tool.mypy.overrides]]
 # module = "lerobot.scripts.*"

src/lerobot/datasets/aggregate.py

@@ -136,40 +136,21 @@ def update_meta_data(
         df["_orig_chunk"] = df[orig_chunk_col].copy()
         df["_orig_file"] = df[orig_file_col].copy()
 
-        # Get mappings for this video key
+        # Update chunk and file indices to point to destination
+        df[orig_chunk_col] = video_idx["chunk"]
+        df[orig_file_col] = video_idx["file"]
+
+        # Apply per-source-file timestamp offsets
         src_to_offset = video_idx.get("src_to_offset", {})
-        src_to_dst = video_idx.get("src_to_dst", {})
-
-        # Apply per-source-file mappings
-        if src_to_dst:
-            # Map each episode to its correct destination file and apply offset
+        if src_to_offset:
+            # Apply offset based on original source file
             for idx in df.index:
-                # Convert to Python int to avoid numpy type mismatch in dict lookup
-                src_key = (int(df.at[idx, "_orig_chunk"]), int(df.at[idx, "_orig_file"]))
-
-                # Get destination chunk/file for this source file
-                dst_chunk, dst_file = src_to_dst.get(src_key, (video_idx["chunk"], video_idx["file"]))
-                df.at[idx, orig_chunk_col] = dst_chunk
-                df.at[idx, orig_file_col] = dst_file
-
-                # Apply timestamp offset
-                offset = src_to_offset.get(src_key, 0)
-                df.at[idx, f"videos/{key}/from_timestamp"] += offset
-                df.at[idx, f"videos/{key}/to_timestamp"] += offset
-        elif src_to_offset:
-            # Fallback: use same destination for all, but apply per-file offsets
-            df[orig_chunk_col] = video_idx["chunk"]
-            df[orig_file_col] = video_idx["file"]
-            for idx in df.index:
-                # Convert to Python int to avoid numpy type mismatch in dict lookup
-                src_key = (int(df.at[idx, "_orig_chunk"]), int(df.at[idx, "_orig_file"]))
+                src_key = (df.at[idx, "_orig_chunk"], df.at[idx, "_orig_file"])
                 offset = src_to_offset.get(src_key, 0)
                 df.at[idx, f"videos/{key}/from_timestamp"] += offset
                 df.at[idx, f"videos/{key}/to_timestamp"] += offset
         else:
             # Fallback to simple offset (for backward compatibility)
-            df[orig_chunk_col] = video_idx["chunk"]
-            df[orig_file_col] = video_idx["file"]
             df[f"videos/{key}/from_timestamp"] = (
                 df[f"videos/{key}/from_timestamp"] + video_idx["latest_duration"]
             )
@@ -287,12 +268,6 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
         videos_idx[key]["episode_duration"] = 0
         # Track offset for each source (chunk, file) pair
         videos_idx[key]["src_to_offset"] = {}
-        # Track destination (chunk, file) for each source (chunk, file) pair
-        videos_idx[key]["src_to_dst"] = {}
-        # Initialize dst_file_durations if not present
-        # dst_file_durations tracks duration of each destination file
-        if "dst_file_durations" not in videos_idx[key]:
-            videos_idx[key]["dst_file_durations"] = {}
 
     for key, video_idx in videos_idx.items():
         unique_chunk_file_pairs = {
@@ -307,13 +282,9 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
 
         chunk_idx = video_idx["chunk"]
         file_idx = video_idx["file"]
-        dst_file_durations = video_idx["dst_file_durations"]
+        current_offset = video_idx["latest_duration"]
 
         for src_chunk_idx, src_file_idx in unique_chunk_file_pairs:
-            # Convert to Python int to ensure consistent dict keys
-            src_chunk_idx = int(src_chunk_idx)
-            src_file_idx = int(src_file_idx)
-
             src_path = src_meta.root / DEFAULT_VIDEO_PATH.format(
                 video_key=key,
                 chunk_index=src_chunk_idx,
@@ -327,17 +298,14 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
             )
 
             src_duration = get_video_duration_in_s(src_path)
-            dst_key = (chunk_idx, file_idx)
 
             if not dst_path.exists():
-                # New destination file: offset is 0
-                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = 0
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                # Store offset before incrementing
+                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_offset
                 dst_path.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copy(str(src_path), str(dst_path))
-                # Track duration of this destination file
-                dst_file_durations[dst_key] = src_duration
                 videos_idx[key]["episode_duration"] += src_duration
+                current_offset += src_duration
                 continue
 
             # Check file sizes before appending
@@ -345,11 +313,10 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
             dst_size = get_file_size_in_mb(dst_path)
 
             if dst_size + src_size >= video_files_size_in_mb:
-                # Rotate to a new file - offset is 0
-                chunk_idx, file_idx = update_chunk_file_indices(chunk_idx, file_idx, chunk_size)
-                dst_key = (chunk_idx, file_idx)
+                # Rotate to a new file, this source becomes start of new destination
+                # So its offset should be 0
                 videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = 0
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                chunk_idx, file_idx = update_chunk_file_indices(chunk_idx, file_idx, chunk_size)
                 dst_path = dst_meta.root / DEFAULT_VIDEO_PATH.format(
                     video_key=key,
                     chunk_index=chunk_idx,
@@ -357,20 +324,16 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
                 )
                 dst_path.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copy(str(src_path), str(dst_path))
-                # Track duration of this new destination file
-                dst_file_durations[dst_key] = src_duration
+                # Reset offset for next file
+                current_offset = src_duration
             else:
-                # Append to existing destination file
-                # Offset is the current duration of this destination file
-                current_dst_duration = dst_file_durations.get(dst_key, 0)
-                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_dst_duration
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                # Append to existing video file - use current accumulated offset
+                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_offset
                 concatenate_video_files(
                     [dst_path, src_path],
                     dst_path,
                 )
-                # Update duration of this destination file
-                dst_file_durations[dst_key] = current_dst_duration + src_duration
+                current_offset += src_duration
 
             videos_idx[key]["episode_duration"] += src_duration
 

src/lerobot/datasets/lerobot_dataset.py

@@ -58,7 +58,6 @@ from lerobot.datasets.utils import (
     load_nested_dataset,
     load_stats,
     load_tasks,
-    load_tasks_high_level,
     update_chunk_file_indices,
     validate_episode_buffer,
     validate_frame,
@@ -162,7 +161,6 @@ class LeRobotDatasetMetadata:
         self.info = load_info(self.root)
         check_version_compatibility(self.repo_id, self._version, CODEBASE_VERSION)
         self.tasks = load_tasks(self.root)
-        self.tasks_high_level = load_tasks_high_level(self.root)
         self.episodes = load_episodes(self.root)
         self.stats = load_stats(self.root)
 
@@ -1052,12 +1050,6 @@ class LeRobotDataset(torch.utils.data.Dataset):
         # Add task as a string
         task_idx = item["task_index"].item()
         item["task"] = self.meta.tasks.iloc[task_idx].name
-        # Optionally add high level task index
-        if "task_index_high_level" in self.features:
-            high_level_task_idx = item["task_index_high_level"].item()
-            item["robot_utterance"] = self.meta.tasks_high_level.iloc[high_level_task_idx]["robot_utterance"]
-            item["user_prompt"] = self.meta.tasks_high_level.iloc[high_level_task_idx]["user_prompt"]
-
         return item
 
     def __repr__(self):

src/lerobot/datasets/utils.py

@@ -60,7 +60,6 @@ VIDEO_DIR = "videos"
 
 CHUNK_FILE_PATTERN = "chunk-{chunk_index:03d}/file-{file_index:03d}"
 DEFAULT_TASKS_PATH = "meta/tasks.parquet"
-DEFAULT_TASKS_HIGH_LEVEL_PATH = "meta/tasks_high_level.parquet"
 DEFAULT_EPISODES_PATH = EPISODES_DIR + "/" + CHUNK_FILE_PATTERN + ".parquet"
 DEFAULT_DATA_PATH = DATA_DIR + "/" + CHUNK_FILE_PATTERN + ".parquet"
 DEFAULT_VIDEO_PATH = VIDEO_DIR + "/{video_key}/" + CHUNK_FILE_PATTERN + ".mp4"
@@ -353,9 +352,6 @@ def load_tasks(local_dir: Path) -> pandas.DataFrame:
     tasks = pd.read_parquet(local_dir / DEFAULT_TASKS_PATH)
     return tasks
 
-def load_tasks_high_level(local_dir: Path) -> pandas.DataFrame:
-    tasks = pd.read_parquet(local_dir / DEFAULT_TASKS_HIGH_LEVEL_PATH)
-    return tasks
 
 def write_episodes(episodes: Dataset, local_dir: Path) -> None:
     """Write episode metadata to a parquet file in the LeRobot v3.0 format.

src/lerobot/optim/optimizers.py

@@ -104,107 +104,6 @@ class SGDConfig(OptimizerConfig):
         return torch.optim.SGD(params, **kwargs)
 
 
-@OptimizerConfig.register_subclass("xvla-adamw")
-@dataclass
-class XVLAAdamWConfig(OptimizerConfig):
-    """Custom AdamW optimizer for XVLA with differential learning rates.
-
-    The Vision-Language Model (VLM) is trained with 1/10 of the base learning rate
-    for stable optimization, while all other components use the full LR.
-
-    This LR ratio is crucial for achieving strong and stable finetuning performance.
-
-    Soft-prompts can optionally use a separate learning rate with warm-up support.
-    Set `soft_prompt_lr_scale` to a value < 1.0 (e.g., 0.1) to start soft-prompts
-    at a lower LR. Combine with a warmup scheduler for optimal results.
-
-    Note:
-        Completely matching official reported performance may require an additional
-        warm-up LR schedule for soft-prompts, which can bring minor improvements.
-        When `soft_prompt_warmup_lr_scale` is set, soft-prompts start at
-        `lr * soft_prompt_warmup_lr_scale` and should be warmed up via the scheduler.
-
-    Parameter Groups:
-        - Group 0 (vlm): VLM parameters at lr * 0.1, weight_decay * 0.1
-        - Group 1 (soft_prompts): Soft-prompt parameters at lr * soft_prompt_lr_scale
-        - Group 2 (other): All other parameters at full lr
-    """
-
-    lr: float = 1e-4
-    betas: tuple[float, float] = (0.9, 0.99)
-    eps: float = 1e-8
-    weight_decay: float = 0.0
-    grad_clip_norm: float = 10.0
-    # Soft-prompt specific settings
-    soft_prompt_lr_scale: float = 1.0  # Scale factor for soft-prompt LR (1.0 = same as base LR)
-    soft_prompt_warmup_lr_scale: float | None = None  # If set, start soft-prompts at this scale (e.g., 0.01)
-
-    def build(self, params: dict) -> torch.optim.Optimizer:
-        """
-        Build AdamW optimizer with differential learning rates.
-
-        Expects `named_parameters()` as input (dict of name -> param).
-        Applies:
-        - lr * 0.1 for all VLM-related parameters
-        - lr * soft_prompt_lr_scale for soft-prompt parameters (with optional warmup)
-        - full lr for all other parameters
-
-        Args:
-            params: Dictionary of parameter names to parameters (from named_parameters())
-
-        Returns:
-            AdamW optimizer with parameter groups for VLM, soft-prompts, and other components
-        """
-        assert isinstance(params, dict), "Custom LR optimizer requires `named_parameters()` as inputs."
-
-        vlm_group, soft_prompt_group, other_group = [], [], []
-        for name, p in params.items():
-            if not p.requires_grad:
-                continue
-            if "vlm" in name.lower():
-                vlm_group.append(p)
-            elif "soft_prompt" in name.lower():
-                soft_prompt_group.append(p)
-            else:
-                other_group.append(p)
-
-        # Determine soft-prompt LR
-        soft_prompt_lr = self.lr * self.soft_prompt_lr_scale
-        if self.soft_prompt_warmup_lr_scale is not None:
-            # Start at warmup scale, scheduler will warm up to soft_prompt_lr
-            soft_prompt_lr = self.lr * self.soft_prompt_warmup_lr_scale
-
-        param_groups = [
-            {
-                "params": vlm_group,
-                "lr": self.lr * 0.1,
-                "weight_decay": self.weight_decay * 0.1,
-                "name": "vlm",
-            },
-            {
-                "params": soft_prompt_group,
-                "lr": soft_prompt_lr,
-                "weight_decay": self.weight_decay,
-                "name": "soft_prompts",
-            },
-            {
-                "params": other_group,
-                "lr": self.lr,
-                "weight_decay": self.weight_decay,
-                "name": "other",
-            },
-        ]
-
-        # Filter out empty groups
-        param_groups = [g for g in param_groups if len(g["params"]) > 0]
-
-        return torch.optim.AdamW(
-            param_groups,
-            betas=self.betas,
-            eps=self.eps,
-        )
-
-
 @OptimizerConfig.register_subclass("multi_adam")
 @dataclass
 class MultiAdamConfig(OptimizerConfig):

src/lerobot/policies/factory.py

@@ -16,7 +16,6 @@
 
 from __future__ import annotations
 
-import importlib
 import logging
 from typing import Any, TypedDict
 
@@ -114,10 +113,7 @@ def get_policy_class(name: str) -> type[PreTrainedPolicy]:
 
         return XVLAPolicy
     else:
-        try:
-            return _get_policy_cls_from_policy_name(name=name)
-        except Exception as e:
-            raise ValueError(f"Policy type '{name}' is not available.") from e
+        raise NotImplementedError(f"Policy with name {name} is not implemented.")
 
 
 def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
@@ -162,11 +158,7 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
     elif policy_type == "xvla":
         return XVLAConfig(**kwargs)
     else:
-        try:
-            config_cls = PreTrainedConfig.get_choice_class(policy_type)
-            return config_cls(**kwargs)
-        except Exception as e:
-            raise ValueError(f"Policy type '{policy_type}' is not available.") from e
+        raise ValueError(f"Policy type '{policy_type}' is not available.")
 
 
 class ProcessorConfigKwargs(TypedDict, total=False):
@@ -355,13 +347,7 @@ def make_pre_post_processors(
         )
 
     else:
-        try:
-            processors = _make_processors_from_policy_config(
-                config=policy_cfg,
-                dataset_stats=kwargs.get("dataset_stats"),
-            )
-        except Exception as e:
-            raise ValueError(f"Processor for policy type '{policy_cfg.type}' is not implemented.") from e
+        raise NotImplementedError(f"Processor for policy type '{policy_cfg.type}' is not implemented.")
 
     return processors
 
@@ -430,7 +416,8 @@ def make_policy(
             raise ValueError("env_cfg cannot be None when ds_meta is not provided")
         features = env_to_policy_features(env_cfg)
 
-    cfg.output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
+    if not cfg.output_features:
+        cfg.output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
     if not cfg.input_features:
         cfg.input_features = {key: ft for key, ft in features.items() if key not in cfg.output_features}
     kwargs["config"] = cfg
@@ -454,65 +441,3 @@ def make_policy(
         # TODO: (jadechoghari) - add a check_state(cfg, features) and check_action(cfg, features)
 
     return policy
-
-
-def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
-    """Get policy class from its registered name using dynamic imports.
-
-    This is used as a helper function to import policies from 3rd party lerobot plugins.
-
-    Args:
-        name: The name of the policy.
-    Returns:
-        The policy class corresponding to the given name.
-    """
-    if name not in PreTrainedConfig.get_known_choices():
-        raise ValueError(
-            f"Unknown policy name '{name}'. Available policies: {PreTrainedConfig.get_known_choices()}"
-        )
-
-    config_cls = PreTrainedConfig.get_choice_class(name)
-    config_cls_name = config_cls.__name__
-
-    model_name = config_cls_name.removesuffix("Config")  # e.g., DiffusionConfig -> Diffusion
-    if model_name == config_cls_name:
-        raise ValueError(
-            f"The config class name '{config_cls_name}' does not follow the expected naming convention."
-            f"Make sure it ends with 'Config'!"
-        )
-    cls_name = model_name + "Policy"  # e.g., DiffusionConfig -> DiffusionPolicy
-    module_path = config_cls.__module__.replace(
-        "configuration_", "modeling_"
-    )  # e.g., configuration_diffusion -> modeling_diffusion
-
-    module = importlib.import_module(module_path)
-    policy_cls = getattr(module, cls_name)
-    return policy_cls
-
-
-def _make_processors_from_policy_config(
-    config: PreTrainedConfig,
-    dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
-) -> tuple[Any, Any]:
-    """Create pre- and post-processors from a policy configuration using dynamic imports.
-
-    This is used as a helper function to import processor factories from 3rd party lerobot plugins.
-
-    Args:
-        config: The policy configuration object.
-        dataset_stats: Dataset statistics for normalization.
-    Returns:
-        A tuple containing the input (pre-processor) and output (post-processor) pipelines.
-    """
-
-    policy_type = config.type
-    function_name = f"make_{policy_type}_pre_post_processors"
-    module_path = config.__class__.__module__.replace(
-        "configuration_", "processor_"
-    )  # e.g., configuration_diffusion -> processor_diffusion
-    logging.debug(
-        f"Instantiating pre/post processors using function '{function_name}' from module '{module_path}'"
-    )
-    module = importlib.import_module(module_path)
-    function = getattr(module, function_name)
-    return function(config, dataset_stats=dataset_stats)

src/lerobot/policies/pi05/configuration_pi05.py

@@ -60,8 +60,8 @@ class PI05Config(PreTrainedConfig):
     normalization_mapping: dict[str, NormalizationMode] = field(
         default_factory=lambda: {
             "VISUAL": NormalizationMode.IDENTITY,
-            "STATE": NormalizationMode.MEAN_STD,  # Pi0.5 uses quantiles for state
-            "ACTION": NormalizationMode.MEAN_STD,  # Pi0.5 uses quantiles for action
+            "STATE": NormalizationMode.QUANTILES,  # Pi0.5 uses quantiles for state
+            "ACTION": NormalizationMode.QUANTILES,  # Pi0.5 uses quantiles for action
         }
     )
 

src/lerobot/policies/pi05/modeling_pi05.py

@@ -48,10 +48,6 @@ from lerobot.utils.constants import (
     ACTION,
     OBS_LANGUAGE_ATTENTION_MASK,
     OBS_LANGUAGE_TOKENS,
-    OBS_LANGUAGE_PROMPT_TOKENS,
-    OBS_LANGUAGE_PROMPT_ATTENTION_MASK,
-    OBS_LANGUAGE_TARGET_TOKENS,
-    OBS_LANGUAGE_TARGET_ATTENTION_MASK,
     OPENPI_ATTENTION_MASK_VALUE,
 )
 
@@ -433,8 +429,6 @@ class PaliGemmaWithExpertModel(
                 adarms_cond=adarms_cond[0] if adarms_cond is not None else None,
             )
             prefix_past_key_values = prefix_output.past_key_values
-            # prefix_output to be used for the language head
-            # shape: [batch_size, seq_len, hidden_size] with hidden_size = 2048
             prefix_output = prefix_output.last_hidden_state
             suffix_output = None
         elif inputs_embeds[0] is None:
@@ -544,8 +538,6 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
         if config.compile_model:
             torch.set_float32_matmul_precision("high")
             self.sample_actions = torch.compile(self.sample_actions, mode=config.compile_mode)
-            # Also compile the main forward pass used during training
-            self.forward = torch.compile(self.forward, mode=config.compile_mode)
 
         msg = """An incorrect transformer version is used, please create an issue on https://github.com/huggingface/lerobot/issues"""
 
@@ -584,13 +576,10 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
             )
         return func(*args, **kwargs)
 
-    def _prepare_attention_masks_4d(self, att_2d_masks, dtype=None):
+    def _prepare_attention_masks_4d(self, att_2d_masks):
         """Helper method to prepare 4D attention masks for transformer."""
         att_2d_masks_4d = att_2d_masks[:, None, :, :]
-        result = torch.where(att_2d_masks_4d, 0.0, OPENPI_ATTENTION_MASK_VALUE)
-        if dtype is not None:
-            result = result.to(dtype=dtype)
-        return result
+        return torch.where(att_2d_masks_4d, 0.0, OPENPI_ATTENTION_MASK_VALUE)
 
     def sample_noise(self, shape, device):
         return torch.normal(
@@ -609,29 +598,13 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
         return time.to(dtype=torch.float32, device=device)
 
     def embed_prefix(
-        self, images, img_masks, prompt_tokens, target_tokens, prompt_masks, target_masks=None
-    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, int]:
-        """Embed images with SigLIP, prompt tokens, and optionally target tokens with embedding layer.
-        
-        Args:
-            images: List of image tensors
-            img_masks: List of image masks
-            prompt_tokens: Prompt tokens (input for generation)
-            target_tokens: Target tokens to predict (can be None for inference)
-            prompt_masks: Attention masks for prompt tokens
-            target_masks: Attention masks for target tokens
-            
-        Returns:
-            embs: Concatenated embeddings [images, prompt_tokens, (target_tokens if provided)]
-            pad_masks: Padding masks
-            att_masks: Attention masks (with causal masking for target prediction if target_tokens provided)
-            total_T_images: Total number of image tokens
-        """
+        self, images, img_masks, tokens, masks
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Embed images with SigLIP and language tokens with embedding layer."""
         embs = []
         pad_masks = []
         att_masks = []
-        total_T_images = 0
-        
+
         # Process images
         for img, img_mask in zip(images, img_masks, strict=True):
 
@@ -643,48 +616,29 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
 
             embs.append(img_emb)
             pad_masks.append(img_mask[:, None].expand(bsize, num_img_embs))
-            att_masks += [0] * num_img_embs  # Images can attend to all previous tokens
-            total_T_images += num_img_embs
-            
-        # Process prompt tokens
-        def prompt_embed_func(prompt_tokens):
-            prompt_emb = self.paligemma_with_expert.embed_language_tokens(prompt_tokens)
-            prompt_emb_dim = prompt_emb.shape[-1]
-            return prompt_emb * math.sqrt(prompt_emb_dim)
+            att_masks += [0] * num_img_embs
 
-        prompt_emb = self._apply_checkpoint(prompt_embed_func, prompt_tokens)
-        embs.append(prompt_emb)
-        pad_masks.append(prompt_masks)
+        # Process language tokens
+        def lang_embed_func(tokens):
+            lang_emb = self.paligemma_with_expert.embed_language_tokens(tokens)
+            lang_emb_dim = lang_emb.shape[-1]
+            return lang_emb * math.sqrt(lang_emb_dim)
 
-        num_prompt_embs = prompt_emb.shape[1]
-        att_masks += [0] * num_prompt_embs  # Prompt tokens can attend to all previous tokens (images + prompt)
+        lang_emb = self._apply_checkpoint(lang_embed_func, tokens)
+        embs.append(lang_emb)
+        pad_masks.append(masks)
 
-        # Process target tokens if provided (these are predicted, so use causal masking)
-        if target_tokens is not None:
-            def target_embed_func(target_tokens):
-                target_emb = self.paligemma_with_expert.embed_language_tokens(target_tokens)
-                target_emb_dim = target_emb.shape[-1]
-                return target_emb * math.sqrt(target_emb_dim)
-
-            target_emb = self._apply_checkpoint(target_embed_func, target_tokens)
-            embs.append(target_emb)
-            
-            # Create target pad masks (non-zero tokens are valid)
-            pad_masks.append(target_masks)
-
-            num_target_embs = target_emb.shape[1]
-            # Causal masking for target tokens: each target token can attend to images, all prompt tokens,
-            # and previous target tokens
-            att_masks += [1] * num_target_embs  # Use 1 for causal attention on target tokens
+        num_lang_embs = lang_emb.shape[1]
+        att_masks += [0] * num_lang_embs
 
         embs = torch.cat(embs, dim=1)
         pad_masks = torch.cat(pad_masks, dim=1)
         att_masks = torch.tensor(att_masks, dtype=torch.bool, device=pad_masks.device)
 
         bsize = pad_masks.shape[0]
-        att_masks = att_masks[None, :].expand(bsize, att_masks.shape[0])
+        att_masks = att_masks[None, :].expand(bsize, len(att_masks))
 
-        return embs, pad_masks, att_masks, total_T_images
+        return embs, pad_masks, att_masks
 
     def embed_suffix(self, noisy_actions, timestep):
         """Embed noisy_actions, timestep to prepare for Expert Gemma processing."""
@@ -733,20 +687,8 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
 
         return embs, pad_masks, att_masks, adarms_cond
 
-    def forward(self, images, img_masks, prompt_tokens, prompt_masks, target_tokens, target_masks, actions, noise=None, time=None) -> Tensor:
-        """Do a full training forward pass and compute the loss.
-        
-        Args:
-            images: List of image tensors
-            img_masks: List of image masks
-            prompt_tokens: Prompt tokens WITHOUT target (e.g., "High level task: X; State: Y; Subtask:")
-            prompt_masks: Attention masks for prompt_tokens
-            target_tokens: Target tokens to predict (e.g., tokens for "pick up the cup")
-            target_masks: Attention masks for target_tokens
-            actions: Ground truth actions
-            noise: Optional noise for flow matching
-            time: Optional time for flow matching
-        """
+    def forward(self, images, img_masks, tokens, masks, actions, noise=None, time=None) -> Tensor:
+        """Do a full training forward pass and compute the loss."""
         if noise is None:
             noise = self.sample_noise(actions.shape, actions.device)
 
@@ -756,57 +698,10 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
         time_expanded = time[:, None, None]
         x_t = time_expanded * noise + (1 - time_expanded) * actions
         u_t = noise - actions
-        
-        # Embed prefix (images + prompt_tokens + target_tokens)
-        prefix_embs, prefix_pad_masks, prefix_att_masks, total_T_images = self.embed_prefix(
-            images, img_masks, prompt_tokens, target_tokens, prompt_masks, target_masks
-        )
-        
+
+        prefix_embs, prefix_pad_masks, prefix_att_masks = self.embed_prefix(images, img_masks, tokens, masks)
         suffix_embs, suffix_pad_masks, suffix_att_masks, adarms_cond = self.embed_suffix(x_t, time)
 
-        # Prepare attention masks for prefix-only pass (for target token prediction)
-        att_2d_prefix = make_att_2d_masks(prefix_pad_masks, prefix_att_masks)
-        position_ids_prefix = torch.cumsum(prefix_pad_masks, dim=1) - 1
-        att_2d_prefix_4d = self._prepare_attention_masks_4d(att_2d_prefix, dtype=prefix_embs.dtype)
-
-        # prefix-only transformer run for target token prediction
-        (prefix_out, _), _ = self.paligemma_with_expert.forward(
-            attention_mask=att_2d_prefix_4d,
-            position_ids=position_ids_prefix,
-            past_key_values=None,
-            inputs_embeds=[prefix_embs, None],  # SUFFIX = None
-            use_cache=False,
-            adarms_cond=[None, None],
-        )
-
-        # LM HEAD → TARGET LOGITS
-        # prefix_out: (B, T_prefix, H) where T_prefix = total_T_images + T_prompt + T_target
-        lm_head = self.paligemma_with_expert.paligemma.lm_head
-        logits = lm_head(prefix_out)  # (B, T_prefix, vocab)
-
-        # Extract logits for target token prediction (shifted by 1 for autoregressive training)
-        # Position i predicts token i+1, so we take logits from positions before target tokens:
-        # - Position (start_index-1) (last prompt token) predicts target_tokens[0]
-        # - Position (start_index) (first target token) predicts target_tokens[1], etc.
-        T_prompt = prompt_tokens.size(1)
-        T_target = target_tokens.size(1)
-        start_index = total_T_images + T_prompt
-        end_index = start_index + T_target
-        logits_target = logits[:, start_index-1:end_index-1, :]  # (B, T_target, vocab)
-
-        # Compute cross-entropy loss
-        loss_fct = torch.nn.CrossEntropyLoss(reduction='none')
-        # Reshape for loss computation
-        logits_flat = logits_target.reshape(-1, logits_target.size(-1))  # (B*T_target, vocab)
-        targets_flat = target_tokens.reshape(-1)  # (B*T_target)
-
-        loss_per_token = loss_fct(logits_flat, targets_flat)  # (B*T_target)
-        loss_per_token = loss_per_token.reshape(target_tokens.shape)  # (B, T_target)
-
-        # Apply mask and compute mean loss over valid tokens
-        masked_loss = loss_per_token * target_masks.float()
-        target_loss = masked_loss.sum() / target_masks.sum().clamp(min=1)
-        # Convert embeddings to bfloat16 if needed for the model
         if (
             self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.q_proj.weight.dtype
             == torch.bfloat16
@@ -814,14 +709,13 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
             suffix_embs = suffix_embs.to(dtype=torch.bfloat16)
             prefix_embs = prefix_embs.to(dtype=torch.bfloat16)
 
-        # Concatenate prefix (images + prompt_tokens + target_tokens) and suffix (actions) masks
         pad_masks = torch.cat([prefix_pad_masks, suffix_pad_masks], dim=1)
         att_masks = torch.cat([prefix_att_masks, suffix_att_masks], dim=1)
 
-        # Prepare attention masks for full forward pass (prefix + suffix)
         att_2d_masks = make_att_2d_masks(pad_masks, att_masks)
         position_ids = torch.cumsum(pad_masks, dim=1) - 1
-        att_2d_masks_4d = self._prepare_attention_masks_4d(att_2d_masks, dtype=prefix_embs.dtype)
+
+        att_2d_masks_4d = self._prepare_attention_masks_4d(att_2d_masks)
 
         def forward_func(prefix_embs, suffix_embs, att_2d_masks_4d, position_ids, adarms_cond):
             (_, suffix_out), _ = self.paligemma_with_expert.forward(
@@ -832,7 +726,6 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
                 use_cache=False,
                 adarms_cond=[None, adarms_cond],
             )
-            # prefix_out to be used for the language head
             return suffix_out
 
         suffix_out = self._apply_checkpoint(
@@ -847,104 +740,25 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
 
         v_t = self._apply_checkpoint(action_out_proj_func, suffix_out)
 
-        fm_loss = F.mse_loss(u_t, v_t, reduction="none")
-
-        return {
-            "flow_loss": fm_loss,
-            "target_loss": target_loss,
-            "loss": 10 * fm_loss.mean() + target_loss,
-        }
-
-    @torch.no_grad()
-    def _generate_target_tokens(
-        self, images, img_masks, prompt_tokens, prompt_masks, tokenizer, max_length, device
-    ):
-        """Generate target tokens autoregressively using next token prediction."""
-        bsize = prompt_tokens.shape[0]
-        
-        # Get lm_head for token generation
-        lm_head = self.paligemma_with_expert.paligemma.lm_head
-        
-        # Embed prefix without target tokens first
-        prefix_embs, prefix_pad_masks, prefix_att_masks, total_T_images = self.embed_prefix(
-            images, img_masks, prompt_tokens, target_tokens=None, prompt_masks=prompt_masks, target_masks=None
-        )
-        
-        # Initialize generated tokens list
-        generated_tokens = torch.zeros((bsize, max_length), dtype=torch.long, device=device)
-        
-        for t in range(max_length):
-            # Prepare attention masks for current prefix
-            att_2d_prefix = make_att_2d_masks(prefix_pad_masks, prefix_att_masks)
-            position_ids_prefix = torch.cumsum(prefix_pad_masks, dim=1) - 1
-            att_2d_prefix_4d = self._prepare_attention_masks_4d(att_2d_prefix, dtype=prefix_embs.dtype)
-            
-            # Forward pass through model to get logits
-            (prefix_out, _), _ = self.paligemma_with_expert.forward(
-                attention_mask=att_2d_prefix_4d,
-                position_ids=position_ids_prefix,
-                past_key_values=None,
-                inputs_embeds=[prefix_embs, None],
-                use_cache=False,
-                adarms_cond=[None, None],
-            )
-            
-            # Get logits from the last position
-            logits = lm_head(prefix_out)  # (B, T_prefix, vocab)
-            next_token_logits = logits[:, -1, :]  # (B, vocab)
-            
-            # Greedy decoding - take the most likely token
-            next_token = torch.argmax(next_token_logits, dim=-1)  # (B,)
-            
-            # Store generated token
-            generated_tokens[:, t] = next_token
-            
-            # Check for EOS token - if all batches have generated EOS, stop
-            if tokenizer.eos_token_id is not None:
-                if (next_token == tokenizer.eos_token_id).all():
-                    break
-            
-            # Embed the generated token and append to prefix
-            next_token_unsqueezed = next_token.unsqueeze(1)  # (B, 1)
-            
-            def next_token_embed_func(next_token_unsqueezed):
-                next_emb = self.paligemma_with_expert.embed_language_tokens(next_token_unsqueezed)
-                next_emb_dim = next_emb.shape[-1]
-                return next_emb * math.sqrt(next_emb_dim)
-            
-            next_emb = self._apply_checkpoint(next_token_embed_func, next_token_unsqueezed)
-            
-            # Append to prefix embeddings
-            prefix_embs = torch.cat([prefix_embs, next_emb], dim=1)
-            
-            # Update masks - new token is valid and uses causal attention
-            prefix_pad_masks = torch.cat([
-                prefix_pad_masks,
-                torch.ones((bsize, 1), dtype=torch.bool, device=device)
-            ], dim=1)
-            prefix_att_masks = torch.cat([prefix_att_masks, torch.ones((bsize, 1), dtype=torch.bool, device=device)], dim=1)
-        
-        return generated_tokens
+        return F.mse_loss(u_t, v_t, reduction="none")
 
     @torch.no_grad()  # see openpi `sample_actions` (slightly adapted)
     def sample_actions(
         self,
         images,
         img_masks,
-        prompt_tokens,
-        prompt_masks,
+        tokens,
+        masks,
         noise=None,
         num_steps=None,
-        tokenizer=None,
-        max_target_tokens=50,
         **kwargs: Unpack[ActionSelectKwargs],
     ) -> Tensor:
         """Do a full inference forward and compute the action."""
         if num_steps is None:
             num_steps = self.config.num_inference_steps
 
-        bsize = prompt_tokens.shape[0]
-        device = prompt_tokens.device
+        bsize = tokens.shape[0]
+        device = tokens.device
 
         if noise is None:
             # Sample noise with padded dimension as expected by action_in_proj
@@ -955,33 +769,11 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
             )  # Use config max_action_dim for internal processing
             noise = self.sample_noise(actions_shape, device)
 
-        # Generate target tokens autoregressively during inference (if tokenizer provided)
-        generated_target_tokens = None
-        target_masks = None
-        if tokenizer is not None:
-            generated_target_tokens = self._generate_target_tokens(
-                images, img_masks, prompt_tokens, prompt_masks, tokenizer, max_target_tokens, device
-            )
-            
-            # Decode and print the generated target tokens
-            for i in range(bsize):
-                # Remove padding tokens (0) and special tokens
-                valid_tokens = generated_target_tokens[i][generated_target_tokens[i] != 0]
-                decoded_text = tokenizer.decode(valid_tokens, skip_special_tokens=True)
-                print(f"[Inference] Generated target {i}: {decoded_text}")
-
-            # Create mask for generated tokens (all valid where token != 0)
-            target_masks = generated_target_tokens != 0
-
-        # Embed prefix with prompt and optionally generated target tokens
-        prefix_embs, prefix_pad_masks, prefix_att_masks, _ = self.embed_prefix(
-            images, img_masks, prompt_tokens, target_tokens=generated_target_tokens, 
-            prompt_masks=prompt_masks, target_masks=target_masks
-        )
+        prefix_embs, prefix_pad_masks, prefix_att_masks = self.embed_prefix(images, img_masks, tokens, masks)
         prefix_att_2d_masks = make_att_2d_masks(prefix_pad_masks, prefix_att_masks)
         prefix_position_ids = torch.cumsum(prefix_pad_masks, dim=1) - 1
 
-        prefix_att_2d_masks_4d = self._prepare_attention_masks_4d(prefix_att_2d_masks, dtype=prefix_embs.dtype)
+        prefix_att_2d_masks_4d = self._prepare_attention_masks_4d(prefix_att_2d_masks)
         self.paligemma_with_expert.paligemma.language_model.config._attn_implementation = "eager"  # noqa: SLF001
 
         _, past_key_values = self.paligemma_with_expert.forward(
@@ -1058,7 +850,7 @@ class PI05Pytorch(nn.Module):  # see openpi `PI0Pytorch`
         prefix_offsets = torch.sum(prefix_pad_masks, dim=-1)[:, None]
         position_ids = prefix_offsets + torch.cumsum(suffix_pad_masks, dim=1) - 1
 
-        full_att_2d_masks_4d = self._prepare_attention_masks_4d(full_att_2d_masks, dtype=suffix_embs.dtype)
+        full_att_2d_masks_4d = self._prepare_attention_masks_4d(full_att_2d_masks)
         self.paligemma_with_expert.gemma_expert.model.config._attn_implementation = "eager"  # noqa: SLF001
 
         outputs_embeds, _ = self.paligemma_with_expert.forward(
@@ -1103,14 +895,6 @@ class PI05Policy(PreTrainedPolicy):
             self.model.gradient_checkpointing_enable()
 
         self.model.to(config.device)
-        
-        # Load tokenizer for subtask decoding
-        try:
-            from transformers import AutoTokenizer
-            self.tokenizer = AutoTokenizer.from_pretrained("google/paligemma-3b-pt-224")
-        except Exception as e:
-            logging.warning(f"Could not load tokenizer for subtask decoding: {e}")
-            self.tokenizer = None
 
         self.reset()
 
@@ -1411,16 +1195,10 @@ class PI05Policy(PreTrainedPolicy):
 
         # Prepare inputs
         images, img_masks = self._preprocess_images(batch)
-        # Use prompt tokens (WITHOUT target) for inference - we'll generate the target
-        prompt_tokens = batch[f"{OBS_LANGUAGE_PROMPT_TOKENS}"]
-        prompt_masks = batch[f"{OBS_LANGUAGE_PROMPT_ATTENTION_MASK}"]
-        
+        tokens, masks = batch[f"{OBS_LANGUAGE_TOKENS}"], batch[f"{OBS_LANGUAGE_ATTENTION_MASK}"]
+
         # Sample actions using the model (pass through RTC kwargs, no separate state needed for PI05)
-        actions = self.model.sample_actions(
-            images, img_masks, prompt_tokens, prompt_masks, 
-            tokenizer=self.tokenizer,
-            **kwargs
-        )
+        actions = self.model.sample_actions(images, img_masks, tokens, masks, **kwargs)
 
         # Unpad actions to actual action dimension
         original_action_dim = self.config.output_features[ACTION].shape[0]
@@ -1433,24 +1211,22 @@ class PI05Policy(PreTrainedPolicy):
 
         # Prepare inputs
         images, img_masks = self._preprocess_images(batch)
-        prompt_tokens = batch[f"{OBS_LANGUAGE_PROMPT_TOKENS}"]
-        prompt_masks = batch[f"{OBS_LANGUAGE_PROMPT_ATTENTION_MASK}"]
-        target_tokens, target_masks = batch[f"{OBS_LANGUAGE_TARGET_TOKENS}"], batch[f"{OBS_LANGUAGE_TARGET_ATTENTION_MASK}"]
-        actions = self.prepare_action(batch)
-        
-        # Compute loss
-        # prompt_tokens = instruction tokens WITHOUT target (e.g., "High level task: X; State: Y; Subtask:")
-        # target_tokens = target tokens to predict (e.g., "pick up the cup")
-        loss_dict = self.model.forward(images, img_masks, prompt_tokens, prompt_masks, target_tokens, target_masks, actions)
+        tokens, masks = batch[f"{OBS_LANGUAGE_TOKENS}"], batch[f"{OBS_LANGUAGE_ATTENTION_MASK}"]
 
-        # Extract the total loss
-        loss = loss_dict["loss"]
-        
-        # Prepare detailed loss dictionary for logging
-        detailed_loss_dict = {
+        actions = self.prepare_action(batch)
+
+        # Compute loss (no separate state needed for PI05)
+        losses = self.model.forward(images, img_masks, tokens, masks, actions)
+
+        # Truncate losses to actual action dimensions
+        original_action_dim = self.config.output_features[ACTION].shape[0]
+        losses = losses[:, :, :original_action_dim]
+
+        loss = losses.mean()
+
+        loss_dict = {
             "loss": loss.item(),
-            "flow_loss": loss_dict["flow_loss"].mean().item(),
-            "target_loss": loss_dict["target_loss"].item(),
+            "loss_per_dim": losses.mean(dim=[0, 1]).detach().cpu().numpy().tolist(),
         }
 
-        return loss, detailed_loss_dict
+        return loss, loss_dict

src/lerobot/policies/pi05/processor_pi05.py

@@ -47,15 +47,13 @@ from lerobot.utils.constants import (
 
 @ProcessorStepRegistry.register(name="pi05_prepare_state_tokenizer_processor_step")
 @dataclass
-class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
+class Pi05PrepareStateTokenizerProcessorStep(ProcessorStep):
     """
     Processor step to prepare the state and tokenize the language input.
     """
 
     max_state_dim: int = 32
     task_key: str = "task"
-    prompt_key: str = "prompt"
-    target_key: str = "target"
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
         transition = transition.copy()
@@ -66,8 +64,6 @@ class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
         tasks = transition.get(TransitionKey.COMPLEMENTARY_DATA, {}).get(self.task_key)
         if tasks is None:
             raise ValueError("No task found in complementary data")
-        
-        high_level_tasks = transition.get(TransitionKey.COMPLEMENTARY_DATA, {}).get("user_prompt")
 
         # TODO: check if this necessary
         state = deepcopy(state)
@@ -80,33 +76,16 @@ class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
         state_np = state.cpu().numpy()
         discretized_states = np.digitize(state_np, bins=np.linspace(-1, 1, 256 + 1)[:-1]) - 1
 
-        # Clean high level tasks first (if available)
-        cleaned_high_level_tasks = []
-        if high_level_tasks is not None:
-            for high_level_task in high_level_tasks:
-                cleaned_high_level_tasks.append(high_level_task.strip().replace("_", " ").replace("\n", " "))
-        
-        # Process tasks to create prompts (input) and targets (what to predict)
-        prompts = []  # Input prompts ending with "Subtask:"
-        targets = []  # Target text to predict (the subtask)
+        full_prompts = []
         for i, task in enumerate(tasks):
             cleaned_text = task.strip().replace("_", " ").replace("\n", " ")
             state_str = " ".join(map(str, discretized_states[i]))
-            
-            # Store the subtask text as target for prediction
-            targets.append(cleaned_text)
-            
-            if cleaned_high_level_tasks:
-                cleaned_high_level_task = cleaned_high_level_tasks[i]
-                # Prompt ends with "Subtask:" - model will predict the target
-                prompt = f"High level task: {cleaned_high_level_task}; State: {state_str}; Subtask:"
-            else:
-                raise ValueError("No high level tasks found")
-            
-            prompts.append(prompt)
+            full_prompt = f"Task: {cleaned_text}, State: {state_str};\nAction: "
+            full_prompts.append(full_prompt)
 
-        transition[TransitionKey.COMPLEMENTARY_DATA][self.prompt_key] = prompts
-        transition[TransitionKey.COMPLEMENTARY_DATA][self.target_key] = targets
+        transition[TransitionKey.COMPLEMENTARY_DATA][self.task_key] = full_prompts
+        # Normalize state to [-1, 1] range if needed (assuming it's already normalized by normalizer processor step!!)
+        # Discretize into 256 bins (see openpi `PaligemmaTokenizer.tokenize()`)
         return transition
 
     def transform_features(
@@ -154,14 +133,14 @@ def make_pi05_pre_post_processors(
     input_steps: list[ProcessorStep] = [
         RenameObservationsProcessorStep(rename_map={}),  # To mimic the same processor as pretrained one
         AddBatchDimensionProcessorStep(),
-        # NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateAndLanguageTokenizerProcessorStep
+        # NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateTokenizerProcessorStep
         # because the tokenizer step expects normalized state in [-1, 1] range for discretization
         NormalizerProcessorStep(
             features={**config.input_features, **config.output_features},
             norm_map=config.normalization_mapping,
             stats=dataset_stats,
         ),
-        Pi05PrepareStateAndLanguageTokenizerProcessorStep(max_state_dim=config.max_state_dim),
+        Pi05PrepareStateTokenizerProcessorStep(max_state_dim=config.max_state_dim),
         TokenizerProcessorStep(
             tokenizer_name="google/paligemma-3b-pt-224",
             max_length=config.tokenizer_max_length,

src/lerobot/policies/xvla/action_hub.py

@@ -269,160 +269,27 @@ class AGIBOTEE6DActionSpace(BaseActionSpace):
 
 @register_action("franka_joint7")
 class FrankaJoint7ActionSpace(BaseActionSpace):
-    """
-    Franka Panda joint-space: 7 joints, with gripper.
-
-    - Real robot action dim: 7
-    - Model-facing dim: 20 (padded with zeros)
-      compatible with pretrained VLA models expecting 20D.
-    """
-
-    dim_action = 20  # model dimension
-    REAL_DIM = 7  # actual Franka joints
+    """Franka Panda joint-space: 7 joints, no gripper."""
 
+    dim_action = 7
     JOINTS_SCALE = 1.0
 
     def __init__(self):
         super().__init__()
         self.mse = nn.MSELoss()
 
-    def _pad_to_model_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Pad 7 → 20 dims (zeros for the dummy channels)."""
-        if x is None:
-            return None
-        if x.size(-1) == self.dim_action:
-            return x
-        if x.size(-1) != self.REAL_DIM:
-            raise ValueError(
-                f"Expected last dim to be {self.REAL_DIM} or {self.dim_action}, got {x.size(-1)}"
-            )
-
-        pad_shape = list(x.shape[:-1]) + [self.dim_action - self.REAL_DIM]  # 13 zeros
-        pad = x.new_zeros(pad_shape)
-        return torch.cat([x, pad], dim=-1)
-
-    def _trim_to_real_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Trim model output 20 → 7 dims."""
-        return x[..., : self.REAL_DIM]
-
     def compute_loss(self, pred, target):
-        """
-        pred :  [B, T, 20]
-        target : [B, T, 7] or [B, T, 20]
-
-        Only compute MSE on the first 7 dims.
-        """
-        pred = self._pad_to_model_dim(pred)
-        target = self._pad_to_model_dim(target)
-
-        assert pred.shape == target.shape
-
-        joints_loss = (
-            self.mse(
-                pred[:, :, : self.REAL_DIM],  # use only the first 7 joints
-                target[:, :, : self.REAL_DIM],
-            )
-            * self.JOINTS_SCALE
-        )
-
+        assert pred.shape == target.shape, "pred/target shapes must match"
+        joints_loss = self.mse(pred, target) * self.JOINTS_SCALE
         return {"joints_loss": joints_loss}
 
     def preprocess(self, proprio, action, mode="train"):
-        """
-        During training:
-        - Pad [7] → [20]
-        """
-        return proprio, self._pad_to_model_dim(action)
+        """No preprocessing needed for 7 joint actions."""
+        return proprio, action
 
     def postprocess(self, action: torch.Tensor) -> torch.Tensor:
-        """
-        After model prediction:
-        - Trim [20] → [7] for real robot control.
-        """
-        return self._trim_to_real_dim(action)
-
-
-@register_action("auto")
-class AutoActionSpace(BaseActionSpace):
-    """
-    Auto-detecting action space that adapts to any action dimension.
-
-    - Auto-detects the real action dimension from the policy feature
-    - Model outputs max_dim for compatibility with pretrained models
-    - Loss is computed only on the first real_dim dimensions
-    - Postprocess trims output back to real_dim
-
-    Args:
-        real_dim: The actual action dimension from the dataset/policy feature
-        max_dim: The model's output dimension for pretrained VLA compatibility
-    """
-
-    JOINTS_SCALE = 1.0
-
-    def __init__(self, real_dim: int, max_dim: int):
-        super().__init__()
-        self.real_dim = real_dim
-        self.dim_action = max_dim  # Model-facing dimension
-        self.mse = nn.MSELoss()
-
-    def _pad_to_model_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Pad real_dim → max_dim (zeros for the dummy channels)."""
-        if x is None:
-            return None
-        if x.size(-1) == self.dim_action:
-            return x
-        if x.size(-1) != self.real_dim:
-            # If dimension doesn't match either, pad/trim to real_dim first
-            if x.size(-1) < self.real_dim:
-                pad_shape = list(x.shape[:-1]) + [self.real_dim - x.size(-1)]
-                pad = x.new_zeros(pad_shape)
-                x = torch.cat([x, pad], dim=-1)
-            else:
-                x = x[..., : self.real_dim]
-
-        pad_shape = list(x.shape[:-1]) + [self.dim_action - self.real_dim]
-        pad = x.new_zeros(pad_shape)
-        return torch.cat([x, pad], dim=-1)
-
-    def _trim_to_real_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Trim model output max_dim → real_dim."""
-        return x[..., : self.real_dim]
-
-    def compute_loss(self, pred: torch.Tensor, target: torch.Tensor) -> dict[str, torch.Tensor]:
-        """
-        Compute loss only on the first real_dim dimensions.
-
-        pred:   [B, T, max_dim] from the model
-        target: [B, T, real_dim] or [B, T, max_dim]
-
-        Loss = MSE(pred[:,:,:real_dim], target[:,:,:real_dim])
-        """
-        pred = self._pad_to_model_dim(pred)
-        target = self._pad_to_model_dim(target)
-        assert pred.shape == target.shape, f"Shape mismatch: pred {pred.shape} vs target {target.shape}"
-
-        # only compute loss on the real dimensions
-        joints_loss = (
-            self.mse(
-                pred[:, :, : self.real_dim],
-                target[:, :, : self.real_dim],
-            )
-            * self.JOINTS_SCALE
-        )
-
-        return {"joints_loss": joints_loss}
-
-    def preprocess(self, proprio: torch.Tensor, action: torch.Tensor, mode: str = "train"):
-        """
-        Pad action from real_dim to max_dim for the model.
-        """
-        return proprio, self._pad_to_model_dim(action)
-
-    def postprocess(self, action: torch.Tensor) -> torch.Tensor:
-        """
-        Trim model output from max_dim to real_dim for real robot control.
-        """
-        return self._trim_to_real_dim(action)
+        """Return directly (no sigmoid since no gripper)."""
+        return action
 
 
 @register_action("so101_bimanual")
@@ -582,7 +449,6 @@ __all__ = [
     "JointActionSpace",
     "AGIBOTEE6DActionSpace",
     "FrankaJoint7ActionSpace",
-    "AutoActionSpace",
     "BimanualSO101ActionSpace",
     "ACTION_REGISTRY",
 ]

src/lerobot/policies/xvla/configuration_xvla.py

@@ -23,7 +23,7 @@ from typing import TYPE_CHECKING, Any
 
 from lerobot.configs.policies import PreTrainedConfig
 from lerobot.configs.types import FeatureType, NormalizationMode, PolicyFeature
-from lerobot.optim.optimizers import XVLAAdamWConfig
+from lerobot.optim.optimizers import AdamWConfig
 from lerobot.optim.schedulers import CosineDecayWithWarmupSchedulerConfig
 from lerobot.utils.constants import OBS_IMAGES
 
@@ -57,7 +57,7 @@ class XVLAConfig(PreTrainedConfig):
         default_factory=lambda: {
             "VISUAL": NormalizationMode.IDENTITY,
             "STATE": NormalizationMode.IDENTITY,
-            "ACTION": NormalizationMode.IDENTITY,
+            "ACTION": NormalizationMode.MEAN_STD,
         }
     )
 
@@ -84,7 +84,6 @@ class XVLAConfig(PreTrainedConfig):
     num_denoising_steps: int = 10
     use_proprio: bool = True
     max_state_dim: int = 32
-    max_action_dim: int = 20  # Maximum action dimension for padding (used by "auto" action mode)
     domain_feature_key: str | None = None
 
     # Vision preprocessing
@@ -94,20 +93,17 @@ class XVLAConfig(PreTrainedConfig):
 
     # Freezing options for VLM components
     # By default, VLM encoders are frozen and only policy transformer + soft prompts train
-    freeze_vision_encoder: bool = False  # Freeze VLM vision encoder weights
-    freeze_language_encoder: bool = False  # Freeze VLM language encoder weights
+    freeze_vision_encoder: bool = True  # Freeze VLM vision encoder weights
+    freeze_language_encoder: bool = True  # Freeze VLM language encoder weights
     train_policy_transformer: bool = True  # Allow policy transformer to train
     train_soft_prompts: bool = True  # Allow soft prompts to train
 
     # Training presets
     optimizer_lr: float = 1e-4
-    optimizer_betas: tuple[float, float] = (0.9, 0.99)
+    optimizer_betas: tuple[float, float] = (0.9, 0.95)
     optimizer_eps: float = 1e-8
-    optimizer_weight_decay: float = 0.0
+    optimizer_weight_decay: float = 1e-4
     optimizer_grad_clip_norm: float = 10.0
-    # Soft-prompt LR settings (for optional warm-up)
-    optimizer_soft_prompt_lr_scale: float = 1.0  # Scale factor for soft-prompt LR
-    optimizer_soft_prompt_warmup_lr_scale: float | None = None  # Start scale for warmup (e.g., 0.01)
 
     scheduler_warmup_steps: int = 1_000
     scheduler_decay_steps: int = 30_000
@@ -164,22 +160,13 @@ class XVLAConfig(PreTrainedConfig):
                         shape=(3, height, width),
                     )
 
-    def get_optimizer_preset(self) -> XVLAAdamWConfig:
-        """Return the XVLA-specific optimizer with differential learning rates.
-
-        This optimizer applies:
-        - 1/10 LR for VLM parameters (stable optimization)
-        - Full LR for transformer/action head
-        - Configurable LR for soft-prompts (with optional warm-up)
-        """
-        return XVLAAdamWConfig(
+    def get_optimizer_preset(self) -> AdamWConfig:
+        return AdamWConfig(
             lr=self.optimizer_lr,
             betas=self.optimizer_betas,
             eps=self.optimizer_eps,
             weight_decay=self.optimizer_weight_decay,
             grad_clip_norm=self.optimizer_grad_clip_norm,
-            soft_prompt_lr_scale=self.optimizer_soft_prompt_lr_scale,
-            soft_prompt_warmup_lr_scale=self.optimizer_soft_prompt_warmup_lr_scale,
         )
 
     def get_scheduler_preset(self) -> CosineDecayWithWarmupSchedulerConfig:

src/lerobot/policies/xvla/modeling_florence2.py

@@ -2508,9 +2508,6 @@ class Florence2ForConditionalGeneration(Florence2PreTrainedModel):
         return model_embeds
 
     def _encode_image(self, pixel_values):
-        # Cast pixel_values to model's dtype
-        pixel_values = pixel_values.to(dtype=self.vision_tower.convs[0].proj.weight.dtype)
-
         if len(pixel_values.shape) == 4:
             batch_size, channels, height, width = pixel_values.shape
             num_frames = 1

src/lerobot/policies/xvla/modeling_xvla.py

@@ -55,23 +55,7 @@ class XVLAModel(nn.Module):
         self.config = config
         self.chunk_size: int = config.chunk_size
         self.use_proprio: bool = config.use_proprio
-
-        # Build action space with auto-detection for "auto" mode
-        if config.action_mode.lower() == "auto":
-            # Auto-detect real action dim from config.action_feature
-            real_dim = (
-                config.action_feature.shape[-1]
-                if config.action_feature is not None
-                else config.max_action_dim
-            )
-            self.action_space = build_action_space(
-                config.action_mode.lower(),
-                real_dim=real_dim,
-                max_dim=config.max_action_dim,
-            )
-        else:
-            self.action_space = build_action_space(config.action_mode.lower())
-
+        self.action_space = build_action_space(config.action_mode.lower())
         self.dim_action = self.action_space.dim_action
         self.dim_proprio = proprio_dim
 
@@ -200,20 +184,12 @@ class XVLAModel(nn.Module):
         proprio: torch.Tensor,
         action: torch.Tensor,
     ) -> dict[str, torch.Tensor]:
-        """
-        Forward pass for the XVLA model.
-        """
-        target_dtype = self._get_target_dtype()
-        image_input = image_input.to(dtype=target_dtype)
-        proprio = proprio.to(dtype=target_dtype)
-        action = action.to(dtype=target_dtype)
-
         enc = self.forward_vlm(input_ids, image_input, image_mask)
 
         batch_size = input_ids.shape[0]
         t = (
-            torch.rand(1, device=input_ids.device, dtype=target_dtype)
-            + torch.arange(batch_size, device=input_ids.device, dtype=target_dtype) / batch_size
+            torch.rand(1, device=input_ids.device)
+            + torch.arange(batch_size, device=input_ids.device) / batch_size
         ) % (1 - 1e-5)
 
         action_noisy = torch.randn_like(action) * t.view(-1, 1, 1) + action * (1 - t).view(-1, 1, 1)
@@ -239,22 +215,17 @@ class XVLAModel(nn.Module):
         steps: int,
     ) -> torch.Tensor:
         self.eval()
-
-        target_dtype = self._get_target_dtype()
-        image_input = image_input.to(dtype=target_dtype)
-        proprio = proprio.to(dtype=target_dtype)
-
         enc = self.forward_vlm(input_ids, image_input, image_mask)
 
         batch_size = input_ids.shape[0]
         action_dim = self.dim_action
 
-        x1 = torch.randn(batch_size, self.chunk_size, action_dim, device=proprio.device, dtype=target_dtype)
+        x1 = torch.randn(batch_size, self.chunk_size, action_dim, device=proprio.device, dtype=proprio.dtype)
         action = torch.zeros_like(x1)
 
         steps = max(1, int(steps))
         for i in range(steps, 0, -1):
-            t = torch.full((batch_size,), i / steps, device=proprio.device, dtype=target_dtype)
+            t = torch.full((batch_size,), i / steps, device=proprio.device, dtype=proprio.dtype)
             x_t = x1 * t.view(-1, 1, 1) + action * (1 - t).view(-1, 1, 1)
             proprio_m, x_t_m = self.action_space.preprocess(proprio, x_t)
             action = self.transformer(
@@ -287,13 +258,8 @@ class XVLAPolicy(PreTrainedPolicy):
         }
 
     def get_optim_params(self) -> dict:
-        """Return trainable named parameters for optimization.
-
-        Returns a dict of name -> param for all trainable parameters.
-        This enables the xvla-adamw optimizer to apply differential learning rates
-        based on parameter names (e.g., 1/10 LR for VLM components).
-        """
-        return dict(filter(lambda kv: kv[1].requires_grad, self.named_parameters()))
+        """Return only trainable parameters for optimization."""
+        return filter(lambda p: p.requires_grad, self.parameters())
 
     def _prepare_state(self, batch: dict[str, Tensor], batch_size: int, device: torch.device) -> Tensor:
         if not self.config.use_proprio or OBS_STATE not in batch:
@@ -383,6 +349,17 @@ class XVLAPolicy(PreTrainedPolicy):
             "proprio": proprio,
         }
 
+    def _trim_action_dim(self, actions: Tensor) -> Tensor:
+        feature = self.config.action_feature
+        if feature is None:
+            return actions
+        desired_dim = self.model.dim_action
+        if desired_dim == actions.shape[-1]:
+            return actions
+        if desired_dim < actions.shape[-1]:
+            return actions[..., :desired_dim]
+        return pad_vector(actions, desired_dim)
+
     def forward(self, batch: dict[str, Tensor]) -> tuple[Tensor, dict]:
         inputs = self._build_model_inputs(batch)
         targets = self._prepare_action_targets(batch)
@@ -396,6 +373,7 @@ class XVLAPolicy(PreTrainedPolicy):
     def _get_action_chunk(self, batch: dict[str, Tensor]) -> Tensor:
         inputs = self._build_model_inputs(batch)
         actions = self.model.generate_actions(**inputs, steps=self.config.num_denoising_steps)
+        actions = self._trim_action_dim(actions)
         return actions
 
     @torch.no_grad()

src/lerobot/policies/xvla/processor_xvla.py

@@ -310,19 +310,16 @@ class XVLAImageToFloatProcessorStep(ProcessorStep):
             if key in obs and isinstance(obs[key], torch.Tensor):
                 tensor = obs[key]
 
-                min_val = tensor.min().item()
-                max_val = tensor.max().item()
-
-                if max_val <= 1.0:
-                    obs[key] = tensor.float()  # ensure float dtype, but no division
-                    continue
                 # Validate that values are in [0, 255] range if requested
-                if self.validate_range and (min_val < 0.0 or max_val > 255.0):
-                    raise ValueError(
-                        f"Image '{key}' has values outside [0, 255] range: "
-                        f"min={min_val:.4f}, max={max_val:.4f}. "
-                        f"Cannot convert to [0, 1] range."
-                    )
+                if self.validate_range:
+                    min_val = tensor.min().item()
+                    max_val = tensor.max().item()
+                    if min_val < 0.0 or max_val > 255.0:
+                        raise ValueError(
+                            f"Image '{key}' has values outside [0, 255] range: "
+                            f"min={min_val:.4f}, max={max_val:.4f}. "
+                            f"Cannot convert to [0, 1] range."
+                        )
 
                 # Convert to float and divide by 255
                 obs[key] = tensor.float() / 255.0

src/lerobot/processor/converters.py

@@ -168,12 +168,10 @@ def _extract_complementary_data(batch: dict[str, Any]) -> dict[str, Any]:
     """
     pad_keys = {k: v for k, v in batch.items() if "_is_pad" in k}
     task_key = {"task": batch["task"]} if "task" in batch else {}
-    user_prompt_key = {"user_prompt": batch["user_prompt"]} if "user_prompt" in batch else {}
-    subtask_key = {"subtask": batch["subtask"]} if "subtask" in batch else {}
     index_key = {"index": batch["index"]} if "index" in batch else {}
     task_index_key = {"task_index": batch["task_index"]} if "task_index" in batch else {}
 
-    return {**pad_keys, **task_key, **index_key, **task_index_key, **user_prompt_key, **subtask_key}
+    return {**pad_keys, **task_key, **index_key, **task_index_key}
 
 
 def create_transition(

src/lerobot/processor/rename_processor.py

@@ -47,6 +47,7 @@ class RenameObservationsProcessorStep(ObservationProcessorStep):
                 processed_obs[self.rename_map[key]] = value
             else:
                 processed_obs[key] = value
+
         return processed_obs
 
     def get_config(self) -> dict[str, Any]:

src/lerobot/processor/tokenizer_processor.py

@@ -29,14 +29,7 @@ from typing import TYPE_CHECKING, Any
 import torch
 
 from lerobot.configs.types import FeatureType, PipelineFeatureType, PolicyFeature
-from lerobot.utils.constants import (
-    OBS_LANGUAGE_ATTENTION_MASK,
-    OBS_LANGUAGE_PROMPT_ATTENTION_MASK,
-    OBS_LANGUAGE_PROMPT_TOKENS,
-    OBS_LANGUAGE_TOKENS,
-    OBS_LANGUAGE_TARGET_TOKENS,
-    OBS_LANGUAGE_TARGET_ATTENTION_MASK,
-)
+from lerobot.utils.constants import OBS_LANGUAGE_ATTENTION_MASK, OBS_LANGUAGE_TOKENS
 from lerobot.utils.import_utils import _transformers_available
 
 from .core import EnvTransition, TransitionKey
@@ -59,9 +52,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
     tokenizes it using a Hugging Face `transformers` tokenizer, and adds the resulting
     token IDs and attention mask to the `observation` dictionary.
 
-    Optionally, this step can also tokenize a prompt (input for generation) and/or
-    a target (text to predict) if present in the complementary data, creating separate tokenized observations.
-
     Requires the `transformers` library to be installed.
 
     Attributes:
@@ -69,8 +59,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
         tokenizer: A pre-initialized tokenizer object. If provided, `tokenizer_name` is ignored.
         max_length: The maximum length to pad or truncate sequences to.
         task_key: The key in `complementary_data` where the task string is stored.
-        prompt_key: The key in `complementary_data` where the prompt (input for generation) is stored.
-        target_key: The key in `complementary_data` where the target (text to predict) is stored.
         padding_side: The side to pad on ('left' or 'right').
         padding: The padding strategy ('max_length', 'longest', etc.).
         truncation: Whether to truncate sequences longer than `max_length`.
@@ -81,8 +69,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
     tokenizer: Any | None = None  # Use `Any` for compatibility without a hard dependency
     max_length: int = 512
     task_key: str = "task"
-    prompt_key: str = "prompt"
-    target_key: str = "target"
     padding_side: str = "right"
     padding: str = "max_length"
     truncation: bool = True
@@ -135,7 +121,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
             raise ValueError("Complementary data is None so no task can be extracted from it")
 
         task = complementary_data[self.task_key]
-        
         if task is None:
             raise ValueError("Task extracted from Complementary data is None")
 
@@ -147,60 +132,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
 
         return None
 
-    def get_prompt(self, transition: EnvTransition) -> list[str] | None:
-        """
-        Extracts the prompt (input for generation) from the transition's complementary data.
-
-        Args:
-            transition: The environment transition.
-
-        Returns:
-            A list of prompt strings, or None if the prompt key is not found or the value is None.
-        """
-        complementary_data = transition.get(TransitionKey.COMPLEMENTARY_DATA)
-        if complementary_data is None:
-            return None
-
-        prompt = complementary_data.get(self.prompt_key)
-        
-        if prompt is None:
-            return None
-
-        # Standardize to a list of strings for the tokenizer
-        if isinstance(prompt, str):
-            return [prompt]
-        elif isinstance(prompt, list) and all(isinstance(t, str) for t in prompt):
-            return prompt
-
-        return None
-
-    def get_target(self, transition: EnvTransition) -> list[str] | None:
-        """
-        Extracts the target (text to predict) from the transition's complementary data.
-
-        Args:
-            transition: The environment transition.
-
-        Returns:
-            A list of target strings, or None if the target key is not found or the value is None.
-        """
-        complementary_data = transition.get(TransitionKey.COMPLEMENTARY_DATA)
-        if complementary_data is None:
-            return None
-
-        target = complementary_data.get(self.target_key)
-        
-        if target is None:
-            return None
-
-        # Standardize to a list of strings for the tokenizer
-        if isinstance(target, str):
-            return [target]
-        elif isinstance(target, list) and all(isinstance(t, str) for t in target):
-            return target
-
-        return None
-
     def observation(self, observation: dict[str, Any]) -> dict[str, Any]:
         """
         Tokenizes the task description and adds it to the observation dictionary.
@@ -238,38 +169,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
         new_observation[OBS_LANGUAGE_TOKENS] = tokenized_prompt["input_ids"]
         new_observation[OBS_LANGUAGE_ATTENTION_MASK] = tokenized_prompt["attention_mask"].to(dtype=torch.bool)
 
-        # Also tokenize prompt (input for generation) if available
-        prompt = self.get_prompt(self.transition)
-        if prompt is not None:
-            tokenized_prompt_input = self._tokenize_text(prompt)
-
-            # Move to the same device
-            if target_device is not None:
-                tokenized_prompt_input = {
-                    k: v.to(target_device) if isinstance(v, torch.Tensor) else v
-                    for k, v in tokenized_prompt_input.items()
-                }
-
-            # Add prompt tokenized data to the observation
-            new_observation[OBS_LANGUAGE_PROMPT_TOKENS] = tokenized_prompt_input["input_ids"]
-            new_observation[OBS_LANGUAGE_PROMPT_ATTENTION_MASK] = tokenized_prompt_input["attention_mask"].to(dtype=torch.bool)
-
-        # Also tokenize target (text to predict) if available
-        target = self.get_target(self.transition)
-        if target is not None:
-            tokenized_target = self._tokenize_text(target)
-
-            # Move to the same device
-            if target_device is not None:
-                tokenized_target = {
-                    k: v.to(target_device) if isinstance(v, torch.Tensor) else v
-                    for k, v in tokenized_target.items()
-                }
-
-            # Add target tokenized data to the observation
-            new_observation[OBS_LANGUAGE_TARGET_TOKENS] = tokenized_target["input_ids"]
-            new_observation[OBS_LANGUAGE_TARGET_ATTENTION_MASK] = tokenized_target["attention_mask"].to(dtype=torch.bool)
-            
         return new_observation
 
     def _detect_device(self, transition: EnvTransition) -> torch.device | None:
@@ -330,8 +229,6 @@ class TokenizerProcessorStep(ObservationProcessorStep):
         config = {
             "max_length": self.max_length,
             "task_key": self.task_key,
-            "prompt_key": self.prompt_key,
-            "target_key": self.target_key,
             "padding_side": self.padding_side,
             "padding": self.padding,
             "truncation": self.truncation,
@@ -370,26 +267,4 @@ class TokenizerProcessorStep(ObservationProcessorStep):
                 type=FeatureType.LANGUAGE, shape=(self.max_length,)
             )
 
-        # Add features for prompt tokens and attention mask if they don't already exist
-        if OBS_LANGUAGE_PROMPT_TOKENS not in features[PipelineFeatureType.OBSERVATION]:
-            features[PipelineFeatureType.OBSERVATION][OBS_LANGUAGE_PROMPT_TOKENS] = PolicyFeature(
-                type=FeatureType.LANGUAGE, shape=(self.max_length,)
-            )
-
-        if OBS_LANGUAGE_PROMPT_ATTENTION_MASK not in features[PipelineFeatureType.OBSERVATION]:
-            features[PipelineFeatureType.OBSERVATION][OBS_LANGUAGE_PROMPT_ATTENTION_MASK] = PolicyFeature(
-                type=FeatureType.LANGUAGE, shape=(self.max_length,)
-            )
-        
-        # Add features for target tokens and attention mask if they don't already exist
-        if OBS_LANGUAGE_TARGET_TOKENS not in features[PipelineFeatureType.OBSERVATION]:
-            features[PipelineFeatureType.OBSERVATION][OBS_LANGUAGE_TARGET_TOKENS] = PolicyFeature(
-                type=FeatureType.LANGUAGE, shape=(self.max_length,)
-            )
-
-        if OBS_LANGUAGE_TARGET_ATTENTION_MASK not in features[PipelineFeatureType.OBSERVATION]:
-            features[PipelineFeatureType.OBSERVATION][OBS_LANGUAGE_TARGET_ATTENTION_MASK] = PolicyFeature(
-                type=FeatureType.LANGUAGE, shape=(self.max_length,)
-            )
-
         return features

src/lerobot/robots/earthrover_mini_plus/__init__.py

@@ -1,20 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .config_earthrover_mini_plus import EarthRoverMiniPlusConfig
-from .robot_earthrover_mini_plus import EarthRoverMiniPlus
-
-__all__ = ["EarthRoverMiniPlus", "EarthRoverMiniPlusConfig"]

src/lerobot/robots/earthrover_mini_plus/config_earthrover_mini_plus.py

@@ -1,35 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Configuration for EarthRover Mini Plus robot."""
-
-from dataclasses import dataclass
-
-from ..config import RobotConfig
-
-
-@RobotConfig.register_subclass("earthrover_mini_plus")
-@dataclass
-class EarthRoverMiniPlusConfig(RobotConfig):
-    """Configuration for EarthRover Mini Plus robot using Frodobots SDK.
-
-    This robot uses cloud-based control via the Frodobots SDK HTTP API.
-    Camera frames are accessed directly through SDK HTTP endpoints.
-
-    Attributes:
-        sdk_url: URL of the Frodobots SDK server (default: http://localhost:8000)
-    """
-
-    sdk_url: str = "http://localhost:8000"

src/lerobot/robots/earthrover_mini_plus/earthrover_mini_plus.mdx

@@ -1 +0,0 @@
-../../../../docs/source/earthrover_mini_plus.mdx

src/lerobot/robots/earthrover_mini_plus/robot_earthrover_mini_plus.py

@@ -1,473 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""EarthRover Mini Plus robot using Frodobots SDK."""
-
-import base64
-import logging
-from functools import cached_property
-from typing import Any
-
-import cv2
-import numpy as np
-import requests
-
-from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
-
-from ..robot import Robot
-from .config_earthrover_mini_plus import EarthRoverMiniPlusConfig
-
-logger = logging.getLogger(__name__)
-
-# Action feature keys
-ACTION_LINEAR_VEL = "linear.vel"
-ACTION_ANGULAR_VEL = "angular.vel"
-
-# Observation feature keys
-OBS_FRONT = "front"
-OBS_REAR = "rear"
-OBS_LINEAR_VEL = "linear.vel"
-OBS_BATTERY_LEVEL = "battery.level"
-OBS_ORIENTATION_DEG = "orientation.deg"
-OBS_GPS_LATITUDE = "gps.latitude"
-OBS_GPS_LONGITUDE = "gps.longitude"
-OBS_GPS_SIGNAL = "gps.signal"
-OBS_SIGNAL_LEVEL = "signal.level"
-OBS_VIBRATION = "vibration"
-OBS_LAMP_STATE = "lamp.state"
-
-
-class EarthRoverMiniPlus(Robot):
-    """
-    EarthRover Mini Plus robot controlled via Frodobots SDK HTTP API.
-
-    This robot uses cloud-based control through the Frodobots SDK instead of direct
-    hardware connection. Cameras stream via WebRTC through Agora cloud, and control
-    commands are sent via HTTP POST requests.
-
-    The robot supports:
-    - Dual cameras (front and rear) accessed via SDK HTTP endpoints
-    - Linear and angular velocity control
-    - Battery and orientation telemetry
-
-    Attributes:
-        config: Robot configuration
-        sdk_base_url: URL of the Frodobots SDK server (default: http://localhost:8000)
-    """
-
-    config_class = EarthRoverMiniPlusConfig
-    name = "earthrover_mini_plus"
-
-    def __init__(self, config: EarthRoverMiniPlusConfig):
-        """Initialize EarthRover Mini Plus robot.
-
-        Args:
-            config: Robot configuration including SDK URL
-        """
-        super().__init__(config)
-        self.config = config
-        self.sdk_base_url = "http://localhost:8000"
-
-        # Empty cameras dict for compatibility with recording script
-        # Cameras are accessed directly via SDK, not through Camera objects
-        self.cameras = {}
-        self._is_connected = False
-
-        # Cache for camera frames (fallback when requests fail)
-        self._last_front_frame = None
-        self._last_rear_frame = None
-
-        # Cache for robot telemetry data (fallback when requests fail)
-        self._last_robot_data = None
-
-        logger.info(f"Initialized {self.name} with SDK at {self.sdk_base_url}")
-
-    @property
-    def is_connected(self) -> bool:
-        """Check if robot is connected to SDK."""
-        return self._is_connected
-
-    def connect(self, calibrate: bool = True) -> None:
-        """Connect to robot via Frodobots SDK.
-
-        Args:
-            calibrate: Not used for SDK-based robot (kept for API compatibility)
-
-        Raises:
-            DeviceAlreadyConnectedError: If robot is already connected
-            DeviceNotConnectedError: If cannot connect to SDK server
-        """
-        if self._is_connected:
-            raise DeviceAlreadyConnectedError(f"{self.name} is already connected")
-
-        # Verify SDK is running and accessible
-        try:
-            response = requests.get(f"{self.sdk_base_url}/data", timeout=10.0)
-            if response.status_code != 200:
-                raise DeviceNotConnectedError(
-                    f"Cannot connect to SDK at {self.sdk_base_url}. "
-                    "Make sure it's running: hypercorn main:app --reload"
-                )
-        except requests.RequestException as e:
-            raise DeviceNotConnectedError(f"Cannot connect to SDK at {self.sdk_base_url}: {e}") from e
-
-        self._is_connected = True
-        logger.info(f"{self.name} connected to SDK")
-
-        if calibrate:
-            self.calibrate()
-
-    def calibrate(self) -> None:
-        """Calibration not needed for SDK-based robot."""
-        logger.info("Calibration not required for SDK-based robot")
-
-    @property
-    def is_calibrated(self) -> bool:
-        """SDK robot doesn't require calibration.
-
-        Returns:
-            bool: Always True for SDK-based robots
-        """
-        return True
-
-    def configure(self) -> None:
-        """Configure robot (no-op for SDK-based robot)."""
-        pass
-
-    @cached_property
-    def observation_features(self) -> dict[str, type | tuple]:
-        """Define the observation space for dataset recording.
-
-        Returns:
-            dict: Observation features with types/shapes:
-                - front: (480, 640, 3) - Front camera RGB image
-                - rear: (480, 640, 3) - Rear camera RGB image
-                - linear.vel: float - Current speed (0-1, SDK reports only positive speeds)
-                - battery.level: float - Battery level (0-1, normalized from 0-100)
-                - orientation.deg: float - Robot orientation (0-1, normalized from raw value)
-                - gps.latitude: float - GPS latitude coordinate
-                - gps.longitude: float - GPS longitude coordinate
-                - gps.signal: float - GPS signal strength (0-1, normalized from percentage)
-                - signal.level: float - Network signal level (0-1, normalized from 0-5)
-                - vibration: float - Vibration sensor reading
-                - lamp.state: float - Lamp state (0=off, 1=on)
-        """
-        return {
-            # Cameras (height, width, channels)
-            OBS_FRONT: (480, 640, 3),
-            OBS_REAR: (480, 640, 3),
-            # Motion state
-            OBS_LINEAR_VEL: float,
-            # Robot state
-            OBS_BATTERY_LEVEL: float,
-            OBS_ORIENTATION_DEG: float,
-            # GPS
-            OBS_GPS_LATITUDE: float,
-            OBS_GPS_LONGITUDE: float,
-            OBS_GPS_SIGNAL: float,
-            # Sensors
-            OBS_SIGNAL_LEVEL: float,
-            OBS_VIBRATION: float,
-            OBS_LAMP_STATE: float,
-        }
-
-    @cached_property
-    def action_features(self) -> dict[str, type]:
-        """Define the action space.
-
-        Returns:
-            dict: Action features with types:
-                - linear.vel: float - Target linear velocity
-                - angular.vel: float - Target angular velocity
-        """
-        return {
-            ACTION_LINEAR_VEL: float,
-            ACTION_ANGULAR_VEL: float,
-        }
-
-    def get_observation(self) -> dict[str, Any]:
-        """Get current robot observation from SDK.
-
-        Returns:
-            dict: Observation containing:
-                - front: Front camera image (480, 640, 3) in RGB format
-                - rear: Rear camera image (480, 640, 3) in RGB format
-                - linear.vel: Current speed (0-1, SDK reports only positive speeds)
-                - battery.level: Battery level (0-1, normalized from 0-100)
-                - orientation.deg: Robot orientation (0-1, normalized from raw value)
-                - gps.latitude: GPS latitude coordinate
-                - gps.longitude: GPS longitude coordinate
-                - gps.signal: GPS signal strength (0-1, normalized from percentage)
-                - signal.level: Network signal level (0-1, normalized from 0-5)
-                - vibration: Vibration sensor reading
-                - lamp.state: Lamp state (0=off, 1=on)
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-
-        Note:
-            Camera frames are retrieved from SDK endpoints /v2/front and /v2/rear.
-            Frames are decoded from base64 and converted from BGR to RGB format.
-            Robot telemetry is retrieved from /data endpoint.
-            All SDK values are normalized to appropriate ranges for dataset recording.
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        observation = {}
-
-        # Get camera images from SDK
-        frames = self._get_camera_frames()
-        observation[OBS_FRONT] = frames["front"]
-        observation[OBS_REAR] = frames["rear"]
-
-        # Get robot state from SDK
-        robot_data = self._get_robot_data()
-
-        # Motion state
-        observation[OBS_LINEAR_VEL] = robot_data["speed"] / 100.0  # Normalize 0-100 to 0-1
-
-        # Robot state
-        observation[OBS_BATTERY_LEVEL] = robot_data["battery"] / 100.0  # Normalize 0-100 to 0-1
-        observation[OBS_ORIENTATION_DEG] = robot_data["orientation"] / 360.0  # Normalize to 0-1
-
-        # GPS data
-        observation[OBS_GPS_LATITUDE] = robot_data["latitude"]
-        observation[OBS_GPS_LONGITUDE] = robot_data["longitude"]
-        observation[OBS_GPS_SIGNAL] = robot_data["gps_signal"] / 100.0  # Normalize percentage to 0-1
-
-        # Sensors
-        observation[OBS_SIGNAL_LEVEL] = robot_data["signal_level"] / 5.0  # Normalize 0-5 to 0-1
-        observation[OBS_VIBRATION] = robot_data["vibration"]
-        observation[OBS_LAMP_STATE] = float(robot_data["lamp"])  # 0 or 1
-
-        return observation
-
-    def send_action(self, action: dict[str, Any]) -> dict[str, Any]:
-        """Send action to robot via SDK.
-
-        Args:
-            action: Action dict with keys:
-                - linear.vel: Target linear velocity (-1 to 1)
-                - angular.vel: Target angular velocity (-1 to 1)
-
-        Returns:
-            dict: The action that was sent (matches action_features keys)
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-
-        Note:
-            Actions are sent to SDK via POST /control endpoint.
-            SDK expects commands in range [-1, 1].
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        # Extract action values and convert to float
-        linear = float(action.get(ACTION_LINEAR_VEL, 0.0))
-        angular = float(action.get(ACTION_ANGULAR_VEL, 0.0))
-
-        # Send command to SDK
-        try:
-            self._send_command_to_sdk(linear, angular)
-        except Exception as e:
-            logger.error(f"Error sending action: {e}")
-
-        # Return action in format matching action_features
-        return {
-            ACTION_LINEAR_VEL: linear,
-            ACTION_ANGULAR_VEL: angular,
-        }
-
-    def disconnect(self) -> None:
-        """Disconnect from robot.
-
-        Stops the robot and closes connection to SDK.
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        # Stop the robot before disconnecting
-        try:
-            self._send_command_to_sdk(0.0, 0.0)
-        except Exception as e:
-            logger.warning(f"Failed to stop robot during disconnect: {e}")
-
-        self._is_connected = False
-        logger.info(f"{self.name} disconnected")
-
-    # Private helper methods for SDK communication
-
-    def _get_camera_frames(self) -> dict[str, np.ndarray]:
-        """Get camera frames from SDK using v2 endpoints with caching fallback.
-
-        Returns:
-            dict: Dictionary with 'front' and 'rear' keys containing:
-                - Current frame (if request succeeds)
-                - Cached frame (if request fails but cache exists)
-                - Zero array (if request fails and no cache exists yet)
-
-        Note:
-            Uses /v2/front and /v2/rear endpoints which are 15x faster than /screenshot.
-            Images are base64 encoded, resized to 640x480, and converted from BGR to RGB.
-            If request fails, returns the last successfully retrieved frame (cached).
-        """
-        frames = {}
-
-        # Get front camera
-        try:
-            response = requests.get(f"{self.sdk_base_url}/v2/front", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                if "front_frame" in data and data["front_frame"]:
-                    front_img = self._decode_base64_image(data["front_frame"])
-                    if front_img is not None:
-                        # Resize and convert BGR to RGB
-                        front_img = cv2.resize(front_img, (640, 480))
-                        front_rgb = cv2.cvtColor(front_img, cv2.COLOR_BGR2RGB)
-                        frames["front"] = front_rgb
-                        # Cache the successful frame
-                        self._last_front_frame = front_rgb
-        except Exception as e:
-            logger.warning(f"Error fetching front camera: {e}")
-
-        # Fallback: use cache or zero array
-        if "front" not in frames:
-            if self._last_front_frame is not None:
-                frames["front"] = self._last_front_frame
-            else:
-                frames["front"] = np.zeros((480, 640, 3), dtype=np.uint8)
-
-        # Get rear camera
-        try:
-            response = requests.get(f"{self.sdk_base_url}/v2/rear", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                if "rear_frame" in data and data["rear_frame"]:
-                    rear_img = self._decode_base64_image(data["rear_frame"])
-                    if rear_img is not None:
-                        # Resize and convert BGR to RGB
-                        rear_img = cv2.resize(rear_img, (640, 480))
-                        rear_rgb = cv2.cvtColor(rear_img, cv2.COLOR_BGR2RGB)
-                        frames["rear"] = rear_rgb
-                        # Cache the successful frame
-                        self._last_rear_frame = rear_rgb
-        except Exception as e:
-            logger.warning(f"Error fetching rear camera: {e}")
-
-        # Fallback: use cache or zero array
-        if "rear" not in frames:
-            if self._last_rear_frame is not None:
-                frames["rear"] = self._last_rear_frame
-            else:
-                frames["rear"] = np.zeros((480, 640, 3), dtype=np.uint8)
-
-        return frames
-
-    def _decode_base64_image(self, base64_string: str) -> np.ndarray | None:
-        """Decode base64 string to image.
-
-        Args:
-            base64_string: Base64 encoded image string
-
-        Returns:
-            np.ndarray: Decoded image in BGR format (OpenCV default), or None if decoding fails
-        """
-        try:
-            img_bytes = base64.b64decode(base64_string)
-            nparr = np.frombuffer(img_bytes, np.uint8)
-            img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
-            return img  # Return in BGR format (OpenCV default)
-        except Exception as e:
-            logger.error(f"Error decoding image: {e}")
-            return None
-
-    def _get_robot_data(self) -> dict:
-        """Get robot telemetry data from SDK.
-
-        Returns:
-            dict: Robot telemetry data including battery, speed, orientation, GPS, etc:
-                - Current data (if request succeeds)
-                - Cached data (if request fails but cache exists)
-                - Default values (if request fails and no cache exists yet)
-
-        Note:
-            Uses /data endpoint which provides comprehensive robot state.
-            If request fails, returns the last successfully retrieved data (cached).
-        """
-        try:
-            response = requests.get(f"{self.sdk_base_url}/data", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                # Cache the successful data
-                self._last_robot_data = data
-                return data
-        except Exception as e:
-            logger.warning(f"Error fetching robot data: {e}")
-
-        # Fallback: use cache or default values
-        if self._last_robot_data is not None:
-            return self._last_robot_data
-        else:
-            # Return dict with default values (used only on first failure before any cache exists)
-            return {
-                "speed": 0,
-                "battery": 0,
-                "orientation": 0,
-                "latitude": 0.0,
-                "longitude": 0.0,
-                "gps_signal": 0,
-                "signal_level": 0,
-                "vibration": 0.0,
-                "lamp": 0,
-            }
-
-    def _send_command_to_sdk(self, linear: float, angular: float, lamp: int = 0) -> bool:
-        """Send control command to SDK.
-
-        Args:
-            linear: Linear velocity command (-1 to 1)
-            angular: Angular velocity command (-1 to 1)
-            lamp: Lamp control (0=off, 1=on)
-
-        Returns:
-            bool: True if command sent successfully, False otherwise
-
-        Note:
-            Uses POST /control endpoint. Commands are sent as JSON payload.
-        """
-        try:
-            payload = {
-                "command": {
-                    "linear": linear,
-                    "angular": angular,
-                    "lamp": lamp,
-                }
-            }
-
-            response = requests.post(
-                f"{self.sdk_base_url}/control",
-                json=payload,
-                timeout=1.0,
-            )
-
-            return response.status_code == 200
-        except Exception as e:
-            logger.error(f"Error sending command: {e}")
-            return False

src/lerobot/robots/unitree_g1/__init__.py

@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .config_unitree_g1 import UnitreeG1Config
-from .unitree_g1 import UnitreeG1

src/lerobot/robots/unitree_g1/config_unitree_g1.py

@@ -1,55 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from dataclasses import dataclass, field
-
-from ..config import RobotConfig
-
-_GAINS: dict[str, dict[str, list[float]]] = {
-    "left_leg": {
-        "kp": [150, 150, 150, 300, 40, 40],
-        "kd": [2, 2, 2, 4, 2, 2],
-    },  # pitch, roll, yaw, knee, ankle_pitch, ankle_roll
-    "right_leg": {"kp": [150, 150, 150, 300, 40, 40], "kd": [2, 2, 2, 4, 2, 2]},
-    "waist": {"kp": [250, 250, 250], "kd": [5, 5, 5]},  # yaw, roll, pitch
-    "left_arm": {"kp": [80, 80, 80, 80], "kd": [3, 3, 3, 3]},  # shoulder_pitch/roll/yaw, elbow
-    "left_wrist": {"kp": [40, 40, 40], "kd": [1.5, 1.5, 1.5]},  # roll, pitch, yaw
-    "right_arm": {"kp": [80, 80, 80, 80], "kd": [3, 3, 3, 3]},
-    "right_wrist": {"kp": [40, 40, 40], "kd": [1.5, 1.5, 1.5]},
-    "other": {"kp": [80, 80, 80, 80, 80, 80], "kd": [3, 3, 3, 3, 3, 3]},
-}
-
-
-def _build_gains() -> tuple[list[float], list[float]]:
-    """Build kp and kd lists from body-part groupings."""
-    kp = [v for g in _GAINS.values() for v in g["kp"]]
-    kd = [v for g in _GAINS.values() for v in g["kd"]]
-    return kp, kd
-
-
-_DEFAULT_KP, _DEFAULT_KD = _build_gains()
-
-
-@RobotConfig.register_subclass("unitree_g1")
-@dataclass
-class UnitreeG1Config(RobotConfig):
-    kp: list[float] = field(default_factory=lambda: _DEFAULT_KP.copy())
-    kd: list[float] = field(default_factory=lambda: _DEFAULT_KD.copy())
-
-    control_dt: float = 1.0 / 250.0  # 250Hz
-
-    # socket config for ZMQ bridge
-    robot_ip: str = "192.168.123.164"

src/lerobot/robots/unitree_g1/g1_utils.py

@@ -1,89 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from enum import IntEnum
-
-# ruff: noqa: N801, N815
-
-NUM_MOTORS = 35
-
-
-class G1_29_JointArmIndex(IntEnum):
-    # Left arm
-    kLeftShoulderPitch = 15
-    kLeftShoulderRoll = 16
-    kLeftShoulderYaw = 17
-    kLeftElbow = 18
-    kLeftWristRoll = 19
-    kLeftWristPitch = 20
-    kLeftWristyaw = 21
-
-    # Right arm
-    kRightShoulderPitch = 22
-    kRightShoulderRoll = 23
-    kRightShoulderYaw = 24
-    kRightElbow = 25
-    kRightWristRoll = 26
-    kRightWristPitch = 27
-    kRightWristYaw = 28
-
-
-class G1_29_JointIndex(IntEnum):
-    # Left leg
-    kLeftHipPitch = 0
-    kLeftHipRoll = 1
-    kLeftHipYaw = 2
-    kLeftKnee = 3
-    kLeftAnklePitch = 4
-    kLeftAnkleRoll = 5
-
-    # Right leg
-    kRightHipPitch = 6
-    kRightHipRoll = 7
-    kRightHipYaw = 8
-    kRightKnee = 9
-    kRightAnklePitch = 10
-    kRightAnkleRoll = 11
-
-    kWaistYaw = 12
-    kWaistRoll = 13
-    kWaistPitch = 14
-
-    # Left arm
-    kLeftShoulderPitch = 15
-    kLeftShoulderRoll = 16
-    kLeftShoulderYaw = 17
-    kLeftElbow = 18
-    kLeftWristRoll = 19
-    kLeftWristPitch = 20
-    kLeftWristyaw = 21
-
-    # Right arm
-    kRightShoulderPitch = 22
-    kRightShoulderRoll = 23
-    kRightShoulderYaw = 24
-    kRightElbow = 25
-    kRightWristRoll = 26
-    kRightWristPitch = 27
-    kRightWristYaw = 28
-
-    # not used
-    kNotUsedJoint0 = 29
-    kNotUsedJoint1 = 30
-    kNotUsedJoint2 = 31
-    kNotUsedJoint3 = 32
-    kNotUsedJoint4 = 33
-    kNotUsedJoint5 = 34

src/lerobot/robots/unitree_g1/run_g1_server.py

@@ -1,212 +0,0 @@
-#!/usr/bin/env python3
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-DDS-to-ZMQ bridge server for Unitree G1 robot.
-
-This server runs on the robot and forwards:
-- Robot state (LowState) from DDS to ZMQ (for remote clients)
-- Robot commands (LowCmd) from ZMQ to DDS (from remote clients)
-
-Uses JSON for secure serialization instead of pickle.
-"""
-
-import base64
-import contextlib
-import json
-import threading
-import time
-from typing import Any
-
-import zmq
-from unitree_sdk2py.comm.motion_switcher.motion_switcher_client import MotionSwitcherClient
-from unitree_sdk2py.core.channel import ChannelFactoryInitialize, ChannelPublisher, ChannelSubscriber
-from unitree_sdk2py.idl.default import unitree_hg_msg_dds__LowCmd_
-from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowCmd_ as hg_LowCmd, LowState_ as hg_LowState
-from unitree_sdk2py.utils.crc import CRC
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"  # action to robot
-kTopicLowState = "rt/lowstate"  # observation from robot
-
-LOWCMD_PORT = 6000
-LOWSTATE_PORT = 6001
-NUM_MOTORS = 35
-
-
-def lowstate_to_dict(msg: hg_LowState) -> dict[str, Any]:
-    """Convert LowState SDK message to a JSON-serializable dictionary."""
-    motor_states = []
-    for i in range(NUM_MOTORS):
-        temp = msg.motor_state[i].temperature
-        avg_temp = float(sum(temp) / len(temp)) if isinstance(temp, list) else float(temp)
-        motor_states.append(
-            {
-                "q": float(msg.motor_state[i].q),
-                "dq": float(msg.motor_state[i].dq),
-                "tau_est": float(msg.motor_state[i].tau_est),
-                "temperature": avg_temp,
-            }
-        )
-
-    return {
-        "motor_state": motor_states,
-        "imu_state": {
-            "quaternion": [float(x) for x in msg.imu_state.quaternion],
-            "gyroscope": [float(x) for x in msg.imu_state.gyroscope],
-            "accelerometer": [float(x) for x in msg.imu_state.accelerometer],
-            "rpy": [float(x) for x in msg.imu_state.rpy],
-            "temperature": float(msg.imu_state.temperature),
-        },
-        # Encode bytes as base64 for JSON compatibility
-        "wireless_remote": base64.b64encode(bytes(msg.wireless_remote)).decode("ascii"),
-        "mode_machine": int(msg.mode_machine),
-    }
-
-
-def dict_to_lowcmd(data: dict[str, Any]) -> hg_LowCmd:
-    """Convert dictionary back to LowCmd SDK message."""
-    cmd = unitree_hg_msg_dds__LowCmd_()
-    cmd.mode_pr = data.get("mode_pr", 0)
-    cmd.mode_machine = data.get("mode_machine", 0)
-
-    for i, motor_data in enumerate(data.get("motor_cmd", [])):
-        cmd.motor_cmd[i].mode = motor_data.get("mode", 0)
-        cmd.motor_cmd[i].q = motor_data.get("q", 0.0)
-        cmd.motor_cmd[i].dq = motor_data.get("dq", 0.0)
-        cmd.motor_cmd[i].kp = motor_data.get("kp", 0.0)
-        cmd.motor_cmd[i].kd = motor_data.get("kd", 0.0)
-        cmd.motor_cmd[i].tau = motor_data.get("tau", 0.0)
-
-    return cmd
-
-
-def state_forward_loop(
-    lowstate_sub: ChannelSubscriber,
-    lowstate_sock: zmq.Socket,
-    state_period: float,
-) -> None:
-    """Read observation from DDS and forward to ZMQ clients."""
-    last_state_time = 0.0
-
-    while True:
-        # read from DDS
-        msg = lowstate_sub.Read()
-        if msg is None:
-            continue
-
-        now = time.time()
-        # optional downsampling (if robot dds rate > state_period)
-        if now - last_state_time >= state_period:
-            # Convert to dict and serialize with JSON
-            state_dict = lowstate_to_dict(msg)
-            payload = json.dumps({"topic": kTopicLowState, "data": state_dict}).encode("utf-8")
-            # if no subscribers / tx buffer full, just drop
-            with contextlib.suppress(zmq.Again):
-                lowstate_sock.send(payload, zmq.NOBLOCK)
-            last_state_time = now
-
-
-def cmd_forward_loop(
-    lowcmd_sock: zmq.Socket,
-    lowcmd_pub_debug: ChannelPublisher,
-    crc: CRC,
-) -> None:
-    """Receive commands from ZMQ and forward to DDS."""
-    while True:
-        payload = lowcmd_sock.recv()
-        msg_dict = json.loads(payload.decode("utf-8"))
-
-        topic = msg_dict.get("topic", "")
-        cmd_data = msg_dict.get("data", {})
-
-        # Reconstruct LowCmd object from dict
-        cmd = dict_to_lowcmd(cmd_data)
-
-        # recompute crc
-        cmd.crc = crc.Crc(cmd)
-
-        if topic == kTopicLowCommand_Debug:
-            lowcmd_pub_debug.Write(cmd)
-
-
-def main() -> None:
-    """Main entry point for the robot server bridge."""
-    # initialize DDS
-    ChannelFactoryInitialize(0)
-
-    # stop all active publishers on the robot
-    msc = MotionSwitcherClient()
-    msc.SetTimeout(5.0)
-    msc.Init()
-
-    status, result = msc.CheckMode()
-    while result is not None and "name" in result and result["name"]:
-        msc.ReleaseMode()
-        status, result = msc.CheckMode()
-        time.sleep(1.0)
-
-    crc = CRC()
-
-    # initialize DDS publisher
-    lowcmd_pub_debug = ChannelPublisher(kTopicLowCommand_Debug, hg_LowCmd)
-    lowcmd_pub_debug.Init()
-
-    # initialize DDS subscriber
-    lowstate_sub = ChannelSubscriber(kTopicLowState, hg_LowState)
-    lowstate_sub.Init()
-
-    # initialize ZMQ
-    ctx = zmq.Context.instance()
-
-    # receive commands from remote client
-    lowcmd_sock = ctx.socket(zmq.PULL)
-    lowcmd_sock.bind(f"tcp://0.0.0.0:{LOWCMD_PORT}")
-
-    # publish state to remote clients
-    lowstate_sock = ctx.socket(zmq.PUB)
-    lowstate_sock.bind(f"tcp://0.0.0.0:{LOWSTATE_PORT}")
-
-    state_period = 0.002  # ~500 hz
-
-    # start observation forwarding thread
-    t_state = threading.Thread(
-        target=state_forward_loop,
-        args=(lowstate_sub, lowstate_sock, state_period),
-        daemon=True,
-    )
-    t_state.start()
-
-    # start action forwarding thread
-    t_cmd = threading.Thread(
-        target=cmd_forward_loop,
-        args=(lowcmd_sock, lowcmd_pub_debug, crc),
-        daemon=True,
-    )
-    t_cmd.start()
-
-    print("bridge running (lowstate -> zmq, lowcmd -> dds)")
-    # keep main thread alive so daemon threads don't exit
-    try:
-        while True:
-            time.sleep(1.0)
-    except KeyboardInterrupt:
-        print("shutting down bridge...")
-
-
-if __name__ == "__main__":
-    main()

src/lerobot/robots/unitree_g1/unitree_g1.py

@@ -1,267 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import struct
-import threading
-import time
-from dataclasses import dataclass, field
-from functools import cached_property
-from typing import Any
-
-import numpy as np
-from unitree_sdk2py.idl.default import unitree_hg_msg_dds__LowCmd_
-from unitree_sdk2py.idl.unitree_hg.msg.dds_ import (
-    LowCmd_ as hg_LowCmd,
-    LowState_ as hg_LowState,
-)
-from unitree_sdk2py.utils.crc import CRC
-
-from lerobot.robots.unitree_g1.g1_utils import G1_29_JointIndex
-from lerobot.robots.unitree_g1.unitree_sdk2_socket import (
-    ChannelFactoryInitialize,
-    ChannelPublisher,
-    ChannelSubscriber,
-)
-
-from ..robot import Robot
-from .config_unitree_g1 import UnitreeG1Config
-
-logger = logging.getLogger(__name__)
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"
-kTopicLowState = "rt/lowstate"
-
-G1_29_Num_Motors = 35
-G1_23_Num_Motors = 35
-H1_2_Num_Motors = 35
-H1_Num_Motors = 20
-
-
-@dataclass
-class MotorState:
-    q: float | None = None  # position
-    dq: float | None = None  # velocity
-    tau_est: float | None = None  # estimated torque
-    temperature: float | None = None  # motor temperature
-
-
-@dataclass
-class IMUState:
-    quaternion: np.ndarray | None = None  # [w, x, y, z]
-    gyroscope: np.ndarray | None = None  # [x, y, z] angular velocity (rad/s)
-    accelerometer: np.ndarray | None = None  # [x, y, z] linear acceleration (m/s²)
-    rpy: np.ndarray | None = None  # [roll, pitch, yaw] (rad)
-    temperature: float | None = None  # IMU temperature
-
-
-# g1 observation class
-@dataclass
-class G1_29_LowState:  # noqa: N801
-    motor_state: list[MotorState] = field(
-        default_factory=lambda: [MotorState() for _ in range(G1_29_Num_Motors)]
-    )
-    imu_state: IMUState = field(default_factory=IMUState)
-    wireless_remote: Any = None  # Raw wireless remote data
-    mode_machine: int = 0  # Robot mode
-
-
-class DataBuffer:
-    def __init__(self):
-        self.data = None
-        self.lock = threading.Lock()
-
-    def get_data(self):
-        with self.lock:
-            return self.data
-
-    def set_data(self, data):
-        with self.lock:
-            self.data = data
-
-
-class UnitreeG1(Robot):
-    config_class = UnitreeG1Config
-    name = "unitree_g1"
-
-    # unitree remote controller
-    class RemoteController:
-        def __init__(self):
-            self.lx = 0
-            self.ly = 0
-            self.rx = 0
-            self.ry = 0
-            self.button = [0] * 16
-
-        def set(self, data):
-            # wireless_remote
-            keys = struct.unpack("H", data[2:4])[0]
-            for i in range(16):
-                self.button[i] = (keys & (1 << i)) >> i
-            self.lx = struct.unpack("f", data[4:8])[0]
-            self.rx = struct.unpack("f", data[8:12])[0]
-            self.ry = struct.unpack("f", data[12:16])[0]
-            self.ly = struct.unpack("f", data[20:24])[0]
-
-    def __init__(self, config: UnitreeG1Config):
-        super().__init__(config)
-
-        logger.info("Initialize UnitreeG1...")
-
-        self.config = config
-
-        self.control_dt = config.control_dt
-
-        # connect robot
-        self.connect()
-
-        # initialize direct motor control interface
-        self.lowcmd_publisher = ChannelPublisher(kTopicLowCommand_Debug, hg_LowCmd)
-        self.lowcmd_publisher.Init()
-        self.lowstate_subscriber = ChannelSubscriber(kTopicLowState, hg_LowState)
-        self.lowstate_subscriber.Init()
-        self.lowstate_buffer = DataBuffer()
-
-        # initialize subscribe thread to read robot state
-        self.subscribe_thread = threading.Thread(target=self._subscribe_motor_state)
-        self.subscribe_thread.daemon = True
-        self.subscribe_thread.start()
-
-        while not self.is_connected:
-            time.sleep(0.1)
-
-        # initialize hg's lowcmd msg
-        self.crc = CRC()
-        self.msg = unitree_hg_msg_dds__LowCmd_()
-        self.msg.mode_pr = 0
-
-        # Wait for first state message to arrive
-        lowstate = None
-        while lowstate is None:
-            lowstate = self.lowstate_buffer.get_data()
-            if lowstate is None:
-                time.sleep(0.01)
-            logger.warning("[UnitreeG1] Waiting for robot state...")
-        logger.warning("[UnitreeG1] Connected to robot.")
-        self.msg.mode_machine = lowstate.mode_machine
-
-        # initialize all motors with unified kp/kd from config
-        self.kp = np.array(config.kp, dtype=np.float32)
-        self.kd = np.array(config.kd, dtype=np.float32)
-
-        for id in G1_29_JointIndex:
-            self.msg.motor_cmd[id].mode = 1
-            self.msg.motor_cmd[id].kp = self.kp[id.value]
-            self.msg.motor_cmd[id].kd = self.kd[id.value]
-            self.msg.motor_cmd[id].q = lowstate.motor_state[id.value].q
-
-        # Initialize remote controller
-        self.remote_controller = self.RemoteController()
-
-    def _subscribe_motor_state(self):  # polls robot state @ 250Hz
-        while True:
-            start_time = time.time()
-            msg = self.lowstate_subscriber.Read()
-            if msg is not None:
-                lowstate = G1_29_LowState()
-
-                # Capture motor states
-                for id in range(G1_29_Num_Motors):
-                    lowstate.motor_state[id].q = msg.motor_state[id].q
-                    lowstate.motor_state[id].dq = msg.motor_state[id].dq
-                    lowstate.motor_state[id].tau_est = msg.motor_state[id].tau_est
-                    lowstate.motor_state[id].temperature = msg.motor_state[id].temperature
-
-                # Capture IMU state
-                lowstate.imu_state.quaternion = list(msg.imu_state.quaternion)
-                lowstate.imu_state.gyroscope = list(msg.imu_state.gyroscope)
-                lowstate.imu_state.accelerometer = list(msg.imu_state.accelerometer)
-                lowstate.imu_state.rpy = list(msg.imu_state.rpy)
-                lowstate.imu_state.temperature = msg.imu_state.temperature
-
-                # Capture wireless remote data
-                lowstate.wireless_remote = msg.wireless_remote
-
-                # Capture mode_machine
-                lowstate.mode_machine = msg.mode_machine
-
-                self.lowstate_buffer.set_data(lowstate)
-
-            current_time = time.time()
-            all_t_elapsed = current_time - start_time
-            sleep_time = max(0, (self.control_dt - all_t_elapsed))  # maintain constant control dt
-            time.sleep(sleep_time)
-
-    @cached_property
-    def action_features(self) -> dict[str, type]:
-        return {f"{G1_29_JointIndex(motor).name}.pos": float for motor in G1_29_JointIndex}
-
-    def calibrate(self) -> None:  # robot is already calibrated
-        pass
-
-    def configure(self) -> None:
-        pass
-
-    def connect(self, calibrate: bool = True) -> None:  # connect to DDS
-        ChannelFactoryInitialize(0)
-
-    def disconnect(self):
-        pass
-
-    def get_observation(self) -> dict[str, Any]:
-        return self.lowstate_buffer.get_data()
-
-    @property
-    def is_calibrated(self) -> bool:
-        return True
-
-    @property
-    def is_connected(self) -> bool:
-        return self.lowstate_buffer.get_data() is not None
-
-    @property
-    def _motors_ft(self) -> dict[str, type]:
-        return {f"{G1_29_JointIndex(motor).name}.pos": float for motor in G1_29_JointIndex}
-
-    @property
-    def _cameras_ft(self) -> dict[str, tuple]:
-        return {
-            cam: (self.config.cameras[cam].height, self.config.cameras[cam].width, 3) for cam in self.cameras
-        }
-
-    @cached_property
-    def observation_features(self) -> dict[str, type | tuple]:
-        return {**self._motors_ft, **self._cameras_ft}
-
-    def send_action(self, action: dict[str, Any]) -> dict[str, Any]:
-        self.msg.crc = self.crc.Crc(action)
-        self.lowcmd_publisher.Write(action)
-        return action
-
-    def get_gravity_orientation(self, quaternion):  # get gravity orientation from quaternion
-        """Get gravity orientation from quaternion."""
-        qw = quaternion[0]
-        qx = quaternion[1]
-        qy = quaternion[2]
-        qz = quaternion[3]
-
-        gravity_orientation = np.zeros(3)
-        gravity_orientation[0] = 2 * (-qz * qx + qw * qy)
-        gravity_orientation[1] = -2 * (qz * qy + qw * qx)
-        gravity_orientation[2] = 1 - 2 * (qw * qw + qz * qz)
-        return gravity_orientation

src/lerobot/robots/unitree_g1/unitree_sdk2_socket.py

@@ -1,168 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import base64
-import json
-from typing import Any
-
-import zmq
-
-from lerobot.robots.unitree_g1.config_unitree_g1 import UnitreeG1Config
-
-_ctx: zmq.Context | None = None
-_lowcmd_sock: zmq.Socket | None = None
-_lowstate_sock: zmq.Socket | None = None
-
-LOWCMD_PORT = 6000
-LOWSTATE_PORT = 6001
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"
-
-
-class LowStateMsg:
-    """
-    Wrapper class that mimics the Unitree SDK LowState_ message structure.
-
-    Reconstructs the message from deserialized JSON data to maintain
-    compatibility with existing code that expects SDK message objects.
-    """
-
-    class MotorState:
-        """Motor state data for a single joint."""
-
-        def __init__(self, data: dict[str, Any]) -> None:
-            self.q: float = data.get("q", 0.0)
-            self.dq: float = data.get("dq", 0.0)
-            self.tau_est: float = data.get("tau_est", 0.0)
-            self.temperature: float = data.get("temperature", 0.0)
-
-    class IMUState:
-        """IMU sensor data."""
-
-        def __init__(self, data: dict[str, Any]) -> None:
-            self.quaternion: list[float] = data.get("quaternion", [1.0, 0.0, 0.0, 0.0])
-            self.gyroscope: list[float] = data.get("gyroscope", [0.0, 0.0, 0.0])
-            self.accelerometer: list[float] = data.get("accelerometer", [0.0, 0.0, 0.0])
-            self.rpy: list[float] = data.get("rpy", [0.0, 0.0, 0.0])
-            self.temperature: float = data.get("temperature", 0.0)
-
-    def __init__(self, data: dict[str, Any]) -> None:
-        """Initialize from deserialized JSON data."""
-        self.motor_state = [self.MotorState(m) for m in data.get("motor_state", [])]
-        self.imu_state = self.IMUState(data.get("imu_state", {}))
-        # Decode base64-encoded wireless_remote bytes
-        wireless_b64 = data.get("wireless_remote", "")
-        self.wireless_remote: bytes = base64.b64decode(wireless_b64) if wireless_b64 else b""
-        self.mode_machine: int = data.get("mode_machine", 0)
-
-
-def lowcmd_to_dict(topic: str, msg: Any) -> dict[str, Any]:
-    """Convert LowCmd message to a JSON-serializable dictionary."""
-    motor_cmds = []
-    # Iterate over all motor commands in the message
-    for i in range(len(msg.motor_cmd)):
-        motor_cmds.append(
-            {
-                "mode": int(msg.motor_cmd[i].mode),
-                "q": float(msg.motor_cmd[i].q),
-                "dq": float(msg.motor_cmd[i].dq),
-                "kp": float(msg.motor_cmd[i].kp),
-                "kd": float(msg.motor_cmd[i].kd),
-                "tau": float(msg.motor_cmd[i].tau),
-            }
-        )
-
-    return {
-        "topic": topic,
-        "data": {
-            "mode_pr": int(msg.mode_pr),
-            "mode_machine": int(msg.mode_machine),
-            "motor_cmd": motor_cmds,
-        },
-    }
-
-
-def ChannelFactoryInitialize(*args: Any, **kwargs: Any) -> None:  # noqa: N802
-    """
-    Initialize ZMQ sockets for robot communication.
-
-    This function mimics the Unitree SDK's ChannelFactoryInitialize but uses
-    ZMQ sockets to connect to the robot server bridge instead of DDS.
-    """
-    global _ctx, _lowcmd_sock, _lowstate_sock
-
-    # read socket config
-    config = UnitreeG1Config()
-    robot_ip = config.robot_ip
-
-    ctx = zmq.Context.instance()
-    _ctx = ctx
-
-    # lowcmd: send robot commands
-    lowcmd_sock = ctx.socket(zmq.PUSH)
-    lowcmd_sock.setsockopt(zmq.CONFLATE, 1)  # keep only last message
-    lowcmd_sock.connect(f"tcp://{robot_ip}:{LOWCMD_PORT}")
-    _lowcmd_sock = lowcmd_sock
-
-    # lowstate: receive robot observations
-    lowstate_sock = ctx.socket(zmq.SUB)
-    lowstate_sock.setsockopt(zmq.CONFLATE, 1)  # keep only last message
-    lowstate_sock.connect(f"tcp://{robot_ip}:{LOWSTATE_PORT}")
-    lowstate_sock.setsockopt_string(zmq.SUBSCRIBE, "")
-    _lowstate_sock = lowstate_sock
-
-
-class ChannelPublisher:
-    """ZMQ-based publisher that sends commands to the robot server."""
-
-    def __init__(self, topic: str, msg_type: type) -> None:
-        self.topic = topic
-        self.msg_type = msg_type
-
-    def Init(self) -> None:  # noqa: N802
-        """Initialize the publisher (no-op for ZMQ)."""
-        pass
-
-    def Write(self, msg: Any) -> None:  # noqa: N802
-        """Serialize and send a command message to the robot."""
-        if _lowcmd_sock is None:
-            raise RuntimeError("ChannelFactoryInitialize must be called first")
-
-        payload = json.dumps(lowcmd_to_dict(self.topic, msg)).encode("utf-8")
-        _lowcmd_sock.send(payload)
-
-
-class ChannelSubscriber:
-    """ZMQ-based subscriber that receives state from the robot server."""
-
-    def __init__(self, topic: str, msg_type: type) -> None:
-        self.topic = topic
-        self.msg_type = msg_type
-
-    def Init(self) -> None:  # noqa: N802
-        """Initialize the subscriber (no-op for ZMQ)."""
-        pass
-
-    def Read(self) -> LowStateMsg:  # noqa: N802
-        """Receive and deserialize a state message from the robot."""
-        if _lowstate_sock is None:
-            raise RuntimeError("ChannelFactoryInitialize must be called first")
-
-        payload = _lowstate_sock.recv()
-        msg_dict = json.loads(payload.decode("utf-8"))
-        return LowStateMsg(msg_dict.get("data", {}))

src/lerobot/scripts/lerobot_calibrate.py

@@ -52,7 +52,7 @@ from lerobot.teleoperators import (  # noqa: F401
     so100_leader,
     so101_leader,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.utils import init_logging
 
 
@@ -84,7 +84,7 @@ def calibrate(cfg: CalibrateConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     calibrate()
 
 

src/lerobot/scripts/lerobot_eval.py

@@ -82,7 +82,6 @@ from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
 from lerobot.processor import PolicyAction, PolicyProcessorPipeline
 from lerobot.utils.constants import ACTION, DONE, OBS_STR, REWARD
-from lerobot.utils.import_utils import register_third_party_plugins
 from lerobot.utils.io_utils import write_video
 from lerobot.utils.random_utils import set_seed
 from lerobot.utils.utils import (
@@ -793,7 +792,6 @@ def eval_policy_all(
 
 def main():
     init_logging()
-    register_third_party_plugins()
     eval_main()
 
 

src/lerobot/scripts/lerobot_find_joint_limits.py

@@ -15,23 +15,18 @@
 # limitations under the License.
 
 """
-Script to find joint limits and end-effector bounds via teleoperation.
+Simple script to control a robot from teleoperation.
 
 Example:
 
 ```shell
 lerobot-find-joint-limits \
   --robot.type=so100_follower \
-  --robot.port=/dev/tty.usbmodem58760432981 \
+  --robot.port=/dev/tty.usbmodem58760431541 \
   --robot.id=black \
   --teleop.type=so100_leader \
-  --teleop.port=/dev/tty.usbmodem58760434471 \
-  --teleop.id=blue \
-  --urdf_path=<user>/SO-ARM100-main/Simulation/SO101/so101_new_calib.urdf \
-  --target_frame_name=gripper \
-  --teleop_time_s=30 \
-  --warmup_time_s=5 \
-  --control_loop_fps=30
+  --teleop.port=/dev/tty.usbmodem58760431551 \
+  --teleop.id=blue
 ```
 """
 
@@ -47,7 +42,6 @@ from lerobot.robots import (  # noqa: F401
     koch_follower,
     make_robot_from_config,
     so100_follower,
-    so101_follower,
 )
 from lerobot.teleoperators import (  # noqa: F401
     TeleoperatorConfig,
@@ -55,7 +49,6 @@ from lerobot.teleoperators import (  # noqa: F401
     koch_leader,
     make_teleoperator_from_config,
     so100_leader,
-    so101_leader,
 )
 from lerobot.utils.robot_utils import precise_sleep
 
@@ -64,19 +57,10 @@ from lerobot.utils.robot_utils import precise_sleep
 class FindJointLimitsConfig:
     teleop: TeleoperatorConfig
     robot: RobotConfig
-
-    # Path to URDF file for kinematics
-    # NOTE: It is highly recommended to use the urdf in the SO-ARM100 repo:
-    # https://github.com/TheRobotStudio/SO-ARM100/blob/main/Simulation/SO101/so101_new_calib.urdf
-    urdf_path: str
-    target_frame_name: str = "gripper"
-
-    # Duration of the recording phase in seconds
+    # Limit the maximum frames per second. By default, no limit.
     teleop_time_s: float = 30
-    # Duration of the warmup phase in seconds
-    warmup_time_s: float = 5
-    # Control loop frequency
-    control_loop_fps: int = 30
+    # Display all cameras on screen
+    display_data: bool = False
 
 
 @draccus.wrap()
@@ -84,127 +68,53 @@ def find_joint_and_ee_bounds(cfg: FindJointLimitsConfig):
     teleop = make_teleoperator_from_config(cfg.teleop)
     robot = make_robot_from_config(cfg.robot)
 
-    print(f"Connecting to robot: {cfg.robot.type}...")
     teleop.connect()
     robot.connect()
-    print("Devices connected.")
 
-    # Initialize Kinematics
-    try:
-        kinematics = RobotKinematics(cfg.urdf_path, cfg.target_frame_name)
-    except Exception as e:
-        print(f"Error initializing kinematics: {e}")
-        print("Ensure URDF path and target frame name are correct.")
-        robot.disconnect()
-        teleop.disconnect()
-        return
+    start_episode_t = time.perf_counter()
+    robot_type = getattr(robot.config, "robot_type", "so101")
+    if "so100" in robot_type or "so101" in robot_type:
+        # Note to be compatible with the rest of the codebase,
+        # we are using the new calibration method for so101 and so100
+        robot_type = "so_new_calibration"
+    kinematics = RobotKinematics(cfg.robot.urdf_path, cfg.robot.target_frame_name)
 
-    # Initialize variables
-    max_pos = None
-    min_pos = None
-    max_ee = None
-    min_ee = None
+    # Initialize min/max values
+    observation = robot.get_observation()
+    joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+    ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
 
-    start_t = time.perf_counter()
-    warmup_done = False
+    max_pos = joint_positions.copy()
+    min_pos = joint_positions.copy()
+    max_ee = ee_pos.copy()
+    min_ee = ee_pos.copy()
 
-    print("\n" + "=" * 40)
-    print(f"  WARMUP PHASE ({cfg.warmup_time_s}s)")
-    print("  Move the robot freely to ensure control works.")
-    print("  Data is NOT being recorded yet.")
-    print("=" * 40 + "\n")
+    while True:
+        action = teleop.get_action()
+        robot.send_action(action)
 
-    try:
-        while True:
-            t0 = time.perf_counter()
+        observation = robot.get_observation()
+        joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+        ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
 
-            # 1. Teleoperation Control Loop
-            action = teleop.get_action()
-            robot.send_action(action)
+        # Skip initial warmup period
+        if (time.perf_counter() - start_episode_t) < 5:
+            continue
 
-            # 2. Read Observations
-            observation = robot.get_observation()
-            joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+        # Update min/max values
+        max_ee = np.maximum(max_ee, ee_pos)
+        min_ee = np.minimum(min_ee, ee_pos)
+        max_pos = np.maximum(max_pos, joint_positions)
+        min_pos = np.minimum(min_pos, joint_positions)
 
-            # 3. Calculate Kinematics
-            # Forward kinematics to get (x, y, z) translation
-            ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
+        if time.perf_counter() - start_episode_t > cfg.teleop_time_s:
+            print(f"Max ee position {np.round(max_ee, 4).tolist()}")
+            print(f"Min ee position {np.round(min_ee, 4).tolist()}")
+            print(f"Max joint pos position {np.round(max_pos, 4).tolist()}")
+            print(f"Min joint pos position {np.round(min_pos, 4).tolist()}")
+            break
 
-            current_time = time.perf_counter()
-            elapsed = current_time - start_t
-
-            # 4. Handle Phases
-            if elapsed < cfg.warmup_time_s:
-                # Still in warmup
-                pass
-
-            else:
-                # Phase Transition: Warmup -> Recording
-                if not warmup_done:
-                    print("\n" + "=" * 40)
-                    print("  RECORDING STARTED")
-                    print("  Move robot to ALL joint limits.")
-                    print("  Press Ctrl+C to stop early and save results.")
-                    print("=" * 40 + "\n")
-
-                    # Initialize limits with current position at start of recording
-                    max_pos = joint_positions.copy()
-                    min_pos = joint_positions.copy()
-                    max_ee = ee_pos.copy()
-                    min_ee = ee_pos.copy()
-                    warmup_done = True
-
-                # Update Limits
-                max_ee = np.maximum(max_ee, ee_pos)
-                min_ee = np.minimum(min_ee, ee_pos)
-                max_pos = np.maximum(max_pos, joint_positions)
-                min_pos = np.minimum(min_pos, joint_positions)
-
-                # Time check
-                recording_time = elapsed - cfg.warmup_time_s
-                remaining = cfg.teleop_time_s - recording_time
-
-                # Simple throttle for print statements (every ~1 sec)
-                if int(recording_time * 100) % 100 == 0:
-                    print(f"Time remaining: {remaining:.1f}s", end="\r")
-
-                if recording_time > cfg.teleop_time_s:
-                    print("\nTime limit reached.")
-                    break
-
-            precise_sleep(max(1.0 / cfg.control_loop_fps - (time.perf_counter() - t0), 0.0))
-
-    except KeyboardInterrupt:
-        print("\n\nInterrupted by user. Stopping safely...")
-
-    finally:
-        # Safety: Disconnect devices
-        print("\nDisconnecting devices...")
-        robot.disconnect()
-        teleop.disconnect()
-
-    # Results Output
-    if max_pos is not None:
-        print("\n" + "=" * 40)
-        print("FINAL RESULTS")
-        print("=" * 40)
-
-        # Rounding for readability
-        r_max_ee = np.round(max_ee, 4).tolist()
-        r_min_ee = np.round(min_ee, 4).tolist()
-        r_max_pos = np.round(max_pos, 4).tolist()
-        r_min_pos = np.round(min_pos, 4).tolist()
-
-        print("\n# End Effector Bounds (x, y, z):")
-        print(f"max_ee = {r_max_ee}")
-        print(f"min_ee = {r_min_ee}")
-
-        print("\n# Joint Position Limits (radians):")
-        print(f"max_pos = {r_max_pos}")
-        print(f"min_pos = {r_min_pos}")
-
-    else:
-        print("No data recorded (exited during warmup).")
+        precise_sleep(0.01)
 
 
 def main():

src/lerobot/scripts/lerobot_record.py

@@ -93,7 +93,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -119,7 +118,7 @@ from lerobot.utils.control_utils import (
     sanity_check_dataset_name,
     sanity_check_dataset_robot_compatibility,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import (
     get_safe_torch_device,
@@ -513,7 +512,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     record()
 
 

src/lerobot/scripts/lerobot_replay.py

@@ -54,7 +54,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -62,7 +61,7 @@ from lerobot.robots import (  # noqa: F401
     so101_follower,
 )
 from lerobot.utils.constants import ACTION
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import (
     init_logging,
@@ -128,7 +127,7 @@ def replay(cfg: ReplayConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     replay()
 
 

src/lerobot/scripts/lerobot_teleoperate.py

@@ -71,7 +71,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -84,13 +83,12 @@ from lerobot.teleoperators import (  # noqa: F401
     bi_so100_leader,
     gamepad,
     homunculus,
-    keyboard,
     koch_leader,
     make_teleoperator_from_config,
     so100_leader,
     so101_leader,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import init_logging, move_cursor_up
 from lerobot.utils.visualization_utils import init_rerun, log_rerun_data
@@ -219,7 +217,7 @@ def teleoperate(cfg: TeleoperateConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     teleoperate()
 
 

src/lerobot/scripts/lerobot_train.py

@@ -36,7 +36,6 @@ from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
 from lerobot.rl.wandb_utils import WandBLogger
 from lerobot.scripts.lerobot_eval import eval_policy_all
-from lerobot.utils.import_utils import register_third_party_plugins
 from lerobot.utils.logging_utils import AverageMeter, MetricsTracker
 from lerobot.utils.random_utils import set_seed
 from lerobot.utils.train_utils import (
@@ -449,7 +448,6 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
 
 
 def main():
-    register_third_party_plugins()
     train()
 
 

src/lerobot/teleoperators/keyboard/__init__.py

@@ -14,18 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from .configuration_keyboard import (
-    KeyboardEndEffectorTeleopConfig,
-    KeyboardRoverTeleopConfig,
-    KeyboardTeleopConfig,
-)
-from .teleop_keyboard import KeyboardEndEffectorTeleop, KeyboardRoverTeleop, KeyboardTeleop
+from .configuration_keyboard import KeyboardEndEffectorTeleopConfig, KeyboardTeleopConfig
+from .teleop_keyboard import KeyboardEndEffectorTeleop, KeyboardTeleop
 
 __all__ = [
     "KeyboardTeleopConfig",
     "KeyboardTeleop",
     "KeyboardEndEffectorTeleopConfig",
     "KeyboardEndEffectorTeleop",
-    "KeyboardRoverTeleopConfig",
-    "KeyboardRoverTeleop",
 ]

src/lerobot/teleoperators/keyboard/configuration_keyboard.py

@@ -13,7 +13,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""Configuration for keyboard teleoperators."""
 
 from dataclasses import dataclass
 
@@ -31,38 +30,4 @@ class KeyboardTeleopConfig(TeleoperatorConfig):
 @TeleoperatorConfig.register_subclass("keyboard_ee")
 @dataclass
 class KeyboardEndEffectorTeleopConfig(KeyboardTeleopConfig):
-    """Configuration for keyboard end-effector teleoperator.
-
-    Used for controlling robot end-effectors with keyboard inputs.
-
-    Attributes:
-        use_gripper: Whether to include gripper control in actions
-    """
-
     use_gripper: bool = True
-
-
-@TeleoperatorConfig.register_subclass("keyboard_rover")
-@dataclass
-class KeyboardRoverTeleopConfig(TeleoperatorConfig):
-    """Configuration for keyboard rover teleoperator.
-
-    Used for controlling mobile robots like EarthRover Mini Plus with WASD controls.
-
-    Attributes:
-        linear_speed: Default linear velocity magnitude (-1 to 1 range for SDK robots)
-        angular_speed: Default angular velocity magnitude (-1 to 1 range for SDK robots)
-        speed_increment: Amount to increase/decrease speed with +/- keys
-        turn_assist_ratio: Forward motion multiplier when turning with A/D keys (0.0-1.0)
-        angular_speed_ratio: Ratio of angular to linear speed for synchronized adjustments
-        min_linear_speed: Minimum linear speed when decreasing (prevents zero speed)
-        min_angular_speed: Minimum angular speed when decreasing (prevents zero speed)
-    """
-
-    linear_speed: float = 1.0
-    angular_speed: float = 1.0
-    speed_increment: float = 0.1
-    turn_assist_ratio: float = 0.3
-    angular_speed_ratio: float = 0.6
-    min_linear_speed: float = 0.1
-    min_angular_speed: float = 0.05

src/lerobot/teleoperators/keyboard/teleop_keyboard.py

@@ -25,11 +25,7 @@ from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnected
 
 from ..teleoperator import Teleoperator
 from ..utils import TeleopEvents
-from .configuration_keyboard import (
-    KeyboardEndEffectorTeleopConfig,
-    KeyboardRoverTeleopConfig,
-    KeyboardTeleopConfig,
-)
+from .configuration_keyboard import KeyboardEndEffectorTeleopConfig, KeyboardTeleopConfig
 
 PYNPUT_AVAILABLE = True
 try:
@@ -293,158 +289,3 @@ class KeyboardEndEffectorTeleop(KeyboardTeleop):
             TeleopEvents.SUCCESS: success,
             TeleopEvents.RERECORD_EPISODE: rerecord_episode,
         }
-
-
-class KeyboardRoverTeleop(KeyboardTeleop):
-    """
-    Keyboard teleoperator for mobile robots like EarthRover Mini Plus.
-
-    Provides intuitive WASD-style controls for driving a mobile robot:
-    - Linear movement (forward/backward)
-    - Angular movement (turning/rotation)
-    - Speed adjustment
-    - Emergency stop
-
-    Keyboard Controls:
-        Movement:
-            - W: Move forward
-            - S: Move backward
-            - A: Turn left (with forward motion)
-            - D: Turn right (with forward motion)
-            - Q: Rotate left in place
-            - E: Rotate right in place
-            - X: Emergency stop
-
-        Speed Control:
-            - +/=: Increase speed
-            - -: Decrease speed
-
-        System:
-            - ESC: Disconnect teleoperator
-
-    Attributes:
-        config: Teleoperator configuration
-        current_linear_speed: Current linear velocity magnitude
-        current_angular_speed: Current angular velocity magnitude
-
-    Example:
-        ```python
-        from lerobot.teleoperators.keyboard import KeyboardRoverTeleop, KeyboardRoverTeleopConfig
-
-        teleop = KeyboardRoverTeleop(
-            KeyboardRoverTeleopConfig(linear_speed=1.0, angular_speed=1.0, speed_increment=0.1)
-        )
-        teleop.connect()
-
-        while teleop.is_connected:
-            action = teleop.get_action()
-            robot.send_action(action)
-        ```
-    """
-
-    config_class = KeyboardRoverTeleopConfig
-    name = "keyboard_rover"
-
-    def __init__(self, config: KeyboardRoverTeleopConfig):
-        super().__init__(config)
-        # Add rover-specific speed settings
-        self.current_linear_speed = config.linear_speed
-        self.current_angular_speed = config.angular_speed
-
-    @property
-    def action_features(self) -> dict:
-        """Return action format for rover (linear and angular velocities)."""
-        return {
-            "linear.vel": float,
-            "angular.vel": float,
-        }
-
-    @property
-    def is_calibrated(self) -> bool:
-        """Rover teleop doesn't require calibration."""
-        return True
-
-    def _drain_pressed_keys(self):
-        """Update current_pressed state from event queue without clearing held keys"""
-        while not self.event_queue.empty():
-            key_char, is_pressed = self.event_queue.get_nowait()
-            if is_pressed:
-                self.current_pressed[key_char] = True
-            else:
-                # Only remove key if it's being released
-                self.current_pressed.pop(key_char, None)
-
-    def get_action(self) -> dict[str, Any]:
-        """
-        Get the current action based on pressed keys.
-
-        Returns:
-            dict with 'linear.vel' and 'angular.vel' keys
-        """
-        before_read_t = time.perf_counter()
-
-        if not self.is_connected:
-            raise DeviceNotConnectedError(
-                "KeyboardRoverTeleop is not connected. You need to run `connect()` before `get_action()`."
-            )
-
-        self._drain_pressed_keys()
-
-        linear_velocity = 0.0
-        angular_velocity = 0.0
-
-        # Check which keys are currently pressed (not released)
-        active_keys = {key for key, is_pressed in self.current_pressed.items() if is_pressed}
-
-        # Linear movement (W/S) - these take priority
-        if "w" in active_keys:
-            linear_velocity = self.current_linear_speed
-        elif "s" in active_keys:
-            linear_velocity = -self.current_linear_speed
-
-        # Turning (A/D/Q/E)
-        if "d" in active_keys:
-            angular_velocity = -self.current_angular_speed
-            if linear_velocity == 0:  # If not moving forward/back, add slight forward motion
-                linear_velocity = self.current_linear_speed * self.config.turn_assist_ratio
-        elif "a" in active_keys:
-            angular_velocity = self.current_angular_speed
-            if linear_velocity == 0:  # If not moving forward/back, add slight forward motion
-                linear_velocity = self.current_linear_speed * self.config.turn_assist_ratio
-        elif "q" in active_keys:
-            angular_velocity = self.current_angular_speed
-            linear_velocity = 0  # Rotate in place
-        elif "e" in active_keys:
-            angular_velocity = -self.current_angular_speed
-            linear_velocity = 0  # Rotate in place
-
-        # Stop (X) - overrides everything
-        if "x" in active_keys:
-            linear_velocity = 0
-            angular_velocity = 0
-
-        # Speed adjustment
-        if "+" in active_keys or "=" in active_keys:
-            self.current_linear_speed += self.config.speed_increment
-            self.current_angular_speed += self.config.speed_increment * self.config.angular_speed_ratio
-            logging.info(
-                f"Speed increased: linear={self.current_linear_speed:.2f}, angular={self.current_angular_speed:.2f}"
-            )
-        if "-" in active_keys:
-            self.current_linear_speed = max(
-                self.config.min_linear_speed, self.current_linear_speed - self.config.speed_increment
-            )
-            self.current_angular_speed = max(
-                self.config.min_angular_speed,
-                self.current_angular_speed - self.config.speed_increment * self.config.angular_speed_ratio,
-            )
-            logging.info(
-                f"Speed decreased: linear={self.current_linear_speed:.2f}, angular={self.current_angular_speed:.2f}"
-            )
-
-        self.logs["read_pos_dt_s"] = time.perf_counter() - before_read_t
-
-        return {
-            "linear.vel": linear_velocity,
-            "angular.vel": angular_velocity,
-        }

src/lerobot/transport/utils.py

@@ -19,7 +19,7 @@ import io
 import json
 import logging
 import pickle  # nosec B403: Safe usage for internal serialization only
-from multiprocessing.synchronize import Event as MpEvent
+from multiprocessing import Event
 from queue import Queue
 from typing import Any
 
@@ -28,9 +28,6 @@ import torch
 from lerobot.transport import services_pb2
 from lerobot.utils.transition import Transition
 
-# FIX for protobuf: Assign the enum to a variable and ignore the type error once
-TransferState = services_pb2.TransferState  # type: ignore[attr-defined]
-
 CHUNK_SIZE = 2 * 1024 * 1024  # 2 MB
 MAX_MESSAGE_SIZE = 4 * 1024 * 1024  # 4 MB
 
@@ -43,8 +40,8 @@ def bytes_buffer_size(buffer: io.BytesIO) -> int:
 
 
 def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = "", silent: bool = True):
-    bytes_buffer: io.BytesIO = io.BytesIO(buffer)
-    size_in_bytes = bytes_buffer_size(bytes_buffer)
+    buffer = io.BytesIO(buffer)
+    size_in_bytes = bytes_buffer_size(buffer)
 
     sent_bytes = 0
 
@@ -53,15 +50,15 @@ def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = ""
     logging_method(f"{log_prefix} Buffer size {size_in_bytes / 1024 / 1024} MB with")
 
     while sent_bytes < size_in_bytes:
-        transfer_state = TransferState.TRANSFER_MIDDLE
+        transfer_state = services_pb2.TransferState.TRANSFER_MIDDLE
 
         if sent_bytes + CHUNK_SIZE >= size_in_bytes:
-            transfer_state = TransferState.TRANSFER_END
+            transfer_state = services_pb2.TransferState.TRANSFER_END
         elif sent_bytes == 0:
-            transfer_state = TransferState.TRANSFER_BEGIN
+            transfer_state = services_pb2.TransferState.TRANSFER_BEGIN
 
         size_to_read = min(CHUNK_SIZE, size_in_bytes - sent_bytes)
-        chunk = bytes_buffer.read(size_to_read)
+        chunk = buffer.read(size_to_read)
 
         yield message_class(transfer_state=transfer_state, data=chunk)
         sent_bytes += size_to_read
@@ -70,7 +67,7 @@ def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = ""
     logging_method(f"{log_prefix} Published {sent_bytes / 1024 / 1024} MB")
 
 
-def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEvent, log_prefix: str = ""):
+def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: Event, log_prefix: str = ""):
     bytes_buffer = io.BytesIO()
     step = 0
 
@@ -81,17 +78,17 @@ def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEve
             logging.info(f"{log_prefix} Shutting down receiver")
             return
 
-        if item.transfer_state == TransferState.TRANSFER_BEGIN:
+        if item.transfer_state == services_pb2.TransferState.TRANSFER_BEGIN:
             bytes_buffer.seek(0)
             bytes_buffer.truncate(0)
             bytes_buffer.write(item.data)
             logging.debug(f"{log_prefix} Received data at step 0")
             step = 0
-        elif item.transfer_state == TransferState.TRANSFER_MIDDLE:
+        elif item.transfer_state == services_pb2.TransferState.TRANSFER_MIDDLE:
             bytes_buffer.write(item.data)
             step += 1
             logging.debug(f"{log_prefix} Received data at step {step}")
-        elif item.transfer_state == TransferState.TRANSFER_END:
+        elif item.transfer_state == services_pb2.TransferState.TRANSFER_END:
             bytes_buffer.write(item.data)
             logging.debug(f"{log_prefix} Received data at step end size {bytes_buffer_size(bytes_buffer)}")
 
@@ -112,17 +109,17 @@ def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEve
 
 def state_to_bytes(state_dict: dict[str, torch.Tensor]) -> bytes:
     """Convert model state dict to flat array for transmission"""
-    bytes_buffer = io.BytesIO()
+    buffer = io.BytesIO()
 
-    torch.save(state_dict, bytes_buffer)
+    torch.save(state_dict, buffer)
 
-    return bytes_buffer.getvalue()
+    return buffer.getvalue()
 
 
 def bytes_to_state_dict(buffer: bytes) -> dict[str, torch.Tensor]:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    return torch.load(bytes_buffer, weights_only=True)
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    return torch.load(buffer, weights_only=True)
 
 
 def python_object_to_bytes(python_object: Any) -> bytes:
@@ -130,24 +127,24 @@ def python_object_to_bytes(python_object: Any) -> bytes:
 
 
 def bytes_to_python_object(buffer: bytes) -> Any:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    obj = pickle.load(bytes_buffer)  # nosec B301: Safe usage of pickle.load
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    obj = pickle.load(buffer)  # nosec B301: Safe usage of pickle.load
     # Add validation checks here
     return obj
 
 
 def bytes_to_transitions(buffer: bytes) -> list[Transition]:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    transitions = torch.load(bytes_buffer, weights_only=True)
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    transitions = torch.load(buffer, weights_only=True)
     return transitions
 
 
 def transitions_to_bytes(transitions: list[Transition]) -> bytes:
-    bytes_buffer = io.BytesIO()
-    torch.save(transitions, bytes_buffer)
-    return bytes_buffer.getvalue()
+    buffer = io.BytesIO()
+    torch.save(transitions, buffer)
+    return buffer.getvalue()
 
 
 def grpc_channel_options(

src/lerobot/utils/constants.py

@@ -26,12 +26,7 @@ OBS_IMAGES = OBS_IMAGE + "s"
 OBS_LANGUAGE = OBS_STR + ".language"
 OBS_LANGUAGE_TOKENS = OBS_LANGUAGE + ".tokens"
 OBS_LANGUAGE_ATTENTION_MASK = OBS_LANGUAGE + ".attention_mask"
-OBS_LANGUAGE_PROMPT = OBS_STR + ".prompt"
-OBS_LANGUAGE_PROMPT_TOKENS = OBS_LANGUAGE_PROMPT + ".tokens"
-OBS_LANGUAGE_PROMPT_ATTENTION_MASK = OBS_LANGUAGE_PROMPT + ".attention_mask"
-OBS_LANGUAGE_TARGET = OBS_STR + ".target"
-OBS_LANGUAGE_TARGET_TOKENS = OBS_LANGUAGE_TARGET + ".tokens"
-OBS_LANGUAGE_TARGET_ATTENTION_MASK = OBS_LANGUAGE_TARGET + ".attention_mask"
+
 ACTION = "action"
 REWARD = "next.reward"
 TRUNCATED = "next.truncated"

src/lerobot/utils/import_utils.py

@@ -130,14 +130,14 @@ def make_device_from_device_class(config: ChoiceRegistry) -> Any:
     )
 
 
-def register_third_party_plugins() -> None:
+def register_third_party_devices() -> None:
     """
     Discover and import third-party lerobot_* plugins so they can register themselves.
 
     Scans top-level modules on sys.path for packages starting with
-    'lerobot_robot_', 'lerobot_camera_', 'lerobot_teleoperator_' or 'lerobot_policy_' and imports them.
+    'lerobot_robot_', 'lerobot_camera_' or 'lerobot_teleoperator_' and imports them.
     """
-    prefixes = ("lerobot_robot_", "lerobot_camera_", "lerobot_teleoperator_", "lerobot_policy_")
+    prefixes = ("lerobot_robot_", "lerobot_camera_", "lerobot_teleoperator_")
     imported: list[str] = []
     failed: list[str] = []
 

tests/policies/pi0_pi05/test_pi05_original_vs_lerobot.py

@@ -266,7 +266,7 @@ def create_original_observation_with_openpi_preprocessing(batch):
     elif len(tasks) == 1:
         tasks = tasks * batch_size
 
-    # Use pi05 state and input tokenizer logic (same as Pi05PrepareStateAndLanguageTokenizerProcessorStep)
+    # Use pi05 state and input tokenizer logic (same as Pi05PrepareStateTokenizerProcessorStep)
     state = batch["observation.state"]
     state = deepcopy(state)
 

tests/policies/xvla/test_xvla_original_vs_lerobot.py

@@ -17,6 +17,7 @@
 """Test script to verify XVLA policy integration with LeRobot vs the original implementation, only meant to be run locally!"""
 # ruff: noqa: E402
 
+import gc
 import random
 from copy import deepcopy
 from typing import Any
@@ -51,6 +52,18 @@ EXPECTED_ACTIONS_STD = 0.245411
 EXPECTED_ACTIONS_FIRST_5 = torch.tensor([0.2742, 0.4977, 0.0500, 0.7040, -0.2653])
 
 
+def cleanup_memory():
+    """Clean up GPU/MPS memory to prevent OOM errors between tests."""
+    print("\nCleaning up memory...")
+    gc.collect()
+    if torch.cuda.is_available():
+        torch.cuda.empty_cache()
+        torch.cuda.synchronize()
+    if torch.backends.mps.is_available():
+        torch.mps.empty_cache()
+    print("Memory cleanup complete.")
+
+
 def set_seed_all(seed: int):
     """Set random seed for all RNG sources to ensure reproducibility."""
     random.seed(seed)
@@ -136,6 +149,7 @@ def xvla_components():
     policy_obj, preprocessor_obj, postprocessor_obj = instantiate_lerobot_xvla(from_pretrained=True)
     print("✔️ Model loaded successfully")
     yield policy_obj, preprocessor_obj, postprocessor_obj
+    cleanup_memory()
 
 
 @pytest.fixture(scope="module")
@@ -316,3 +330,32 @@ def test_xvla_inference_reproducibility(policy, preprocessor):
     assert torch.allclose(actions_1, actions_2, atol=1e-6), "Inference should be reproducible!"
 
     print("\nInference is reproducible!")
+
+
+if __name__ == "__main__":
+    print("\n" + "=" * 80)
+    print("XVLA LeRobot Validation Test Suite")
+    print("=" * 80)
+
+    try:
+        # Initialize model once for all tests
+        print("\n[Setup] Instantiating LeRobot XVLA policy...")
+        policy, preprocessor, postprocessor = instantiate_lerobot_xvla(from_pretrained=True)
+        print("✔️ Model loaded successfully")
+
+        # Run all tests with the same model instance
+        test_xvla_preprocessor_alignment(policy, preprocessor)
+        test_xvla_action_generation(policy, preprocessor)
+        test_xvla_inference_reproducibility(policy, preprocessor)
+
+        print("\n" + "=" * 80)
+        print("All tests passed!")
+        print("=" * 80)
+
+        cleanup_memory()
+    except Exception as e:
+        print("\n" + "=" * 80)
+        print(f"Test failed with error: {e}")
+        print("=" * 80)
+        cleanup_memory()
+        raise