Merge branch 'fix/storage-ci-runners' into fix/add-xvla-ci-main

Added detailed instructions for implementing a custom optimizer and modifying parameter retrieval for X-VLA finetuning.

Signed-off-by: Jinliang Zheng <54488861+2toinf@users.noreply.github.com>

.github/workflows/fast_tests.yml

@@ -68,8 +68,6 @@ jobs:
           persist-credentials: false
           lfs: true
 
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 
@@ -87,8 +85,14 @@ jobs:
           version: ${{ env.UV_VERSION }}
           python-version: ${{ env.PYTHON_VERSION }}
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Install lerobot with test extras
         run: uv sync --extra "test"
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run pytest
         run: uv run pytest tests -vv --maxfail=10

.github/workflows/full_tests.yml

@@ -66,8 +66,6 @@ jobs:
           lfs: true
           persist-credentials: false
 
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 
@@ -87,12 +85,21 @@ jobs:
       - name: Install lerobot with all extras
         run: uv sync --all-extras --no-extra groot # TODO(Steven): Make flash-attn optional
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run pytest (all extras)
         run: uv run pytest tests -vv --maxfail=10
 
+      - name: Check disk usage
+        run: df -h
+
       - name: Run end-to-end tests
         run: uv run make test-end-to-end
 
+      - name: Check disk usage
+        run: df -h
+
   # This job builds a GPU enabled image for testing
   # It runs everytime a PR is approved or a push to main
   # TODO(Steven): For now we skip this job for community PRs

.github/workflows/unbound_deps_tests.yml

@@ -52,9 +52,6 @@ jobs:
         with:
           lfs: true
           persist-credentials: false
-
-      # NOTE(Steven): Mount to `/mnt` to avoid the limited storage on `/home`. Consider cleaning default SDKs or using self-hosted runners for more space.
-      # (As of 2024-06-10, the runner's `/home` has only 6.2 GB free—8% of its 72 GB total.)
       - name: Setup /mnt storage
         run: sudo chown -R $USER:$USER /mnt
 

docs/source/_toctree.yml

@@ -9,8 +9,6 @@
     title: Imitation Learning for Robots
   - local: cameras
     title: Cameras
-  - local: bring_your_own_policies
-    title: Bring Your Own Policies
   - local: integrate_hardware
     title: Bring Your Own Hardware
   - local: hilserl
@@ -83,19 +81,11 @@
     title: Hope Jr
   - local: reachy2
     title: Reachy 2
-  - local: unitree_g1
-    title: Unitree G1
-  - local: earthrover_mini_plus
-    title: Earth Rover Mini
   title: "Robots"
 - sections:
   - local: phone_teleop
     title: Phone
   title: "Teleoperators"
-- sections:
-  - local: torch_accelerators
-    title: PyTorch accelerators
-  title: "Supported Hardware"
 - sections:
   - local: notebooks
     title: Notebooks

docs/source/async.mdx

@@ -278,7 +278,7 @@ We found the default values of `actions_per_chunk` and `chunk_size_threshold` to
 2. **Adjust your `fps` based on inference latency.** While the server generates a new action chunk, the client is not idle and is stepping through its current action queue. If the two processes happen at fundamentally different speeds, the client might end up with an empty queue. As such, you should reduce your fps if you consistently run out of actions in queue.
 3. **Adjust `chunk_size_threshold`**.
    - Values closer to `0.0` result in almost sequential behavior. Values closer to `1.0` → send observation every step (more bandwidth, relies on good world-model).
-   - We found values around 0.5-0.6 to work well. If you want to tweak this, spin up a `RobotClient` setting the `--debug_visualize_queue_size` to `True`. This will plot the action queue size evolution at runtime, and you can use it to find the value of `chunk_size_threshold` that works best for your setup.
+   - We found values around 0.5-0.6 to work well. If you want to tweak this, spin up a `RobotClient` setting the `--debug-visualize-queue-size` to `True`. This will plot the action queue size evolution at runtime, and you can use it to find the value of `chunk_size_threshold` that works best for your setup.
 
 <p align="center">
   <img
@@ -289,7 +289,7 @@ We found the default values of `actions_per_chunk` and `chunk_size_threshold` to
 <p align="center">
   <i>
     The action queue size is plotted at runtime when the
-    `--debug_visualize_queue_size` flag is passed, for various levels of
+    `--debug-visualize-queue-size` flag is passed, for various levels of
     `chunk_size_threshold` (`g` in the SmolVLA paper).
   </i>
 </p>

docs/source/bring_your_own_policies.mdx

@@ -1,175 +0,0 @@
-# Bring Your Own Policies
-
-This tutorial explains how to integrate your own custom policy implementations into the LeRobot ecosystem, allowing you to leverage all LeRobot tools for training, evaluation, and deployment while using your own algorithms.
-
-## Step 1: Create a Policy Package
-
-Your custom policy should be organized as an installable Python package following LeRobot's plugin conventions.
-
-### Package Structure
-
-Create a package with the prefix `lerobot_policy_` (IMPORTANT!) followed by your policy name:
-
-```bash
-lerobot_policy_my_custom_policy/
-├── pyproject.toml
-└── src/
-    └── lerobot_policy_my_custom_policy/
-        ├── __init__.py
-        ├── configuration_my_custom_policy.py
-        ├── modeling_my_custom_policy.py
-        └── processor_my_custom_policy.py
-```
-
-### Package Configuration
-
-Set up your `pyproject.toml`:
-
-```toml
-[project]
-name = "lerobot_policy_my_custom_policy"
-version = "0.1.0"
-dependencies = [
-    # your policy-specific dependencies
-]
-requires-python = ">= 3.11"
-
-[build-system]
-build-backend = # your-build-backend
-requires = # your-build-system
-```
-
-## Step 2: Define the Policy Configuration
-
-Create a configuration class that inherits from `PreTrainedConfig` and registers your policy type:
-
-```python
-# configuration_my_custom_policy.py
-from dataclasses import dataclass, field
-from lerobot.configs.policies import PreTrainedConfig
-from lerobot.configs.types import NormalizationMode
-
-@PreTrainedConfig.register_subclass("my_custom_policy")
-@dataclass
-class MyCustomPolicyConfig(PreTrainedConfig):
-    """Configuration class for MyCustomPolicy.
-
-    Args:
-        n_obs_steps: Number of observation steps to use as input
-        horizon: Action prediction horizon
-        n_action_steps: Number of action steps to execute
-        hidden_dim: Hidden dimension for the policy network
-        # Add your policy-specific parameters here
-    """
-    # ...PreTrainedConfig fields...
-    pass
-
-    def __post_init__(self):
-        super().__post_init__()
-        # Add any validation logic here
-
-    def validate_features(self) -> None:
-        """Validate input/output feature compatibility."""
-        # Implement validation logic for your policy's requirements
-        pass
-```
-
-## Step 3: Implement the Policy Class
-
-Create your policy implementation by inheriting from LeRobot's base `PreTrainedPolicy` class:
-
-```python
-# modeling_my_custom_policy.py
-import torch
-import torch.nn as nn
-from typing import Dict, Any
-
-from lerobot.policies.pretrained import PreTrainedPolicy
-from .configuration_my_custom_policy import MyCustomPolicyConfig
-
-class MyCustomPolicy(PreTrainedPolicy):
-    config_class = MyCustomPolicyConfig
-    name = "my_custom_policy"
-
-    def __init__(self, config: MyCustomPolicyConfig, dataset_stats: Dict[str, Any] = None):
-        super().__init__(config, dataset_stats)
-        ...
-```
-
-## Step 4: Add Data Processors
-
-Create processor functions:
-
-```python
-# processor_my_custom_policy.py
-from typing import Dict, Any
-import torch
-
-
-def make_my_custom_policy_pre_post_processors(
-    config,
-) -> tuple[
-    PolicyProcessorPipeline[dict[str, Any], dict[str, Any]],
-    PolicyProcessorPipeline[PolicyAction, PolicyAction],
-]:
-    """Create preprocessing and postprocessing functions for your policy."""
-    pass  # Define your preprocessing and postprocessing logic here
-
-```
-
-## Step 5: Package Initialization
-
-Expose your classes in the package's `__init__.py`:
-
-```python
-# __init__.py
-"""Custom policy package for LeRobot."""
-
-try:
-    import lerobot  # noqa: F401
-except ImportError:
-    raise ImportError(
-        "lerobot is not installed. Please install lerobot to use this policy package."
-    )
-
-from .configuration_my_custom_policy import MyCustomPolicyConfig
-from .modeling_my_custom_policy import MyCustomPolicy
-from .processor_my_custom_policy import make_my_custom_policy_pre_post_processors
-
-__all__ = [
-    "MyCustomPolicyConfig",
-    "MyCustomPolicy",
-    "make_my_custom_policy_pre_post_processors",
-]
-```
-
-## Step 6: Installation and Usage
-
-### Install Your Policy Package
-
-```bash
-cd lerobot_policy_my_custom_policy
-pip install -e .
-
-# Or install from PyPI if published
-pip install lerobot_policy_my_custom_policy
-```
-
-### Use Your Policy
-
-Once installed, your policy automatically integrates with LeRobot's training and evaluation tools:
-
-```bash
-lerobot-train \
-    --policy.type my_custom_policy \
-    --env.type pusht \
-    --steps 200000
-```
-
-## Examples and Community Contributions
-
-Check out these example policy implementations:
-
-- [DiTFlow Policy](https://github.com/danielsanjosepro/lerobot_policy_ditflow) - Diffusion Transformer policy with flow-matching objective. Try it out in this example: [DiTFlow Example](https://github.com/danielsanjosepro/test_lerobot_policy_ditflow)
-
-Share your policy implementations with the community! 🤗

docs/source/earthrover_mini_plus.mdx

@@ -1,206 +0,0 @@
-# EarthRover Mini Plus
-
-The EarthRover Mini Plus is a fully open source mobile robot that connects through the cloud using the Frodobots SDK. This lets you control the robot and record datasets for training AI models.
-
-## What You Need
-
-### Hardware
-
-- EarthRover Mini robot
-- Computer with Python 3.10 or newer
-- Internet connection
-
-### Setting Up the Frodobots SDK
-
-The robot needs the [Frodobots SDK](https://github.com/Frodobots/earth-rovers-sdk) running on your computer. Here's how:
-
-1. Download and install the SDK:
-
-```bash
-git clone https://github.com/Frodobots/earth-rovers-sdk.git
-cd earth-rovers-sdk
-pip install -r requirements.txt
-```
-
-2. Start the SDK:
-
-```bash
-hypercorn main:app --reload
-```
-
-3. Open your web browser and go to `http://localhost:8000`, then click "Join"
-
-The SDK gives you:
-
-- Live video from front and rear cameras
-
-> [!IMPORTANT]
-> The SDK must be running before you can use the robot.
-
-## Install LeRobot
-
-Follow our [Installation Guide](./installation) to install LeRobot.
-
-In addition to the base installation, install the EarthRover Mini dependencies:
-
-```bash
-pip install -e .
-```
-
-## How It Works
-
-The robot uses the internet to communicate:
-
-- **Movement commands**: Sent through the SDK
-- **Camera video**: Received from the SDK
-- **Robot info**: Battery, location, speed from the SDK
-
-You don't need to plug anything in - it all works through the SDK.
-
-## Calibration
-
-No calibration needed! The robot is ready to use as soon as the SDK is running.
-
-## Controlling the Robot
-
-You control the robot using your keyboard - just like playing a video game with WASD keys.
-
-### Keyboard Controls
-
-| Key | Action                           |
-| --- | -------------------------------- |
-| W   | Move forward                     |
-| S   | Move backward                    |
-| A   | Turn left (with forward motion)  |
-| D   | Turn right (with forward motion) |
-| Q   | Rotate left in place             |
-| E   | Rotate right in place            |
-| X   | Stop all movement                |
-| +/= | Increase speed                   |
-| -   | Decrease speed                   |
-| ESC | Disconnect                       |
-
-### Speed Settings
-
-You can adjust how fast the robot moves:
-
-- **Forward/backward speed**: Default is full speed (1.0)
-- **Turning speed**: Default is full speed (1.0)
-- **Speed changes**: Use +/- keys to adjust by 0.1 each time
-
-### Try It Out
-
-Test driving the robot before recording data:
-
-```python
-from lerobot.robots.earthrover_mini_plus import EarthRoverMiniPlus, EarthRoverMiniPlusConfig
-from lerobot.teleoperators.keyboard import KeyboardRoverTeleop, KeyboardRoverTeleopConfig
-
-# Initialize robot
-robot_config = EarthRoverMiniPlusConfig()
-robot = EarthRoverMiniPlus(robot_config)
-
-# Initialize teleoperator
-teleop_config = KeyboardRoverTeleopConfig(
-    linear_speed=1.0,
-    angular_speed=1.0,
-    speed_increment=0.1
-)
-teleop = KeyboardRoverTeleop(teleop_config)
-
-# Connect
-robot.connect()
-teleop.connect()
-
-# Teleoperate (use keyboard controls)
-try:
-    while True:
-        action = teleop.get_action()
-        robot.send_action(action)
-except KeyboardInterrupt:
-    pass
-finally:
-    robot.disconnect()
-    teleop.disconnect()
-```
-
-> [!TIP]
-> If you're using a Mac, you might need to give Terminal permission to access your keyboard for teleoperation. Go to System Preferences > Security & Privacy > Input Monitoring and check the box for Terminal.
-
-## Recording Data
-
-Once you can drive the robot well, you can start recording data to train AI models. The system records:
-
-- **What you do**: How you move the robot (forward, backward, turning)
-- **What the robot sees**:
-  - Videos from both cameras
-  - Robot speed and direction
-  - Battery level and location
-  - GPS position and signal
-  - Other sensor data
-- **When it happened**: Timestamps for everything
-
-### Setting Up Hugging Face
-
-We use Hugging Face to store your data online. First, log in with your token from [Hugging Face settings](https://huggingface.co/settings/tokens):
-
-```bash
-huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
-```
-
-Store your Hugging Face username:
-
-```bash
-HF_USER=$(huggingface-cli whoami | head -n 1)
-echo $HF_USER
-```
-
-### Start Recording
-
-Use the standard recording command:
-
-```bash
-python src/lerobot/scripts/lerobot_record.py \
-    --robot.type=earthrover_mini_plus \
-    --teleop.type=keyboard_rover \
-    --dataset.repo_id=your_username/dataset_name \
-    --dataset.num_episodes=2 \
-    --dataset.fps=10 \
-    --dataset.single_task="Navigate around obstacles" \
-    --display_data=true
-```
-
-Replace `your_username/dataset_name` with your Hugging Face username and a name for your dataset.
-
-### What Gets Saved
-
-Your dataset includes:
-
-**Your Actions (2 things)**:
-
-- How much you moved forward/backward
-- How much you turned left/right
-
-**Robot Observations (12 things)**:
-
-- Front camera video
-- Rear camera video
-- Current speed
-- Battery level
-- Which way the robot is facing
-- GPS location (latitude, longitude, signal strength)
-- Network signal strength
-- Vibration level
-- Lamp status (on/off)
-
-### Where Your Data Goes
-
-On your computer: `~/.cache/huggingface/lerobot/{repo-id}`
-
-After recording, your data automatically uploads to your Hugging Face page:
-
-```bash
-echo https://huggingface.co/datasets/${HF_USER}/earthrover-navigation
-```
-
-Your dataset will be tagged with `LeRobot` for community discovery.

docs/source/il_robots.mdx

@@ -428,7 +428,7 @@ Your robot should replicate movements similar to those you recorded. For example
 
 ## Train a policy
 
-To train a policy to control your robot, use the [`lerobot-train`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/lerobot_train.py) script. A few arguments are required. Here is an example command:
+To train a policy to control your robot, use the [`lerobot-train`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/train.py) script. A few arguments are required. Here is an example command:
 
 ```bash
 lerobot-train \
@@ -485,7 +485,7 @@ huggingface-cli upload ${HF_USER}/act_so101_test${CKPT} \
 
 ## Run inference and evaluate your policy
 
-You can use the `record` script from [`lerobot-record`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/scripts/lerobot_record.py) with a policy checkpoint as input, to run inference and evaluate your policy. For instance, run this command or API example to run inference and record 10 evaluation episodes:
+You can use the `record` script from [`lerobot/record.py`](https://github.com/huggingface/lerobot/blob/main/src/lerobot/record.py) with a policy checkpoint as input, to run inference and evaluate your policy. For instance, run this command or API example to run inference and record 10 evaluation episodes:
 
 <hfoptions id="eval">
 <hfoption id="Command">

docs/source/installation.mdx

@@ -90,7 +90,7 @@ If you encounter build errors, you may need to install additional dependencies:
 To install these for linux run:
 
 ```bash
-sudo apt-get install cmake build-essential python3-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev
+sudo apt-get install cmake build-essential python-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev pkg-config
 ```
 
 For other systems, see: [Compiling PyAV](https://pyav.org/docs/develop/overview/installation.html#bring-your-own-ffmpeg)

docs/source/so101.mdx

@@ -30,6 +30,131 @@ The follower arm uses 6x STS3215 motors with 1/345 gearing. The leader, however,
 | Wrist Roll          |   5   |  1 / 147   |
 | Gripper             |   6   |  1 / 147   |
 
+### Clean Parts
+
+Remove all support material from the 3D-printed parts. The easiest way to do this is using a small screwdriver to get underneath the support material.
+
+It is advisable to install one 3-pin cable in the motor after placing them before continuing assembly.
+
+### Joint 1
+
+- Place the first motor into the base.
+- Fasten the motor with 4 M2x6mm screws (smallest screws). Two from the top and two from the bottom.
+- Slide over the first motor holder and fasten it using two M2x6mm screws (one on each side).
+- Install both motor horns, securing the top horn with a M3x6mm screw.
+- Attach the shoulder part.
+- Tighten the shoulder part with 4 M3x6mm screws on top and 4 M3x6mm screws on the bottom
+- Add the shoulder motor holder.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint1_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 2
+
+- Slide the second motor in from the top.
+- Fasten the second motor with 4 M2x6mm screws.
+- Attach both motor horns to motor 2, again use the M3x6mm horn screw.
+- Attach the upper arm with 4 M3x6mm screws on each side.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint2_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 3
+
+- Insert motor 3 and fasten using 4 M2x6mm screws
+- Attach both motor horns to motor 3 and secure one again with a M3x6mm horn screw.
+- Connect the forearm to motor 3 using 4 M3x6mm screws on each side.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint3_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 4
+
+- Slide over motor holder 4.
+- Slide in motor 4.
+- Fasten motor 4 with 4 M2x6mm screws and attach its motor horns, use a M3x6mm horn screw.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint4_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Joint 5
+
+- Insert motor 5 into the wrist holder and secure it with 2 M2x6mm front screws.
+- Install only one motor horn on the wrist motor and secure it with a M3x6mm horn screw.
+- Secure the wrist to motor 4 using 4 M3x6mm screws on both sides.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint5_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+### Gripper / Handle
+
+<hfoptions id="assembly">
+<hfoption id="Follower">
+
+- Attach the gripper to motor 5, attach it to the motor horn on the wrist using 4 M3x6mm screws.
+- Insert the gripper motor and secure it with 2 M2x6mm screws on each side.
+- Attach the motor horns and again use a M3x6mm horn screw.
+- Install the gripper claw and secure it with 4 M3x6mm screws on both sides.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Gripper_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+</hfoption>
+<hfoption id="Leader">
+
+- Mount the leader holder onto the wrist and secure it with 4 M3x6mm screws.
+- Attach the handle to motor 5 using 1 M2x6mm screw.
+- Insert the gripper motor, secure it with 2 M2x6mm screws on each side, attach a motor horn using a M3x6mm horn screw.
+- Attach the follower trigger with 4 M3x6mm screws.
+
+<div class="video-container">
+  <video controls width="600">
+    <source
+      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Leader_v2.mp4"
+      type="video/mp4"
+    />
+  </video>
+</div>
+
+</hfoption>
+</hfoptions>
+
 ## Configure the motors
 
 ### 1. Find the USB ports associated with each arm
@@ -215,131 +340,6 @@ leader.setup_motors()
 </hfoption>
 </hfoptions>
 
-### Clean Parts
-
-Remove all support material from the 3D-printed parts. The easiest way to do this is using a small screwdriver to get underneath the support material.
-
-It is advisable to install one 3-pin cable in the motor after placing them before continuing assembly.
-
-### Joint 1
-
-- Place the first motor into the base.
-- Fasten the motor with 4 M2x6mm screws (smallest screws). Two from the top and two from the bottom.
-- Slide over the first motor holder and fasten it using two M2x6mm screws (one on each side).
-- Install both motor horns, securing the top horn with a M3x6mm screw.
-- Attach the shoulder part.
-- Tighten the shoulder part with 4 M3x6mm screws on top and 4 M3x6mm screws on the bottom
-- Add the shoulder motor holder.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint1_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 2
-
-- Slide the second motor in from the top.
-- Fasten the second motor with 4 M2x6mm screws.
-- Attach both motor horns to motor 2, again use the M3x6mm horn screw.
-- Attach the upper arm with 4 M3x6mm screws on each side.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint2_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 3
-
-- Insert motor 3 and fasten using 4 M2x6mm screws
-- Attach both motor horns to motor 3 and secure one again with a M3x6mm horn screw.
-- Connect the forearm to motor 3 using 4 M3x6mm screws on each side.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint3_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 4
-
-- Slide over motor holder 4.
-- Slide in motor 4.
-- Fasten motor 4 with 4 M2x6mm screws and attach its motor horns, use a M3x6mm horn screw.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint4_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Joint 5
-
-- Insert motor 5 into the wrist holder and secure it with 2 M2x6mm front screws.
-- Install only one motor horn on the wrist motor and secure it with a M3x6mm horn screw.
-- Secure the wrist to motor 4 using 4 M3x6mm screws on both sides.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Joint5_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-### Gripper / Handle
-
-<hfoptions id="assembly">
-<hfoption id="Follower">
-
-- Attach the gripper to motor 5, attach it to the motor horn on the wrist using 4 M3x6mm screws.
-- Insert the gripper motor and secure it with 2 M2x6mm screws on each side.
-- Attach the motor horns and again use a M3x6mm horn screw.
-- Install the gripper claw and secure it with 4 M3x6mm screws on both sides.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Gripper_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-</hfoption>
-<hfoption id="Leader">
-
-- Mount the leader holder onto the wrist and secure it with 4 M3x6mm screws.
-- Attach the handle to motor 5 using 1 M2x6mm screw.
-- Insert the gripper motor, secure it with 2 M2x6mm screws on each side, attach a motor horn using a M3x6mm horn screw.
-- Attach the follower trigger with 4 M3x6mm screws.
-
-<div class="video-container">
-  <video controls width="600">
-    <source
-      src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/Leader_v2.mp4"
-      type="video/mp4"
-    />
-  </video>
-</div>
-
-</hfoption>
-</hfoptions>
-
 ## Calibrate
 
 Next, you'll need to calibrate your robot to ensure that the leader and follower arms have the same position values when they are in the same physical position.

docs/source/torch_accelerators.mdx

@@ -1,42 +0,0 @@
-# PyTorch accelerators
-
-LeRobot supports multiple hardware acceleration options for both training and inference.
-
-These options include:
-
-- **CPU**: CPU executes all computations, no dedicated accelerator is used
-- **CUDA**: acceleration with NVIDIA & AMD GPUs
-- **MPS**: acceleration with Apple Silicon GPUs
-- **XPU**: acceleration with Intel integrated and discrete GPUs
-
-## Getting Started
-
-To use particular accelerator, a suitable version of PyTorch should be installed.
-
-For CPU, CUDA, and MPS backends follow instructions provided on [PyTorch installation page](https://pytorch.org/get-started/locally).
-For XPU backend, follow instructions from [PyTorch documentation](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
-
-### Verifying the installation
-
-After installation, accelerator availability can be verified by running
-
-```python
-import torch
-print(torch.<backend_name>.is_available())  # <backend_name> is cuda, mps, or xpu
-```
-
-## How to run training or evaluation
-
-To select the desired accelerator, use the `--policy.device` flag when running `lerobot-train` or `lerobot-eval`. For example, to use MPS on Apple Silicon, run:
-
-```bash
-lerobot-train
-    --policy.device=mps ...
-```
-
-```bash
-lerobot-eval \
-    --policy.device=mps ...
-```
-
-However, in most cases, presence of an accelerator is detected automatically and `policy.device` parameter can be omitted from CLI commands.

docs/source/unitree_g1.mdx

@@ -1,203 +0,0 @@
-# Unitree G1 Robot Setup and Control
-
-This guide covers the complete setup process for the Unitree G1 humanoid, from initial connection to running gr00t_wbc locomotion.
-
-## About the Unitree G1
-
-We offer support for both 29 and 23 DOF G1. In this first PR we introduce:
-
-- **`unitree g1` robot class, handling low level communication with the humanoid**
-- **ZMQ socket bridge** for remote communication over WiFi, allowing one to deploy policies remotely instead of over ethernet or directly on the Orin
-- **GR00T locomotion policy** for bipedal walking and balance
-
----
-
-## Part 1: Connect to Robot over Ethernet
-
-### Step 1: Configure Your Computer's Ethernet Interface
-
-Set a static IP on the same subnet as the robot:
-
-```bash
-# Replace 'enp131s0' with your ethernet interface name (check with `ip a`)
-sudo ip addr flush dev enp131s0
-sudo ip addr add 192.168.123.200/24 dev enp131s0
-sudo ip link set enp131s0 up
-```
-
-**Note**: The robot's Ethernet IP is fixed at `192.168.123.164`. Your computer must use `192.168.123.x` where x ≠ 164.
-
-### Step 2: SSH into the Robot
-
-```bash
-ssh unitree@192.168.123.164
-# Password: 123
-```
-
-You should now be connected to the robot's onboard computer.
-
----
-
-## Part 2: Enable WiFi on the Robot
-
-Once connected via Ethernet, follow these steps to enable WiFi:
-
-### Step 1: Enable WiFi Hardware
-
-```bash
-# Unblock WiFi radio
-sudo rfkill unblock wifi
-sudo rfkill unblock all
-
-# Bring up WiFi interface
-sudo ip link set wlan0 up
-
-# Enable NetworkManager control
-sudo nmcli radio wifi on
-sudo nmcli device set wlan0 managed yes
-sudo systemctl restart NetworkManager
-```
-
-### Step 2: Enable Internet Forwarding
-
-**On your laptop:**
-
-```bash
-# Enable IP forwarding
-sudo sysctl -w net.ipv4.ip_forward=1
-
-# Set up NAT (replace wlp132s0f0 with your WiFi interface)
-sudo iptables -t nat -A POSTROUTING -o wlp132s0f0 -s 192.168.123.0/24 -j MASQUERADE
-sudo iptables -A FORWARD -i wlp132s0f0 -o enp131s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-sudo iptables -A FORWARD -i enp131s0 -o wlp132s0f0 -j ACCEPT
-```
-
-**On the robot:**
-
-```bash
-# Add laptop as default gateway
-sudo ip route del default 2>/dev/null || true
-sudo ip route add default via 192.168.123.200 dev eth0
-echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
-
-# Test connection
-ping -c 3 8.8.8.8
-```
-
-### Step 3: Connect to WiFi Network
-
-```bash
-# List available networks
-nmcli device wifi list
-
-# Connect to your WiFi (example)
-sudo nmcli connection add type wifi ifname wlan0 con-name "YourNetwork" ssid "YourNetwork"
-sudo nmcli connection modify "YourNetwork" wifi-sec.key-mgmt wpa-psk
-sudo nmcli connection modify "YourNetwork" wifi-sec.psk "YourPassword"
-sudo nmcli connection modify "YourNetwork" connection.autoconnect yes
-sudo nmcli connection up "YourNetwork"
-
-# Check WiFi IP address
-ip a show wlan0
-```
-
-### Step 4: SSH Over WiFi
-
-Once connected to WiFi, note the robot's IP address and disconnect the Ethernet cable. You can now SSH over WiFi:
-
-```bash
-ssh unitree@<YOUR_ROBOT_IP>
-# Password: 123
-```
-
-Replace `<YOUR_ROBOT_IP>` with your robot's actual WiFi IP address (e.g., `172.18.129.215`).
-
----
-
-## Part 3: Robot Server Setup
-
-### Step 1: Install LeRobot on the Orin
-
-SSH into the robot and install LeRobot:
-
-```bash
-ssh unitree@<YOUR_ROBOT_IP>
-
-conda create -y -n lerobot python=3.10
-conda activate lerobot
-git clone https://github.com/huggingface/lerobot.git
-cd lerobot
-pip install -e '.[unitree_g1]'
-git clone https://github.com/unitreerobotics/unitree_sdk2_python.git
-cd unitree_sdk2_python  && pip install -e .
-```
-
-**Note**: The Unitree SDK requires CycloneDDS v0.10.2 to be installed. See the [Unitree SDK documentation](https://github.com/unitreerobotics/unitree_sdk2_python) for details.
-
-### Step 2: Run the Robot Server
-
-On the robot:
-
-```bash
-python src/lerobot/robots/unitree_g1/run_g1_server.py
-```
-
-**Important**: Keep this terminal running. The server must be active for remote control.
-
----
-
-## Part 4: Running GR00T Locomotion
-
-With the robot server running, you can now control the robot from your laptop.
-
-### Step 1: Install LeRobot on your machine
-
-```bash
-conda create -y -n lerobot python=3.10
-conda activate lerobot
-git clone https://github.com/huggingface/lerobot.git
-cd lerobot
-pip install -e '.[unitree_g1]'
-git clone https://github.com/unitreerobotics/unitree_sdk2_python.git
-cd unitree_sdk2_python  && pip install -e .
-```
-
-### Step 2: Update Robot IP in Config
-
-Edit the config file to match your robot's WiFi IP:
-
-```python
-# In src/lerobot/robots/unitree_g1/config_unitree_g1.py
-robot_ip: str = "<YOUR_ROBOT_IP>"  # Replace with your robot's WiFi IP.
-```
-
-**Note**: When running directly on the G1 (not remotely), set `robot_ip: str = "127.0.0.1"` instead.
-
-### Step 3: Run the Locomotion Policy
-
-```bash
-# Run GR00T locomotion controller
-python examples/unitree_g1/gr00t_locomotion.py --repo-id "nepyope/GR00T-WholeBodyControl_g1"
-```
-
-### Step 4: Control with Remote
-
-- **Left stick**: Forward/backward and left/right movement
-- **Right stick**: Rotation
-- **R1 button**: Raise waist height
-- **R2 button**: Lower waist height
-
-Press `Ctrl+C` to stop the policy.
-
----
-
-## Additional Resources
-
-- [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python)
-- [GR00T Policy Repository](https://huggingface.co/nepyope/GR00T-WholeBodyControl_g1)
-- [LeRobot Documentation](https://github.com/huggingface/lerobot)
-- [Unitree_IL_Lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot)
-
----
-
-_Last updated: December 2025_

docs/source/xvla.mdx

@@ -10,36 +10,20 @@ Inspired by meta-learning and prompt learning, we ask: **"What if a VLA model co
 
 **X-VLA** is a soft-prompted, flow-matching VLA framework that treats each hardware setup as a "task" and encodes it using a small set of learnable embeddings. These **Soft Prompts** capture embodiment and domain-specific variations, guiding the Transformer from the earliest stages of multimodal fusion. With this mechanism, X-VLA can reconcile diverse robot morphologies, data types, and sensor setups within a single unified architecture.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture.png"
-    alt="XVLA Architecture"
-    style="max-width: 100%; height: auto; width: 800px;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture.png" width="400">
 
 Built from pure Transformer encoders, X-VLA scales naturally with model size and dataset diversity. Across 6 simulation benchmarks and 3 real robots, Soft Prompts consistently outperform existing methods in handling hardware and domain differences. X-VLA-0.9B, trained on 290K episodes spanning seven robotic platforms, learns an embodiment-agnostic generalist policy in Phase I, and adapts efficiently to new robots in Phase II simply by learning a new set of prompts, while keeping the backbone frozen.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture2.png"
-    alt="XVLA Architecture 2"
-    style="width: 32%; max-width: 450px; height: auto;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-architecture2.png" width="400">
 
 With only 1% of parameters tuned (9M), X-VLA-0.9B achieves near-π₀ performance on LIBERO and Simpler-WidowX, despite using **300× fewer trainable parameters**. It also demonstrates strong real-world dexterity with minimal demonstrations, including folding cloths in under two minutes.
 
-<p align="center">
-  <img
-    src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-fold.png"
-    alt="XVLA fold visualization"
-    style="width: 95%; max-width: 1100px; height: auto;"
-  />
-</p>
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot/xvla-fold.png" width="400">
 
 X-VLA shows that generalist robot intelligence does not require increasingly complex architectures, only the right way to absorb heterogeneity. Soft Prompts offer a simple, scalable mechanism for unifying diverse robotic data, paving the way toward adaptable, cross-embodiment robot foundation models.
 
+---
+
 ## Installation
 
 After installing LeRobot, install the X-VLA dependencies:
@@ -54,6 +38,8 @@ After the new release, you'll be able to do:
 pip install lerobot[xvla]
 ```
 
+---
+
 ## Quick Start
 
 ### Basic Usage
@@ -80,6 +66,8 @@ lerobot-eval \
   --seed=142
 ```
 
+---
+
 ## Available Checkpoints
 
 ### 🎯 Base Model
@@ -92,7 +80,7 @@ A 0.9B parameter instantiation of X-VLA, trained with a carefully designed data
 
 - **Phase II: Domain Adaptation** - Adapted to deployable policies for target domains. A new set of soft prompts is introduced and optimized to encode the hardware configuration of the novel domain, while the pretrained backbone remains frozen.
 
-### Simulation Checkpoints
+### 🎮 Simulation Checkpoints
 
 **[lerobot/xvla-libero](https://huggingface.co/lerobot/xvla-libero)**
 
@@ -116,6 +104,8 @@ Optimized for AgileX robot dexterous manipulation tasks.
 
 Adapted for Google Robot platforms.
 
+---
+
 ## Training X-VLA
 
 ### Recommended Training Configuration
@@ -242,6 +232,8 @@ This ensures the optimizer receives a dict of named parameters, allowing it to c
 Completely matching the official reported performance may require an additional warm-up LR schedule for soft-prompts, which can bring minor improvements.
 We encourage implementing this in your customized training pipeline for optimal results.
 
+---
+
 ## Core Concepts
 
 ### 1. Action Modes
@@ -250,13 +242,13 @@ X-VLA uses an **Action Registry** system to handle different action spaces and e
 
 #### Available Action Modes
 
-| Action Mode      | Action Dim              | Description                                 | Use Case                             |
-| ---------------- | ----------------------- | ------------------------------------------- | ------------------------------------ |
-| `ee6d`           | 20                      | End-effector with xyz, 6D rotation, gripper | Dual-arm setups with spatial control |
-| `joint`          | 14                      | Joint-space with gripper                    | Direct joint control robots          |
-| `agibot_ee6d`    | 20                      | AGI-bot variant with MSE loss               | AGI-bot platforms                    |
-| `so101_bimanual` | 20 (model), 12 (real)   | SO101 bimanual robot                        | Bimanual manipulation tasks          |
-| `auto`           | 20 (model), auto (real) | Auto-detects action dim from dataset        | **Recommended** for new robots       |
+| Action Mode      | Action Dim            | Description                                 | Use Case                             |
+| ---------------- | --------------------- | ------------------------------------------- | ------------------------------------ |
+| `ee6d`           | 20                    | End-effector with xyz, 6D rotation, gripper | Dual-arm setups with spatial control |
+| `joint`          | 14                    | Joint-space with gripper                    | Direct joint control robots          |
+| `agibot_ee6d`    | 20                    | AGI-bot variant with MSE loss               | AGI-bot platforms                    |
+| `franka_joint7`  | 7                     | Franka Panda 7-joint control                | Franka robots without gripper        |
+| `so101_bimanual` | 20 (model), 12 (real) | SO101 bimanual robot                        | Bimanual manipulation tasks          |
 
 #### Why Action Modes Matter
 
@@ -284,27 +276,6 @@ REAL_DIM = 12
 
 See the [action_hub.py](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/action_hub.py) implementation for details.
 
-#### Auto Action Mode (Recommended)
-
-The `auto` action mode is the easiest way to use X-VLA with any robot. It automatically detects your dataset's action dimension and handles padding/trimming:
-
-```bash
-lerobot-train \
-  --policy.path="lerobot/xvla-base" \
-  --policy.action_mode=auto \
-  --policy.max_action_dim=20 \
-  ...
-```
-
-**How it works:**
-
-- Reads `action_feature.shape[-1]` from your dataset (e.g., 7 for Franka)
-- Model outputs `max_action_dim` (default 20) for pretrained compatibility
-- Loss is computed **only on the real dimensions**: `MSE(pred[:,:,:real_dim], target[:,:,:real_dim])`
-- Postprocess trims output back to `real_dim` for robot control
-
-This eliminates the need to create custom action modes for most robots.
-
 ### 2. Domain IDs
 
 Domain IDs are learnable identifiers for different robot configurations and camera setups. They allow X-VLA to distinguish between:
@@ -366,10 +337,9 @@ num_domains: int = 30          # Maximum number of domain IDs
 len_soft_prompts: int = 32     # Length of soft prompt embeddings
 
 # Action space
-action_mode: str = "ee6d"      # Action space type (use "auto" for auto-detection)
+action_mode: str = "ee6d"      # Action space type
 use_proprio: bool = True       # Use proprioceptive state
 max_state_dim: int = 32        # Maximum state dimension
-max_action_dim: int = 20       # Max action dim for padding (used by "auto" mode)
 
 # Vision
 num_image_views: int | None    # Number of camera views
@@ -379,6 +349,8 @@ resize_imgs_with_padding: tuple[int, int] | None  # Target image size with paddi
 num_denoising_steps: int = 10  # Flow matching denoising steps
 ```
 
+---
+
 ## Creating Custom Action Modes
 
 If your robot has a unique action space, you can create a custom action mode:
@@ -440,6 +412,8 @@ lerobot-train \
   ...
 ```
 
+---
+
 ## Advanced Topics
 
 ### Multi-Camera Support
@@ -481,16 +455,7 @@ preprocessor = PolicyProcessorPipeline(
 
 When your dataset has fewer action dimensions than the pretrained model:
 
-**Option 1 (Recommended)**: Use `auto` action mode
-
-```bash
-# Automatically detects your dataset's action dimension
-# Works with any robot without custom code
-policy.action_mode=auto
-policy.max_action_dim=20  # Match pretrained model
-```
-
-**Option 2**: Use a predefined action mode with built-in padding
+**Option 1**: Use padding (automatic in most action modes)
 
 ```python
 # Model expects 20D, dataset has 12D
@@ -511,6 +476,8 @@ class MappedActionSpace(BaseActionSpace):
         ...
 ```
 
+---
+
 ## Troubleshooting
 
 ### Common Issues
@@ -543,6 +510,8 @@ class MappedActionSpace(BaseActionSpace):
   3. Reduce batch size
   4. Freeze more components
 
+---
+
 ## Citation
 
 If you use X-VLA in your research, please cite:
@@ -557,13 +526,17 @@ If you use X-VLA in your research, please cite:
 }
 ```
 
+---
+
 ## Additional Resources
 
-- [X-VLA Paper](https://arxiv.org/pdf/2510.10274)
+- [X-VLA Paper](https://arxiv.org) (coming soon)
 - [LeRobot Documentation](https://github.com/huggingface/lerobot)
-- [Action Registry Implementation](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/action_hub.py)
-- [Processor Implementation](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/processor_xvla.py)
-- [Model Configuration](https://github.com/huggingface/lerobot/src/lerobot/policies/xvla/configuration_xvla.py)
+- [Action Registry Implementation](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/action_hub.py)
+- [Processor Implementation](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/processor_xvla.py)
+- [Model Configuration](/home/jade_choghari/robot/lerobot/src/lerobot/policies/xvla/configuration_xvla.py)
+
+---
 
 ## Contributing
 

examples/dataset/PGEN_SUMMARY.md

@@ -1,243 +0,0 @@
-# Synthetic Data Generation Script - Summary
-
-## ✅ What Was Created
-
-### Main Script: `annotate_pgen.py` (717 lines)
-A production-ready script implementing the Hi-Robot synthetic data generation pipeline.
-
-**Key Features:**
-- ✅ Loads LeRobot datasets with skill annotations
-- ✅ Generates synthetic user prompts and robot utterances using Qwen VLM
-- ✅ **Temporal sampling** - generates dialogue every N seconds (default: 1s)
-- ✅ Adds `task_index_high_level` feature to dataset parquets
-- ✅ Saves high-level tasks to `meta/tasks_high_level.parquet`
-- ✅ Exports debug JSONL for quality analysis
-- ✅ Supports both Qwen2-VL and Qwen3-VL models
-- ✅ Multi-view camera support
-- ✅ Episode-aware processing with automatic first-frame sampling
-- ✅ Modular architecture for easy extension
-
-### Supporting Files Created
-
-1. **`run_pgen.sh`** - Convenience script with sensible defaults
-2. **`README_PGEN.md`** - Comprehensive documentation with examples
-3. **`example_pgen_usage.md`** - Practical examples and performance estimates
-4. **`SAMPLING_DIAGRAM.md`** - Visual explanation of temporal sampling strategy
-5. **`PGEN_SUMMARY.md`** - This file
-
-## 🚀 Key Innovation: Temporal Sampling
-
-The script processes **ALL episodes** in the dataset efficiently via `--sample-interval`:
-
-```bash
-# Instead of calling VLM for every frame (expensive):
-# 15,000 frames × VLM call = ~5 hours
-
-# Generate dialogue every 1 second (efficient):
-python annotate_pgen.py --repo-id dataset --model qwen --sample-interval 1.0
-# 15,000 frames processed, only ~500 VLM calls (30x speedup!)
-```
-
-**How it works:**
-- Process ALL frames in ALL episodes (complete coverage)
-- Generate dialogue at sampled timepoints (e.g., every 1 second)
-- Propagate task indices to intermediate frames
-- Always sample first frame of each episode
-- All frames get labeled, but VLM is only called for samples
-- No dummy values or skipped episodes
-
-**Benefits:**
-- 30-100x speedup depending on interval
-- Maintains temporal coherence
-- Reduces cost without losing quality
-- Configurable based on skill duration
-
-## 📊 Efficiency Comparison
-
-For a typical 15,000 frame dataset at 30 fps:
-
-| Method | VLM Calls | Time | Cost |
-|--------|-----------|------|------|
-| Every frame | 15,000 | ~5 hours | $$$$ |
-| Every 0.5s | 1,000 | ~20 min | $$$ |
-| **Every 1s** (default) | **500** | **~10 min** | **$$** |
-| Every 2s | 250 | ~5 min | $ |
-
-## 🎯 Usage
-
-### Quick Test (5s sampling for fast iteration)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 5.0 \
-    --output-dir ./outputs/test_quick
-```
-
-### Production Run (Recommended Settings)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/full_pgen
-```
-
-### High-Quality with Qwen3
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --sample-interval 0.5 \
-    --temperature 0.6 \
-    --output-dir ./outputs/high_quality
-```
-
-## 📦 Output Structure
-
-After running, you'll have:
-
-```
-dataset_root/
-├── meta/
-│   ├── tasks_high_level.parquet      # High-level tasks with prompts/utterances
-│   └── syn_annotations.jsonl         # Debug: full context for each sample
-└── data/
-    └── chunk-000/
-        └── file-000.parquet           # Updated with task_index_high_level
-```
-
-**New feature added to all parquet files:**
-- `task_index_high_level` (int64): Links to tasks_high_level.parquet
-
-## 🔧 All Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--repo-id` / `--data-dir` | - | Dataset source |
-| `--model` | Qwen/Qwen2-VL-7B-Instruct | VLM model |
-| `--device` | cuda | Device to use |
-| `--dtype` | bfloat16 | Model precision |
-| `--temperature` | 0.7 | Sampling temperature |
-| **`--sample-interval`** | **1.0** | **Generate every N seconds (all episodes processed)** |
-| `--num-image-views-per-sample` | 1 | Number of cameras |
-| `--batch-size` | 1 | Batch size (currently unused) |
-| `--output-dir` | None | Output directory |
-| `--push-to-hub` | False | Push to HuggingFace |
-
-## 🎨 Generated Data Format
-
-Each sampled frame produces:
-
-```json
-{
-  "scenario_type": "specific_object",
-  "response_type": "confirmation",
-  "user_prompt": "Can you pick up the pink brick?",
-  "robot_utterance": "Sure, I'll grab the pink lego brick.",
-  "skill": "robot arm picks up pink lego brick",
-  "episode_id": 0,
-  "frame_index": 45,
-  "timestamp": 1.5,
-  "skill_history": ["robot arm moves towards pink lego brick"],
-  "task_description": "pink lego brick into the transparent box"
-}
-```
-
-**Scenario Types:**
-- specific_object, negative_task, situated_correction, implicit_request, constraint_based
-
-**Response Types:**
-- confirmation, clarification, acknowledgment, constraint_acknowledgment
-
-## 🔬 Code Architecture
-
-```python
-# Main components (modular design)
-
-class QwenPgen:
-    """VLM wrapper supporting Qwen2/3"""
-    def call_qwen(images, prompt) -> dict
-
-def construct_prompt(task, history, skill) -> str:
-    """Build contextual prompt with history"""
-
-def annotate_sample(pgen, images, ...) -> dict:
-    """Generate dialogue for one sample"""
-
-def generate_synthetic_data(dataset, pgen, ...) -> tuple:
-    """Process entire dataset with temporal sampling"""
-    # Core sampling logic:
-    # - Track last_sample_timestamp per episode
-    # - Sample if time_elapsed >= sample_interval
-    # - Always sample first frame of episodes
-    # - Propagate task_index to intermediate frames
-
-def main():
-    """CLI entrypoint with argparse"""
-```
-
-## ✨ Next Steps
-
-1. **Quick test with large interval:**
-   ```bash
-   # Fast iteration - samples every 5 seconds
-   python examples/dataset/annotate_pgen.py \
-       --data-dir /path/to/dataset \
-       --model Qwen/Qwen2-VL-7B-Instruct \
-       --sample-interval 5.0 \
-       --output-dir ./outputs/quick_test
-   ```
-
-2. **Verify output quality:**
-   ```bash
-   head outputs/quick_test/meta/syn_annotations.jsonl
-   ```
-
-3. **Production run:**
-   ```bash
-   # Standard 1 second sampling for production
-   bash examples/dataset/run_pgen.sh
-   ```
-
-4. **Use in training:**
-   ```python
-   from lerobot.datasets.lerobot_dataset import LeRobotDataset
-   
-   ds = LeRobotDataset(repo_id="...", root="outputs/pgen_annotations")
-   
-   # Access high-level task for each frame
-   frame = ds[100]
-   task_idx = frame["task_index_high_level"].item()
-   ```
-
-## 📚 Documentation Files
-
-- **`README_PGEN.md`**: Full API reference and troubleshooting
-- **`example_pgen_usage.md`**: Practical examples with performance estimates
-- **`SAMPLING_DIAGRAM.md`**: Visual explanation of temporal sampling
-- **`PGEN_SUMMARY.md`**: This overview document
-
-## 🎯 Success Criteria
-
-✅ Script generates synthetic dialogue using Qwen VLM  
-✅ Adds `task_index_high_level` feature to dataset  
-✅ Saves tasks to `tasks_high_level.parquet`  
-✅ Implements efficient temporal sampling (30-100x speedup)  
-✅ Handles episode boundaries correctly  
-✅ Produces diverse interaction types (scenarios + responses)  
-✅ Maintains temporal coherence within episodes  
-✅ Includes comprehensive documentation and examples  
-✅ Ready for production use on real datasets  
-
-## 💡 Key Takeaway
-
-**The script processes ALL episodes with intelligent sampling:**
-- `--sample-interval` controls how often VLM is called (default: 1.0s)
-- ALL frames in ALL episodes get labeled (complete coverage)
-- Intermediate frames inherit from most recent sample (temporal coherence)
-- Achieves 30-100x speedup while maintaining quality
-- Adjust interval based on use case: 5.0s for testing, 1.0s for production, 0.5s for fine detail
-
-This makes the synthetic data generation **practical, scalable, and complete** for real-world datasets!
-

examples/dataset/README_PGEN.md

@@ -1,243 +0,0 @@
-# Synthetic Data Generation for Hierarchical Robot Policies
-
-This directory contains `annotate_pgen.py`, a script for generating synthetic user prompts and robot utterances for hierarchical policy training using Vision-Language Models (VLMs).
-
-## Overview
-
-The script implements the synthetic data generation pipeline described in the Hi-Robot paper:
-
-1. **Load** a LeRobot dataset with skill annotations (from `annotate.py`)
-2. **Generate** synthetic dialogue using Qwen VLM:
-   - User prompts (ℓ_t): Natural requests that lead to specific skills
-   - Robot utterances (u_t): Acknowledgments and clarifications
-3. **Save** results as a new dataset feature `task_index_high_level`
-
-## Prerequisites
-
-1. First, annotate your dataset with skills using `annotate.py`:
-
-```bash
-python examples/dataset/annotate.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --video-key observation.images.base \
-    --model Qwen/Qwen2-VL-7B-Instruct
-```
-
-This creates `meta/skills.json` with skill segmentation for each episode.
-
-## Usage
-
-### Basic Usage
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/pgen_dataset
-```
-
-**Note**: The script processes **all episodes** in the dataset. It generates dialogue every 1 second (`--sample-interval 1.0`) using temporal sampling. Frames between samples reuse the last generated dialogue. This makes the process efficient while ensuring complete dataset coverage.
-
-### Advanced Options
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --temperature 0.8 \
-    --sample-interval 0.5 \
-    --num-image-views-per-sample 2 \
-    --output-dir ./outputs/pgen_dataset \
-    --push-to-hub
-```
-
-This example uses a more powerful model and samples every 0.5 seconds for finer granularity.
-
-### Fast Testing (larger interval)
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --repo-id lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 5.0 \
-    --output-dir ./outputs/pgen_quick_test
-```
-
-Use a larger interval (5.0 seconds) for rapid iteration during development. All episodes are still processed.
-
-### Using Local Dataset
-
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --output-dir ./outputs/pgen_dataset
-```
-
-## Output Files
-
-The script produces several outputs:
-
-1. **`meta/tasks_high_level.parquet`**: High-level tasks with user prompts and robot utterances
-   - Columns: task_index, user_prompt, robot_utterance, skill, scenario_type, response_type
-
-2. **`meta/syn_annotations.jsonl`**: Debug file with all generated dialogues
-   - One JSON object per line with full context for each frame
-
-3. **Modified dataset**: New dataset with `task_index_high_level` feature added to all parquet files
-
-## Scenario and Response Types
-
-The generator produces diverse interaction types:
-
-### Scenario Types
-- **specific_object**: Direct specification of objects/actions
-- **negative_task**: Instructions about what NOT to do
-- **situated_correction**: Adjustments based on current state
-- **implicit_request**: Implied needs without direct commands
-- **constraint_based**: Specific constraints or preferences
-
-### Response Types
-- **confirmation**: Simple acknowledgment ("OK, I'll do X")
-- **clarification**: Seeking confirmation ("Just to confirm...")
-- **acknowledgment**: Action acknowledgment ("Got it, doing X")
-- **constraint_acknowledgment**: Acknowledging constraints ("Sure, I'll X while Y")
-
-## Example Generated Data
-
-```json
-{
-  "episode_id": 0,
-  "frame_index": 45,
-  "timestamp": 2.5,
-  "skill_current": "robot arm picks up pink lego brick",
-  "skill_history": ["robot arm moves towards pink lego brick"],
-  "task_description": "pink lego brick into the transparent box",
-  "scenario_type": "specific_object",
-  "response_type": "confirmation",
-  "user_prompt": "Can you grab the pink brick?",
-  "robot_utterance": "Sure, I'll pick up the pink lego brick."
-}
-```
-
-## Accessing the Data
-
-After running the script, access the synthetic data in your code:
-
-```python
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-import pandas as pd
-
-# Load modified dataset
-dataset = LeRobotDataset(repo_id="lerobot/svla_so101_pickplace_with_high_level_tasks")
-
-# Access frame with high-level task
-frame = dataset[100]
-high_level_task_idx = frame["task_index_high_level"].item()
-
-# Load high-level tasks
-tasks_df = pd.read_parquet(dataset.root / "meta" / "tasks_high_level.parquet")
-task_info = tasks_df.iloc[high_level_task_idx]
-
-print(f"User prompt: {task_info['user_prompt']}")
-print(f"Robot utterance: {task_info['robot_utterance']}")
-print(f"Skill: {task_info['skill']}")
-```
-
-## Architecture
-
-The script is modular and extensible:
-
-```python
-# Core components
-class QwenPgen:
-    """VLM wrapper for generation"""
-    def call_qwen(images, prompt) -> dict
-    
-def construct_prompt(task, history, skill) -> str
-    """Build prompt for VLM"""
-    
-def annotate_sample(pgen, images, ...) -> dict
-    """Generate dialogue for one sample"""
-    
-def generate_synthetic_data(dataset, pgen, ...) -> tuple
-    """Process entire dataset"""
-```
-
-## Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--repo-id` | - | HuggingFace dataset ID |
-| `--data-dir` | - | Local dataset path |
-| `--model` | Qwen/Qwen2-VL-7B-Instruct | VLM model name |
-| `--device` | cuda | Device (cuda/cpu) |
-| `--dtype` | bfloat16 | Model precision |
-| `--temperature` | 0.7 | Sampling temperature |
-| `--sample-interval` | 1.0 | Generate dialogue every N seconds (all episodes processed) |
-| `--num-image-views-per-sample` | 1 | Number of cameras |
-| `--output-dir` | None | Output directory |
-| `--push-to-hub` | False | Push to HuggingFace Hub |
-
-## Sampling Strategy
-
-The script uses **temporal sampling** to efficiently generate dialogue:
-
-- **Default**: Generate dialogue every 1 second (`--sample-interval 1.0`)
-- **Efficiency**: If a dataset runs at 30fps, this samples ~3% of frames
-- **Propagation**: Frames between samples reuse the last generated task_index
-- **Episode-aware**: Always samples the first frame of each episode
-
-### Example with 30 fps dataset:
-```bash
-# Sample every 1 second (every 30 frames)
---sample-interval 1.0  # ~3,000 generations for a 100 episode dataset (3 sec/episode)
-
-# Sample every 0.5 seconds (every 15 frames)
---sample-interval 0.5  # ~6,000 generations (more granular)
-
-# Sample every 2 seconds (every 60 frames)
---sample-interval 2.0  # ~1,500 generations (more efficient)
-```
-
-### Why sampling works:
-- Skills typically last 1-3 seconds
-- Dialogue doesn't need to change every frame
-- Reduces computational cost by 30-100x
-- Still provides good coverage for training
-
-## Tips
-
-1. **Quick testing**: Use larger `--sample-interval` (e.g., 5.0 or 10.0) for rapid iteration
-2. **Monitor GPU**: VLM inference is memory-intensive
-3. **Check outputs**: Review `syn_annotations.jsonl` for quality
-4. **Adjust temperature**: Higher = more diverse, lower = more consistent
-5. **Multiple views**: Use `--num-image-views-per-sample 2+` for better context
-6. **Tune sampling**: Start with 1.0s, increase for speed (testing), decrease for granularity (production)
-
-## Troubleshooting
-
-### No skills.json found
-Run `annotate.py` first to generate skill annotations.
-
-### Out of memory
-- Reduce batch size to 1
-- Use smaller model (Qwen2-VL-7B instead of Qwen3-VL-30B)
-- Process fewer samples at a time
-
-### Poor quality generations
-- Adjust temperature (try 0.6-0.9)
-- Check that skills.json has good annotations
-- Ensure images are loading correctly
-
-## Citation
-
-Based on the Hi-Robot paper's synthetic data generation approach:
-```
-@article{hirobot2024,
-  title={Hi-Robot: Hierarchical Robot Learning with Vision-Language Models},
-  year={2024}
-}
-```
-

examples/dataset/SAMPLING_DIAGRAM.md

@@ -1,141 +0,0 @@
-# Temporal Sampling Strategy Visualization
-
-## How `--sample-interval` Works
-
-### Example: 30 fps dataset, `--sample-interval 1.0` (1 second)
-
-```
-Timeline (seconds):  0.0      0.5      1.0      1.5      2.0      2.5      3.0
-                     │        │        │        │        │        │        │
-Frames:              0───15───30───45───60───75───90───105──120──135──150
-                     │        │        │        │        │        │        │
-                     ▼                 ▼                 ▼                 ▼
-Sampled:            YES      NO       YES      NO       YES      NO       YES
-                     │                 │                 │                 │
-Task Index:         [0]──────────────>[1]──────────────>[2]──────────────>[3]
-                     │                 │                 │                 │
-VLM Called:         ✓ Gen             ✓ Gen             ✓ Gen             ✓ Gen
-                    dialogue          dialogue          dialogue          dialogue
-                     │                 │                 │                 │
-Frames 0-29    ─────┘                 │                 │                 │
-get task 0                             │                 │                 │
-                                       │                 │                 │
-Frames 30-59  ────────────────────────┘                 │                 │
-get task 1                                               │                 │
-                                                         │                 │
-Frames 60-89  ──────────────────────────────────────────┘                 │
-get task 2                                                                 │
-                                                                           │
-Frames 90-119 ────────────────────────────────────────────────────────────┘
-get task 3
-```
-
-## Comparison: Different Sampling Intervals
-
-### `--sample-interval 2.0` (every 2 seconds)
-```
-Timeline:    0.0      1.0      2.0      3.0      4.0      5.0      6.0
-             │        │        │        │        │        │        │
-Sampled:    YES      NO       YES      NO       YES      NO       YES
-             │                 │                 │                 │
-Tasks:      [0]───────────────>[1]───────────────>[2]───────────────>[3]
-             
-VLM Calls:   4 (fewer calls, faster but less granular)
-```
-
-### `--sample-interval 1.0` (every 1 second) - **DEFAULT**
-```
-Timeline:    0.0   0.5   1.0   1.5   2.0   2.5   3.0   3.5   4.0   4.5   5.0   5.5   6.0
-             │     │     │     │     │     │     │     │     │     │     │     │     │
-Sampled:    YES   NO   YES   NO   YES   NO   YES   NO   YES   NO   YES   NO   YES
-             │           │           │           │           │           │           │
-Tasks:      [0]─────────>[1]─────────>[2]─────────>[3]─────────>[4]─────────>[5]─────>[6]
-             
-VLM Calls:   7 (balanced coverage and speed)
-```
-
-### `--sample-interval 0.5` (every 0.5 seconds)
-```
-Timeline:    0.0  0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0  5.5  6.0
-             │    │    │    │    │    │    │    │    │    │    │    │    │
-Sampled:    YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES  YES
-             │    │    │    │    │    │    │    │    │    │    │    │    │
-Tasks:      [0]─>[1]─>[2]─>[3]─>[4]─>[5]─>[6]─>[7]─>[8]─>[9]─>[10]>[11]>[12]
-             
-VLM Calls:   13 (high granularity, slower but more detailed)
-```
-
-## Episode Boundaries
-
-The script always samples the **first frame** of each episode:
-
-```
-Episode 0                          Episode 1                          Episode 2
-├─────────────────────────────────┤├─────────────────────────────────┤├──────...
-│                                 ││                                 ││
-Frame: 0    30    60    90   120  130   160   190   220  250  260   290  320
-Time:  0.0  1.0   2.0   3.0  4.0  0.0   1.0   2.0   3.0  4.0  0.0   1.0  2.0
-       │    │     │     │    │    │     │     │     │    │    │     │    │
-       ▼    ▼     ▼     ▼    ▼    ▼     ▼     ▼     ▼    ▼    ▼     ▼    ▼
-Sample:YES  YES   YES   YES  YES  YES   YES   YES   YES  YES  YES   YES  YES
-       │    │     │     │    │    │     │     │     │    │    │     │    │
-Task:  0────1─────2─────3────4    5─────6─────7─────8────9    10────11───12
-
-Note: Frames 0, 130, 260 are ALWAYS sampled (episode starts)
-      Even if they're within the sample-interval window
-```
-
-## Real-World Example: svla_so101_pickplace Dataset
-
-Typical stats:
-- **Total episodes**: 50
-- **Avg episode length**: 300 frames (10 seconds at 30 fps)
-- **Total frames**: 15,000
-
-### Without Sampling (every frame)
-```
-Frames processed:    15,000
-VLM calls:           15,000
-Time estimate:       ~5 hours
-Unique tasks:        ~12,000 (lots of duplicates)
-```
-
-### With `--sample-interval 1.0` (every 1 second)
-```
-Frames processed:    15,000 ✓
-VLM calls:           500
-Time estimate:       ~10 minutes
-Unique tasks:        ~450 (meaningful variety)
-Efficiency gain:     30x faster
-```
-
-### With `--sample-interval 2.0` (every 2 seconds)
-```
-Frames processed:    15,000 ✓
-VLM calls:           250
-Time estimate:       ~5 minutes
-Unique tasks:        ~220
-Efficiency gain:     60x faster
-```
-
-## Key Points
-
-1. **All frames get labeled**: Every frame gets a `task_index_high_level`
-2. **Only sampled frames call VLM**: Huge efficiency gain
-3. **Temporal coherence**: Nearby frames share the same task
-4. **Episode-aware**: Always samples episode starts
-5. **Configurable**: Adjust `--sample-interval` based on your needs
-
-## Choosing Your Sampling Interval
-
-| Use Case | Recommended Interval | Why |
-|----------|---------------------|-----|
-| Quick testing | 2.0s | Fastest iteration |
-| Standard training | 1.0s | Good balance |
-| High-quality dataset | 0.5s | Better coverage |
-| Fine-grained control | 0.33s | Very detailed |
-| Dense annotations | 0.1s | Nearly every frame |
-
-**Rule of thumb**: Match your sampling interval to your typical skill duration.
-If skills last 1-3 seconds, sampling every 1 second captures each skill multiple times.
-

examples/dataset/action_tokenizer_example.py

@@ -1,138 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Example demonstrating how to use the ActionTokenizerProcessorStep to tokenize actions.
-
-This example shows how to:
-1. Load a dataset with action data
-2. Apply the action tokenizer processor to tokenize actions with proper padding/truncation
-3. Access both the tokenized actions and the attention mask
-4. Decode tokenized actions back to their original form
-"""
-
-import torch
-from transformers import AutoProcessor
-
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.processor.core import EnvTransition, TransitionKey
-from lerobot.processor.tokenizer_processor import ActionTokenizerProcessorStep
-from lerobot.utils.constants import ACTION_TOKEN_MASK
-
-# Define delta timestamps for the dataset
-delta_timestamps = {
-    'action': [
-        0.0, 0.03333333333333333, 0.06666666666666667, 0.1, 0.13333333333333333,
-        0.16666666666666666, 0.2, 0.23333333333333334, 0.26666666666666666, 0.3,
-        0.3333333333333333, 0.36666666666666664, 0.4, 0.43333333333333335,
-        0.4666666666666667, 0.5, 0.5333333333333333, 0.5666666666666667, 0.6,
-        0.6333333333333333, 0.6666666666666666, 0.7, 0.7333333333333333,
-        0.7666666666666667, 0.8, 0.8333333333333334, 0.8666666666666667, 0.9,
-        0.9333333333333333, 0.9666666666666667, 1.0, 1.0333333333333334,
-        1.0666666666666667, 1.1, 1.1333333333333333, 1.1666666666666667, 1.2,
-        1.2333333333333334, 1.2666666666666666, 1.3, 1.3333333333333333,
-        1.3666666666666667, 1.4, 1.4333333333333333, 1.4666666666666666, 1.5,
-        1.5333333333333334, 1.5666666666666667, 1.6, 1.6333333333333333
-    ]
-}
-
-# Load the dataset
-print("Loading dataset...")
-dataset = LeRobotDataset(
-    repo_id="local",
-    root="/fsx/jade_choghari/outputs/pgen_annotations1",
-    delta_timestamps=delta_timestamps
-)
-
-# Create a dataloader
-dataloader = torch.utils.data.DataLoader(
-    dataset,
-    num_workers=0,
-    batch_size=4,
-    shuffle=True,
-)
-
-# Get a batch of data
-batch = next(iter(dataloader))
-action_data = batch["action"]  # Shape: (batch_size, action_horizon, action_dim)
-
-print(f"\nOriginal action shape: {action_data.shape}")
-print(f"Original action data (first sample, first timestep):\n{action_data[0, 0]}")
-
-# Method 1: Using the tokenizer directly (as in fast_tokenize.py)
-print("\n" + "="*80)
-print("Method 1: Direct tokenizer usage")
-print("="*80)
-
-tokenizer = AutoProcessor.from_pretrained("physical-intelligence/fast", trust_remote_code=True)
-
-# Tokenize directly
-tokens = tokenizer(action_data)
-print(f"\nDirect tokenization result type: {type(tokens)}")
-print(f"Tokens shape/length: {tokens.shape if isinstance(tokens, torch.Tensor) else len(tokens)}")
-
-# Decode
-decoded_actions = tokenizer.decode(tokens)
-print(f"Decoded actions shape: {decoded_actions.shape}")
-reconstruction_error = torch.abs(action_data - decoded_actions).mean()
-print(f"Mean absolute reconstruction error: {reconstruction_error.item():.6f}")
-
-# Method 2: Using the ActionTokenizerProcessorStep with proper padding/truncation
-print("\n" + "="*80)
-print("Method 2: Using ActionTokenizerProcessorStep (with padding & mask)")
-print("="*80)
-
-# Create the action tokenizer processor step
-action_tokenizer_processor = ActionTokenizerProcessorStep(
-    tokenizer_name="physical-intelligence/fast",
-    trust_remote_code=True,
-    max_action_tokens=32,  # Maximum number of tokens per action
-)
-
-# Create a transition with the action data
-transition = {
-    TransitionKey.ACTION: action_data,
-    TransitionKey.OBSERVATION: {},  # Empty for this example
-}
-
-# Apply the processor
-processed_transition = action_tokenizer_processor(transition)
-
-# Extract tokenized actions and mask
-tokenized_actions = processed_transition[TransitionKey.ACTION]
-complementary_data = processed_transition[TransitionKey.COMPLEMENTARY_DATA]
-action_mask = complementary_data[ACTION_TOKEN_MASK]
-
-print(f"\nTokenized actions shape: {tokenized_actions.shape}")  # (batch_size, max_action_tokens)
-print(f"Action mask shape: {action_mask.shape}")  # (batch_size, max_action_tokens)
-print(f"Tokenized actions dtype: {tokenized_actions.dtype}")
-print(f"Action mask dtype: {action_mask.dtype}")
-
-# Show token statistics
-print(f"\nFirst sample tokens: {tokenized_actions[0]}")
-print(f"First sample mask: {action_mask[0]}")
-num_real_tokens = action_mask[0].sum().item()
-print(f"Number of real tokens (non-padding): {num_real_tokens}")
-print(f"Number of padding tokens: {action_mask.shape[1] - num_real_tokens}")
-
-# Decode using the mask
-print("\nDecoding tokenized actions...")
-decoded_with_processor = tokenizer.decode(tokenized_actions)
-print(f"Decoded actions shape: {decoded_with_processor.shape}")
-
-# Calculate reconstruction error
-reconstruction_error_processor = torch.abs(action_data - decoded_with_processor).mean()
-print(f"Mean absolute reconstruction error: {reconstruction_error_processor.item():.6f}")
-
-# Show that masking works correctly
-print("\n" + "="*80)
-print("Mask demonstration")
-print("="*80)
-for i in range(min(4, tokenized_actions.shape[0])):
-    mask_i = action_mask[i]
-    num_real = mask_i.sum().item()
-    print(f"Sample {i}: {num_real} real tokens, {len(mask_i) - num_real} padding tokens")
-
-print("\n" + "="*80)
-print("Action tokenization example completed successfully!")
-print("="*80)
-

examples/dataset/example_pgen_usage.md

@@ -1,143 +0,0 @@
-# Example: Synthetic Data Generation with Sampling
-
-## Quick Start
-
-### 1. Test with 100 frames and 1 second sampling
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --num-samples 100 \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/test_pgen
-```
-
-**Expected behavior** (assuming 30 fps):
-- Total frames: 100
-- Frames sampled: ~4 (every 30 frames = 1 second)
-- Efficiency: 96% fewer VLM calls
-- Output: All 100 frames get `task_index_high_level`, but only 4 unique dialogues generated
-
-### 2. Process full dataset with different sampling rates
-
-#### Conservative (every 2 seconds)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 2.0 \
-    --output-dir ./outputs/pgen_2s
-```
-
-#### Standard (every 1 second) - **RECOMMENDED**
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 1.0 \
-    --output-dir ./outputs/pgen_1s
-```
-
-#### Fine-grained (every 0.5 seconds)
-```bash
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen2-VL-7B-Instruct \
-    --sample-interval 0.5 \
-    --output-dir ./outputs/pgen_0.5s
-```
-
-## Performance Estimates
-
-For a dataset with:
-- 100 episodes
-- 10 seconds per episode (average)
-- 30 fps
-- Total frames: 30,000
-
-| Sampling Interval | Frames Sampled | % Sampled | Speedup | Time Estimate |
-|-------------------|----------------|-----------|---------|---------------|
-| Every frame (0.033s) | 30,000 | 100% | 1x | ~10 hours |
-| 0.5 seconds | 2,000 | 6.7% | 15x | ~40 min |
-| **1.0 seconds** | **1,000** | **3.3%** | **30x** | **~20 min** |
-| 2.0 seconds | 500 | 1.7% | 60x | ~10 min |
-
-*Note: Times are approximate and depend on GPU, model size, and generation speed*
-
-## Understanding the Output
-
-### Console Output Example
-```
-[cyan]Generating synthetic data for 30000 frames...[/cyan]
-[cyan]Sampling interval: 1.0s (fps: 30)[/cyan]
-Generating synthetic dialogue: 100%|████████| 30000/30000 [20:15<00:00, 24.68it/s]
-[green]✓ Sampled 1000 frames out of 30000 (3.3%)[/green]
-[green]✓ Generated 450 unique high-level tasks[/green]
-```
-
-### What happens:
-1. **Frame 0 (t=0.0s)**: Generate dialogue → Task index 0
-2. **Frames 1-29 (t=0.033s-0.967s)**: Reuse task index 0
-3. **Frame 30 (t=1.0s)**: Generate new dialogue → Task index 1
-4. **Frames 31-59 (t=1.033s-1.967s)**: Reuse task index 1
-5. And so on...
-
-### Result:
-- Every frame has a `task_index_high_level`
-- Only sampled frames have unique dialogues generated
-- Intermediate frames inherit from the most recent sample
-- Maintains temporal coherence within episodes
-
-## Checking Your Results
-
-After running, verify the output:
-
-```bash
-# Check the generated tasks
-python -c "
-import pandas as pd
-from pathlib import Path
-
-tasks = pd.read_parquet('outputs/test_pgen/meta/tasks_high_level.parquet')
-print(f'Total unique tasks: {len(tasks)}')
-print(f'Sample tasks:')
-print(tasks[['user_prompt', 'robot_utterance', 'skill']].head())
-"
-
-# Check debug output
-head outputs/test_pgen/meta/syn_annotations.jsonl
-
-# Load and verify dataset
-python -c "
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-
-ds = LeRobotDataset(repo_id='local_with_high_level_tasks', 
-                    root='outputs/test_pgen')
-print(f'Dataset has {len(ds)} frames')
-print(f'Features: {list(ds.features.keys())}')
-assert 'task_index_high_level' in ds.features
-print('✓ task_index_high_level feature added successfully!')
-"
-```
-
-## Common Use Cases
-
-### Development/Testing
-```bash
---sample-interval 2.0  # Fast iteration
---num-samples 500      # Small subset
-```
-
-### Production Training
-```bash
---sample-interval 1.0  # Good coverage
-# Process all samples (no --num-samples)
-```
-
-### High-Quality Dataset
-```bash
---sample-interval 0.5  # Fine-grained
---temperature 0.6      # More consistent
---model Qwen/Qwen3-VL-30B-A3B-Instruct  # Larger model
-```
-

examples/dataset/fast_tokenize.py

@@ -1,25 +0,0 @@
-import numpy as np
-from transformers import AutoProcessor
-import torch
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-
-delta_timestamps = {'action': [0.0, 0.03333333333333333, 0.06666666666666667, 0.1, 0.13333333333333333, 0.16666666666666666, 0.2, 0.23333333333333334, 0.26666666666666666, 0.3, 0.3333333333333333, 0.36666666666666664, 0.4, 0.43333333333333335, 0.4666666666666667, 0.5, 0.5333333333333333, 0.5666666666666667, 0.6, 0.6333333333333333, 0.6666666666666666, 0.7, 0.7333333333333333, 0.7666666666666667, 0.8, 0.8333333333333334, 0.8666666666666667, 0.9, 0.9333333333333333, 0.9666666666666667, 1.0, 1.0333333333333334, 1.0666666666666667, 1.1, 1.1333333333333333, 1.1666666666666667, 1.2, 1.2333333333333334, 1.2666666666666666, 1.3, 1.3333333333333333, 1.3666666666666667, 1.4, 1.4333333333333333, 1.4666666666666666, 1.5, 1.5333333333333334, 1.5666666666666667, 1.6, 1.6333333333333333]}
-dataset = LeRobotDataset(repo_id="local", root="/fsx/jade_choghari/outputs/pgen_annotations1", delta_timestamps=delta_timestamps)
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=4,
-        shuffle=True,
-)
-
-batch = next(iter(dataloader))
-
-# Load the tokenizer from the Hugging Face hub
-tokenizer = AutoProcessor.from_pretrained("physical-intelligence/fast", trust_remote_code=True)
-
-# Tokenize & decode action chunks (we use dummy data here)
-action_data = batch["action"]    # one batch of action chunks
-tokens = tokenizer(action_data)              # tokens = list[int]
-decoded_actions = tokenizer.decode(tokens)
-print("tokenized actions: ", tokens)

examples/dataset/inference_gemma.py

@@ -1,17 +0,0 @@
-from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
-
-model_id = "google/paligemma-3b-pt-224"
-model = PaliGemmaForConditionalGeneration.from_pretrained(model_id)
-processor = AutoProcessor.from_pretrained(model_id)
-
-breakpoint()
-prefix_output = model.language_model.forward(
-    inputs_embeds=inputs_embeds[0],
-    attention_mask=attention_mask,
-    position_ids=position_ids,
-    adarms_cond=adarms_cond[0] if adarms_cond is not None else None,
-)
-prefix_past_key_values = prefix_output.past_key_values
-# prefix_output to be used for the language head
-# shape: [batch_size, seq_len, hidden_size] with hidden_size = 2048
-prefix_output = prefix_output.last_hidden_state

examples/dataset/inference_pi05.py

@@ -1,91 +0,0 @@
-import torch
-from huggingface_hub import HfApi
-
-import lerobot
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-# import make_pre_post_processors
-from lerobot.policies.factory import make_pre_post_processors
-from lerobot.policies.pi05.configuration_pi05 import PI05Config
-from lerobot.policies.factory import make_policy, make_policy_config
-from lerobot.configs.policies import PreTrainedConfig
-
-cfg = PreTrainedConfig.from_pretrained(
-    pretrained_name_or_path="/fsx/jade_choghari/outputs/pi0_training/checkpoints/last/pretrained_model",
-)
-cfg.dtype = "bfloat16"
-
-pre_processor, post_processor = make_pre_post_processors(
-    policy_cfg=cfg,
-    pretrained_path="/fsx/jade_choghari/outputs/pi0_training/checkpoints/last/pretrained_model",
-)
-
-delta_timestamps = {'action': [0.0, 0.03333333333333333, 0.06666666666666667, 0.1, 0.13333333333333333, 0.16666666666666666, 0.2, 0.23333333333333334, 0.26666666666666666, 0.3, 0.3333333333333333, 0.36666666666666664, 0.4, 0.43333333333333335, 0.4666666666666667, 0.5, 0.5333333333333333, 0.5666666666666667, 0.6, 0.6333333333333333, 0.6666666666666666, 0.7, 0.7333333333333333, 0.7666666666666667, 0.8, 0.8333333333333334, 0.8666666666666667, 0.9, 0.9333333333333333, 0.9666666666666667, 1.0, 1.0333333333333334, 1.0666666666666667, 1.1, 1.1333333333333333, 1.1666666666666667, 1.2, 1.2333333333333334, 1.2666666666666666, 1.3, 1.3333333333333333, 1.3666666666666667, 1.4, 1.4333333333333333, 1.4666666666666666, 1.5, 1.5333333333333334, 1.5666666666666667, 1.6, 1.6333333333333333]}
-
-dataset = LeRobotDataset(repo_id="local", root="/fsx/jade_choghari/outputs/pgen_annotations1", delta_timestamps=delta_timestamps)
-
-# rename map --rename_map='{
-#         "observation.images.side": "observation.images.base_0_rgb",
-#         "observation.images.up": "observation.images.left_wrist_0_rgb"
-#         }'
-rename_map = {
-    "observation.images.side": "observation.images.base_0_rgb",
-    "observation.images.up": "observation.images.left_wrist_0_rgb"
-}
-policy = make_policy(
-    cfg=cfg,
-    ds_meta=dataset.meta,
-    rename_map=rename_map,
-)
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=4,
-        shuffle=True,
-)
-
-batch = next(iter(dataloader))
-batch = pre_processor(batch)
-policy.train()
-# run inference
-# action = policy.select_action(batch)
-loss, loss_dict = policy.forward(batch)
-breakpoint()
-# import requests
-# from PIL import Image
-# from transformers import AutoProcessor
-# model = policy.model.paligemma_with_expert.paligemma
-# model = model.to(device="cuda", dtype=torch.bfloat16)
-# model.eval()
-# prompt = "Describe this image."
-# url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
-# image = Image.open(requests.get(url, stream=True).raw)
-# processor = AutoProcessor.from_pretrained(
-#     "google/paligemma-3b-pt-224",
-# )
-# inputs = processor(image, prompt, return_tensors="pt").to(model.device)
-# print("generating...")
-# output = model.generate(
-#     **inputs,
-#     max_new_tokens=50,
-#     use_cache=True,  # default dynamic cache
-# )
-# print(processor.decode(output[0], skip_special_tokens=True))
-
-
-# # other model
-# from transformers import PaliGemmaForConditionalGeneration
-# model = PaliGemmaForConditionalGeneration.from_pretrained(
-#     "google/paligemma2-3b-pt-224",
-#     torch_dtype=torch.bfloat16,
-#     device_map="auto",
-# )
-# model.eval()
-# print("generating...")
-# output = model.generate(
-#     **inputs,
-#     max_new_tokens=100,
-#     use_cache=True,  # default dynamic cache
-# )
-# print("Model 2 output:")
-# print(processor.decode(output[0], skip_special_tokens=True))

examples/dataset/load_lerobot_high.py

@@ -1,23 +0,0 @@
-import torch
-from huggingface_hub import HfApi
-
-import lerobot
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-
-dataset = LeRobotDataset(repo_id="local", root="/fsx/jade_choghari/outputs/pgen_annotations1")
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=32,
-        shuffle=True,
-)
-
-batch = next(iter(dataloader))
-print(batch.keys())
-print(batch['task_index_high_level'].shape)
-print(batch['task_index_high_level'])
-print(batch['user_prompt'][0])
-print(batch['robot_utterance'][0])
-print(batch['task'][0])
-breakpoint()

examples/dataset/load_libero.py

@@ -1,18 +0,0 @@
-import torch
-from huggingface_hub import HfApi
-
-import lerobot
-from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
-
-dataset = LeRobotDataset(repo_id="lerobot/libero")
-
-dataloader = torch.utils.data.DataLoader(
-        dataset,
-        num_workers=0,
-        batch_size=4,
-        shuffle=True,
-)
-batch = next(iter(dataloader))
-print(batch.keys())
-
-breakpoint()

examples/dataset/mask.md

@@ -1,159 +0,0 @@
-## One-sentence answer
-
-> `make_att_2d_masks(prefix_pad_masks, prefix_att_masks)` builds the **actual 2D attention mask** `[B, L, L]` that tells the transformer **which token positions may attend to which others**, combining **padding** and **causality**.
-
-Everything else you’ve seen so far was just metadata.
-
----
-
-## What goes in
-
-### Inputs
-
-```python
-prefix_pad_masks   # shape [B, L]
-prefix_att_masks   # shape [B, L]
-```
-
-Where:
-
-* `prefix_pad_masks[b, i] = True`
-  → token `i` exists (not padding)
-
-* `prefix_att_masks[b, i] = False`
-  → token `i` is **bidirectional**
-
-* `prefix_att_masks[b, i] = True`
-  → token `i` is **causal (autoregressive)**
-
----
-
-## What comes out
-
-```python
-att_2d_prefix  # shape [B, L, L]
-```
-
-Each entry:
-
-```text
-att_2d_prefix[b, i, j] = True
-```
-
-means:
-
-> “In batch `b`, **token i (query)** is allowed to attend to **token j (key)**.”
-
----
-
-## How it is constructed (conceptually)
-
-For **each batch b**, **each query position i**, **each key position j**:
-
-```python
-if not prefix_pad_masks[b, j]:
-    att[b, i, j] = False           # cannot attend to padding
-else if not prefix_att_masks[b, i]:
-    att[b, i, j] = True            # bidirectional token → can see all real tokens
-else:
-    att[b, i, j] = (j <= i)        # causal token → can see only past + itself
-```
-
-That’s it.
-
----
-
-## Tiny concrete example (exactly matching your code)
-
-Suppose:
-
-```python
-prefix_pad_masks[0] = [T, T, T, T, T, F]
-prefix_att_masks[0] = [F, F, F, T, T, T]
-```
-
-Tokens:
-
-```
-0: IMG
-1: IMG
-2: LANG
-3: SUB0
-4: SUB1
-5: PAD
-```
-
----
-
-### Resulting `att_2d_prefix[0]`
-
-`✓ = True, ✗ = False`
-
-| Q \ K      | 0 | 1 | 2 | 3 | 4 | 5 |
-| ---------- | - | - | - | - | - | - |
-| 0 (bi)     | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
-| 1 (bi)     | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
-| 2 (bi)     | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
-| 3 (causal) | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
-| 4 (causal) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
-| 5 (pad)    | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
-
----
-
-## Why this matters for your training code
-
-This line:
-
-```python
-att_2d_prefix_4d = self._prepare_attention_masks_4d(att_2d_prefix)
-```
-
-Converts `[B, L, L] → [B, 1, L, L]` and possibly flips True/False to `0/-inf`.
-
-This is **exactly what Paligemma uses inside self-attention**.
-
----
-
-## Key implications (VERY important)
-
-### 1️⃣ This mask does **not isolate token groups**
-
-* Bidirectional tokens can attend to **everything**
-* Causal tokens only restrict *their own row*
-
-So **flow/action tokens must be blocked separately**.
-
----
-
-### 2️⃣ This is why your AR subtask prediction works
-
-* Subtask tokens are causal
-* Output at position `i` predicts token `i+1`
-* Padding is fully ignored
-
----
-
-### 3️⃣ Inference behavior
-
-When `subtask_tokens = None`:
-
-* `prefix_att_masks` contains only `False`
-* `att_2d_prefix` becomes **fully bidirectional**
-* No AR behavior remains
-
-Exactly what you want.
-
----
-
-## One-sentence takeaway (commit this)
-
-> `make_att_2d_masks` fuses **padding** and **causality** into a concrete `[B, L, L]` attention matrix that the transformer actually uses.
-
-If you want next, I can:
-
-* inspect `make_att_2d_masks()` source with you
-* show how to block **flow → subtask** attention
-* explain how this changes when suffix tokens are added
-* help you refactor this into a cleaner “grouped attention” API
-
-You’re now at the point where the model’s behavior should feel *predictable*, not magical.

examples/dataset/prompt.txt

@@ -1,334 +0,0 @@
-Generate annotate_pgen.py using Qwen for synthetic data generation
-
-You are writing a Python script called annotate_pgen.py.
-This script generates synthetic user prompts (ℓ_t) and robot utterances (u_t) for Hi Robot–style hierarchical policy training, using Qwen 3vl as the generator model (pgen).
-
-SCRIPT PURPOSE
-
-The script must:
-
-Load Dlabeled which is a LeRobot Dataset that has been annotate using the annotate.py script, which contains:
-
-images: list of image paths at time t
-
-skill_current: the annotated skill label (ℓ̂_t)
-
-skill_history: list of previous skill labels (ℓ̂₀ … ℓ̂_{t−1}), those where annotated, and you can find details on them stored in teh dataset inside the the DATA_PATH/meta/skills.json
-
-you will find something like 
-
-{
-  "coarse_description": "pink lego brick into the transparent box",
-  "skill_to_task_index": {
-    "robot arm picks up pink lego brick": 19,
-    "robot arm approaches transparent box": 3,
-    "robot arm retracts from transparent box": 28,
-    "robot arm moves towards pink lego brick": 12,
-    "robot arm releases red lego brick into box": 26,
-    "robot arm releases red lego brick into transparent box": 27,
-    "robot arm closes gripper to pick up the pink lego brick": 5,
-    "robot arm lifts the pink lego brick": 7,
-    etc..
-  },
-  "episodes": {
-    "0": {
-      "episode_index": 0,
-      "description": "pink lego brick into the transparent box",
-      "skills": [
-        {
-          "name": "robot arm moves towards pink lego brick",
-          "start": 0.0,
-          "end": 1.8
-        },
-        {
-          "name": "robot arm picks up pink lego brick",
-          "start": 1.8,
-          "end": 3.1
-        },
-        {
-          "name": "robot arm moves towards transparent box",
-          "start": 3.1,
-          "end": 5.5
-        },
-        {
-          "name": "robot arm releases pink lego brick into transparent box",
-          "start": 5.5,
-          "end": 7.0
-        },
-        {
-          "name": "robot arm retracts from transparent box",
-          "start": 7.0,
-          "end": 10.1
-        }
-      ]
-    },
-    "1": {
-      "episode_index": 1,
-      "description": "pink lego brick into the transparent box",
-      "skills": [
-        {
-          "name": "robot arm moves towards red lego brick",
-          "start": 0.0,
-          "end": 1.2
-        },
-        {
-          "name": "robot arm picks up red lego brick",
-          "start": 1.2,
-          "end": 2.0
-        },
-        {
-          "name": "robot arm moves towards transparent box",
-          "start": 2.0,
-          "end": 3.8
-        },
-        {
-          "name": "robot arm places red lego brick into transparent box",
-          "start": 3.8,
-          "end": 5.0
-        },
-        {
-          "name": "robot arm moves away from transparent box",
-          "start": 5.0,
-          "end": 8.9
-        }
-      ]
-    },
-
-notice how task_description: is a high-level description (e.g., "make a sandwich") stored in description for each episode
-
-For each sample, call Qwen VLM to generate:
-
-synthetic user prompt ℓ_t
-
-synthetic robot response u_t
-
-Save results to D_syn in Parquet format insdie DATA_PATH/meta/tasks.parquet ; note tasks.parquet already contains the other tasks, so you need to update
-
-Should be modular, clean, easy to extend, with:
-
-a PGEN_PROMPT_TEMPLATE
-
-a construct_prompt() method
-
-a call_qwen() method
-
-a annotate_sample() method
-
-a CLI entrypoint (if __name__ == "__main__":)
-
-📦 INPUT FORMAT (Dlabeled)
-
-The script should expect Dlabeled as a .jsonl file where each line has:
-
-{
-  "episode_id": "ep_001",
-  "t": 37,
-  "images": ["path/to/cam0_t.jpg", "path/to/cam1_t.jpg"],
-  "skill_current": "pick up the KitKat",
-  "skill_history": ["open fridge", "pick up lettuce", "place lettuce"],
-  "task_description": "making a sandwich"
-}
-
-📤 OUTPUT FORMAT (D_syn)
-
-Each line of synthetically generated data should be:
-
-{
-  "episode_id": "ep_001",
-  "t": 37,
-  "images": ["path/to/cam0_t.jpg", "path/to/cam1_t.jpg"],
-  "skill_current": "pick up the KitKat",
-  "skill_history": [...],
-  "user_prompt": "Can you grab me something sweet?",
-  "robot_utterance": "Sure, I can pick up the KitKat.",
-  "task_description": "making a sandwich"
-}
-
-
-Store as syn_annotations.jsonl. for debugging
-
-🧠 pgen MODEL (Qwen) REQUIREMENTS
-
-Use HuggingFace Transformers:
-
-Qwen/Qwen2-VL-7B-Instruct (or any Qwen2-VL Vision-Language model available)
-
-Use the image + text chat interface
-
-Vision inputs should be loaded with PIL
-
-Use a single forward pass that outputs BOTH ℓ_t and u_t in a structured JSON
-
-📝 PROMPT FORMAT FOR pgen
-
-Create a template like:
-
-You are a robot-assistant dialogue generator for hierarchical robot policies.
-
-You will receive:
-- A list of images showing the current robot scene.
-- The high-level task: {task_description}
-- Previous skill steps completed: {skill_history}
-- The next skill to be performed by the robot: {skill_current}
-
-Generate two things in JSON:
-1. "user_prompt": a natural-sounding user request that logically leads to the robot performing the skill "{skill_current}" given the task and history.
-2. "robot_utterance": a natural robot reply acknowledging or clarifying the request.
-
-The responses must be grounded in the visual scene, the task, and the skill history.
-
-Respond ONLY in JSON:
-{
-  "user_prompt": "...",
-  "robot_utterance": "..."
-}
-
-This resposne will have a corresponsing task_index, and the task will be saved in task.parqeut and you must update each dataset parquet in for example /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace/data/chunk-000/
-file-000.parquet to include this new feature called task_index_high_level consider udpatign the metadata in info.json as well
-📌 LOGIC REQUIRED
-construct_prompt(sample)
-
-Loads sample dict
-
-Inserts:
-
-task_description
-
-skill_history
-
-skill_current
-
-Returns a full text prompt string
-
-call_qwen(images, prompt)
-
-Loads images into Qwen-VL multimodal input format
-
-Calls model.generate
-
-Parses JSON output
-
-annotate_sample(sample)
-
-Builds prompt
-
-Calls Qwen
-
-Returns augmented sample with user_prompt + robot_utterance
-
-🚀 CLI Usage
-
-The script should run as:
-
-python annotate_pgen.py \
-  --output-dir PATH \
-  --model Qwen/Qwen2-VL-7B-Instruct \
-  --repo-id lerobot/svla_so101_pickplace \
-  --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-  --batch-size 1
-
-
-Include arguments via argparse.
-
-🔧 OTHER REQUIREMENTS
-
-Use tqdm for progress bars
-
-Log errors gracefully and continue
-
-Support GPU acceleration (device="cuda")
-
-Cache model loading so it's not reloaded every call
-
-Make the prompt deterministic but allow temperature parameter
-
-Add a flag --num-image-views-per-sample
-
-Add automatic JSON parsing with helpful error messages
-
-🎯 FINAL DELIVERABLE
-
-Cursor must now generate:
-A full Python file named annotate_pgen.py implementing the above functionality end-to-end.
-
-It should be production-ready, runnable on real data, cleanly structured, and easy to modify.
-
-
-from the paper:
-Next, we use a large vision-language model (VLM) pgen
-to produce synthetic user prompts and interjections ℓt,
-and corresponding robot utterance ut. Given Dlabeled, we
-prompt pgen with both the visual context I1
-t ,...,In
-t and the
-skill labelˆ
-ℓt (e.g., pick up the lettuce). pgen then imag-
-ines an appropriate interaction that might have led toˆ
-ℓt in a
-real user interaction: it generates possible user prompts ℓt
-(e.g., “Can you add some lettuce for me?”) along with the
-robot’s verbal responses and clarifications ut. We detail the
-A. Synthetic Data Generation
-A.1. Scenario and Response Categorization
-To ensure the quality and diversity of the synthetic data,
-we incorporate structured scenario classification and re-
-sponse categorization into the prompt design for pgen, fol-
-lowing (Stephan et al., 2024). Specifically, we classify
-interactions into different scenario types, such as nega-
-tive task (where the user instructs the robot what not to
-do), situated correction (where the user adjusts an earlier
-command based on the evolving task state), and specific
-constraint (where the user specifies particular constraints,
-such as dietary preferences). In addition, we categorize
-the robot’s responses into types such as simple confirma-
-tions, clarifications, and error handling. These classifica-
-tions guide the generation process to ensure a broad range
-of user-robot interactions.
-A.2. Prompt Construction for Contextual Grounding
-In prompt P, we include a detailed description of the task
-(e.g., bussing a table, making a sandwich, grocery shop-
-ping) and instruct the model to ground responses in visual
-observations and prior context. A key advantage of lever-
-aging large pretrained VLMs is their ability to incorporate
-world knowledge when generating interactions. For in-
-stance, the model can infer dietary constraints when gener-
-ating prompts for sandwich-making, producing user com-
-mands such as “Can you make a sandwich for me? I’m
-lactose intolerant” and an appropriate robot response like
-“Sure, I won’t put cheese on it.” Similarly, it can reason
-over ambiguous or implicit requests, such as inferring that
-“I want something sweet” in a grocery shopping scenario
-should lead to suggestions like chocolate or candy.
-To maintain consistency in multi-step tasks, we condition
-pgen on prior skill labels within an episodeˆ
-ˆ
-ℓ0,...,
-ℓt−1,
-allowing it to generate coherent user commands that
-account for past actions. For instance, if the robot
-has already placed lettuce and tomato on a sandwich,
-the generated user prompt might request additional in-
-gredients that logically follow. This ensures that the
-synthetic interactions reflect realistic task progression
-rather than isolated commands. As such, we leverage
-ˆ
-ˆ
-ˆ
-pgen(ℓt,ut|I1
-t ,...,In
-t ,
-ℓ0,...,
-ℓt−1,
-ℓt,P) to produce a richer,
-more diverse synthetic dataset Dsyn that provides mean-
-ingful supervision for training our high-level policy.
-While in this work we generate a separate Dsyn and train
-a separate high-level policy for each task (e.g., sandwich
-making vs. table cleaning) for clarity and ease of bench-
-marking, the architecture is readily amenable to a unified
-multi-task formulation. In principle, the same hierarchical
-approach could be used to train a single high-level policy
-across a multitude of tasks, facilitating knowledge transfer
-
-
-The result should be a new LeRobotDataset with a new feature called task_index_high_level inside each dataset parquet

examples/dataset/run.sh

@@ -1,11 +0,0 @@
-python examples/dataset/annotate.py \
-    --repo-id jadechoghari/collect-data \
-    --video-key observation.images.base \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --episodes 16 22
-
-# python examples/dataset/annotate.py \
-#     --repo-id lerobot/svla_so101_pickplace \
-#     --video-key observation.images.side \
-#     --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-#     --episodes 5

examples/dataset/run_pgen.sh

@@ -1,43 +0,0 @@
-#!/bin/bash
-
-# Example script to run synthetic data generation with Qwen VLM
-# This generates user prompts and robot utterances for hierarchical policy training
-
-# Configuration
-REPO_ID="jadechoghari/collect-data"
-MODEL="Qwen/Qwen3-VL-30B-A3B-Instruct"
-# Alternative: MODEL="Qwen/Qwen2-VL-7B-Instruct"
-
-
-OUTPUT_DIR="/fsx/jade_choghari/outputs/collect-data-pgen"
-BATCH_SIZE=32
-TEMPERATURE=0.9
-SAMPLE_INTERVAL=5.0  # Generate dialogue every 1 second (all episodes processed)
-
-# Run synthetic data generation (processes ALL episodes)
-python examples/dataset/annotate_pgen.py \
-    --repo-id "$REPO_ID" \
-    --model "$MODEL" \
-    --output-dir "$OUTPUT_DIR" \
-    --temperature "$TEMPERATURE" \
-    --batch-size "$BATCH_SIZE" \
-    --sample-interval "$SAMPLE_INTERVAL" \
-    --image-key observation.images.base \
-    --num-image-views-per-sample 1
-
-# For faster testing, increase sample interval:
-# --sample-interval 5.0  # Samples every 5 seconds (much faster)
-
-# To push to hub after generation:
-# Add --push-to-hub flag
-
-# Efficient batch processing: 4 episodes at once
-# python examples/dataset/annotate_pgen.py \
-#     --repo-id "$REPO_ID" \
-#     --model "$MODEL" \
-#     --output-dir "$OUTPUT_DIR" \
-#     --video-mode \
-#     --video-key observation.images.up \
-#     --video-batch-size "$BATCH_SIZE" \
-#     --sample-interval 1.0
-

examples/dataset/subtask_annotation.py

@@ -1,802 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-SARM Subtask Annotation using local GPU (Qwen3-VL).
-
-This script implements the annotation approach from the SARM paper using local GPU inference:
-"SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation"
-Paper: https://arxiv.org/pdf/2509.25358
-
-What it does:
-1. Takes videos from a LeRobot dataset
-2. Uses Qwen3-VL running locally on GPU to identify when subtasks occur
-3. Saves subtask timestamps to the dataset metadata
-4. Optionally pushes the annotated dataset to HuggingFace Hub
-
-SARM trains reward models that predict:
-  - Stage: Which subtask is currently being executed (discrete classification)
-  - Progress: How far along the subtask we are (continuous 0-1)
-
-Supports three annotation modes:
-  1. No annotations (no args): Auto-creates single sparse "task" stage covering full episode.
-     Use with SARM config annotation_mode="single_stage" for simple tasks.
-
-  2. Dense-only (--dense-only --dense-subtasks): Dense annotations from VLM, auto-generated
-     single sparse "task" stage. Use with annotation_mode="dense_only".
-
-  3. Dual mode (--sparse-subtasks + --dense-subtasks): Both sparse and dense annotations
-     from VLM. Use with annotation_mode="dual".
-
-Requirements:
-  - GPU with sufficient VRAM (16GB+ recommended for 30B model)
-  - `pip install transformers, torch, qwen-vl-utils`
-
-Run with:
-```bash
-python examples/dataset_annotation/subtask_annotation.py \
-  --repo-id your-username/your-dataset \
-  --sparse-subtasks "Do ..." \
-  --dense-subtasks "Do task 1, Do task 2, Do task 3" \
-  --video-key observation.images.base \
-  --push-to-hub
-```
-"""
-
-import argparse
-import json
-import multiprocessing as mp
-import re
-import subprocess
-import tempfile
-import textwrap
-import time
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-import cv2
-import pandas as pd
-import torch
-from qwen_vl_utils import process_vision_info
-from rich.console import Console
-from transformers import AutoProcessor, Qwen3VLMoeForConditionalGeneration
-
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-from lerobot.policies.sarm.sarm_utils import (
-    Subtask,
-    SubtaskAnnotation,
-    Timestamp,
-    compute_temporal_proportions,
-)
-
-
-def create_sarm_prompt(subtask_list: list[str]) -> str:
-    subtask_str = "\n".join([f"  - {name}" for name in subtask_list])
-
-    return textwrap.dedent(f"""\
-        # Role
-        You are a Robotics Vision System specializing in temporal action localization for robot manipulation. Your job is to segment a single demonstration video into distinct, non-overlapping atomic actions from a fixed subtask list.
-
-        # Subtask Label Set (Closed Vocabulary)
-        You must strictly identify the video segments using ONLY the following labels. Do not create new labels or modify existing ones:
-
-        [
-        {subtask_str}
-        ]
-
-        The video shows one successful execution of all subtasks in a logical order.
-
-        # Ground-Truth Semantics (Very Important)
-        Use **visual state changes** to define when a subtask starts and ends. Do NOT assume equal durations for the subtasks.
-
-        - A subtask **starts** at the first frame where the robot's motion clearly initiates that subtask.
-        - A subtask **ends** at the first frame where that specific action is visually completed and the manipulated object reaches a temporary, stable configuration.
-
-        If there are short pauses or micro-motions that don't clearly correspond to a new subtask, they belong to the **current** subtask.
-
-        # Hard Constraints & Logic
-        1. **Continuous Coverage (No Gaps):**
-           - The entire video duration from "00:00" to the final timestamp must be covered by subtasks.
-           - There can be no gaps between subtasks.
-           - If there is any idle or ambiguous time between clear actions, extend the *preceding* subtask to cover it.
-
-        2. **Boundary Consistency:**
-           - The `"end"` timestamp of one subtask must be exactly equal to the `"start"` timestamp of the next subtask.
-           - Boundaries must coincide with a real visual state transition, not just a convenient time split.
-
-        3. **Chronological Order, One Occurrence Each:**
-           - This is a single successful demonstration.
-           - Each subtask from the vocabulary appears **exactly once**, in the correct logical order.
-           - **Durations may be very different** between subtasks. Never assume they are similar lengths. Base all boundaries only on the video.
-
-        4. **Reject Uniform Segmentation (Important):**
-           - Do NOT simply divide the video into equal or nearly equal time chunks.
-           - If your boundaries would result in subtasks with similar durations (e.g. all around 5 seconds), treat this as evidence that your segmentation is wrong and refine the boundaries.
-           - Only use nearly equal durations if the video truly shows each subtask taking the same amount of time (this is very rare).
-
-        5. **Timestamps:**
-           - Timestamps must be in `"MM:SS"` format.
-           - The first subtask always starts at `"00:00"`.
-           - The last subtask ends at the final visible frame of the video.
-
-        # Step 1 — Textual Timeline (must do this first)
-        First, write a extensive and detailed textual timeline describing what happens in the video with approximate timestamps.
-        For each subtask, include:
-        - its name
-        - an approximate start and end time,
-        - an description of the visual event at the boundary (e.g. "shirt fully folded to the left", "robot rotates folded shirt 90 degrees").
-
-        Format this as a bullet list.
-
-        # Step 2 — JSON Output (final answer)
-        After the textual timeline, output **only** valid JSON with this structure.
-        The JSON **must** be consistent with the textual timeline above:
-
-        {{
-          "subtasks": [
-            {{
-              "name": "EXACT_NAME_FROM_LIST",
-              "timestamps": {{
-                "start": "MM:SS",
-                "end":   "MM:SS"
-              }}
-            }},
-            {{
-              "name": "EXACT_NAME_FROM_LIST",
-              "timestamps": {{
-                "start": "MM:SS",
-                "end":   "MM:SS"
-              }}
-            }}
-          ]
-        }}
-
-        Do not add any extra keys to the JSON.
-        """)
-
-
-class VideoAnnotator:
-    """Annotates robot manipulation videos using local Qwen3-VL model on GPU"""
-
-    def __init__(
-        self,
-        subtask_list: list[str],
-        model_name: str = "Qwen/Qwen3-VL-30B-A3B-Instruct",
-        device: str = "cuda",
-        torch_dtype: torch.dtype = torch.bfloat16,
-        model: "Qwen3VLMoeForConditionalGeneration | None" = None,
-        processor: "AutoProcessor | None" = None,
-    ):
-        """
-        Initialize the video annotator with local model.
-
-        Args:
-            subtask_list: List of allowed subtask names (for consistency)
-            model_name: Hugging Face model name (default: Qwen/Qwen3-VL-30B-A3B-Instruct)
-            device: Device to use (cuda, cpu)
-            torch_dtype: Data type for model (bfloat16, float16, float32)
-            model: Pre-loaded model instance (optional, to share between annotators)
-            processor: Pre-loaded processor instance (optional, to share between annotators)
-        """
-        self.subtask_list = subtask_list
-        self.prompt = create_sarm_prompt(subtask_list)
-        self.console = Console()
-        self.device = device
-
-        # Use provided model/processor or load new ones
-        if model is not None and processor is not None:
-            self.model = model
-            self.processor = processor
-            self.console.print(f"[green]✓ Using shared model on {device}[/green]")
-        else:
-            self.console.print(f"[cyan]Loading model: {model_name}...[/cyan]")
-
-            self.model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
-                model_name, torch_dtype=torch_dtype, device_map=device, trust_remote_code=True
-            )
-
-            self.processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
-
-            self.console.print(f"[green]✓ Model loaded successfully on {device}[/green]")
-
-    def extract_episode_segment(
-        self, file_path: Path, start_timestamp: float, end_timestamp: float, target_fps: int = 1
-    ) -> Path:
-        """
-        Extract a specific episode segment from concatenated video.
-        Uses minimal compression to preserve quality for local inference.
-
-        Args:
-            file_path: Path to the concatenated video file
-            start_timestamp: Starting timestamp in seconds (within this video file)
-            end_timestamp: Ending timestamp in seconds (within this video file)
-            target_fps: Target FPS (default: 1 for faster processing)
-
-        Returns:
-            Path to extracted video file
-        """
-        # Create temporary file for extracted video
-        tmp_file = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False)
-        tmp_path = Path(tmp_file.name)
-        tmp_file.close()
-
-        try:
-            # Check if ffmpeg is available
-            subprocess.run(
-                ["ffmpeg", "-version"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True
-            )
-        except (subprocess.CalledProcessError, FileNotFoundError):
-            raise RuntimeError("ffmpeg not found, cannot extract episode segment") from e
-
-        try:
-            # Calculate duration
-            duration = end_timestamp - start_timestamp
-
-            self.console.print(
-                f"[cyan]Extracting episode: {start_timestamp:.1f}s-{end_timestamp:.1f}s ({duration:.1f}s)[/cyan]"
-            )
-
-            # Use ffmpeg to extract segment with minimal quality loss
-            cmd = [
-                "ffmpeg",
-                "-i",
-                str(file_path),
-                "-ss",
-                str(start_timestamp),
-                "-t",
-                str(duration),
-                "-r",
-                str(target_fps),
-                "-c:v",
-                "libx264",
-                "-preset",
-                "ultrafast",
-                "-crf",
-                "23",
-                "-an",
-                "-y",
-                str(tmp_path),
-            ]
-
-            subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True)
-
-            # Verify the output file was created and is not empty
-            if not tmp_path.exists() or tmp_path.stat().st_size == 0:
-                self.console.print("[red]✗ Video extraction failed (0 bytes) - skipping episode[/red]")
-                if tmp_path.exists():
-                    tmp_path.unlink()
-                raise RuntimeError("FFmpeg produced empty video file")
-
-            # Show extraction results
-            file_size_mb = tmp_path.stat().st_size / (1024 * 1024)
-
-            # Fail if file is too small (< 100KB likely means extraction failed)
-            if file_size_mb < 0.1:
-                self.console.print(
-                    f"[red]✗ Extracted video too small ({file_size_mb:.2f}MB) - skipping episode[/red]"
-                )
-                tmp_path.unlink()
-                raise RuntimeError(f"Video extraction produced invalid file ({file_size_mb:.2f}MB)")
-
-            self.console.print(f"[green]✓ Extracted: {file_size_mb:.1f}MB ({target_fps} FPS)[/green]")
-
-            return tmp_path
-
-        except subprocess.CalledProcessError as e:
-            raise RuntimeError(f"ffmpeg failed ({e})") from e
-
-    def annotate(
-        self,
-        file_path: str | Path,
-        fps: int,
-        start_timestamp: float = 0.0,
-        end_timestamp: float | None = None,
-        max_retries: int = 3,
-    ) -> SubtaskAnnotation:
-        """Annotate a video segment using local GPU."""
-        file_path = Path(file_path)
-
-        if end_timestamp is None:
-            cap = cv2.VideoCapture(str(file_path))
-            end_timestamp = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) / (cap.get(cv2.CAP_PROP_FPS) or 1)
-            cap.release()
-
-        duration = end_timestamp - start_timestamp
-        duration_str = f"{int(duration // 60):02d}:{int(duration % 60):02d}"
-
-        extracted_path = self.extract_episode_segment(file_path, start_timestamp, end_timestamp, 1)
-        is_extracted = extracted_path != file_path
-
-        try:
-            messages = [
-                {"role": "system", "content": [{"type": "text", "text": self.prompt}]},
-                {
-                    "role": "user",
-                    "content": [
-                        {"type": "video", "video": str(extracted_path), "fps": 1.0},
-                        {
-                            "type": "text",
-                            "text": f"Video is {duration_str} (~{duration:.1f}s). Follow instructions.",
-                        },
-                    ],
-                },
-            ]
-
-            for attempt in range(max_retries):
-                try:
-                    text = self.processor.apply_chat_template(
-                        messages, tokenize=False, add_generation_prompt=True
-                    )
-                    image_inputs, video_inputs = process_vision_info(messages)
-                    inputs = self.processor(
-                        text=[text],
-                        images=image_inputs,
-                        videos=video_inputs,
-                        padding=True,
-                        return_tensors="pt",
-                    ).to(self.device)
-
-                    with torch.no_grad():
-                        generated_ids = self.model.generate(
-                            **inputs, max_new_tokens=1024, do_sample=True, temperature=0.7
-                        )
-
-                    response = self.processor.batch_decode(
-                        [out[len(inp) :] for inp, out in zip(inputs.input_ids, generated_ids)],
-                        skip_special_tokens=True,
-                    )[0].strip()
-
-                    # Extract JSON
-                    if "```json" in response:
-                        response = response.split("```json")[1].split("```")[0]
-                    elif "```" in response:
-                        response = response.split("```")[1].split("```")[0]
-
-                    try:
-                        return SubtaskAnnotation.model_validate(json.loads(response))
-                    except json.JSONDecodeError:
-                        match = re.search(r"\{.*\}", response, re.DOTALL)
-                        if match:
-                            return SubtaskAnnotation.model_validate(json.loads(match.group()))
-                        raise ValueError("No JSON found")
-                except Exception as e:
-                    if attempt == max_retries - 1:
-                        raise RuntimeError(f"Failed after {max_retries} attempts") from e
-                    time.sleep(1)
-        finally:
-            if is_extracted and extracted_path.exists():
-                extracted_path.unlink()
-
-
-def display_annotation(
-    annotation: SubtaskAnnotation, console: Console, episode_idx: int, fps: int, prefix: str = ""
-):
-    """Display annotation summary."""
-    subtask_summary = ", ".join(
-        f"{s.name}({s.timestamps.start}-{s.timestamps.end})" for s in annotation.subtasks
-    )
-    console.print(
-        f"[green]Episode {episode_idx} {prefix}: {len(annotation.subtasks)} subtasks - {subtask_summary}[/green]"
-    )
-
-
-def timestamp_to_seconds(timestamp: str) -> float:
-    """Convert MM:SS or SS timestamp to seconds"""
-    parts = timestamp.split(":")
-    if len(parts) == 2:
-        return int(parts[0]) * 60 + int(parts[1])
-    else:
-        return int(parts[0])
-
-
-def save_annotations_to_dataset(
-    dataset_path: Path, annotations: dict[int, SubtaskAnnotation], fps: int, prefix: str = "sparse"
-):
-    """Save annotations to LeRobot dataset parquet format."""
-    from lerobot.datasets.utils import DEFAULT_EPISODES_PATH, load_episodes
-
-    episodes_dataset = load_episodes(dataset_path)
-    if not episodes_dataset or len(episodes_dataset) == 0:
-        return
-
-    episodes_df = episodes_dataset.to_pandas()
-    cols = [
-        f"{prefix}_{c}"
-        for c in [
-            "subtask_names",
-            "subtask_start_times",
-            "subtask_end_times",
-            "subtask_start_frames",
-            "subtask_end_frames",
-        ]
-    ]
-    for col in cols:
-        episodes_df[col] = None
-
-    for ep_idx, ann in annotations.items():
-        if ep_idx >= len(episodes_df):
-            continue
-        names, starts, ends, start_frames, end_frames = [], [], [], [], []
-        for s in ann.subtasks:
-            names.append(s.name)
-            st, et = timestamp_to_seconds(s.timestamps.start), timestamp_to_seconds(s.timestamps.end)
-            starts.append(st)
-            ends.append(et)
-            start_frames.append(int(st * fps))
-            end_frames.append(int(et * fps))
-        episodes_df.at[ep_idx, cols[0]] = names
-        episodes_df.at[ep_idx, cols[1]] = starts
-        episodes_df.at[ep_idx, cols[2]] = ends
-        episodes_df.at[ep_idx, cols[3]] = start_frames
-        episodes_df.at[ep_idx, cols[4]] = end_frames
-
-    # Group by file and write
-    for ep_idx in episodes_df.index:
-        key = (
-            episodes_df.loc[ep_idx, "meta/episodes/chunk_index"],
-            episodes_df.loc[ep_idx, "meta/episodes/file_index"],
-        )
-        path = dataset_path / DEFAULT_EPISODES_PATH.format(chunk_index=key[0], file_index=key[1])
-        if path.exists():
-            file_df = pd.read_parquet(path)
-            for col in cols + (
-                [
-                    "subtask_names",
-                    "subtask_start_times",
-                    "subtask_end_times",
-                    "subtask_start_frames",
-                    "subtask_end_frames",
-                ]
-                if prefix == "sparse"
-                else []
-            ):
-                if col not in file_df.columns:
-                    file_df[col] = None
-            if ep_idx in annotations:
-                for col in cols:
-                    file_df.at[ep_idx, col] = episodes_df.loc[ep_idx, col]
-                if prefix == "sparse":  # Legacy columns
-                    for i, legacy in enumerate(
-                        [
-                            "subtask_names",
-                            "subtask_start_times",
-                            "subtask_end_times",
-                            "subtask_start_frames",
-                            "subtask_end_frames",
-                        ]
-                    ):
-                        file_df.at[ep_idx, legacy] = episodes_df.loc[ep_idx, cols[i]]
-            file_df.to_parquet(path, engine="pyarrow", compression="snappy")
-
-
-def generate_auto_sparse_annotations(
-    dataset: LeRobotDataset, episode_indices: list[int], video_key: str
-) -> dict[int, SubtaskAnnotation]:
-    """Auto-generate single 'task' stage annotations for all episodes."""
-    annotations = {}
-    for ep_idx in episode_indices:
-        start = float(dataset.meta.episodes[f"videos/{video_key}/from_timestamp"][ep_idx])
-        end = float(dataset.meta.episodes[f"videos/{video_key}/to_timestamp"][ep_idx])
-        duration = end - start
-        end_str = f"{int(duration // 60):02d}:{int(duration % 60):02d}"
-        annotations[ep_idx] = SubtaskAnnotation(
-            subtasks=[Subtask(name="task", timestamps=Timestamp(start="00:00", end=end_str))]
-        )
-    return annotations
-
-
-def load_annotations_from_dataset(dataset_path: Path, prefix: str = "sparse") -> dict[int, SubtaskAnnotation]:
-    """Load annotations from LeRobot dataset parquet files."""
-    from lerobot.datasets.utils import load_episodes
-
-    episodes_dataset = load_episodes(dataset_path)
-    if not episodes_dataset or len(episodes_dataset) == 0:
-        return {}
-
-    col_names = f"{prefix}_subtask_names"
-    col_start = f"{prefix}_subtask_start_times"
-    col_end = f"{prefix}_subtask_end_times"
-
-    # Fall back to legacy columns for sparse
-    if col_names not in episodes_dataset.column_names:
-        if prefix == "sparse" and "subtask_names" in episodes_dataset.column_names:
-            col_names, col_start, col_end = "subtask_names", "subtask_start_times", "subtask_end_times"
-        else:
-            return {}
-
-    df = episodes_dataset.to_pandas()
-    annotations = {}
-    for ep_idx in df.index:
-        names = df.loc[ep_idx, col_names]
-        if names is None or (isinstance(names, float) and pd.isna(names)):
-            continue
-        starts, ends = df.loc[ep_idx, col_start], df.loc[ep_idx, col_end]
-        annotations[int(ep_idx)] = SubtaskAnnotation(
-            subtasks=[
-                Subtask(
-                    name=n,
-                    timestamps=Timestamp(
-                        start=f"{int(s) // 60:02d}:{int(s) % 60:02d}",
-                        end=f"{int(e) // 60:02d}:{int(e) % 60:02d}",
-                    ),
-                )
-                for n, s, e in zip(names, starts, ends)
-            ]
-        )
-    return annotations
-
-
-def process_single_episode(
-    ep_idx: int,
-    dataset_root: Path,
-    dataset_meta,
-    video_key: str,
-    fps: int,
-    annotator: VideoAnnotator,
-    console: Console,
-) -> tuple[int, SubtaskAnnotation | None, str | None]:
-    """Process a single episode annotation."""
-    try:
-        video_path = dataset_root / dataset_meta.get_video_file_path(ep_idx, video_key)
-        if not video_path.exists():
-            return ep_idx, None, f"Video not found: {video_path}"
-
-        start = float(dataset_meta.episodes[f"videos/{video_key}/from_timestamp"][ep_idx])
-        end = float(dataset_meta.episodes[f"videos/{video_key}/to_timestamp"][ep_idx])
-        return ep_idx, annotator.annotate(video_path, fps, start, end), None
-    except Exception as e:
-        return ep_idx, None, str(e)
-
-
-def worker_process_episodes(
-    worker_id: int,
-    gpu_id: int,
-    episode_indices: list[int],
-    repo_id: str,
-    video_key: str,
-    sparse_subtask_list: list[str],
-    dense_subtask_list: list[str] | None,
-    model_name: str,
-    torch_dtype: torch.dtype,
-) -> tuple[dict, dict | None]:
-    """Worker for parallel processing across GPUs."""
-    device = f"cuda:{gpu_id}"
-    console = Console()
-    dataset = LeRobotDataset(repo_id, download_videos=False)
-
-    sparse_annotator = VideoAnnotator(sparse_subtask_list, model_name, device, torch_dtype)
-    dense_annotator = (
-        VideoAnnotator(
-            dense_subtask_list,
-            model_name,
-            device,
-            torch_dtype,
-            sparse_annotator.model,
-            sparse_annotator.processor,
-        )
-        if dense_subtask_list
-        else None
-    )
-
-    sparse_annotations, dense_annotations = {}, {} if dense_subtask_list else None
-
-    for ep_idx in episode_indices:
-        _, sparse_ann, err = process_single_episode(
-            ep_idx, dataset.root, dataset.meta, video_key, dataset.fps, sparse_annotator, console
-        )
-        if sparse_ann:
-            sparse_annotations[ep_idx] = sparse_ann
-
-        if dense_annotator:
-            _, dense_ann, _ = process_single_episode(
-                ep_idx, dataset.root, dataset.meta, video_key, dataset.fps, dense_annotator, console
-            )
-            if dense_ann:
-                dense_annotations[ep_idx] = dense_ann
-
-    return sparse_annotations, dense_annotations
-
-
-def main():
-    parser = argparse.ArgumentParser(description="SARM-style subtask annotation using local GPU (Qwen3-VL)")
-    parser.add_argument("--repo-id", type=str, required=True, help="HuggingFace dataset repository ID")
-    parser.add_argument(
-        "--sparse-subtasks", type=str, default=None, help="Comma-separated sparse subtask names"
-    )
-    parser.add_argument(
-        "--dense-subtasks", type=str, default=None, help="Comma-separated dense subtask names"
-    )
-    parser.add_argument(
-        "--dense-only", action="store_true", help="Dense-only mode with auto-generated sparse 'task' stage"
-    )
-    parser.add_argument("--episodes", type=int, nargs="+", default=None, help="Episode indices to annotate")
-    parser.add_argument("--model", type=str, default="Qwen/Qwen3-VL-30B-A3B-Instruct", help="VLM model")
-    parser.add_argument("--skip-existing", action="store_true", help="Skip already annotated episodes")
-    parser.add_argument("--video-key", type=str, default=None, help="Video key (default: first available)")
-    parser.add_argument("--push-to-hub", action="store_true", help="Push to HuggingFace Hub")
-    parser.add_argument("--output-repo-id", type=str, default=None, help="Output repo ID for push")
-    parser.add_argument("--device", type=str, default="cuda", help="Device (cuda/cpu)")
-    parser.add_argument("--dtype", type=str, default="bfloat16", choices=["bfloat16", "float16", "float32"])
-    parser.add_argument("--num-workers", type=int, default=1, help="Parallel workers for multi-GPU")
-    parser.add_argument("--gpu-ids", type=int, nargs="+", default=None, help="GPU IDs to use")
-
-    args = parser.parse_args()
-    console = Console()
-
-    # Validate arguments
-    if args.dense_only and not args.dense_subtasks:
-        return console.print("[red]Error: --dense-only requires --dense-subtasks[/red]")
-    if args.dense_subtasks and not args.sparse_subtasks and not args.dense_only:
-        return console.print("[red]Error: --dense-subtasks requires --sparse-subtasks or --dense-only[/red]")
-
-    sparse_subtask_list = (
-        [s.strip() for s in args.sparse_subtasks.split(",")] if args.sparse_subtasks else None
-    )
-    dense_subtask_list = [s.strip() for s in args.dense_subtasks.split(",")] if args.dense_subtasks else None
-    auto_sparse = sparse_subtask_list is None
-    dense_mode = dense_subtask_list is not None
-    torch_dtype = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}[args.dtype]
-
-    console.print(f"[cyan]Loading dataset: {args.repo_id}[/cyan]")
-    dataset = LeRobotDataset(args.repo_id, download_videos=True)
-    fps = dataset.fps
-
-    if not dataset.meta.video_keys:
-        raise ValueError("No video keys found")
-
-    video_key = (
-        args.video_key if args.video_key in (dataset.meta.video_keys or []) else dataset.meta.video_keys[0]
-    )
-    console.print(f"[cyan]Using camera: {video_key}, FPS: {fps}[/cyan]")
-
-    # Determine episodes
-    episode_indices = args.episodes or list(range(dataset.meta.total_episodes))
-
-    existing_annotations = load_annotations_from_dataset(dataset.root, prefix="sparse")
-    if args.skip_existing:
-        episode_indices = [ep for ep in episode_indices if ep not in existing_annotations]
-
-    if not episode_indices:
-        return console.print("[green]All episodes already annotated![/green]")
-    console.print(f"[cyan]Annotating {len(episode_indices)} episodes[/cyan]")
-
-    # GPU setup
-    gpu_ids = args.gpu_ids or list(
-        range(min(args.num_workers, torch.cuda.device_count() if torch.cuda.is_available() else 1))
-    )
-    args.num_workers = len(gpu_ids)
-
-    sparse_annotations = existing_annotations.copy()
-    dense_annotations = {} if dense_mode else None
-
-    # Auto-sparse mode
-    if auto_sparse:
-        sparse_annotations.update(generate_auto_sparse_annotations(dataset, episode_indices, video_key))
-        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-        console.print(f"[green]Auto-generated {len(episode_indices)} sparse 'task' annotations[/green]")
-
-    # VLM annotation (for sparse if not auto, and for dense)
-    need_vlm = (not auto_sparse) or dense_mode
-
-    if need_vlm:
-        if args.num_workers > 1 and not auto_sparse:
-            # Parallel processing
-            console.print(f"[cyan]Parallel processing with {args.num_workers} workers[/cyan]")
-            episodes_per_worker = [[] for _ in range(args.num_workers)]
-            for i, ep_idx in enumerate(episode_indices):
-                episodes_per_worker[i % args.num_workers].append(ep_idx)
-
-            with ProcessPoolExecutor(
-                max_workers=args.num_workers, mp_context=mp.get_context("spawn")
-            ) as executor:
-                futures = [
-                    executor.submit(
-                        worker_process_episodes,
-                        w,
-                        gpu_ids[w],
-                        episodes_per_worker[w],
-                        args.repo_id,
-                        video_key,
-                        sparse_subtask_list,
-                        dense_subtask_list,
-                        args.model,
-                        torch_dtype,
-                    )
-                    for w in range(args.num_workers)
-                    if episodes_per_worker[w]
-                ]
-
-                for future in as_completed(futures):
-                    try:
-                        worker_sparse, worker_dense = future.result()
-                        sparse_annotations.update(worker_sparse)
-                        if dense_mode and worker_dense:
-                            dense_annotations.update(worker_dense)
-                        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-                        if dense_mode:
-                            save_annotations_to_dataset(dataset.root, dense_annotations, fps, prefix="dense")
-                    except Exception as e:
-                        raise RuntimeError(f"Worker failed: {e}") from e
-        else:
-            # Sequential processing
-            sparse_annotator = (
-                VideoAnnotator(sparse_subtask_list, args.model, args.device, torch_dtype)
-                if not auto_sparse and sparse_subtask_list
-                else None
-            )
-            dense_annotator = (
-                VideoAnnotator(
-                    dense_subtask_list,
-                    args.model,
-                    args.device,
-                    torch_dtype,
-                    sparse_annotator.model if sparse_annotator else None,
-                    sparse_annotator.processor if sparse_annotator else None,
-                )
-                if dense_mode
-                else None
-            )
-
-            for i, ep_idx in enumerate(episode_indices):
-                console.print(f"[cyan]Episode {ep_idx} ({i + 1}/{len(episode_indices)})[/cyan]")
-
-                if sparse_annotator:
-                    _, sparse_ann, err = process_single_episode(
-                        ep_idx, dataset.root, dataset.meta, video_key, fps, sparse_annotator, console
-                    )
-                    if sparse_ann:
-                        sparse_annotations[ep_idx] = sparse_ann
-                        save_annotations_to_dataset(dataset.root, sparse_annotations, fps, prefix="sparse")
-                    elif err:
-                        console.print(f"[red]Sparse failed: {err}[/red]")
-
-                if dense_annotator:
-                    _, dense_ann, err = process_single_episode(
-                        ep_idx, dataset.root, dataset.meta, video_key, fps, dense_annotator, console
-                    )
-                    if dense_ann:
-                        dense_annotations[ep_idx] = dense_ann
-                        save_annotations_to_dataset(dataset.root, dense_annotations, fps, prefix="dense")
-                    elif err:
-                        console.print(f"[red]Dense failed: {err}[/red]")
-
-    # Save temporal proportions
-    def save_proportions(annotations, prefix, is_auto=False):
-        props: dict[str, float] = {"task": 1.0} if is_auto else compute_temporal_proportions(annotations, fps)
-        path = dataset.root / "meta" / f"temporal_proportions_{prefix}.json"
-        path.parent.mkdir(parents=True, exist_ok=True)
-        with open(path, "w") as f:
-            json.dump(props, f, indent=2)
-        console.print(f"[green]Saved {prefix} temporal proportions[/green]")
-
-    save_proportions(sparse_annotations, "sparse", auto_sparse)
-    if dense_mode and dense_annotations:
-        save_proportions(dense_annotations, "dense")
-
-    console.print(
-        f"\n[bold green]Complete! {len(sparse_annotations)} sparse, {len(dense_annotations or {})} dense annotations[/bold green]"
-    )
-
-    if args.push_to_hub:
-        try:
-            dataset.push_to_hub(push_videos=True)
-            console.print(f"[green]Pushed to {args.output_repo_id or args.repo_id}[/green]")
-        except Exception as e:
-            console.print(f"[red]Push failed: {e}[/red]")
-
-
-if __name__ == "__main__":
-    main()

examples/dataset/test.txt

@@ -1 +0,0 @@
-srun --time 12:00:00     --qos=high     --gres=gpu:1     --mem=24G     --partition=hopper-prod     --container-image /fsx/michel_aractingi/docker_images/huggingface+lerobot-gpu+dev.sqsh     --container-mounts /fsx/jade_choghari   

examples/dataset/test_pgen_quick.sh

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-# Quick test to verify the fix for task_indices length mismatch
-# This should now work correctly even with --num-samples < full dataset length
-
-echo "Testing annotate_pgen.py with --num-samples=100 on full dataset..."
-
-python examples/dataset/annotate_pgen.py \
-    --data-dir /fsx/jade_choghari/.cache/huggingface/lerobot/lerobot/svla_so101_pickplace \
-    --model Qwen/Qwen3-VL-30B-A3B-Instruct \
-    --num-samples 100 \
-    --sample-interval 1.0 \
-    --output-dir /fsx/jade_choghari/outputs/pgen_test_fixed
-
-if [ $? -eq 0 ]; then
-    echo "✓ SUCCESS: Script completed without errors!"
-    echo ""
-    echo "Verifying output..."
-    
-    # Check that all frames have task_index_high_level
-    python -c "
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-import numpy as np
-
-ds = LeRobotDataset(repo_id='local_test', root='/fsx/jade_choghari/outputs/pgen_test_fixed')
-print(f'Dataset has {len(ds)} frames')
-print(f'Features: {list(ds.features.keys())}')
-
-# Check that task_index_high_level exists
-assert 'task_index_high_level' in ds.features, 'task_index_high_level not in features!'
-
-# Sample some frames
-for idx in [0, 50, 99, 100, 500, 1000, 11938]:
-    if idx < len(ds):
-        frame = ds[idx]
-        task_idx = frame['task_index_high_level'].item()
-        print(f'Frame {idx}: task_index_high_level = {task_idx}')
-
-print('✓ All checks passed!')
-"
-else
-    echo "✗ FAILED: Script exited with error code $?"
-fi
-

examples/unitree_g1/gr00t_locomotion.py

@@ -1,347 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""
-Example: GR00T Locomotion with Pre-loaded Policies
-
-This example demonstrates the NEW pattern for loading GR00T policies externally
-and passing them to the robot class.
-"""
-
-import argparse
-import logging
-import threading
-import time
-from collections import deque
-
-import numpy as np
-import onnxruntime as ort
-from huggingface_hub import hf_hub_download
-
-from lerobot.robots.unitree_g1.config_unitree_g1 import UnitreeG1Config
-from lerobot.robots.unitree_g1.unitree_g1 import UnitreeG1
-
-logger = logging.getLogger(__name__)
-
-GROOT_DEFAULT_ANGLES = np.zeros(29, dtype=np.float32)
-GROOT_DEFAULT_ANGLES[[0, 6]] = -0.1  # hip pitch
-GROOT_DEFAULT_ANGLES[[3, 9]] = 0.3  # knee
-GROOT_DEFAULT_ANGLES[[4, 10]] = -0.2  # ankle pitch
-
-MISSING_JOINTS = []
-G1_MODEL = "g1_23"  # or "g1_29"
-if G1_MODEL == "g1_23":
-    MISSING_JOINTS = [12, 14, 20, 21, 27, 28]  # waist yaw/pitch, wrist pitch/yaw
-
-LOCOMOTION_ACTION_SCALE = 0.25
-
-LOCOMOTION_CONTROL_DT = 0.02
-
-ANG_VEL_SCALE: float = 0.25
-DOF_POS_SCALE: float = 1.0
-DOF_VEL_SCALE: float = 0.05
-CMD_SCALE: list = [2.0, 2.0, 0.25]
-
-
-DEFAULT_GROOT_REPO_ID = "nepyope/GR00T-WholeBodyControl_g1"
-
-
-def load_groot_policies(
-    repo_id: str = DEFAULT_GROOT_REPO_ID,
-) -> tuple[ort.InferenceSession, ort.InferenceSession]:
-    """Load GR00T dual-policy system (Balance + Walk) from Hugging Face Hub.
-
-    Args:
-        repo_id: Hugging Face Hub repository ID containing the ONNX policies.
-    """
-    logger.info(f"Loading GR00T dual-policy system from Hugging Face Hub ({repo_id})...")
-
-    # Download ONNX policies from Hugging Face Hub
-    balance_path = hf_hub_download(
-        repo_id=repo_id,
-        filename="GR00T-WholeBodyControl-Balance.onnx",
-    )
-    walk_path = hf_hub_download(
-        repo_id=repo_id,
-        filename="GR00T-WholeBodyControl-Walk.onnx",
-    )
-
-    # Load ONNX policies
-    policy_balance = ort.InferenceSession(balance_path)
-    policy_walk = ort.InferenceSession(walk_path)
-
-    logger.info("GR00T policies loaded successfully")
-
-    return policy_balance, policy_walk
-
-
-class GrootLocomotionController:
-    """
-    Handles GR00T-style locomotion control for the Unitree G1 robot.
-
-    This controller manages:
-    - Dual-policy system (Balance + Walk)
-    - 29-joint observation processing
-    - 15D action output (legs + waist)
-    - Policy inference and motor command generation
-    """
-
-    def __init__(self, policy_balance, policy_walk, robot, config):
-        self.policy_balance = policy_balance
-        self.policy_walk = policy_walk
-        self.robot = robot
-        self.config = config
-
-        self.locomotion_cmd = np.array([0.0, 0.0, 0.0], dtype=np.float32)  # vx, vy, theta_dot
-
-        # GR00T-specific state
-        self.groot_qj_all = np.zeros(29, dtype=np.float32)
-        self.groot_dqj_all = np.zeros(29, dtype=np.float32)
-        self.groot_action = np.zeros(15, dtype=np.float32)
-        self.groot_obs_single = np.zeros(86, dtype=np.float32)
-        self.groot_obs_history = deque(maxlen=6)
-        self.groot_obs_stacked = np.zeros(516, dtype=np.float32)
-        self.groot_height_cmd = 0.74  # Default base height
-        self.groot_orientation_cmd = np.array([0.0, 0.0, 0.0], dtype=np.float32)
-
-        # input to gr00t is 6 frames (6*86D=516)
-        for _ in range(6):
-            self.groot_obs_history.append(np.zeros(86, dtype=np.float32))
-
-        # Thread management
-        self.locomotion_running = False
-        self.locomotion_thread = None
-
-        logger.info("GrootLocomotionController initialized")
-
-    def groot_locomotion_run(self):
-        # get current observation
-        robot_state = self.robot.get_observation()
-
-        if robot_state is None:
-            return
-
-        # get command from remote controller
-        if robot_state.wireless_remote is not None:
-            self.robot.remote_controller.set(robot_state.wireless_remote)
-            if self.robot.remote_controller.button[0]:  # R1 - raise waist
-                self.groot_height_cmd += 0.001
-                self.groot_height_cmd = np.clip(self.groot_height_cmd, 0.50, 1.00)
-            if self.robot.remote_controller.button[4]:  # R2 - lower waist
-                self.groot_height_cmd -= 0.001
-                self.groot_height_cmd = np.clip(self.groot_height_cmd, 0.50, 1.00)
-        else:
-            self.robot.remote_controller.lx = 0.0
-            self.robot.remote_controller.ly = 0.0
-            self.robot.remote_controller.rx = 0.0
-            self.robot.remote_controller.ry = 0.0
-
-        self.locomotion_cmd[0] = self.robot.remote_controller.ly  # forward/backward
-        self.locomotion_cmd[1] = self.robot.remote_controller.lx * -1  # left/right
-        self.locomotion_cmd[2] = self.robot.remote_controller.rx * -1  # rotation rate
-
-        for i in range(29):
-            self.groot_qj_all[i] = robot_state.motor_state[i].q
-            self.groot_dqj_all[i] = robot_state.motor_state[i].dq
-
-        # adapt observation for g1_23dof
-        for idx in MISSING_JOINTS:
-            self.groot_qj_all[idx] = 0.0
-            self.groot_dqj_all[idx] = 0.0
-
-        # Scale joint positions and velocities
-        qj_obs = self.groot_qj_all.copy()
-        dqj_obs = self.groot_dqj_all.copy()
-
-        # express imu data in gravity frame of reference
-        quat = robot_state.imu_state.quaternion
-        ang_vel = np.array(robot_state.imu_state.gyroscope, dtype=np.float32)
-        gravity_orientation = self.robot.get_gravity_orientation(quat)
-
-        # scale joint positions and velocities before policy inference
-        qj_obs = (qj_obs - GROOT_DEFAULT_ANGLES) * DOF_POS_SCALE
-        dqj_obs = dqj_obs * DOF_VEL_SCALE
-        ang_vel_scaled = ang_vel * ANG_VEL_SCALE
-
-        # build single frame observation
-        self.groot_obs_single[:3] = self.locomotion_cmd * np.array(CMD_SCALE)
-        self.groot_obs_single[3] = self.groot_height_cmd
-        self.groot_obs_single[4:7] = self.groot_orientation_cmd
-        self.groot_obs_single[7:10] = ang_vel_scaled
-        self.groot_obs_single[10:13] = gravity_orientation
-        self.groot_obs_single[13:42] = qj_obs
-        self.groot_obs_single[42:71] = dqj_obs
-        self.groot_obs_single[71:86] = self.groot_action  # 15D previous actions
-
-        # Add to history and stack observations (6 frames × 86D = 516D)
-        self.groot_obs_history.append(self.groot_obs_single.copy())
-
-        # Stack all 6 frames into 516D vector
-        for i, obs_frame in enumerate(self.groot_obs_history):
-            start_idx = i * 86
-            end_idx = start_idx + 86
-            self.groot_obs_stacked[start_idx:end_idx] = obs_frame
-
-        # Run policy inference (ONNX) with 516D stacked observation
-
-        cmd_magnitude = np.linalg.norm(self.locomotion_cmd)
-
-        selected_policy = (
-            self.policy_balance if cmd_magnitude < 0.05 else self.policy_walk
-        )  # balance/standing policy for small commands, walking policy for movement commands
-
-        # run policy inference
-        ort_inputs = {selected_policy.get_inputs()[0].name: np.expand_dims(self.groot_obs_stacked, axis=0)}
-        ort_outs = selected_policy.run(None, ort_inputs)
-        self.groot_action = ort_outs[0].squeeze()
-
-        # transform action back to target joint positions
-        target_dof_pos_15 = GROOT_DEFAULT_ANGLES[:15] + self.groot_action * LOCOMOTION_ACTION_SCALE
-
-        # command motors
-        for i in range(15):
-            motor_idx = i
-            self.robot.msg.motor_cmd[motor_idx].q = target_dof_pos_15[i]
-            self.robot.msg.motor_cmd[motor_idx].qd = 0
-            self.robot.msg.motor_cmd[motor_idx].kp = self.robot.kp[motor_idx]
-            self.robot.msg.motor_cmd[motor_idx].kd = self.robot.kd[motor_idx]
-            self.robot.msg.motor_cmd[motor_idx].tau = 0
-
-        # adapt action for g1_23dof
-        for joint_idx in MISSING_JOINTS:
-            self.robot.msg.motor_cmd[joint_idx].q = 0.0
-            self.robot.msg.motor_cmd[joint_idx].qd = 0
-            self.robot.msg.motor_cmd[joint_idx].kp = self.robot.kp[joint_idx]
-            self.robot.msg.motor_cmd[joint_idx].kd = self.robot.kd[joint_idx]
-            self.robot.msg.motor_cmd[joint_idx].tau = 0
-
-        # send action to robot
-        self.robot.send_action(self.robot.msg)
-
-    def _locomotion_thread_loop(self):
-        """Background thread that runs the locomotion policy at specified rate."""
-        logger.info("Locomotion thread started")
-        while self.locomotion_running:
-            start_time = time.time()
-            try:
-                self.groot_locomotion_run()
-            except Exception as e:
-                logger.error(f"Error in locomotion loop: {e}")
-
-            # Sleep to maintain control rate
-            elapsed = time.time() - start_time
-            sleep_time = max(0, LOCOMOTION_CONTROL_DT - elapsed)
-            time.sleep(sleep_time)
-        logger.info("Locomotion thread stopped")
-
-    def start_locomotion_thread(self):
-        if self.locomotion_running:
-            logger.warning("Locomotion thread already running")
-            return
-
-        logger.info("Starting locomotion control thread...")
-        self.locomotion_running = True
-        self.locomotion_thread = threading.Thread(target=self._locomotion_thread_loop, daemon=True)
-        self.locomotion_thread.start()
-
-        logger.info("Locomotion control thread started!")
-
-    def stop_locomotion_thread(self):
-        if not self.locomotion_running:
-            return
-
-        logger.info("Stopping locomotion control thread...")
-        self.locomotion_running = False
-        if self.locomotion_thread:
-            self.locomotion_thread.join(timeout=2.0)
-        logger.info("Locomotion control thread stopped")
-
-    def reset_robot(self):
-        """Move robot legs to default standing position over 2 seconds (arms are not moved)."""
-        total_time = 3.0
-        num_step = int(total_time / self.robot.control_dt)
-
-        # Only control legs, not arms (first 12 joints)
-        default_pos = GROOT_DEFAULT_ANGLES  # First 12 values are leg angles
-        dof_size = len(default_pos)
-
-        # Get current lowstate
-        robot_state = self.robot.get_observation()
-
-        # Record the current leg positions
-        init_dof_pos = np.zeros(dof_size, dtype=np.float32)
-        for i in range(dof_size):
-            init_dof_pos[i] = robot_state.motor_state[i].q
-
-        # Move legs to default pos
-        for i in range(num_step):
-            alpha = i / num_step
-            for motor_idx in range(dof_size):
-                target_pos = default_pos[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].q = (
-                    init_dof_pos[motor_idx] * (1 - alpha) + target_pos * alpha
-                )
-                self.robot.msg.motor_cmd[motor_idx].qd = 0
-                self.robot.msg.motor_cmd[motor_idx].kp = self.robot.kp[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].kd = self.robot.kd[motor_idx]
-                self.robot.msg.motor_cmd[motor_idx].tau = 0
-            self.robot.msg.crc = self.robot.crc.Crc(self.robot.msg)
-            self.robot.lowcmd_publisher.Write(self.robot.msg)
-            time.sleep(self.robot.control_dt)
-        logger.info("Reached default position (legs only)")
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="GR00T Locomotion Controller for Unitree G1")
-    parser.add_argument(
-        "--repo-id",
-        type=str,
-        default=DEFAULT_GROOT_REPO_ID,
-        help=f"Hugging Face Hub repo ID for GR00T policies (default: {DEFAULT_GROOT_REPO_ID})",
-    )
-    args = parser.parse_args()
-
-    # load policies
-    policy_balance, policy_walk = load_groot_policies(repo_id=args.repo_id)
-
-    # initialize robot
-    config = UnitreeG1Config()
-    robot = UnitreeG1(config)
-
-    # initialize gr00t locomotion controller
-    groot_controller = GrootLocomotionController(
-        policy_balance=policy_balance,
-        policy_walk=policy_walk,
-        robot=robot,
-        config=config,
-    )
-
-    # reset legs and start locomotion thread
-    try:
-        groot_controller.reset_robot()
-        groot_controller.start_locomotion_thread()
-
-        # log status
-        logger.info("Robot initialized with GR00T locomotion policies")
-        logger.info("Locomotion controller running in background thread")
-        logger.info("Press Ctrl+C to stop")
-
-        # keep robot alive
-        while True:
-            time.sleep(1.0)
-    except KeyboardInterrupt:
-        print("\nStopping locomotion...")
-        groot_controller.stop_locomotion_thread()
-        print("Done!")

examples/voice_control/README.md

@@ -1,47 +0,0 @@
-# Voice Assistant Examples
-
-Voice-enabled robot assistant examples using speech-to-text (STT), and text-to-speech (TTS).
-
-## Overview
-
-These examples demonstrate how to build a voice interface for robot control:
-
-1. **Hold SPACE** → Push-to-talk recording starts
-2. **Release SPACE** → Recording stops
-3. **STT (Whisper)** → Converts speech to text (high-level task prompt)
-4. **Pi0.5** → Generates robot response/utterance
-5. **TTS (Kokoro)** → Speaks the response back
-
-## Requirements
-
-```bash
-pip install torch transformers sounddevice numpy pynput kokoro>=0.9.2
-```
-
-## Usage
-
-### With Pi0.5 Model
-
-```bash
-python examples/voice_assistant/voice_assistant_pi05.py \
-    --pretrained_path path/to/pi05/checkpoint
-```
-
-## How It Works
-
-### Pi0.5 Voice Integration
-
-Pi0.5 can generate robot utterances as part of its subtask prediction. The flow:
-
-1. **High-level prompt**: User voice command is transcribed and formatted as a task prompt
-2. **Subtask generation**: Pi0.5 autoregressively generates a response
-3. **Utterance extraction**: If the response contains `<utterance>...</utterance>` tags, the content is extracted
-4. **TTS output**: The response is spoken back to the user
-
-## Configuration Options
-
-| Option | Default | Description |
-|--------|---------|-------------|
-| `--pretrained_path` | None | Path to Pi0.5 checkpoint |
-| `--record_seconds` | 5.0 | Audio recording duration |
-| `--max_response_tokens` | 100 | Max tokens in generated response |

examples/voice_control/voice_assistant_pi05.py

@@ -1,336 +0,0 @@
-#!/usr/bin/env python
-"""
-Voice Assistant with Pi0.5: Microphone → STT → Pi0.5 → TTS → Speaker
-
-This example demonstrates how to use Pi0.5 as a conversational robot assistant:
-1. Hold SPACE to record your voice command
-2. Speech-to-text (Whisper) converts speech to text
-3. Text is fed as a high-level prompt to Pi0.5
-4. Pi0.5 generates a response (robot utterance)
-5. Text-to-speech (Kokoro) speaks the response back
-
-Requirements:
-    pip install torch transformers sounddevice numpy pynput kokoro>=0.9.2
-
-Usage:
-    python examples/voice_assistant/voice_assistant_pi05.py \
-        --pretrained_path lerobot/pi0.5-base
-"""
-
-import os
-
-os.environ["TOKENIZERS_PARALLELISM"] = "false"
-
-import argparse
-import re
-import subprocess
-import threading
-import time
-
-import numpy as np
-import sounddevice as sd
-import torch
-from pynput import keyboard
-from transformers import AutoTokenizer, WhisperForConditionalGeneration, WhisperProcessor
-
-from lerobot.policies.pi05.configuration_pi05 import PI05Config
-from lerobot.policies.pi05.modeling_pi05 import PI05Pytorch
-
-SAMPLE_RATE = 16000
-
-
-def get_device():
-    if torch.cuda.is_available():
-        return torch.device("cuda")
-    elif torch.backends.mps.is_available():
-        return torch.device("mps")
-    return torch.device("cpu")
-
-
-class Pi05VoiceAssistant:
-    """Voice assistant using Pi0.5 for generating robot utterances."""
-
-    def __init__(
-        self,
-        pretrained_path: str | None = None,
-        max_response_tokens: int = 100,
-        max_record_seconds: float = 30.0,
-    ):
-        self.device = get_device()
-        self.dtype = torch.float32 if self.device.type == "mps" else torch.bfloat16
-        self.max_response_tokens = max_response_tokens
-        self.max_record_seconds = max_record_seconds
-
-        # Push-to-talk state
-        self._recording = False
-        self._audio_chunks: list[np.ndarray] = []
-        self._stream: sd.InputStream | None = None
-
-        print(f"Using device: {self.device}")
-        self._load_models(pretrained_path)
-
-    def _load_models(self, pretrained_path: str | None):
-        print("Loading STT (Whisper tiny)...")
-        self.stt_processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
-        self.stt_model = WhisperForConditionalGeneration.from_pretrained(
-            "openai/whisper-tiny.en", torch_dtype=self.dtype
-        ).to(self.device)
-
-        print("Loading Pi0.5 model...")
-        self._load_pi05(pretrained_path)
-
-        print("Loading tokenizer...")
-        self.tokenizer = AutoTokenizer.from_pretrained("google/paligemma-3b-pt-224")
-
-        self._load_tts()
-        print("Ready!\n")
-
-    def _load_pi05(self, pretrained_path: str | None):
-        """Load Pi0.5 model for utterance generation."""
-        config = PI05Config()
-        config.dtype = "float32" if self.device.type == "mps" else "bfloat16"
-
-        self.pi05_model = PI05Pytorch(config)
-
-        if pretrained_path:
-            try:
-                from safetensors.torch import load_file
-                state_dict = load_file(f"{pretrained_path}/model.safetensors")
-                self.pi05_model.load_state_dict(state_dict, strict=False)
-                print(f"✓ Loaded Pi0.5 weights from {pretrained_path}")
-            except Exception as e:
-                print(f"Warning: Could not load pretrained weights: {e}")
-                print("Using randomly initialized model for demo purposes")
-
-        self.pi05_model = self.pi05_model.to(self.device)
-        self.pi05_model.eval()
-
-    def _load_tts(self):
-        try:
-            print("Loading TTS (Kokoro 82M)...")
-            from kokoro import KPipeline
-
-            self.tts_pipeline = KPipeline(lang_code="a")  # American English
-            self.tts_voice = "af_heart"
-            self.tts_type = "kokoro"
-            print("Kokoro loaded!")
-        except Exception as e:
-            print(f"Kokoro not available ({e})")
-            print("Using macOS `say` for TTS")
-            self.tts_pipeline = None
-            self.tts_type = "system"
-
-    def _audio_callback(self, indata, frames, time_info, status):
-        """Callback for audio stream - collects chunks while recording."""
-        if self._recording:
-            self._audio_chunks.append(indata.copy())
-
-    def _start_recording(self):
-        """Start recording audio."""
-        if self._recording:
-            return
-        self._recording = True
-        self._audio_chunks = []
-        print("🎤 Recording... (release SPACE to stop)")
-
-    def _stop_recording(self) -> np.ndarray | None:
-        """Stop recording and return the audio."""
-        if not self._recording:
-            return None
-        self._recording = False
-
-        if not self._audio_chunks:
-            return None
-
-        audio = np.concatenate(self._audio_chunks, axis=0).flatten()
-        duration = len(audio) / SAMPLE_RATE
-        volume = np.abs(audio).max()
-        print(f"Recorded {duration:.1f}s, volume: {volume:.4f}")
-
-        if volume < 0.001:
-            print("⚠️  Very low audio - check microphone permissions!")
-            return None
-
-        return audio
-
-    def wait_for_spacebar(self) -> np.ndarray | None:
-        """Wait for spacebar press, record while held, return audio on release."""
-        audio_result = None
-        recording_done = threading.Event()
-
-        def on_press(key):
-            if key == keyboard.Key.space:
-                self._start_recording()
-
-        def on_release(key):
-            nonlocal audio_result
-            if key == keyboard.Key.space and self._recording:
-                audio_result = self._stop_recording()
-                recording_done.set()
-                return False  # Stop listener
-
-        # Start audio stream
-        self._stream = sd.InputStream(
-            samplerate=SAMPLE_RATE,
-            channels=1,
-            dtype="float32",
-            callback=self._audio_callback,
-            blocksize=int(SAMPLE_RATE * 0.1),  # 100ms blocks
-        )
-
-        with self._stream:
-            print("\n⏳ Press and hold SPACE to speak...")
-            with keyboard.Listener(on_press=on_press, on_release=on_release) as listener:
-                # Wait for recording to complete or timeout
-                recording_done.wait(timeout=self.max_record_seconds)
-                if self._recording:
-                    audio_result = self._stop_recording()
-
-        return audio_result
-
-    def transcribe(self, audio: np.ndarray) -> str:
-        start = time.perf_counter()
-        inputs = self.stt_processor(audio, sampling_rate=SAMPLE_RATE, return_tensors="pt")
-        input_features = inputs.input_features.to(self.device, dtype=self.dtype)
-        tokens = self.stt_model.generate(input_features)
-        text = self.stt_processor.batch_decode(tokens, skip_special_tokens=True)[0]
-        print(f"STT: {time.perf_counter() - start:.2f}s")
-        return text.strip()
-
-    def _create_dummy_images(self, batch_size: int = 1) -> tuple[list[torch.Tensor], list[torch.Tensor]]:
-        """Create placeholder images for Pi0.5 when no camera is available."""
-        image_shape = (batch_size, 3, 224, 224)
-        dummy_image = torch.zeros(image_shape, dtype=torch.float32, device=self.device)
-        dummy_mask = torch.ones(batch_size, dtype=torch.bool, device=self.device)
-        return [dummy_image], [dummy_mask]
-
-    def _tokenize_prompt(self, text: str) -> tuple[torch.Tensor, torch.Tensor]:
-        """Tokenize the user prompt for Pi0.5."""
-        prompt = f"User request: {text}\nRobot response:"
-        tokenized = self.tokenizer(
-            [prompt],
-            max_length=200,
-            truncation=True,
-            padding="max_length",
-            return_tensors="pt",
-        )
-        tokens = tokenized["input_ids"].to(self.device)
-        masks = tokenized["attention_mask"].to(self.device, dtype=torch.bool)
-        return tokens, masks
-
-    def generate_response(self, user_text: str) -> str:
-        """Generate robot utterance using Pi0.5's language generation."""
-        start = time.perf_counter()
-
-        images, img_masks = self._create_dummy_images()
-        tokens, masks = self._tokenize_prompt(user_text)
-
-        with torch.no_grad():
-            generated_tokens = self.pi05_model._generate_subtask_tokens(
-                images=images,
-                img_masks=img_masks,
-                tokens=tokens,
-                masks=masks,
-                tokenizer=self.tokenizer,
-                max_length=self.max_response_tokens,
-                device=self.device,
-            )
-
-        # Decode generated tokens
-        valid_tokens = generated_tokens[0][generated_tokens[0] != 0]
-        response = self.tokenizer.decode(valid_tokens, skip_special_tokens=True)
-
-        # Extract utterance if marked with special tokens
-        response = self._extract_utterance(response)
-
-        print(f"Pi0.5: {time.perf_counter() - start:.2f}s")
-        return response.strip()
-
-    def _extract_utterance(self, text: str) -> str:
-        """Extract utterance from between <utterance> tokens if present."""
-        pattern = r"<utterance>(.*?)</utterance>"
-        match = re.search(pattern, text, re.DOTALL)
-        if match:
-            return match.group(1).strip()
-        return text
-
-    def speak(self, text: str):
-        start = time.perf_counter()
-        if self.tts_type == "kokoro":
-            generator = self.tts_pipeline(text, voice=self.tts_voice)
-            audio_chunks = [audio for _, _, audio in generator]
-            if audio_chunks:
-                audio = np.concatenate(audio_chunks)
-                sd.play(audio, 24000)
-                sd.wait()
-        else:
-            subprocess.run(["say", text], check=True)
-        print(f"TTS: {time.perf_counter() - start:.2f}s")
-
-    def run(self):
-        print("=" * 50)
-        print("Pi0.5 Voice Assistant")
-        print("=" * 50)
-        print("• Hold SPACE to record your voice command")
-        print("• Release SPACE when done speaking")
-        print("• Press Ctrl+C to exit")
-        print("=" * 50)
-
-        while True:
-            try:
-                audio = self.wait_for_spacebar()
-
-                if audio is None:
-                    print("(no audio captured)\n")
-                    continue
-
-                user_text = self.transcribe(audio)
-
-                if not user_text:
-                    print("(no speech detected)\n")
-                    continue
-
-                print(f"You: {user_text}")
-
-                response = self.generate_response(user_text)
-                print(f"Robot: {response}\n")
-
-                self.speak(response)
-
-            except KeyboardInterrupt:
-                print("\nGoodbye!")
-                break
-
-
-def main():
-    parser = argparse.ArgumentParser(description="Pi0.5 Voice Assistant")
-    parser.add_argument(
-        "--pretrained_path",
-        type=str,
-        default=None,
-        help="Path to pretrained Pi0.5 model (optional)",
-    )
-    parser.add_argument(
-        "--max_response_tokens",
-        type=int,
-        default=100,
-        help="Maximum tokens in generated response",
-    )
-    parser.add_argument(
-        "--max_record_seconds",
-        type=float,
-        default=30.0,
-        help="Maximum recording duration in seconds",
-    )
-    args = parser.parse_args()
-
-    assistant = Pi05VoiceAssistant(
-        pretrained_path=args.pretrained_path,
-        max_response_tokens=args.max_response_tokens,
-        max_record_seconds=args.max_record_seconds,
-    )
-    assistant.run()
-
-
-if __name__ == "__main__":
-    main()

fast_tokenizer_local/metadata.json

@@ -1,27 +0,0 @@
-{
-  "repo_id": "local",
-  "vocab_size": 1024,
-  "scale": 10.0,
-  "encoded_dims": "0:7",
-  "encoded_dim_ranges": [
-    [
-      0,
-      7
-    ]
-  ],
-  "total_encoded_dims": 7,
-  "delta_dims": null,
-  "delta_dim_list": null,
-  "use_delta_transform": false,
-  "state_key": "observation.state",
-  "normalization_mode": "QUANTILES",
-  "action_horizon": 10,
-  "num_training_chunks": 25065,
-  "compression_stats": {
-    "compression_ratio": 3.464660463274599,
-    "mean_token_length": 20.204,
-    "p99_token_length": 36.00999999999999,
-    "min_token_length": 5.0,
-    "max_token_length": 38.0
-  }
-}

fast_tokenizer_local/processing_action_tokenizer.py

@@ -1,158 +0,0 @@
-import logging
-from typing import ClassVar
-
-import numpy as np
-from scipy.fft import dct
-from scipy.fft import idct
-from tokenizers import ByteLevelBPETokenizer
-from tokenizers.trainers import BpeTrainer
-from transformers import PreTrainedTokenizerFast
-from transformers.processing_utils import ProcessorMixin
-
-
-class UniversalActionProcessor(ProcessorMixin):
-    attributes: ClassVar[list[str]] = ["bpe_tokenizer"]
-    bpe_tokenizer_class: str = "AutoTokenizer"
-
-    def __init__(
-        self,
-        bpe_tokenizer: PreTrainedTokenizerFast,
-        scale: float = 10,
-        vocab_size: int = 1024,
-        min_token: int = 0,
-        *,
-        action_dim: int | None = None,
-        time_horizon: int | None = None,
-    ):
-        self.scale = scale
-        self.vocab_size = vocab_size
-        self.min_token = min_token
-
-        # Action horizon and dimension needed during decoding. These can be specified
-        # in three ways (in order of priority):
-        # 1. passed in as kwargs to decode()
-        # 2. in the constructor
-        # 3. cached from the last time decode() was called
-        self.time_horizon = time_horizon
-        self.action_dim = action_dim
-        self.called_time_horizon = time_horizon
-        self.called_action_dim = action_dim
-
-        super().__init__(bpe_tokenizer)
-
-    def __call__(self, action_chunk: np.array) -> np.array:
-        assert action_chunk.ndim <= 3, "Only 3 dimensions supported: [batch, timesteps, action_dim]"
-        if action_chunk.ndim == 2:
-            action_chunk = action_chunk[None, ...]
-
-        # Cache the time horizon and action dimension for decoding
-        self.called_time_horizon = action_chunk.shape[-2]
-        self.called_action_dim = action_chunk.shape[-1]
-
-        dct_coeff = dct(action_chunk, axis=1, norm="ortho")
-        dct_coeff = np.around(dct_coeff * self.scale)
-        tokens = []
-        for elem in dct_coeff:
-            token_str = "".join(map(chr, np.maximum(elem.flatten() - self.min_token, 0).astype(int)))
-            tokens.append(self.bpe_tokenizer(token_str)["input_ids"])
-        return tokens
-
-    def decode(
-        self,
-        tokens: list[list[int]],
-        *,
-        time_horizon: int | None = None,
-        action_dim: int | None = None,
-    ) -> np.array:
-        self.time_horizon = time_horizon or self.time_horizon or self.called_time_horizon
-        self.action_dim = action_dim or self.action_dim or self.called_action_dim
-
-        # Cache the time horizon and action dimension for the next call
-        self.called_time_horizon = self.time_horizon
-        self.called_action_dim = self.action_dim
-
-        assert (
-            self.time_horizon is not None and self.action_dim is not None
-        ), "Tokenizer not initialized, call encode() once or pass in time_horizon and action_dim."
-
-        decoded_actions = []
-        for token in tokens:
-            try:
-                decoded_tokens = self.bpe_tokenizer.decode(token)
-                decoded_dct_coeff = np.array(list(map(ord, decoded_tokens))) + self.min_token
-                decoded_dct_coeff = decoded_dct_coeff.reshape(-1, self.action_dim)
-                assert (
-                    decoded_dct_coeff.shape
-                    == (
-                        self.time_horizon,
-                        self.action_dim,
-                    )
-                ), f"Decoded DCT coefficients have shape {decoded_dct_coeff.shape}, expected ({self.time_horizon}, {self.action_dim})"
-            except Exception as e:
-                print(f"Error decoding tokens: {e}")
-                print(f"Tokens: {token}")
-                decoded_dct_coeff = np.zeros((self.time_horizon, self.action_dim))
-            decoded_actions.append(idct(decoded_dct_coeff / self.scale, axis=0, norm="ortho"))
-        return np.stack(decoded_actions)
-
-    @classmethod
-    def fit(
-        cls,
-        action_data: list[np.array],
-        scale: float = 10,
-        vocab_size: int = 1024,
-        *,
-        time_horizon: int | None = None,
-        action_dim: int | None = None,
-    ) -> "UniversalActionProcessor":
-        # Run DCT over all inputs
-        dct_tokens = [dct(a, axis=0, norm="ortho").flatten() for a in action_data]
-
-        # Quantize and find min token
-        max_token = int(np.around(np.concatenate(dct_tokens) * scale).max())
-        min_token = int(np.around(np.concatenate(dct_tokens) * scale).min())
-        min_vocab_size = max_token - min_token
-
-        assert (
-            min_vocab_size <= vocab_size
-        ), f"Vocab size {vocab_size} is too small for the range of tokens {min_vocab_size}"
-        if min_vocab_size + 100 > vocab_size:
-            logging.warning(
-                f"Initial alphabet size {min_vocab_size} is almost as large as the vocab"
-                f"size {vocab_size}, consider increasing vocab size"
-            )
-
-        # Make token iterator for BPE training
-        def _token_iter():
-            for tokens in dct_tokens:
-                rounded_tokens = np.around(tokens * scale) - min_token
-                rounded_tokens = rounded_tokens.astype(int)
-                string = "".join(map(chr, rounded_tokens))
-                yield string
-
-        # Train BPE tokenizer
-        bpe = ByteLevelBPETokenizer()
-
-        # Set up the entire range of possible tokens as the initial alphabet
-        alphabet = [chr(i) for i in range(max_token - min_token + 1)]
-        trainer = BpeTrainer(
-            vocab_size=vocab_size,
-            min_frequency=2,
-            show_progress=True,
-            special_tokens=[],
-            initial_alphabet=alphabet,
-            max_token_length=10000,
-        )
-
-        # Train the inner tokenizer (don't use ByteLevelBPETokenizer.train_from_iterator()
-        # because it doesn't support custom alphabets)
-        bpe._tokenizer.train_from_iterator(_token_iter(), trainer=trainer)
-
-        return cls(
-            PreTrainedTokenizerFast(tokenizer_object=bpe, clean_up_tokenization_spaces=False),
-            scale=scale,
-            vocab_size=vocab_size,
-            min_token=min_token,
-            time_horizon=time_horizon,
-            action_dim=action_dim,
-        )

fast_tokenizer_local/processor_config.json

@@ -1,11 +0,0 @@
-{
-  "action_dim": 7,
-  "auto_map": {
-    "AutoProcessor": "processing_action_tokenizer.UniversalActionProcessor"
-  },
-  "min_token": -32,
-  "processor_class": "UniversalActionProcessor",
-  "scale": 10.0,
-  "time_horizon": 10,
-  "vocab_size": 1024
-}

fast_tokenizer_local/special_tokens_map.json

@@ -1 +0,0 @@
-{}

fast_tokenizer_local/tokenizer_config.json

@@ -1,11 +0,0 @@
-{
-  "added_tokens_decoder": {},
-  "auto_map": {
-    "AutoProcessor": "processing_action_tokenizer.UniversalActionProcessor"
-  },
-  "clean_up_tokenization_spaces": false,
-  "extra_special_tokens": {},
-  "model_max_length": 1000000000000000019884624838656,
-  "processor_class": "UniversalActionProcessor",
-  "tokenizer_class": "PreTrainedTokenizerFast"
-}

pyproject.toml

@@ -107,10 +107,6 @@ dynamixel = ["dynamixel-sdk>=3.7.31,<3.9.0"]
 gamepad = ["lerobot[pygame-dep]", "hidapi>=0.14.0,<0.15.0"]
 hopejr = ["lerobot[feetech]", "lerobot[pygame-dep]"]
 lekiwi = ["lerobot[feetech]", "pyzmq>=26.2.1,<28.0.0"]
-unitree_g1 = [
-    "pyzmq>=26.2.1,<28.0.0",
-    "onnxruntime>=1.16.0"
-]
 reachy2 = ["reachy2_sdk>=1.0.14,<1.1.0"]
 kinematics = ["lerobot[placo-dep]"]
 intelrealsense = [
@@ -362,9 +358,9 @@ ignore_errors = false
 # module = "lerobot.async_inference.*"
 # ignore_errors = false
 
-[[tool.mypy.overrides]]
-module = "lerobot.transport.*"
-ignore_errors = false
+# [[tool.mypy.overrides]]
+# module = "lerobot.transport.*"
+# ignore_errors = false
 
 # [[tool.mypy.overrides]]
 # module = "lerobot.scripts.*"

src/lerobot/datasets/aggregate.py

@@ -136,40 +136,21 @@ def update_meta_data(
         df["_orig_chunk"] = df[orig_chunk_col].copy()
         df["_orig_file"] = df[orig_file_col].copy()
 
-        # Get mappings for this video key
+        # Update chunk and file indices to point to destination
+        df[orig_chunk_col] = video_idx["chunk"]
+        df[orig_file_col] = video_idx["file"]
+
+        # Apply per-source-file timestamp offsets
         src_to_offset = video_idx.get("src_to_offset", {})
-        src_to_dst = video_idx.get("src_to_dst", {})
-
-        # Apply per-source-file mappings
-        if src_to_dst:
-            # Map each episode to its correct destination file and apply offset
+        if src_to_offset:
+            # Apply offset based on original source file
             for idx in df.index:
-                # Convert to Python int to avoid numpy type mismatch in dict lookup
-                src_key = (int(df.at[idx, "_orig_chunk"]), int(df.at[idx, "_orig_file"]))
-
-                # Get destination chunk/file for this source file
-                dst_chunk, dst_file = src_to_dst.get(src_key, (video_idx["chunk"], video_idx["file"]))
-                df.at[idx, orig_chunk_col] = dst_chunk
-                df.at[idx, orig_file_col] = dst_file
-
-                # Apply timestamp offset
-                offset = src_to_offset.get(src_key, 0)
-                df.at[idx, f"videos/{key}/from_timestamp"] += offset
-                df.at[idx, f"videos/{key}/to_timestamp"] += offset
-        elif src_to_offset:
-            # Fallback: use same destination for all, but apply per-file offsets
-            df[orig_chunk_col] = video_idx["chunk"]
-            df[orig_file_col] = video_idx["file"]
-            for idx in df.index:
-                # Convert to Python int to avoid numpy type mismatch in dict lookup
-                src_key = (int(df.at[idx, "_orig_chunk"]), int(df.at[idx, "_orig_file"]))
+                src_key = (df.at[idx, "_orig_chunk"], df.at[idx, "_orig_file"])
                 offset = src_to_offset.get(src_key, 0)
                 df.at[idx, f"videos/{key}/from_timestamp"] += offset
                 df.at[idx, f"videos/{key}/to_timestamp"] += offset
         else:
             # Fallback to simple offset (for backward compatibility)
-            df[orig_chunk_col] = video_idx["chunk"]
-            df[orig_file_col] = video_idx["file"]
             df[f"videos/{key}/from_timestamp"] = (
                 df[f"videos/{key}/from_timestamp"] + video_idx["latest_duration"]
             )
@@ -287,12 +268,6 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
         videos_idx[key]["episode_duration"] = 0
         # Track offset for each source (chunk, file) pair
         videos_idx[key]["src_to_offset"] = {}
-        # Track destination (chunk, file) for each source (chunk, file) pair
-        videos_idx[key]["src_to_dst"] = {}
-        # Initialize dst_file_durations if not present
-        # dst_file_durations tracks duration of each destination file
-        if "dst_file_durations" not in videos_idx[key]:
-            videos_idx[key]["dst_file_durations"] = {}
 
     for key, video_idx in videos_idx.items():
         unique_chunk_file_pairs = {
@@ -307,13 +282,9 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
 
         chunk_idx = video_idx["chunk"]
         file_idx = video_idx["file"]
-        dst_file_durations = video_idx["dst_file_durations"]
+        current_offset = video_idx["latest_duration"]
 
         for src_chunk_idx, src_file_idx in unique_chunk_file_pairs:
-            # Convert to Python int to ensure consistent dict keys
-            src_chunk_idx = int(src_chunk_idx)
-            src_file_idx = int(src_file_idx)
-
             src_path = src_meta.root / DEFAULT_VIDEO_PATH.format(
                 video_key=key,
                 chunk_index=src_chunk_idx,
@@ -327,17 +298,14 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
             )
 
             src_duration = get_video_duration_in_s(src_path)
-            dst_key = (chunk_idx, file_idx)
 
             if not dst_path.exists():
-                # New destination file: offset is 0
-                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = 0
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                # Store offset before incrementing
+                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_offset
                 dst_path.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copy(str(src_path), str(dst_path))
-                # Track duration of this destination file
-                dst_file_durations[dst_key] = src_duration
                 videos_idx[key]["episode_duration"] += src_duration
+                current_offset += src_duration
                 continue
 
             # Check file sizes before appending
@@ -345,11 +313,10 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
             dst_size = get_file_size_in_mb(dst_path)
 
             if dst_size + src_size >= video_files_size_in_mb:
-                # Rotate to a new file - offset is 0
-                chunk_idx, file_idx = update_chunk_file_indices(chunk_idx, file_idx, chunk_size)
-                dst_key = (chunk_idx, file_idx)
+                # Rotate to a new file, this source becomes start of new destination
+                # So its offset should be 0
                 videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = 0
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                chunk_idx, file_idx = update_chunk_file_indices(chunk_idx, file_idx, chunk_size)
                 dst_path = dst_meta.root / DEFAULT_VIDEO_PATH.format(
                     video_key=key,
                     chunk_index=chunk_idx,
@@ -357,20 +324,16 @@ def aggregate_videos(src_meta, dst_meta, videos_idx, video_files_size_in_mb, chu
                 )
                 dst_path.parent.mkdir(parents=True, exist_ok=True)
                 shutil.copy(str(src_path), str(dst_path))
-                # Track duration of this new destination file
-                dst_file_durations[dst_key] = src_duration
+                # Reset offset for next file
+                current_offset = src_duration
             else:
-                # Append to existing destination file
-                # Offset is the current duration of this destination file
-                current_dst_duration = dst_file_durations.get(dst_key, 0)
-                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_dst_duration
-                videos_idx[key]["src_to_dst"][(src_chunk_idx, src_file_idx)] = dst_key
+                # Append to existing video file - use current accumulated offset
+                videos_idx[key]["src_to_offset"][(src_chunk_idx, src_file_idx)] = current_offset
                 concatenate_video_files(
                     [dst_path, src_path],
                     dst_path,
                 )
-                # Update duration of this destination file
-                dst_file_durations[dst_key] = current_dst_duration + src_duration
+                current_offset += src_duration
 
             videos_idx[key]["episode_duration"] += src_duration
 

src/lerobot/datasets/lerobot_dataset.py

@@ -58,7 +58,6 @@ from lerobot.datasets.utils import (
     load_nested_dataset,
     load_stats,
     load_tasks,
-    load_tasks_high_level,
     update_chunk_file_indices,
     validate_episode_buffer,
     validate_frame,
@@ -162,7 +161,6 @@ class LeRobotDatasetMetadata:
         self.info = load_info(self.root)
         check_version_compatibility(self.repo_id, self._version, CODEBASE_VERSION)
         self.tasks = load_tasks(self.root)
-        # self.tasks_high_level = load_tasks_high_level(self.root)
         self.episodes = load_episodes(self.root)
         self.stats = load_stats(self.root)
 
@@ -1052,12 +1050,6 @@ class LeRobotDataset(torch.utils.data.Dataset):
         # Add task as a string
         task_idx = item["task_index"].item()
         item["task"] = self.meta.tasks.iloc[task_idx].name
-        # Optionally add high level task index
-        if "task_index_high_level" in self.features:
-            high_level_task_idx = item["task_index_high_level"].item()
-            item["robot_utterance"] = self.meta.tasks_high_level.iloc[high_level_task_idx]["robot_utterance"]
-            item["user_prompt"] = self.meta.tasks_high_level.iloc[high_level_task_idx]["user_prompt"]
-
         return item
 
     def __repr__(self):

src/lerobot/datasets/utils.py

@@ -60,7 +60,6 @@ VIDEO_DIR = "videos"
 
 CHUNK_FILE_PATTERN = "chunk-{chunk_index:03d}/file-{file_index:03d}"
 DEFAULT_TASKS_PATH = "meta/tasks.parquet"
-DEFAULT_TASKS_HIGH_LEVEL_PATH = "meta/tasks_high_level.parquet"
 DEFAULT_EPISODES_PATH = EPISODES_DIR + "/" + CHUNK_FILE_PATTERN + ".parquet"
 DEFAULT_DATA_PATH = DATA_DIR + "/" + CHUNK_FILE_PATTERN + ".parquet"
 DEFAULT_VIDEO_PATH = VIDEO_DIR + "/{video_key}/" + CHUNK_FILE_PATTERN + ".mp4"
@@ -353,9 +352,6 @@ def load_tasks(local_dir: Path) -> pandas.DataFrame:
     tasks = pd.read_parquet(local_dir / DEFAULT_TASKS_PATH)
     return tasks
 
-def load_tasks_high_level(local_dir: Path) -> pandas.DataFrame:
-    tasks = pd.read_parquet(local_dir / DEFAULT_TASKS_HIGH_LEVEL_PATH)
-    return tasks
 
 def write_episodes(episodes: Dataset, local_dir: Path) -> None:
     """Write episode metadata to a parquet file in the LeRobot v3.0 format.

src/lerobot/optim/optimizers.py

@@ -104,107 +104,6 @@ class SGDConfig(OptimizerConfig):
         return torch.optim.SGD(params, **kwargs)
 
 
-@OptimizerConfig.register_subclass("xvla-adamw")
-@dataclass
-class XVLAAdamWConfig(OptimizerConfig):
-    """Custom AdamW optimizer for XVLA with differential learning rates.
-
-    The Vision-Language Model (VLM) is trained with 1/10 of the base learning rate
-    for stable optimization, while all other components use the full LR.
-
-    This LR ratio is crucial for achieving strong and stable finetuning performance.
-
-    Soft-prompts can optionally use a separate learning rate with warm-up support.
-    Set `soft_prompt_lr_scale` to a value < 1.0 (e.g., 0.1) to start soft-prompts
-    at a lower LR. Combine with a warmup scheduler for optimal results.
-
-    Note:
-        Completely matching official reported performance may require an additional
-        warm-up LR schedule for soft-prompts, which can bring minor improvements.
-        When `soft_prompt_warmup_lr_scale` is set, soft-prompts start at
-        `lr * soft_prompt_warmup_lr_scale` and should be warmed up via the scheduler.
-
-    Parameter Groups:
-        - Group 0 (vlm): VLM parameters at lr * 0.1, weight_decay * 0.1
-        - Group 1 (soft_prompts): Soft-prompt parameters at lr * soft_prompt_lr_scale
-        - Group 2 (other): All other parameters at full lr
-    """
-
-    lr: float = 1e-4
-    betas: tuple[float, float] = (0.9, 0.99)
-    eps: float = 1e-8
-    weight_decay: float = 0.0
-    grad_clip_norm: float = 10.0
-    # Soft-prompt specific settings
-    soft_prompt_lr_scale: float = 1.0  # Scale factor for soft-prompt LR (1.0 = same as base LR)
-    soft_prompt_warmup_lr_scale: float | None = None  # If set, start soft-prompts at this scale (e.g., 0.01)
-
-    def build(self, params: dict) -> torch.optim.Optimizer:
-        """
-        Build AdamW optimizer with differential learning rates.
-
-        Expects `named_parameters()` as input (dict of name -> param).
-        Applies:
-        - lr * 0.1 for all VLM-related parameters
-        - lr * soft_prompt_lr_scale for soft-prompt parameters (with optional warmup)
-        - full lr for all other parameters
-
-        Args:
-            params: Dictionary of parameter names to parameters (from named_parameters())
-
-        Returns:
-            AdamW optimizer with parameter groups for VLM, soft-prompts, and other components
-        """
-        assert isinstance(params, dict), "Custom LR optimizer requires `named_parameters()` as inputs."
-
-        vlm_group, soft_prompt_group, other_group = [], [], []
-        for name, p in params.items():
-            if not p.requires_grad:
-                continue
-            if "vlm" in name.lower():
-                vlm_group.append(p)
-            elif "soft_prompt" in name.lower():
-                soft_prompt_group.append(p)
-            else:
-                other_group.append(p)
-
-        # Determine soft-prompt LR
-        soft_prompt_lr = self.lr * self.soft_prompt_lr_scale
-        if self.soft_prompt_warmup_lr_scale is not None:
-            # Start at warmup scale, scheduler will warm up to soft_prompt_lr
-            soft_prompt_lr = self.lr * self.soft_prompt_warmup_lr_scale
-
-        param_groups = [
-            {
-                "params": vlm_group,
-                "lr": self.lr * 0.1,
-                "weight_decay": self.weight_decay * 0.1,
-                "name": "vlm",
-            },
-            {
-                "params": soft_prompt_group,
-                "lr": soft_prompt_lr,
-                "weight_decay": self.weight_decay,
-                "name": "soft_prompts",
-            },
-            {
-                "params": other_group,
-                "lr": self.lr,
-                "weight_decay": self.weight_decay,
-                "name": "other",
-            },
-        ]
-
-        # Filter out empty groups
-        param_groups = [g for g in param_groups if len(g["params"]) > 0]
-
-        return torch.optim.AdamW(
-            param_groups,
-            betas=self.betas,
-            eps=self.eps,
-        )
-
-
 @OptimizerConfig.register_subclass("multi_adam")
 @dataclass
 class MultiAdamConfig(OptimizerConfig):

src/lerobot/policies/factory.py

@@ -16,7 +16,6 @@
 
 from __future__ import annotations
 
-import importlib
 import logging
 from typing import Any, TypedDict
 
@@ -114,10 +113,7 @@ def get_policy_class(name: str) -> type[PreTrainedPolicy]:
 
         return XVLAPolicy
     else:
-        try:
-            return _get_policy_cls_from_policy_name(name=name)
-        except Exception as e:
-            raise ValueError(f"Policy type '{name}' is not available.") from e
+        raise NotImplementedError(f"Policy with name {name} is not implemented.")
 
 
 def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
@@ -162,11 +158,7 @@ def make_policy_config(policy_type: str, **kwargs) -> PreTrainedConfig:
     elif policy_type == "xvla":
         return XVLAConfig(**kwargs)
     else:
-        try:
-            config_cls = PreTrainedConfig.get_choice_class(policy_type)
-            return config_cls(**kwargs)
-        except Exception as e:
-            raise ValueError(f"Policy type '{policy_type}' is not available.") from e
+        raise ValueError(f"Policy type '{policy_type}' is not available.")
 
 
 class ProcessorConfigKwargs(TypedDict, total=False):
@@ -355,13 +347,7 @@ def make_pre_post_processors(
         )
 
     else:
-        try:
-            processors = _make_processors_from_policy_config(
-                config=policy_cfg,
-                dataset_stats=kwargs.get("dataset_stats"),
-            )
-        except Exception as e:
-            raise ValueError(f"Processor for policy type '{policy_cfg.type}' is not implemented.") from e
+        raise NotImplementedError(f"Processor for policy type '{policy_cfg.type}' is not implemented.")
 
     return processors
 
@@ -430,7 +416,8 @@ def make_policy(
             raise ValueError("env_cfg cannot be None when ds_meta is not provided")
         features = env_to_policy_features(env_cfg)
 
-    cfg.output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
+    if not cfg.output_features:
+        cfg.output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
     if not cfg.input_features:
         cfg.input_features = {key: ft for key, ft in features.items() if key not in cfg.output_features}
     kwargs["config"] = cfg
@@ -454,65 +441,3 @@ def make_policy(
         # TODO: (jadechoghari) - add a check_state(cfg, features) and check_action(cfg, features)
 
     return policy
-
-
-def _get_policy_cls_from_policy_name(name: str) -> type[PreTrainedConfig]:
-    """Get policy class from its registered name using dynamic imports.
-
-    This is used as a helper function to import policies from 3rd party lerobot plugins.
-
-    Args:
-        name: The name of the policy.
-    Returns:
-        The policy class corresponding to the given name.
-    """
-    if name not in PreTrainedConfig.get_known_choices():
-        raise ValueError(
-            f"Unknown policy name '{name}'. Available policies: {PreTrainedConfig.get_known_choices()}"
-        )
-
-    config_cls = PreTrainedConfig.get_choice_class(name)
-    config_cls_name = config_cls.__name__
-
-    model_name = config_cls_name.removesuffix("Config")  # e.g., DiffusionConfig -> Diffusion
-    if model_name == config_cls_name:
-        raise ValueError(
-            f"The config class name '{config_cls_name}' does not follow the expected naming convention."
-            f"Make sure it ends with 'Config'!"
-        )
-    cls_name = model_name + "Policy"  # e.g., DiffusionConfig -> DiffusionPolicy
-    module_path = config_cls.__module__.replace(
-        "configuration_", "modeling_"
-    )  # e.g., configuration_diffusion -> modeling_diffusion
-
-    module = importlib.import_module(module_path)
-    policy_cls = getattr(module, cls_name)
-    return policy_cls
-
-
-def _make_processors_from_policy_config(
-    config: PreTrainedConfig,
-    dataset_stats: dict[str, dict[str, torch.Tensor]] | None = None,
-) -> tuple[Any, Any]:
-    """Create pre- and post-processors from a policy configuration using dynamic imports.
-
-    This is used as a helper function to import processor factories from 3rd party lerobot plugins.
-
-    Args:
-        config: The policy configuration object.
-        dataset_stats: Dataset statistics for normalization.
-    Returns:
-        A tuple containing the input (pre-processor) and output (post-processor) pipelines.
-    """
-
-    policy_type = config.type
-    function_name = f"make_{policy_type}_pre_post_processors"
-    module_path = config.__class__.__module__.replace(
-        "configuration_", "processor_"
-    )  # e.g., configuration_diffusion -> processor_diffusion
-    logging.debug(
-        f"Instantiating pre/post processors using function '{function_name}' from module '{module_path}'"
-    )
-    module = importlib.import_module(module_path)
-    function = getattr(module, function_name)
-    return function(config, dataset_stats=dataset_stats)

src/lerobot/policies/pi05/README_TOKENIZER.md

@@ -1,196 +0,0 @@
-# FAST Tokenizer Training for LeRobotDataset
-
-This directory contains tools for training a FAST (Factorized Action Sequence Tokenizer) on LeRobot datasets.
-
-## Files
-
-- **`train_fast_tokenizer.py`**: Main training script (refactored for LeRobotDataset)
-- **`train_fast_tokenizer_example.md`**: Usage examples and parameter documentation
-- **`MIGRATION_NOTES.md`**: Migration guide from B1K to LeRobotDataset
-
-## Quick Start
-
-```bash
-# Basic usage
-python train_fast_tokenizer.py \
-    --repo_id "lerobot/aloha_sim_insertion_human" \
-    --action_horizon 10 \
-    --encoded_dims "0:14"
-
-# With delta transform
-python train_fast_tokenizer.py \
-    --repo_id "lerobot/aloha_sim_insertion_human" \
-    --action_horizon 10 \
-    --encoded_dims "0:14" \
-    --delta_dims "0,1,2,3,4,5,6,7,8,9,10,11,12,13" \
-    --state_key "observation.state" \
-    --vocab_size 1024
-```
-
-## What is FAST?
-
-FAST is a tokenizer for robotic action sequences that:
-1. Applies DCT (Discrete Cosine Transform) to action chunks
-2. Quantizes DCT coefficients 
-3. Uses BPE (Byte-Pair Encoding) to compress the quantized sequence
-4. Achieves high compression ratios (e.g., 10-20x) while maintaining accuracy
-
-This enables efficient storage and processing of long action sequences in vision-language-action models.
-
-## Requirements
-
-- Python 3.10+
-- LeRobot dataset (either local or from HuggingFace Hub)
-- transformers (for AutoProcessor)
-- numpy
-- torch
-- tyro
-
-## Workflow
-
-```
-LeRobotDataset → Extract Episodes → Apply Delta Transform 
-    ↓
-Select Dimensions → Normalize (q01, q99) → Create Chunks
-    ↓
-Train FAST Tokenizer → Compute Stats → Save
-```
-
-## Parameters Guide
-
-### Essential Parameters
-
-- **`repo_id`**: HuggingFace dataset repository ID
-  - Example: `"lerobot/aloha_sim_insertion_human"`
-  
-- **`action_horizon`**: Length of action sequences to tokenize
-  - Typical: 10-16 steps
-  
-- **`encoded_dims`**: Which action dimensions to encode
-  - Format: `"start:end,start:end"`
-  - Example: `"0:7"` = dimensions 0-6
-  - Example: `"0:3,7:10"` = dimensions 0-2 and 7-9
-
-### Optional Parameters
-
-- **`delta_dims`**: Apply delta transform (action - state) to these dimensions
-  - Format: `"0,1,2,3,4,5"`
-  - Use for position-based actions
-  
-- **`state_key`**: Dataset key containing state observations
-  - Default: `"observation.state"`
-  
-- **`vocab_size`**: BPE vocabulary size
-  - Default: 1024
-  - Larger = better compression but more memory
-  
-- **`scale`**: DCT quantization scale
-  - Default: 10.0
-  - Smaller = finer quantization, larger = coarser
-
-- **`sample_fraction`**: Fraction of action chunks to use per episode
-  - Default: 0.1 (10%)
-  - Increase for small datasets, decrease for large datasets
-
-## Output
-
-The script creates a directory (default: `./fast_tokenizer_{repo_id}`) containing:
-
-1. **Tokenizer files**: Can be loaded with `AutoProcessor.from_pretrained()`
-2. **`metadata.json`**: Contains:
-   - Training configuration
-   - Compression statistics
-   - Dataset information
-
-## Example Output
-
-```
-Loading dataset: lerobot/aloha_sim_insertion_human
-Dataset loaded: 50 episodes, 5000 frames
-Encoding 14 dimensions: 0:14
-Delta dimensions: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
-Action horizon: 10
-Processing 50 episodes...
-Collected 4500 action chunks
-Extracted 14 encoded dimensions
-
-Before normalization - overall stats:
-  Min: -2.3451, Max: 3.1234, Mean: 0.0234, Std: 0.8765
-
-Applied quantile normalization [q01, q99] → [-1, 1]
-
-After normalization - overall stats:
-  Min: -1.0000, Max: 1.0000, Mean: 0.0156, Std: 0.4321
-
-Training FAST tokenizer on 4500 action chunks...
-Action chunk shape: (4500, 10, 14)
-Vocab size: 1024
-DCT scale: 10.0
-✓ Tokenizer training complete!
-
-Compression Statistics:
-  Average compression ratio: 14.23x
-  Mean token length: 9.8
-  P99 token length: 15
-  Min token length: 6
-  Max token length: 18
-
-✅ Saved FAST tokenizer to ./fast_tokenizer_lerobot_aloha_sim_insertion_human
-```
-
-## Using the Trained Tokenizer
-
-```python
-from transformers import AutoProcessor
-
-# Load tokenizer
-tokenizer = AutoProcessor.from_pretrained(
-    "./fast_tokenizer_lerobot_aloha_sim_insertion_human",
-    trust_remote_code=True
-)
-
-# Encode action chunk [horizon, action_dim]
-action_chunk = np.random.randn(10, 14)  # Example
-tokens = tokenizer(action_chunk[None])[0]  # Returns token IDs
-
-# Decode tokens back to actions
-reconstructed = tokenizer.decode(tokens)
-```
-
-## Tips
-
-1. **Start Small**: Use `--max_episodes 10` for initial testing
-2. **Check Dimensions**: Verify encoded dimensions match your robot's action space
-3. **Delta Transform**: Use for position-based actions, not velocity-based
-4. **Normalization**: Ensure dataset has proper statistics computed
-5. **Compression Ratio**: Aim for 10-20x for good balance of compression and accuracy
-
-## Troubleshooting
-
-**Issue**: "No normalization stats found"
-- **Solution**: Compute dataset statistics first, or use raw actions
-
-**Issue**: "Episode too short for action horizon"
-- **Solution**: Reduce `--action_horizon` or filter short episodes
-
-**Issue**: "State key not found"
-- **Solution**: Check dataset features and use correct `--state_key`
-
-**Issue**: Memory error with large datasets
-- **Solution**: Reduce `--sample_fraction` or `--max_episodes`
-
-## Citation
-
-If you use FAST in your research, please cite:
-
-```bibtex
-@article{black2023fast,
-  title={FAST: Factorized Action Sequence Tokenizer for Vision-Language-Action Models},
-  author={Black, Kevin and others},
-  journal={arXiv preprint},
-  year={2023}
-}
-```
-
-
-

src/lerobot/policies/pi05/configuration_pi05.py

@@ -37,11 +37,6 @@ class PI05Config(PreTrainedConfig):
     # Shorter state and action vectors will be padded to these dimensions
     max_state_dim: int = 32
     max_action_dim: int = 32
-    max_action_tokens: int = 32
-    fast_vocab_size: int = 2048
-    
-    # FAST-only mode: train with only discrete action token prediction (no flow matching, no subtask)
-    fast_only: bool = False
 
     # Flow matching parameters: see openpi `PI0Pytorch`
     num_inference_steps: int = 10
@@ -65,8 +60,8 @@ class PI05Config(PreTrainedConfig):
     normalization_mapping: dict[str, NormalizationMode] = field(
         default_factory=lambda: {
             "VISUAL": NormalizationMode.IDENTITY,
-            "STATE": NormalizationMode.MEAN_STD,  # Pi0.5 uses quantiles for state
-            "ACTION": NormalizationMode.MEAN_STD,  # Pi0.5 uses quantiles for action
+            "STATE": NormalizationMode.QUANTILES,  # Pi0.5 uses quantiles for state
+            "ACTION": NormalizationMode.QUANTILES,  # Pi0.5 uses quantiles for action
         }
     )
 

src/lerobot/policies/pi05/finetune_pi0.sh

@@ -1,21 +0,0 @@
-lerobot-train \
-    --dataset.repo_id=lerobot \
-    --dataset.root=/fsx/jade_choghari/outputs/collect-data-pgen \
-    --output_dir=/fsx/jade_choghari/outputs/pi0test1 \
-    --job_name=pi0_training \
-    --policy.repo_id=jade_choghari/pi0-base \
-    --policy.path=/fsx/jade_choghari/outputs/pi0_fast_fruit1/checkpoints/last/pretrained_model \
-    --policy.dtype=bfloat16 \
-    --steps=3000 \
-    --save_freq=1000 \
-    --rename_map='{
-        "observation.images.base": "observation.images.base_0_rgb",
-        "observation.images.left_wrist": "observation.images.left_wrist_0_rgb",
-        "observation.images.right_wrist": "observation.images.right_wrist_0_rgb",
-        }' \
-    --batch_size=4 \
-    --policy.device=cuda \
-    # --wandb.enable=true \
-    # --wandb.disable_artifact=true \
-    # --wandb.project=pi05hi-training \
-

src/lerobot/policies/pi05/processor_pi05.py

@@ -33,7 +33,6 @@ from lerobot.processor import (
     ProcessorStep,
     ProcessorStepRegistry,
     RenameObservationsProcessorStep,
-    ActionTokenizerProcessorStep,
     TokenizerProcessorStep,
     UnnormalizerProcessorStep,
 )
@@ -48,15 +47,13 @@ from lerobot.utils.constants import (
 
 @ProcessorStepRegistry.register(name="pi05_prepare_state_tokenizer_processor_step")
 @dataclass
-class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
+class Pi05PrepareStateTokenizerProcessorStep(ProcessorStep):
     """
     Processor step to prepare the state and tokenize the language input.
     """
 
     max_state_dim: int = 32
     task_key: str = "task"
-    high_level_task_key: str = "user_prompt"
-    subtask_only_key: str = "subtask"
 
     def __call__(self, transition: EnvTransition) -> EnvTransition:
         transition = transition.copy()
@@ -67,8 +64,6 @@ class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
         tasks = transition.get(TransitionKey.COMPLEMENTARY_DATA, {}).get(self.task_key)
         if tasks is None:
             raise ValueError("No task found in complementary data")
-        
-        high_level_tasks = transition.get(TransitionKey.COMPLEMENTARY_DATA, {}).get(self.high_level_task_key)
 
         # TODO: check if this necessary
         state = deepcopy(state)
@@ -81,42 +76,16 @@ class Pi05PrepareStateAndLanguageTokenizerProcessorStep(ProcessorStep):
         state_np = state.cpu().numpy()
         discretized_states = np.digitize(state_np, bins=np.linspace(-1, 1, 256 + 1)[:-1]) - 1
 
-        # Clean high level tasks first (if available)
-        cleaned_high_level_tasks = []
-        if high_level_tasks is not None:
-            for high_level_task in high_level_tasks:
-                cleaned_high_level_tasks.append(high_level_task.strip().replace("_", " ").replace("\n", " "))
-        
-        # Process low level tasks with state information
-        low_level_prompts = []
-        subtask_only_prompts = []  # Store only the subtask text for prediction
+        full_prompts = []
         for i, task in enumerate(tasks):
             cleaned_text = task.strip().replace("_", " ").replace("\n", " ")
             state_str = " ".join(map(str, discretized_states[i]))
-            
-            # Store only the subtask text (used as prediction target)
-            subtask_only_prompts.append(cleaned_text)
-            
-            if cleaned_high_level_tasks:
-                cleaned_high_level_task = cleaned_high_level_tasks[i]
-                full_prompt = f"High level task: {cleaned_high_level_task}; State: {state_str}; Subtask: {cleaned_text}"
-            else:
-                full_prompt = f"Task: {cleaned_text}, State: {state_str};\n" #remove Action by jade
+            full_prompt = f"Task: {cleaned_text}, State: {state_str};\nAction: "
+            full_prompts.append(full_prompt)
 
-            low_level_prompts.append(full_prompt)
-
-        transition[TransitionKey.COMPLEMENTARY_DATA][self.task_key] = low_level_prompts
-        transition[TransitionKey.COMPLEMENTARY_DATA][self.subtask_only_key] = subtask_only_prompts
-        
-        # Process high level tasks without state information (if available)
-        if high_level_tasks is not None:
-            high_level_prompts = []
-            for i, cleaned_high_level_task in enumerate(cleaned_high_level_tasks):
-                state_str = " ".join(map(str, discretized_states[i]))
-                full_prompt = f"High level task: {cleaned_high_level_task}; State: {state_str}; Subtask:"
-                high_level_prompts.append(full_prompt)
-            
-            transition[TransitionKey.COMPLEMENTARY_DATA][self.high_level_task_key] = high_level_prompts
+        transition[TransitionKey.COMPLEMENTARY_DATA][self.task_key] = full_prompts
+        # Normalize state to [-1, 1] range if needed (assuming it's already normalized by normalizer processor step!!)
+        # Discretize into 256 bins (see openpi `PaligemmaTokenizer.tokenize()`)
         return transition
 
     def transform_features(
@@ -159,27 +128,25 @@ def make_pi05_pre_post_processors(
     Returns:
         A tuple containing the configured pre-processor and post-processor pipelines.
     """
+
     # Add remaining processors
     input_steps: list[ProcessorStep] = [
         RenameObservationsProcessorStep(rename_map={}),  # To mimic the same processor as pretrained one
         AddBatchDimensionProcessorStep(),
-        # NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateAndLanguageTokenizerProcessorStep
+        # NOTE: NormalizerProcessorStep MUST come before Pi05PrepareStateTokenizerProcessorStep
         # because the tokenizer step expects normalized state in [-1, 1] range for discretization
         NormalizerProcessorStep(
             features={**config.input_features, **config.output_features},
             norm_map=config.normalization_mapping,
             stats=dataset_stats,
         ),
-        Pi05PrepareStateAndLanguageTokenizerProcessorStep(max_state_dim=config.max_state_dim),
+        Pi05PrepareStateTokenizerProcessorStep(max_state_dim=config.max_state_dim),
         TokenizerProcessorStep(
             tokenizer_name="google/paligemma-3b-pt-224",
             max_length=config.tokenizer_max_length,
             padding_side="right",
             padding="max_length",
         ),
-        ActionTokenizerProcessorStep(
-            tokenizer_name="/fsx/jade_choghari/outputs/fast_tokenizer", # TODO: jade put the PI
-        ),
         DeviceProcessorStep(device=config.device),
     ]
 
@@ -189,7 +156,7 @@ def make_pi05_pre_post_processors(
         ),
         DeviceProcessorStep(device="cpu"),
     ]
-    
+
     return (
         PolicyProcessorPipeline[dict[str, Any], dict[str, Any]](
             steps=input_steps,

src/lerobot/policies/pi05/train.sh

@@ -1,23 +0,0 @@
-export CUDA_LAUNCH_BLOCKING=1 
-lerobot-train \
-    --dataset.repo_id=local \
-    --dataset.root=/fsx/jade_choghari/outputs/collect-data-pgen \
-    --output_dir=/fsx/jade_choghari/outputs/pi0_fast_fruit2 \
-    --job_name=pi0_training \
-    --policy.repo_id=jade_choghari/pi0-base1 \
-    --policy.path=lerobot/pi05_base \
-    --policy.dtype=bfloat16 \
-    --steps=200000 \
-    --save_freq=5000 \
-    --rename_map='{
-        "observation.images.base": "observation.images.base_0_rgb",
-        "observation.images.left_wrist": "observation.images.left_wrist_0_rgb",
-        "observation.images.right_wrist": "observation.images.right_wrist_0_rgb",
-        }' \
-    --batch_size=16 \
-    --policy.device=cuda \
-    --policy.fast_only=true \
-    # --wandb.enable=true \
-    # --wandb.disable_artifact=true \
-    # --wandb.project=pi05hi-training \
-# /fsx/jade_choghari/.cache/huggingface/lerobot/jadechoghari/collect-data

src/lerobot/policies/pi05/train2.sh

@@ -1,13 +0,0 @@
-rm -rf /fsx/jade_choghari/outputs/pi0_multi_training
-lerobot-train \
-    --dataset.repo_id=local\
-    --dataset.root=/fsx/jade_choghari/data/libero \
-    --output_dir=/fsx/jade_choghari/outputs/pi0_multi_training \
-    --job_name=pi0_multi_training \
-    --policy.repo_id=jadechoghari/pi0-base1 \
-    --policy.path=/fsx/jade_choghari/outputs/libero_training_fast_6/checkpoints/last/pretrained_model/ \
-    --policy.dtype=bfloat16 \
-    --steps=50000 \
-    --save_freq=5000 \
-    --batch_size=4 \
-    --policy.device=cuda \

src/lerobot/policies/pi05/train_fast.sh

@@ -1,12 +0,0 @@
-python src/lerobot/policies/pi05/train_fast_tokenizer.py \
-    --repo_id "local" \
-    --root /fsx/jade_choghari/data/libero \
-    --action_horizon 10 \
-    --encoded_dims "0:7" \
-    --vocab_size 1024 \
-    --push_to_hub \
-    --hub_repo_id jadechoghari/fast-libero-tokenizer-quantiles \
-    --normalization_mode QUANTILES \
-
-
-# python train_fast_tokenizer.py --repo_id my_dataset

src/lerobot/policies/pi05/train_fast_tokenizer.py

@@ -1,533 +0,0 @@
-"""Train FAST tokenizer for action encoding.
-
-This script:
-1. Loads action chunks from LeRobotDataset (with sampling)
-2. Applies delta transforms and per-timestamp normalization
-3. Trains FAST tokenizer on specified action dimensions
-4. Saves tokenizer to assets directory
-5. Reports compression statistics
-"""
-
-import json
-import numpy as np
-import tyro
-from pathlib import Path
-from transformers import AutoProcessor
-import torch
-
-from huggingface_hub import HfApi
-from lerobot.configs.types import NormalizationMode
-from lerobot.datasets.lerobot_dataset import LeRobotDataset
-
-
-def apply_delta_transform(state: np.ndarray, actions: np.ndarray, delta_dims: list[int] | None) -> np.ndarray:
-    """Apply delta transform to specified dimensions.
-    
-    Args:
-        state: Current state [D]
-        actions: Future actions [D]
-        delta_dims: List of dimension indices to apply delta transform to
-    
-    Returns:
-        Transformed actions [D]
-    """
-    if delta_dims is None or len(delta_dims) == 0:
-        return actions
-    
-    delta_actions = actions.copy()
-    for dim in delta_dims:
-        delta_actions[dim] = actions[dim] - state[dim]
-    
-    return delta_actions
-
-
-def apply_normalization(
-    data: np.ndarray,
-    stats: dict[str, np.ndarray],
-    mode: NormalizationMode,
-    eps: float = 1e-8,
-) -> np.ndarray:
-    """Apply normalization to data based on the specified mode.
-    
-    Args:
-        data: Data to normalize [N, H, D] or [D]
-        stats: Dictionary of statistics (mean, std, min, max, q01, q99, q10, q90)
-        mode: Normalization mode to apply
-        eps: Small epsilon for numerical stability
-    
-    Returns:
-        Normalized data with the same shape as input
-    """
-    if mode == NormalizationMode.IDENTITY:
-        return data
-    
-    if mode == NormalizationMode.MEAN_STD:
-        mean = stats.get("mean")
-        std = stats.get("std")
-        if mean is None or std is None:
-            raise ValueError("MEAN_STD mode requires 'mean' and 'std' in stats")
-        return (data - mean) / np.maximum(std, eps)
-    
-    if mode == NormalizationMode.MIN_MAX:
-        min_val = stats.get("min")
-        max_val = stats.get("max")
-        if min_val is None or max_val is None:
-            raise ValueError("MIN_MAX mode requires 'min' and 'max' in stats")
-        denom = np.maximum(max_val - min_val, eps)
-        return 2.0 * (data - min_val) / denom - 1.0
-    
-    if mode == NormalizationMode.QUANTILES:
-        q01 = stats.get("q01")
-        q99 = stats.get("q99")
-        if q01 is None or q99 is None:
-            raise ValueError("QUANTILES mode requires 'q01' and 'q99' in stats")
-        denom = np.maximum(q99 - q01, eps)
-        # Clip to quantile range then normalize to [-1, 1]
-        clipped = np.clip(data, q01, q99)
-        return 2.0 * (clipped - q01) / denom - 1.0
-    
-    if mode == NormalizationMode.QUANTILE10:
-        q10 = stats.get("q10")
-        q90 = stats.get("q90")
-        if q10 is None or q90 is None:
-            raise ValueError("QUANTILE10 mode requires 'q10' and 'q90' in stats")
-        denom = np.maximum(q90 - q10, eps)
-        # Clip to quantile range then normalize to [-1, 1]
-        clipped = np.clip(data, q10, q90)
-        return 2.0 * (clipped - q10) / denom - 1.0
-    
-    raise ValueError(f"Unsupported normalization mode: {mode}")
-
-
-def process_episode(args):
-    """Process single episode and return action chunks."""
-    dataset, ep_idx, action_horizon, delta_dims, sample_fraction, state_key, use_delta_transform = args
-    
-    try:
-        # Get episode info
-        ep_info = dataset.meta.episodes[ep_idx]
-        from_idx = ep_info["dataset_from_index"]
-        to_idx = ep_info["dataset_to_index"]
-        ep_length = to_idx - from_idx
-        
-        if ep_length < action_horizon:
-            return None
-        
-        # Load all frames in episode
-        # If dataset has episode filtering, we need to use the mapping
-        states = []
-        actions = []
-        
-        for abs_idx in range(from_idx, to_idx):
-            # Map absolute index to relative index if needed
-            if dataset._absolute_to_relative_idx is not None:
-                if abs_idx not in dataset._absolute_to_relative_idx:
-                    # This episode's frames aren't in the filtered dataset
-                    return None
-                rel_idx = dataset._absolute_to_relative_idx[abs_idx]
-            else:
-                rel_idx = abs_idx
-            
-            frame = dataset.hf_dataset[rel_idx]
-            
-            # Get state (could be from observation.state or other state key)
-            if state_key in frame:
-                state = frame[state_key].numpy() if torch.is_tensor(frame[state_key]) else np.array(frame[state_key])
-            else:
-                # If no state key, use zeros (no delta transform)
-                state = np.zeros_like(frame["action"].numpy() if torch.is_tensor(frame["action"]) else np.array(frame["action"]))
-            
-            action = frame["action"].numpy() if torch.is_tensor(frame["action"]) else np.array(frame["action"])
-            
-            states.append(state)
-            actions.append(action)
-        
-        states = np.array(states)
-        actions = np.array(actions)
-        
-        # Create action chunks (sliding window)
-        # All actions in a chunk are relative to the FIRST state in that chunk
-        action_chunks = []
-        
-        for i in range(len(states) - action_horizon + 1):
-            current_state = states[i]  # First state in chunk
-            future_absolute_actions = actions[i:i + action_horizon]
-            
-            if use_delta_transform:
-                # Relative actions
-                delta_chunk = np.zeros_like(future_absolute_actions)
-                for t in range(action_horizon):
-                    delta_chunk[t] = apply_delta_transform(
-                        current_state,
-                        future_absolute_actions[t],
-                        delta_dims,
-                    )
-                action_chunks.append(delta_chunk)
-            else:
-                # Absolute actions (NO delta)
-                action_chunks.append(future_absolute_actions)
-        
-        if len(action_chunks) == 0:
-            return None
-        
-        action_chunks = np.array(action_chunks)
-        
-        # Sample chunks
-        if sample_fraction < 1.0:
-            n_chunks = len(action_chunks)
-            n_samples = max(1, int(n_chunks * sample_fraction))
-            episode_seed = hash(ep_idx) % (2**31)
-            rng = np.random.RandomState(episode_seed)
-            indices = rng.choice(n_chunks, size=n_samples, replace=False)
-            action_chunks = action_chunks[indices]
-        
-        return action_chunks
-        
-    except Exception as e:
-        print(f"Error processing episode {ep_idx}: {e}")
-        import traceback
-        traceback.print_exc()
-        return None
-
-
-def train_fast_tokenizer(
-    action_chunks: np.ndarray,
-    vocab_size: int = 1024,
-    scale: float = 10.0,
-) -> AutoProcessor:
-    """
-    Train FAST tokenizer (BPE on DCT coefficients) on action chunks.
-    
-    Uses the .fit() method to train a new tokenizer on the provided data.
-    
-    Args:
-        action_chunks: Array of action chunks [N, H, D] where N=num_chunks, H=horizon, D=action_dim
-        vocab_size: BPE vocabulary size
-        scale: DCT scaling factor for quantization
-    
-    Returns:
-        Trained FAST tokenizer
-    """
-    print(f"Training FAST tokenizer on {len(action_chunks)} action chunks...")
-    print(f"Action chunk shape: {action_chunks.shape}")
-    print(f"Vocab size: {vocab_size}")
-    print(f"DCT scale: {scale}")
-    
-    # Download the tokenizer source code (not pretrained weights)
-    # We'll train a new tokenizer on our own data
-    base_tokenizer = AutoProcessor.from_pretrained(
-        "physical-intelligence/fast",
-        trust_remote_code=True
-    )
-    
-    # Convert action_chunks array to list of arrays (expected by .fit())
-    action_data_list = [action_chunks[i] for i in range(len(action_chunks))]
-    
-    # Train the new tokenizer on our action data using .fit()
-    # This trains the BPE tokenizer on DCT coefficients
-    print("Training new tokenizer (this may take a few minutes)...")
-    tokenizer = base_tokenizer.fit(
-        action_data_list,
-        scale=scale,
-        vocab_size=vocab_size,
-        time_horizon=action_chunks.shape[1],  # action_horizon
-        action_dim=action_chunks.shape[2],     # encoded dimensions
-    )
-    print("✓ Tokenizer training complete!")
-    
-    # Validate it works
-    sample_chunk = action_chunks[0]
-    encoded = tokenizer(sample_chunk[None])[0]
-    if isinstance(encoded, list):
-        encoded = np.array(encoded)
-    print(f"Sample encoding: {len(encoded)} tokens for chunk shape {sample_chunk.shape}")
-    
-    return tokenizer
-
-
-def compute_compression_stats(tokenizer, action_chunks: np.ndarray):
-    """Compute compression statistics."""
-    print("\nComputing compression statistics...")
-    
-    # Sample for stats (use max 1000 chunks for speed)
-    sample_size = min(1000, len(action_chunks))
-    sample_indices = np.random.RandomState(42).choice(len(action_chunks), size=sample_size, replace=False)
-    sample_chunks = action_chunks[sample_indices]
-    
-    token_lengths = []
-    for chunk in sample_chunks:
-        encoded = tokenizer(chunk[None])[0]
-        if isinstance(encoded, list):
-            token_lengths.append(len(encoded))
-        else:
-            token_lengths.append(encoded.shape[0] if hasattr(encoded, 'shape') else len(encoded))
-    
-    token_lengths = np.array(token_lengths)
-    
-    # Compression ratio: (H * D) / avg_tokens
-    input_size = action_chunks.shape[1] * action_chunks.shape[2]
-    avg_tokens = np.mean(token_lengths)
-    compression_ratio = input_size / avg_tokens
-    
-    stats = {
-        'compression_ratio': float(compression_ratio),
-        'mean_token_length': float(np.mean(token_lengths)),
-        'p99_token_length': float(np.percentile(token_lengths, 99)),
-        'min_token_length': float(np.min(token_lengths)),
-        'max_token_length': float(np.max(token_lengths)),
-    }
-    
-    print(f"Compression Statistics:")
-    print(f"  Average compression ratio: {stats['compression_ratio']:.2f}x")
-    print(f"  Mean token length: {stats['mean_token_length']:.1f}")
-    print(f"  P99 token length: {stats['p99_token_length']:.0f}")
-    print(f"  Min token length: {stats['min_token_length']:.0f}")
-    print(f"  Max token length: {stats['max_token_length']:.0f}")
-    
-    return stats
-
-
-def main(
-    repo_id: str,
-    root: str | None = None,
-    action_horizon: int = 10,
-    max_episodes: int | None = None,
-    sample_fraction: float = 0.1,
-    encoded_dims: str = "0:6,7:23",
-    delta_dims: str | None = None,
-    use_delta_transform: bool = False,
-    state_key: str = "observation.state",
-    normalization_mode: str = "QUANTILES",
-    vocab_size: int = 1024,
-    scale: float = 10.0,
-    output_dir: str | None = None,
-    push_to_hub: bool = False,
-    hub_repo_id: str | None = None,
-    hub_private: bool = False,
-):
-    """
-    Train FAST tokenizer for action encoding.
-    
-    Args:
-        repo_id: LeRobot dataset repository ID
-        root: Root directory for dataset (default: ~/.cache/huggingface/lerobot)
-        action_horizon: Number of future actions in each chunk
-        max_episodes: Max episodes to use (None = all episodes in dataset)
-        sample_fraction: Fraction of chunks to sample per episode
-        encoded_dims: Comma-separated dimension ranges to encode (e.g., "0:6,7:23")
-        delta_dims: Comma-separated dimension indices for delta transform (e.g., "0,1,2,3,4,5")
-        use_delta_transform: Whether to apply delta transform (relative actions vs absolute actions)
-        state_key: Dataset key for state observations (default: "observation.state")
-        normalization_mode: Normalization mode (MEAN_STD, MIN_MAX, QUANTILES, QUANTILE10, IDENTITY)
-        vocab_size: FAST vocabulary size (BPE vocab size)
-        scale: DCT scaling factor (default: 10.0)
-        output_dir: Directory to save tokenizer (default: ./fast_tokenizer_{repo_id})
-        push_to_hub: Whether to push the tokenizer to Hugging Face Hub
-        hub_repo_id: Hub repository ID (e.g., "username/tokenizer-name"). If None, uses output_dir name
-        hub_private: Whether to create a private repository on the Hub
-    """
-    # Load dataset
-    print(f"Loading dataset: {repo_id}")
-    dataset = LeRobotDataset(repo_id=repo_id, root=root)
-    print(f"Dataset loaded: {dataset.num_episodes} episodes, {dataset.num_frames} frames")
-    
-    # Parse normalization mode
-    try:
-        norm_mode = NormalizationMode(normalization_mode)
-    except ValueError:
-        raise ValueError(
-            f"Invalid normalization_mode: {normalization_mode}. "
-            f"Must be one of: {', '.join([m.value for m in NormalizationMode])}"
-        )
-    print(f"Normalization mode: {norm_mode.value}")
-    
-    # Parse encoded dimensions
-    encoded_dim_ranges = []
-    for range_str in encoded_dims.split(','):
-        start, end = map(int, range_str.strip().split(':'))
-        encoded_dim_ranges.append((start, end))
-    
-    total_encoded_dims = sum(end - start for start, end in encoded_dim_ranges)
-    print(f"Encoding {total_encoded_dims} dimensions: {encoded_dims}")
-    
-    # Parse delta dimensions
-    delta_dim_list = None
-    if delta_dims is not None and delta_dims.strip():
-        delta_dim_list = [int(d.strip()) for d in delta_dims.split(',')]
-        print(f"Delta dimensions: {delta_dim_list}")
-    else:
-        print("No delta dimensions specified")
-    
-    print(f"Use delta transform: {use_delta_transform}")
-    if use_delta_transform and (delta_dim_list is None or len(delta_dim_list) == 0):
-        print("Warning: use_delta_transform=True but no delta_dims specified. No delta will be applied.")
-    
-    print(f"Action horizon: {action_horizon}")
-    print(f"State key: {state_key}")
-    
-    # Determine episodes to process
-    num_episodes = dataset.num_episodes
-    if max_episodes is not None:
-        num_episodes = min(max_episodes, num_episodes)
-    
-    print(f"Processing {num_episodes} episodes...")
-    
-    # Process episodes sequentially (to avoid pickling issues with dataset)
-    all_chunks = []
-    for ep_idx in range(num_episodes):
-        if ep_idx % 10 == 0:
-            print(f"  Processing episode {ep_idx}/{num_episodes}...")
-        
-        chunks = process_episode(
-            (dataset, ep_idx, action_horizon, delta_dim_list, sample_fraction, state_key, use_delta_transform)
-        )
-        if chunks is not None:
-            all_chunks.append(chunks)
-    
-    # Concatenate all chunks
-    all_chunks = np.concatenate(all_chunks, axis=0)
-    print(f"Collected {len(all_chunks)} action chunks")
-    
-    # Extract only encoded dimensions FIRST (before normalization)
-    encoded_chunks = []
-    for start, end in encoded_dim_ranges:
-        encoded_chunks.append(all_chunks[:, :, start:end])
-    encoded_chunks = np.concatenate(encoded_chunks, axis=-1)  # [N, H, D_encoded]
-    print(f"Extracted {encoded_chunks.shape[-1]} encoded dimensions")
-    
-    # Apply normalization to encoded dimensions
-    print(f"\nBefore normalization - overall stats:")
-    print(f"  Min: {np.min(encoded_chunks):.4f}, Max: {np.max(encoded_chunks):.4f}")
-    print(f"  Mean: {np.mean(encoded_chunks):.4f}, Std: {np.std(encoded_chunks):.4f}")
-    
-    # Get normalization stats from dataset
-    norm_stats = dataset.meta.stats
-    if norm_stats is not None and "action" in norm_stats:
-        action_stats = norm_stats["action"]
-        
-        # Build encoded dimension indices
-        encoded_dim_indices = []
-        for start, end in encoded_dim_ranges:
-            encoded_dim_indices.extend(range(start, end))
-        encoded_dim_indices = np.array(encoded_dim_indices)
-        
-        # Extract stats for encoded dimensions only
-        encoded_stats = {}
-        for stat_name, stat_values in action_stats.items():
-            if isinstance(stat_values, (list, np.ndarray)):
-                stat_array = np.array(stat_values)
-                if len(stat_array) > max(encoded_dim_indices):
-                    encoded_stats[stat_name] = stat_array[encoded_dim_indices]
-        
-        if encoded_stats:
-            print(f"\nNormalization stats for encoded dimensions (mode: {norm_mode.value}):")
-            for stat_name, stat_values in encoded_stats.items():
-                print(f"  {stat_name}: shape={stat_values.shape}, "
-                      f"range=[{np.min(stat_values):.4f}, {np.max(stat_values):.4f}]")
-            
-            # Apply normalization based on mode
-            try:
-                encoded_chunks = apply_normalization(
-                    encoded_chunks, 
-                    encoded_stats, 
-                    norm_mode,
-                    eps=1e-8
-                )
-                print(f"\nApplied {norm_mode.value} normalization")
-            except ValueError as e:
-                print(f"Warning: {e}. Using raw actions without normalization.")
-            
-            print(f"\nAfter normalization - overall stats:")
-            print(f"  Min: {np.min(encoded_chunks):.4f}, Max: {np.max(encoded_chunks):.4f}")
-            print(f"  Mean: {np.mean(encoded_chunks):.4f}, Std: {np.std(encoded_chunks):.4f}")
-            
-            print(f"\nPer-dimension stats (after normalization):")
-            for d in range(encoded_chunks.shape[-1]):
-                dim_data = encoded_chunks[:, :, d]
-                print(f"  Dim {d}: min={np.min(dim_data):7.4f}, max={np.max(dim_data):7.4f}, "
-                      f"mean={np.mean(dim_data):7.4f}, std={np.std(dim_data):7.4f}")
-        else:
-            print("Warning: Could not extract stats for encoded dimensions, using raw actions")
-    else:
-        print("Warning: No normalization stats found in dataset, using raw actions")
-    
-    print(f"Encoded chunks shape: {encoded_chunks.shape}")
-    
-    # Train FAST tokenizer
-    tokenizer = train_fast_tokenizer(
-        encoded_chunks,
-        vocab_size=vocab_size,
-        scale=scale,
-    )
-    
-    # Compute compression statistics
-    compression_stats = compute_compression_stats(tokenizer, encoded_chunks)
-    
-    # Save tokenizer
-    if output_dir is None:
-        output_dir = f"fast_tokenizer_{repo_id.replace('/', '_')}"
-    output_path = Path(output_dir)
-    output_path.mkdir(parents=True, exist_ok=True)
-    
-    tokenizer.save_pretrained(output_path)
-    
-    # Save metadata
-    metadata = {
-        'repo_id': repo_id,
-        'vocab_size': vocab_size,
-        'scale': scale,
-        'encoded_dims': encoded_dims,
-        'encoded_dim_ranges': encoded_dim_ranges,
-        'total_encoded_dims': total_encoded_dims,
-        'delta_dims': delta_dims,
-        'delta_dim_list': delta_dim_list,
-        'use_delta_transform': use_delta_transform,
-        'state_key': state_key,
-        'normalization_mode': norm_mode.value,
-        'action_horizon': action_horizon,
-        'num_training_chunks': len(encoded_chunks),
-        'compression_stats': compression_stats,
-    }
-    
-    with open(output_path / "metadata.json", 'w') as f:
-        json.dump(metadata, f, indent=2)
-    
-    print(f"\nSaved FAST tokenizer to {output_path}")
-    print(f"Metadata: {json.dumps(metadata, indent=2)}")
-    
-    # Push to Hugging Face Hub if requested
-    if push_to_hub:
-        # Determine the hub repository ID
-        if hub_repo_id is None:
-            hub_repo_id = output_path.name
-            print(f"\nNo hub_repo_id provided, using: {hub_repo_id}")
-        
-        print(f"\nPushing tokenizer to Hugging Face Hub: {hub_repo_id}")
-        print(f"   Private: {hub_private}")
-        
-        try:
-            # Use the tokenizer's push_to_hub method
-            tokenizer.push_to_hub(
-                repo_id=hub_repo_id,
-                private=hub_private,
-                commit_message=f"Upload FAST tokenizer trained on {repo_id}"
-            )
-            
-            # Also upload the metadata.json file separately
-            api = HfApi()
-            api.upload_file(
-                path_or_fileobj=str(output_path / "metadata.json"),
-                path_in_repo="metadata.json",
-                repo_id=hub_repo_id,
-                repo_type="model",
-                commit_message="Upload tokenizer metadata"
-            )
-            
-            print(f"Successfully pushed tokenizer to: https://huggingface.co/{hub_repo_id}")
-        except Exception as e:
-            print(f"Error pushing to hub: {e}")
-            print("   Make sure you're logged in with `huggingface-cli login`")
-
-
-if __name__ == "__main__":
-    tyro.cli(main)

src/lerobot/policies/pi05/train_fast_tokenizer_example.md

@@ -1,101 +0,0 @@
-# Train FAST Tokenizer - Usage Examples
-
-This script trains a FAST (Factorized Action Sequence Tokenizer) on LeRobotDataset action data.
-
-## Basic Usage
-
-```bash
-python src/lerobot/policies/pi05/train_fast_tokenizer.py \
-    --repo_id "lerobot/aloha_sim_insertion_human" \
-    --action_horizon 10 \
-    --encoded_dims "0:7" \
-    --vocab_size 1024 \
-    --scale 10.0
-```
-
-## Parameters
-
-### Required
-- `--repo_id`: LeRobot dataset repository ID (e.g., "lerobot/aloha_sim_insertion_human")
-
-### Optional
-- `--root`: Root directory for dataset (default: ~/.cache/huggingface/lerobot)
-- `--action_horizon`: Number of future actions in each chunk (default: 10)
-- `--max_episodes`: Maximum number of episodes to use (default: None = all)
-- `--sample_fraction`: Fraction of chunks to sample per episode (default: 0.1)
-- `--encoded_dims`: Comma-separated dimension ranges to encode (default: "0:6,7:23")
-  - Example: "0:7" encodes dimensions 0-6
-  - Example: "0:3,6:9" encodes dimensions 0-2 and 6-8
-- `--delta_dims`: Comma-separated dimension indices for delta transform (default: None)
-  - Example: "0,1,2,3,4,5" applies delta transform to first 6 dimensions
-  - Delta transform: action[i] - state[i] for specified dimensions
-- `--state_key`: Dataset key for state observations (default: "observation.state")
-- `--vocab_size`: FAST vocabulary size / BPE vocab size (default: 1024)
-- `--scale`: DCT scaling factor (default: 10.0)
-- `--output_dir`: Directory to save tokenizer (default: ./fast_tokenizer_{repo_id})
-
-## Examples
-
-### Example 1: Train on full action space
-
-```bash
-python src/lerobot/policies/pi05/train_fast_tokenizer.py \
-    --repo_id "lerobot/pusht" \
-    --action_horizon 16 \
-    --encoded_dims "0:2" \
-    --vocab_size 512 \
-    --max_episodes 100
-```
-
-### Example 2: Train with delta transform
-
-```bash
-python src/lerobot/policies/pi05/train_fast_tokenizer.py \
-    --repo_id "lerobot/aloha_sim_insertion_human" \
-    --action_horizon 10 \
-    --encoded_dims "0:14" \
-    --delta_dims "0,1,2,3,4,5,6,7,8,9,10,11,12,13" \
-    --state_key "observation.state" \
-    --vocab_size 1024 \
-    --scale 10.0 \
-    --sample_fraction 0.2
-```
-
-### Example 3: Train on subset of dimensions
-
-```bash
-python src/lerobot/policies/pi05/train_fast_tokenizer.py \
-    --repo_id "lerobot/aloha_sim_insertion_human" \
-    --action_horizon 10 \
-    --encoded_dims "0:7" \
-    --vocab_size 1024 \
-    --output_dir "./my_tokenizer"
-```
-
-## Output
-
-The script saves:
-1. **Tokenizer files**: Trained FAST tokenizer (can be loaded with `AutoProcessor.from_pretrained()`)
-2. **metadata.json**: Contains:
-   - Configuration parameters
-   - Compression statistics (compression ratio, token lengths)
-   - Training dataset information
-
-## Understanding the Process
-
-1. **Load Dataset**: Loads the LeRobotDataset from HuggingFace
-2. **Extract Action Chunks**: Creates sliding windows of actions with specified horizon
-3. **Apply Delta Transform**: (Optional) Computes action deltas relative to current state
-4. **Select Encoded Dimensions**: Extracts only the dimensions to be encoded
-5. **Normalize**: Applies quantile normalization ([q01, q99] → [-1, 1])
-6. **Train Tokenizer**: Trains BPE tokenizer on DCT coefficients
-7. **Compute Stats**: Reports compression ratio and token length statistics
-8. **Save**: Saves tokenizer and metadata
-
-## Notes
-
-- **Normalization**: The script uses quantile normalization (q01, q99) from the dataset's statistics
-- **Sampling**: To speed up training, you can sample a fraction of chunks per episode
-- **Delta Transform**: Applied per-dimension to make actions relative to current state
-- **Compression**: FAST uses DCT + BPE to compress action sequences efficiently
-

src/lerobot/policies/pi05/train_fsdp.sh

@@ -1,28 +0,0 @@
-#!/bin/bash
-
-# FSDP training script for PI05 with aggressive memory optimization
-# Use this for large models that OOM with standard DDP
-
-accelerate launch --config_file /admin/home/jade_choghari/lerobot/fsdp_config.yaml \
-    $(which lerobot-train) \
-    --dataset.repo_id=local \
-    --dataset.root=/fsx/jade_choghari/data/libero \
-    --output_dir=/fsx/jade_choghari/outputs/libero_training_fsdp \
-    --job_name=libero_training_fsdp \
-    --policy.repo_id=jade_choghari/pi05-fast-libero-fsdp \
-    --policy.path=/fsx/jade_choghari/models/libero-pi-fast \
-    --policy.dtype=bfloat16 \
-    --steps=100000 \
-    --save_freq=10 \
-    --batch_size=8 \
-    --policy.device=cuda \
-    --policy.fast_only=true \
-    --policy.scheduler_warmup_steps=2000 \
-    --policy.scheduler_decay_steps=60000 \
-    --policy.scheduler_decay_lr=1e-5 \
-    --policy.gradient_checkpointing=false \
-    --wandb.enable=true \
-    --wandb.disable_artifact=true \
-    --wandb.project=pi05-libero-training-fsdp
-
-

src/lerobot/policies/pi05/train_libero.sh

@@ -1,24 +0,0 @@
-export CUDA_LAUNCH_BLOCKING=1 
-lerobot-train \
-    --dataset.repo_id=local \
-    --dataset.root=/fsx/jade_choghari/data/libero \
-    --output_dir=/fsx/jade_choghari/outputs/libero_training_fast_4 \
-    --job_name=libero_training_fast \
-    --policy.repo_id=jade_choghari/pi05-fast-libero \
-    --policy.path=/fsx/jade_choghari/models/pi05-base \
-    --policy.dtype=bfloat16 \
-    --steps=100000 \
-    --save_freq=20000 \
-    --batch_size=4 \
-    --policy.device=cuda \
-    --policy.fast_only=true \
-    --policy.scheduler_warmup_steps=1000 \
-    --policy.scheduler_decay_steps=30000 \
-    --policy.scheduler_decay_lr=1e-5 \
-    --policy.gradient_checkpointing=true \
-    --rename_map='{
-        "observation.images.image1": "observation.images.base_0_rgb",
-        "observation.images.image2": "observation.images.left_wrist_0_rgb",
-        }' \
-    --policy.empty_cameras=1 \
-# /fsx/jade_choghari/.cache/huggingface/lerobot/jadechoghari/collect-data

src/lerobot/policies/pi05/train_multi.sbatch

@@ -1,15 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=pi05-train
-#SBATCH --time=24:00:00
-#SBATCH --qos=high
-#SBATCH --gres=gpu:8
-#SBATCH --mem=256G
-#SBATCH --partition=hopper-prod
-#SBATCH --output=/fsx/jade_choghari/logs/%x-%j.out
-#SBATCH --error=/fsx/jade_choghari/logs/%x-%j.err
-
-srun \
-  --container-image=/fsx/michel_aractingi/docker_images/huggingface+lerobot-gpu+dev.sqsh \
-  --container-mounts=/fsx/jade_choghari \
-  --container-workdir=$HOME/lerobot \
-  bash /admin/home/jade_choghari/lerobot/src/lerobot/policies/pi05/train_multi.sh

src/lerobot/policies/pi05/train_multi.sh

@@ -1,36 +0,0 @@
-#!/bin/bash
-set -euxo pipefail
-
-# Source YOUR Miniforge conda (mounted from FSX)
-source /fsx/jade_choghari/miniforge3/etc/profile.d/conda.sh
-
-conda activate lerobot
-accelerate launch --mixed_precision=bf16 --multi_gpu --num_processes=8 \
-    $(which lerobot-train) \
-    --dataset.repo_id=local \
-    --dataset.root=/fsx/jade_choghari/data/libero \
-    --output_dir=/fsx/jade_choghari/outputs/libero_training_fast_mean_1 \
-    --job_name=libero_training_fast \
-    --policy.repo_id=jade_choghari/pi05-fast-libero \
-    --policy.path=/fsx/jade_choghari/models/pi05-base \
-    --policy.dtype=bfloat16 \
-    --steps=100000 \
-    --save_freq=20000 \
-    --batch_size=4 \
-    --policy.device=cuda \
-    --policy.fast_only=true \
-    --policy.scheduler_warmup_steps=4000 \
-    --policy.scheduler_decay_steps=100000 \
-    --policy.scheduler_decay_lr=1e-5 \
-    --policy.gradient_checkpointing=true \
-    --policy.chunk_size=10 \
-    --policy.n_action_steps=10 \
-    --policy.max_action_tokens=256 \
-    --rename_map='{
-        "observation.images.image1": "observation.images.base_0_rgb",
-        "observation.images.image2": "observation.images.left_wrist_0_rgb",
-        }' \
-    --policy.empty_cameras=1 \
-    --wandb.enable=true \
-    --wandb.disable_artifact=true \
-    --wandb.project=pi05-libero-training \

src/lerobot/policies/xvla/action_hub.py

@@ -269,160 +269,27 @@ class AGIBOTEE6DActionSpace(BaseActionSpace):
 
 @register_action("franka_joint7")
 class FrankaJoint7ActionSpace(BaseActionSpace):
-    """
-    Franka Panda joint-space: 7 joints, with gripper.
-
-    - Real robot action dim: 7
-    - Model-facing dim: 20 (padded with zeros)
-      compatible with pretrained VLA models expecting 20D.
-    """
-
-    dim_action = 20  # model dimension
-    REAL_DIM = 7  # actual Franka joints
+    """Franka Panda joint-space: 7 joints, no gripper."""
 
+    dim_action = 7
     JOINTS_SCALE = 1.0
 
     def __init__(self):
         super().__init__()
         self.mse = nn.MSELoss()
 
-    def _pad_to_model_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Pad 7 → 20 dims (zeros for the dummy channels)."""
-        if x is None:
-            return None
-        if x.size(-1) == self.dim_action:
-            return x
-        if x.size(-1) != self.REAL_DIM:
-            raise ValueError(
-                f"Expected last dim to be {self.REAL_DIM} or {self.dim_action}, got {x.size(-1)}"
-            )
-
-        pad_shape = list(x.shape[:-1]) + [self.dim_action - self.REAL_DIM]  # 13 zeros
-        pad = x.new_zeros(pad_shape)
-        return torch.cat([x, pad], dim=-1)
-
-    def _trim_to_real_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Trim model output 20 → 7 dims."""
-        return x[..., : self.REAL_DIM]
-
     def compute_loss(self, pred, target):
-        """
-        pred :  [B, T, 20]
-        target : [B, T, 7] or [B, T, 20]
-
-        Only compute MSE on the first 7 dims.
-        """
-        pred = self._pad_to_model_dim(pred)
-        target = self._pad_to_model_dim(target)
-
-        assert pred.shape == target.shape
-
-        joints_loss = (
-            self.mse(
-                pred[:, :, : self.REAL_DIM],  # use only the first 7 joints
-                target[:, :, : self.REAL_DIM],
-            )
-            * self.JOINTS_SCALE
-        )
-
+        assert pred.shape == target.shape, "pred/target shapes must match"
+        joints_loss = self.mse(pred, target) * self.JOINTS_SCALE
         return {"joints_loss": joints_loss}
 
     def preprocess(self, proprio, action, mode="train"):
-        """
-        During training:
-        - Pad [7] → [20]
-        """
-        return proprio, self._pad_to_model_dim(action)
+        """No preprocessing needed for 7 joint actions."""
+        return proprio, action
 
     def postprocess(self, action: torch.Tensor) -> torch.Tensor:
-        """
-        After model prediction:
-        - Trim [20] → [7] for real robot control.
-        """
-        return self._trim_to_real_dim(action)
-
-
-@register_action("auto")
-class AutoActionSpace(BaseActionSpace):
-    """
-    Auto-detecting action space that adapts to any action dimension.
-
-    - Auto-detects the real action dimension from the policy feature
-    - Model outputs max_dim for compatibility with pretrained models
-    - Loss is computed only on the first real_dim dimensions
-    - Postprocess trims output back to real_dim
-
-    Args:
-        real_dim: The actual action dimension from the dataset/policy feature
-        max_dim: The model's output dimension for pretrained VLA compatibility
-    """
-
-    JOINTS_SCALE = 1.0
-
-    def __init__(self, real_dim: int, max_dim: int):
-        super().__init__()
-        self.real_dim = real_dim
-        self.dim_action = max_dim  # Model-facing dimension
-        self.mse = nn.MSELoss()
-
-    def _pad_to_model_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Pad real_dim → max_dim (zeros for the dummy channels)."""
-        if x is None:
-            return None
-        if x.size(-1) == self.dim_action:
-            return x
-        if x.size(-1) != self.real_dim:
-            # If dimension doesn't match either, pad/trim to real_dim first
-            if x.size(-1) < self.real_dim:
-                pad_shape = list(x.shape[:-1]) + [self.real_dim - x.size(-1)]
-                pad = x.new_zeros(pad_shape)
-                x = torch.cat([x, pad], dim=-1)
-            else:
-                x = x[..., : self.real_dim]
-
-        pad_shape = list(x.shape[:-1]) + [self.dim_action - self.real_dim]
-        pad = x.new_zeros(pad_shape)
-        return torch.cat([x, pad], dim=-1)
-
-    def _trim_to_real_dim(self, x: torch.Tensor) -> torch.Tensor:
-        """Trim model output max_dim → real_dim."""
-        return x[..., : self.real_dim]
-
-    def compute_loss(self, pred: torch.Tensor, target: torch.Tensor) -> dict[str, torch.Tensor]:
-        """
-        Compute loss only on the first real_dim dimensions.
-
-        pred:   [B, T, max_dim] from the model
-        target: [B, T, real_dim] or [B, T, max_dim]
-
-        Loss = MSE(pred[:,:,:real_dim], target[:,:,:real_dim])
-        """
-        pred = self._pad_to_model_dim(pred)
-        target = self._pad_to_model_dim(target)
-        assert pred.shape == target.shape, f"Shape mismatch: pred {pred.shape} vs target {target.shape}"
-
-        # only compute loss on the real dimensions
-        joints_loss = (
-            self.mse(
-                pred[:, :, : self.real_dim],
-                target[:, :, : self.real_dim],
-            )
-            * self.JOINTS_SCALE
-        )
-
-        return {"joints_loss": joints_loss}
-
-    def preprocess(self, proprio: torch.Tensor, action: torch.Tensor, mode: str = "train"):
-        """
-        Pad action from real_dim to max_dim for the model.
-        """
-        return proprio, self._pad_to_model_dim(action)
-
-    def postprocess(self, action: torch.Tensor) -> torch.Tensor:
-        """
-        Trim model output from max_dim to real_dim for real robot control.
-        """
-        return self._trim_to_real_dim(action)
+        """Return directly (no sigmoid since no gripper)."""
+        return action
 
 
 @register_action("so101_bimanual")
@@ -582,7 +449,6 @@ __all__ = [
     "JointActionSpace",
     "AGIBOTEE6DActionSpace",
     "FrankaJoint7ActionSpace",
-    "AutoActionSpace",
     "BimanualSO101ActionSpace",
     "ACTION_REGISTRY",
 ]

src/lerobot/policies/xvla/configuration_xvla.py

@@ -23,7 +23,7 @@ from typing import TYPE_CHECKING, Any
 
 from lerobot.configs.policies import PreTrainedConfig
 from lerobot.configs.types import FeatureType, NormalizationMode, PolicyFeature
-from lerobot.optim.optimizers import XVLAAdamWConfig
+from lerobot.optim.optimizers import AdamWConfig
 from lerobot.optim.schedulers import CosineDecayWithWarmupSchedulerConfig
 from lerobot.utils.constants import OBS_IMAGES
 
@@ -57,7 +57,7 @@ class XVLAConfig(PreTrainedConfig):
         default_factory=lambda: {
             "VISUAL": NormalizationMode.IDENTITY,
             "STATE": NormalizationMode.IDENTITY,
-            "ACTION": NormalizationMode.IDENTITY,
+            "ACTION": NormalizationMode.MEAN_STD,
         }
     )
 
@@ -84,7 +84,6 @@ class XVLAConfig(PreTrainedConfig):
     num_denoising_steps: int = 10
     use_proprio: bool = True
     max_state_dim: int = 32
-    max_action_dim: int = 20  # Maximum action dimension for padding (used by "auto" action mode)
     domain_feature_key: str | None = None
 
     # Vision preprocessing
@@ -94,20 +93,17 @@ class XVLAConfig(PreTrainedConfig):
 
     # Freezing options for VLM components
     # By default, VLM encoders are frozen and only policy transformer + soft prompts train
-    freeze_vision_encoder: bool = False  # Freeze VLM vision encoder weights
-    freeze_language_encoder: bool = False  # Freeze VLM language encoder weights
+    freeze_vision_encoder: bool = True  # Freeze VLM vision encoder weights
+    freeze_language_encoder: bool = True  # Freeze VLM language encoder weights
     train_policy_transformer: bool = True  # Allow policy transformer to train
     train_soft_prompts: bool = True  # Allow soft prompts to train
 
     # Training presets
     optimizer_lr: float = 1e-4
-    optimizer_betas: tuple[float, float] = (0.9, 0.99)
+    optimizer_betas: tuple[float, float] = (0.9, 0.95)
     optimizer_eps: float = 1e-8
-    optimizer_weight_decay: float = 0.0
+    optimizer_weight_decay: float = 1e-4
     optimizer_grad_clip_norm: float = 10.0
-    # Soft-prompt LR settings (for optional warm-up)
-    optimizer_soft_prompt_lr_scale: float = 1.0  # Scale factor for soft-prompt LR
-    optimizer_soft_prompt_warmup_lr_scale: float | None = None  # Start scale for warmup (e.g., 0.01)
 
     scheduler_warmup_steps: int = 1_000
     scheduler_decay_steps: int = 30_000
@@ -164,22 +160,13 @@ class XVLAConfig(PreTrainedConfig):
                         shape=(3, height, width),
                     )
 
-    def get_optimizer_preset(self) -> XVLAAdamWConfig:
-        """Return the XVLA-specific optimizer with differential learning rates.
-
-        This optimizer applies:
-        - 1/10 LR for VLM parameters (stable optimization)
-        - Full LR for transformer/action head
-        - Configurable LR for soft-prompts (with optional warm-up)
-        """
-        return XVLAAdamWConfig(
+    def get_optimizer_preset(self) -> AdamWConfig:
+        return AdamWConfig(
             lr=self.optimizer_lr,
             betas=self.optimizer_betas,
             eps=self.optimizer_eps,
             weight_decay=self.optimizer_weight_decay,
             grad_clip_norm=self.optimizer_grad_clip_norm,
-            soft_prompt_lr_scale=self.optimizer_soft_prompt_lr_scale,
-            soft_prompt_warmup_lr_scale=self.optimizer_soft_prompt_warmup_lr_scale,
         )
 
     def get_scheduler_preset(self) -> CosineDecayWithWarmupSchedulerConfig:

src/lerobot/policies/xvla/modeling_florence2.py

@@ -2508,9 +2508,6 @@ class Florence2ForConditionalGeneration(Florence2PreTrainedModel):
         return model_embeds
 
     def _encode_image(self, pixel_values):
-        # Cast pixel_values to model's dtype
-        pixel_values = pixel_values.to(dtype=self.vision_tower.convs[0].proj.weight.dtype)
-
         if len(pixel_values.shape) == 4:
             batch_size, channels, height, width = pixel_values.shape
             num_frames = 1

src/lerobot/policies/xvla/modeling_xvla.py

@@ -55,23 +55,7 @@ class XVLAModel(nn.Module):
         self.config = config
         self.chunk_size: int = config.chunk_size
         self.use_proprio: bool = config.use_proprio
-
-        # Build action space with auto-detection for "auto" mode
-        if config.action_mode.lower() == "auto":
-            # Auto-detect real action dim from config.action_feature
-            real_dim = (
-                config.action_feature.shape[-1]
-                if config.action_feature is not None
-                else config.max_action_dim
-            )
-            self.action_space = build_action_space(
-                config.action_mode.lower(),
-                real_dim=real_dim,
-                max_dim=config.max_action_dim,
-            )
-        else:
-            self.action_space = build_action_space(config.action_mode.lower())
-
+        self.action_space = build_action_space(config.action_mode.lower())
         self.dim_action = self.action_space.dim_action
         self.dim_proprio = proprio_dim
 
@@ -200,20 +184,12 @@ class XVLAModel(nn.Module):
         proprio: torch.Tensor,
         action: torch.Tensor,
     ) -> dict[str, torch.Tensor]:
-        """
-        Forward pass for the XVLA model.
-        """
-        target_dtype = self._get_target_dtype()
-        image_input = image_input.to(dtype=target_dtype)
-        proprio = proprio.to(dtype=target_dtype)
-        action = action.to(dtype=target_dtype)
-
         enc = self.forward_vlm(input_ids, image_input, image_mask)
 
         batch_size = input_ids.shape[0]
         t = (
-            torch.rand(1, device=input_ids.device, dtype=target_dtype)
-            + torch.arange(batch_size, device=input_ids.device, dtype=target_dtype) / batch_size
+            torch.rand(1, device=input_ids.device)
+            + torch.arange(batch_size, device=input_ids.device) / batch_size
         ) % (1 - 1e-5)
 
         action_noisy = torch.randn_like(action) * t.view(-1, 1, 1) + action * (1 - t).view(-1, 1, 1)
@@ -239,22 +215,17 @@ class XVLAModel(nn.Module):
         steps: int,
     ) -> torch.Tensor:
         self.eval()
-
-        target_dtype = self._get_target_dtype()
-        image_input = image_input.to(dtype=target_dtype)
-        proprio = proprio.to(dtype=target_dtype)
-
         enc = self.forward_vlm(input_ids, image_input, image_mask)
 
         batch_size = input_ids.shape[0]
         action_dim = self.dim_action
 
-        x1 = torch.randn(batch_size, self.chunk_size, action_dim, device=proprio.device, dtype=target_dtype)
+        x1 = torch.randn(batch_size, self.chunk_size, action_dim, device=proprio.device, dtype=proprio.dtype)
         action = torch.zeros_like(x1)
 
         steps = max(1, int(steps))
         for i in range(steps, 0, -1):
-            t = torch.full((batch_size,), i / steps, device=proprio.device, dtype=target_dtype)
+            t = torch.full((batch_size,), i / steps, device=proprio.device, dtype=proprio.dtype)
             x_t = x1 * t.view(-1, 1, 1) + action * (1 - t).view(-1, 1, 1)
             proprio_m, x_t_m = self.action_space.preprocess(proprio, x_t)
             action = self.transformer(
@@ -287,13 +258,8 @@ class XVLAPolicy(PreTrainedPolicy):
         }
 
     def get_optim_params(self) -> dict:
-        """Return trainable named parameters for optimization.
-
-        Returns a dict of name -> param for all trainable parameters.
-        This enables the xvla-adamw optimizer to apply differential learning rates
-        based on parameter names (e.g., 1/10 LR for VLM components).
-        """
-        return dict(filter(lambda kv: kv[1].requires_grad, self.named_parameters()))
+        """Return only trainable parameters for optimization."""
+        return filter(lambda p: p.requires_grad, self.parameters())
 
     def _prepare_state(self, batch: dict[str, Tensor], batch_size: int, device: torch.device) -> Tensor:
         if not self.config.use_proprio or OBS_STATE not in batch:
@@ -383,6 +349,17 @@ class XVLAPolicy(PreTrainedPolicy):
             "proprio": proprio,
         }
 
+    def _trim_action_dim(self, actions: Tensor) -> Tensor:
+        feature = self.config.action_feature
+        if feature is None:
+            return actions
+        desired_dim = self.model.dim_action
+        if desired_dim == actions.shape[-1]:
+            return actions
+        if desired_dim < actions.shape[-1]:
+            return actions[..., :desired_dim]
+        return pad_vector(actions, desired_dim)
+
     def forward(self, batch: dict[str, Tensor]) -> tuple[Tensor, dict]:
         inputs = self._build_model_inputs(batch)
         targets = self._prepare_action_targets(batch)
@@ -396,6 +373,7 @@ class XVLAPolicy(PreTrainedPolicy):
     def _get_action_chunk(self, batch: dict[str, Tensor]) -> Tensor:
         inputs = self._build_model_inputs(batch)
         actions = self.model.generate_actions(**inputs, steps=self.config.num_denoising_steps)
+        actions = self._trim_action_dim(actions)
         return actions
 
     @torch.no_grad()

src/lerobot/policies/xvla/processor_xvla.py

@@ -310,19 +310,16 @@ class XVLAImageToFloatProcessorStep(ProcessorStep):
             if key in obs and isinstance(obs[key], torch.Tensor):
                 tensor = obs[key]
 
-                min_val = tensor.min().item()
-                max_val = tensor.max().item()
-
-                if max_val <= 1.0:
-                    obs[key] = tensor.float()  # ensure float dtype, but no division
-                    continue
                 # Validate that values are in [0, 255] range if requested
-                if self.validate_range and (min_val < 0.0 or max_val > 255.0):
-                    raise ValueError(
-                        f"Image '{key}' has values outside [0, 255] range: "
-                        f"min={min_val:.4f}, max={max_val:.4f}. "
-                        f"Cannot convert to [0, 1] range."
-                    )
+                if self.validate_range:
+                    min_val = tensor.min().item()
+                    max_val = tensor.max().item()
+                    if min_val < 0.0 or max_val > 255.0:
+                        raise ValueError(
+                            f"Image '{key}' has values outside [0, 255] range: "
+                            f"min={min_val:.4f}, max={max_val:.4f}. "
+                            f"Cannot convert to [0, 1] range."
+                        )
 
                 # Convert to float and divide by 255
                 obs[key] = tensor.float() / 255.0

src/lerobot/processor/__init__.py

@@ -75,7 +75,7 @@ from .policy_robot_bridge import (
     RobotActionToPolicyActionProcessorStep,
 )
 from .rename_processor import RenameObservationsProcessorStep
-from .tokenizer_processor import TokenizerProcessorStep, ActionTokenizerProcessorStep
+from .tokenizer_processor import TokenizerProcessorStep
 
 __all__ = [
     "ActionProcessorStep",

src/lerobot/processor/converters.py

@@ -168,12 +168,10 @@ def _extract_complementary_data(batch: dict[str, Any]) -> dict[str, Any]:
     """
     pad_keys = {k: v for k, v in batch.items() if "_is_pad" in k}
     task_key = {"task": batch["task"]} if "task" in batch else {}
-    user_prompt_key = {"user_prompt": batch["user_prompt"]} if "user_prompt" in batch else {}
-    subtask_key = {"subtask": batch["subtask"]} if "subtask" in batch else {}
     index_key = {"index": batch["index"]} if "index" in batch else {}
     task_index_key = {"task_index": batch["task_index"]} if "task_index" in batch else {}
 
-    return {**pad_keys, **task_key, **index_key, **task_index_key, **user_prompt_key, **subtask_key}
+    return {**pad_keys, **task_key, **index_key, **task_index_key}
 
 
 def create_transition(

src/lerobot/processor/rename_processor.py

@@ -47,6 +47,7 @@ class RenameObservationsProcessorStep(ObservationProcessorStep):
                 processed_obs[self.rename_map[key]] = value
             else:
                 processed_obs[key] = value
+
         return processed_obs
 
     def get_config(self) -> dict[str, Any]:

src/lerobot/robots/earthrover_mini_plus/__init__.py

@@ -1,20 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .config_earthrover_mini_plus import EarthRoverMiniPlusConfig
-from .robot_earthrover_mini_plus import EarthRoverMiniPlus
-
-__all__ = ["EarthRoverMiniPlus", "EarthRoverMiniPlusConfig"]

src/lerobot/robots/earthrover_mini_plus/config_earthrover_mini_plus.py

@@ -1,35 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Configuration for EarthRover Mini Plus robot."""
-
-from dataclasses import dataclass
-
-from ..config import RobotConfig
-
-
-@RobotConfig.register_subclass("earthrover_mini_plus")
-@dataclass
-class EarthRoverMiniPlusConfig(RobotConfig):
-    """Configuration for EarthRover Mini Plus robot using Frodobots SDK.
-
-    This robot uses cloud-based control via the Frodobots SDK HTTP API.
-    Camera frames are accessed directly through SDK HTTP endpoints.
-
-    Attributes:
-        sdk_url: URL of the Frodobots SDK server (default: http://localhost:8000)
-    """
-
-    sdk_url: str = "http://localhost:8000"

src/lerobot/robots/earthrover_mini_plus/earthrover_mini_plus.mdx

@@ -1 +0,0 @@
-../../../../docs/source/earthrover_mini_plus.mdx

src/lerobot/robots/earthrover_mini_plus/robot_earthrover_mini_plus.py

@@ -1,473 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""EarthRover Mini Plus robot using Frodobots SDK."""
-
-import base64
-import logging
-from functools import cached_property
-from typing import Any
-
-import cv2
-import numpy as np
-import requests
-
-from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnectedError
-
-from ..robot import Robot
-from .config_earthrover_mini_plus import EarthRoverMiniPlusConfig
-
-logger = logging.getLogger(__name__)
-
-# Action feature keys
-ACTION_LINEAR_VEL = "linear.vel"
-ACTION_ANGULAR_VEL = "angular.vel"
-
-# Observation feature keys
-OBS_FRONT = "front"
-OBS_REAR = "rear"
-OBS_LINEAR_VEL = "linear.vel"
-OBS_BATTERY_LEVEL = "battery.level"
-OBS_ORIENTATION_DEG = "orientation.deg"
-OBS_GPS_LATITUDE = "gps.latitude"
-OBS_GPS_LONGITUDE = "gps.longitude"
-OBS_GPS_SIGNAL = "gps.signal"
-OBS_SIGNAL_LEVEL = "signal.level"
-OBS_VIBRATION = "vibration"
-OBS_LAMP_STATE = "lamp.state"
-
-
-class EarthRoverMiniPlus(Robot):
-    """
-    EarthRover Mini Plus robot controlled via Frodobots SDK HTTP API.
-
-    This robot uses cloud-based control through the Frodobots SDK instead of direct
-    hardware connection. Cameras stream via WebRTC through Agora cloud, and control
-    commands are sent via HTTP POST requests.
-
-    The robot supports:
-    - Dual cameras (front and rear) accessed via SDK HTTP endpoints
-    - Linear and angular velocity control
-    - Battery and orientation telemetry
-
-    Attributes:
-        config: Robot configuration
-        sdk_base_url: URL of the Frodobots SDK server (default: http://localhost:8000)
-    """
-
-    config_class = EarthRoverMiniPlusConfig
-    name = "earthrover_mini_plus"
-
-    def __init__(self, config: EarthRoverMiniPlusConfig):
-        """Initialize EarthRover Mini Plus robot.
-
-        Args:
-            config: Robot configuration including SDK URL
-        """
-        super().__init__(config)
-        self.config = config
-        self.sdk_base_url = "http://localhost:8000"
-
-        # Empty cameras dict for compatibility with recording script
-        # Cameras are accessed directly via SDK, not through Camera objects
-        self.cameras = {}
-        self._is_connected = False
-
-        # Cache for camera frames (fallback when requests fail)
-        self._last_front_frame = None
-        self._last_rear_frame = None
-
-        # Cache for robot telemetry data (fallback when requests fail)
-        self._last_robot_data = None
-
-        logger.info(f"Initialized {self.name} with SDK at {self.sdk_base_url}")
-
-    @property
-    def is_connected(self) -> bool:
-        """Check if robot is connected to SDK."""
-        return self._is_connected
-
-    def connect(self, calibrate: bool = True) -> None:
-        """Connect to robot via Frodobots SDK.
-
-        Args:
-            calibrate: Not used for SDK-based robot (kept for API compatibility)
-
-        Raises:
-            DeviceAlreadyConnectedError: If robot is already connected
-            DeviceNotConnectedError: If cannot connect to SDK server
-        """
-        if self._is_connected:
-            raise DeviceAlreadyConnectedError(f"{self.name} is already connected")
-
-        # Verify SDK is running and accessible
-        try:
-            response = requests.get(f"{self.sdk_base_url}/data", timeout=10.0)
-            if response.status_code != 200:
-                raise DeviceNotConnectedError(
-                    f"Cannot connect to SDK at {self.sdk_base_url}. "
-                    "Make sure it's running: hypercorn main:app --reload"
-                )
-        except requests.RequestException as e:
-            raise DeviceNotConnectedError(f"Cannot connect to SDK at {self.sdk_base_url}: {e}") from e
-
-        self._is_connected = True
-        logger.info(f"{self.name} connected to SDK")
-
-        if calibrate:
-            self.calibrate()
-
-    def calibrate(self) -> None:
-        """Calibration not needed for SDK-based robot."""
-        logger.info("Calibration not required for SDK-based robot")
-
-    @property
-    def is_calibrated(self) -> bool:
-        """SDK robot doesn't require calibration.
-
-        Returns:
-            bool: Always True for SDK-based robots
-        """
-        return True
-
-    def configure(self) -> None:
-        """Configure robot (no-op for SDK-based robot)."""
-        pass
-
-    @cached_property
-    def observation_features(self) -> dict[str, type | tuple]:
-        """Define the observation space for dataset recording.
-
-        Returns:
-            dict: Observation features with types/shapes:
-                - front: (480, 640, 3) - Front camera RGB image
-                - rear: (480, 640, 3) - Rear camera RGB image
-                - linear.vel: float - Current speed (0-1, SDK reports only positive speeds)
-                - battery.level: float - Battery level (0-1, normalized from 0-100)
-                - orientation.deg: float - Robot orientation (0-1, normalized from raw value)
-                - gps.latitude: float - GPS latitude coordinate
-                - gps.longitude: float - GPS longitude coordinate
-                - gps.signal: float - GPS signal strength (0-1, normalized from percentage)
-                - signal.level: float - Network signal level (0-1, normalized from 0-5)
-                - vibration: float - Vibration sensor reading
-                - lamp.state: float - Lamp state (0=off, 1=on)
-        """
-        return {
-            # Cameras (height, width, channels)
-            OBS_FRONT: (480, 640, 3),
-            OBS_REAR: (480, 640, 3),
-            # Motion state
-            OBS_LINEAR_VEL: float,
-            # Robot state
-            OBS_BATTERY_LEVEL: float,
-            OBS_ORIENTATION_DEG: float,
-            # GPS
-            OBS_GPS_LATITUDE: float,
-            OBS_GPS_LONGITUDE: float,
-            OBS_GPS_SIGNAL: float,
-            # Sensors
-            OBS_SIGNAL_LEVEL: float,
-            OBS_VIBRATION: float,
-            OBS_LAMP_STATE: float,
-        }
-
-    @cached_property
-    def action_features(self) -> dict[str, type]:
-        """Define the action space.
-
-        Returns:
-            dict: Action features with types:
-                - linear.vel: float - Target linear velocity
-                - angular.vel: float - Target angular velocity
-        """
-        return {
-            ACTION_LINEAR_VEL: float,
-            ACTION_ANGULAR_VEL: float,
-        }
-
-    def get_observation(self) -> dict[str, Any]:
-        """Get current robot observation from SDK.
-
-        Returns:
-            dict: Observation containing:
-                - front: Front camera image (480, 640, 3) in RGB format
-                - rear: Rear camera image (480, 640, 3) in RGB format
-                - linear.vel: Current speed (0-1, SDK reports only positive speeds)
-                - battery.level: Battery level (0-1, normalized from 0-100)
-                - orientation.deg: Robot orientation (0-1, normalized from raw value)
-                - gps.latitude: GPS latitude coordinate
-                - gps.longitude: GPS longitude coordinate
-                - gps.signal: GPS signal strength (0-1, normalized from percentage)
-                - signal.level: Network signal level (0-1, normalized from 0-5)
-                - vibration: Vibration sensor reading
-                - lamp.state: Lamp state (0=off, 1=on)
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-
-        Note:
-            Camera frames are retrieved from SDK endpoints /v2/front and /v2/rear.
-            Frames are decoded from base64 and converted from BGR to RGB format.
-            Robot telemetry is retrieved from /data endpoint.
-            All SDK values are normalized to appropriate ranges for dataset recording.
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        observation = {}
-
-        # Get camera images from SDK
-        frames = self._get_camera_frames()
-        observation[OBS_FRONT] = frames["front"]
-        observation[OBS_REAR] = frames["rear"]
-
-        # Get robot state from SDK
-        robot_data = self._get_robot_data()
-
-        # Motion state
-        observation[OBS_LINEAR_VEL] = robot_data["speed"] / 100.0  # Normalize 0-100 to 0-1
-
-        # Robot state
-        observation[OBS_BATTERY_LEVEL] = robot_data["battery"] / 100.0  # Normalize 0-100 to 0-1
-        observation[OBS_ORIENTATION_DEG] = robot_data["orientation"] / 360.0  # Normalize to 0-1
-
-        # GPS data
-        observation[OBS_GPS_LATITUDE] = robot_data["latitude"]
-        observation[OBS_GPS_LONGITUDE] = robot_data["longitude"]
-        observation[OBS_GPS_SIGNAL] = robot_data["gps_signal"] / 100.0  # Normalize percentage to 0-1
-
-        # Sensors
-        observation[OBS_SIGNAL_LEVEL] = robot_data["signal_level"] / 5.0  # Normalize 0-5 to 0-1
-        observation[OBS_VIBRATION] = robot_data["vibration"]
-        observation[OBS_LAMP_STATE] = float(robot_data["lamp"])  # 0 or 1
-
-        return observation
-
-    def send_action(self, action: dict[str, Any]) -> dict[str, Any]:
-        """Send action to robot via SDK.
-
-        Args:
-            action: Action dict with keys:
-                - linear.vel: Target linear velocity (-1 to 1)
-                - angular.vel: Target angular velocity (-1 to 1)
-
-        Returns:
-            dict: The action that was sent (matches action_features keys)
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-
-        Note:
-            Actions are sent to SDK via POST /control endpoint.
-            SDK expects commands in range [-1, 1].
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        # Extract action values and convert to float
-        linear = float(action.get(ACTION_LINEAR_VEL, 0.0))
-        angular = float(action.get(ACTION_ANGULAR_VEL, 0.0))
-
-        # Send command to SDK
-        try:
-            self._send_command_to_sdk(linear, angular)
-        except Exception as e:
-            logger.error(f"Error sending action: {e}")
-
-        # Return action in format matching action_features
-        return {
-            ACTION_LINEAR_VEL: linear,
-            ACTION_ANGULAR_VEL: angular,
-        }
-
-    def disconnect(self) -> None:
-        """Disconnect from robot.
-
-        Stops the robot and closes connection to SDK.
-
-        Raises:
-            DeviceNotConnectedError: If robot is not connected
-        """
-        if not self._is_connected:
-            raise DeviceNotConnectedError(f"{self.name} is not connected")
-
-        # Stop the robot before disconnecting
-        try:
-            self._send_command_to_sdk(0.0, 0.0)
-        except Exception as e:
-            logger.warning(f"Failed to stop robot during disconnect: {e}")
-
-        self._is_connected = False
-        logger.info(f"{self.name} disconnected")
-
-    # Private helper methods for SDK communication
-
-    def _get_camera_frames(self) -> dict[str, np.ndarray]:
-        """Get camera frames from SDK using v2 endpoints with caching fallback.
-
-        Returns:
-            dict: Dictionary with 'front' and 'rear' keys containing:
-                - Current frame (if request succeeds)
-                - Cached frame (if request fails but cache exists)
-                - Zero array (if request fails and no cache exists yet)
-
-        Note:
-            Uses /v2/front and /v2/rear endpoints which are 15x faster than /screenshot.
-            Images are base64 encoded, resized to 640x480, and converted from BGR to RGB.
-            If request fails, returns the last successfully retrieved frame (cached).
-        """
-        frames = {}
-
-        # Get front camera
-        try:
-            response = requests.get(f"{self.sdk_base_url}/v2/front", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                if "front_frame" in data and data["front_frame"]:
-                    front_img = self._decode_base64_image(data["front_frame"])
-                    if front_img is not None:
-                        # Resize and convert BGR to RGB
-                        front_img = cv2.resize(front_img, (640, 480))
-                        front_rgb = cv2.cvtColor(front_img, cv2.COLOR_BGR2RGB)
-                        frames["front"] = front_rgb
-                        # Cache the successful frame
-                        self._last_front_frame = front_rgb
-        except Exception as e:
-            logger.warning(f"Error fetching front camera: {e}")
-
-        # Fallback: use cache or zero array
-        if "front" not in frames:
-            if self._last_front_frame is not None:
-                frames["front"] = self._last_front_frame
-            else:
-                frames["front"] = np.zeros((480, 640, 3), dtype=np.uint8)
-
-        # Get rear camera
-        try:
-            response = requests.get(f"{self.sdk_base_url}/v2/rear", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                if "rear_frame" in data and data["rear_frame"]:
-                    rear_img = self._decode_base64_image(data["rear_frame"])
-                    if rear_img is not None:
-                        # Resize and convert BGR to RGB
-                        rear_img = cv2.resize(rear_img, (640, 480))
-                        rear_rgb = cv2.cvtColor(rear_img, cv2.COLOR_BGR2RGB)
-                        frames["rear"] = rear_rgb
-                        # Cache the successful frame
-                        self._last_rear_frame = rear_rgb
-        except Exception as e:
-            logger.warning(f"Error fetching rear camera: {e}")
-
-        # Fallback: use cache or zero array
-        if "rear" not in frames:
-            if self._last_rear_frame is not None:
-                frames["rear"] = self._last_rear_frame
-            else:
-                frames["rear"] = np.zeros((480, 640, 3), dtype=np.uint8)
-
-        return frames
-
-    def _decode_base64_image(self, base64_string: str) -> np.ndarray | None:
-        """Decode base64 string to image.
-
-        Args:
-            base64_string: Base64 encoded image string
-
-        Returns:
-            np.ndarray: Decoded image in BGR format (OpenCV default), or None if decoding fails
-        """
-        try:
-            img_bytes = base64.b64decode(base64_string)
-            nparr = np.frombuffer(img_bytes, np.uint8)
-            img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
-            return img  # Return in BGR format (OpenCV default)
-        except Exception as e:
-            logger.error(f"Error decoding image: {e}")
-            return None
-
-    def _get_robot_data(self) -> dict:
-        """Get robot telemetry data from SDK.
-
-        Returns:
-            dict: Robot telemetry data including battery, speed, orientation, GPS, etc:
-                - Current data (if request succeeds)
-                - Cached data (if request fails but cache exists)
-                - Default values (if request fails and no cache exists yet)
-
-        Note:
-            Uses /data endpoint which provides comprehensive robot state.
-            If request fails, returns the last successfully retrieved data (cached).
-        """
-        try:
-            response = requests.get(f"{self.sdk_base_url}/data", timeout=2.0)
-            if response.status_code == 200:
-                data = response.json()
-                # Cache the successful data
-                self._last_robot_data = data
-                return data
-        except Exception as e:
-            logger.warning(f"Error fetching robot data: {e}")
-
-        # Fallback: use cache or default values
-        if self._last_robot_data is not None:
-            return self._last_robot_data
-        else:
-            # Return dict with default values (used only on first failure before any cache exists)
-            return {
-                "speed": 0,
-                "battery": 0,
-                "orientation": 0,
-                "latitude": 0.0,
-                "longitude": 0.0,
-                "gps_signal": 0,
-                "signal_level": 0,
-                "vibration": 0.0,
-                "lamp": 0,
-            }
-
-    def _send_command_to_sdk(self, linear: float, angular: float, lamp: int = 0) -> bool:
-        """Send control command to SDK.
-
-        Args:
-            linear: Linear velocity command (-1 to 1)
-            angular: Angular velocity command (-1 to 1)
-            lamp: Lamp control (0=off, 1=on)
-
-        Returns:
-            bool: True if command sent successfully, False otherwise
-
-        Note:
-            Uses POST /control endpoint. Commands are sent as JSON payload.
-        """
-        try:
-            payload = {
-                "command": {
-                    "linear": linear,
-                    "angular": angular,
-                    "lamp": lamp,
-                }
-            }
-
-            response = requests.post(
-                f"{self.sdk_base_url}/control",
-                json=payload,
-                timeout=1.0,
-            )
-
-            return response.status_code == 200
-        except Exception as e:
-            logger.error(f"Error sending command: {e}")
-            return False

src/lerobot/robots/unitree_g1/__init__.py

@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .config_unitree_g1 import UnitreeG1Config
-from .unitree_g1 import UnitreeG1

src/lerobot/robots/unitree_g1/config_unitree_g1.py

@@ -1,55 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from dataclasses import dataclass, field
-
-from ..config import RobotConfig
-
-_GAINS: dict[str, dict[str, list[float]]] = {
-    "left_leg": {
-        "kp": [150, 150, 150, 300, 40, 40],
-        "kd": [2, 2, 2, 4, 2, 2],
-    },  # pitch, roll, yaw, knee, ankle_pitch, ankle_roll
-    "right_leg": {"kp": [150, 150, 150, 300, 40, 40], "kd": [2, 2, 2, 4, 2, 2]},
-    "waist": {"kp": [250, 250, 250], "kd": [5, 5, 5]},  # yaw, roll, pitch
-    "left_arm": {"kp": [80, 80, 80, 80], "kd": [3, 3, 3, 3]},  # shoulder_pitch/roll/yaw, elbow
-    "left_wrist": {"kp": [40, 40, 40], "kd": [1.5, 1.5, 1.5]},  # roll, pitch, yaw
-    "right_arm": {"kp": [80, 80, 80, 80], "kd": [3, 3, 3, 3]},
-    "right_wrist": {"kp": [40, 40, 40], "kd": [1.5, 1.5, 1.5]},
-    "other": {"kp": [80, 80, 80, 80, 80, 80], "kd": [3, 3, 3, 3, 3, 3]},
-}
-
-
-def _build_gains() -> tuple[list[float], list[float]]:
-    """Build kp and kd lists from body-part groupings."""
-    kp = [v for g in _GAINS.values() for v in g["kp"]]
-    kd = [v for g in _GAINS.values() for v in g["kd"]]
-    return kp, kd
-
-
-_DEFAULT_KP, _DEFAULT_KD = _build_gains()
-
-
-@RobotConfig.register_subclass("unitree_g1")
-@dataclass
-class UnitreeG1Config(RobotConfig):
-    kp: list[float] = field(default_factory=lambda: _DEFAULT_KP.copy())
-    kd: list[float] = field(default_factory=lambda: _DEFAULT_KD.copy())
-
-    control_dt: float = 1.0 / 250.0  # 250Hz
-
-    # socket config for ZMQ bridge
-    robot_ip: str = "192.168.123.164"

src/lerobot/robots/unitree_g1/g1_utils.py

@@ -1,89 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from enum import IntEnum
-
-# ruff: noqa: N801, N815
-
-NUM_MOTORS = 35
-
-
-class G1_29_JointArmIndex(IntEnum):
-    # Left arm
-    kLeftShoulderPitch = 15
-    kLeftShoulderRoll = 16
-    kLeftShoulderYaw = 17
-    kLeftElbow = 18
-    kLeftWristRoll = 19
-    kLeftWristPitch = 20
-    kLeftWristyaw = 21
-
-    # Right arm
-    kRightShoulderPitch = 22
-    kRightShoulderRoll = 23
-    kRightShoulderYaw = 24
-    kRightElbow = 25
-    kRightWristRoll = 26
-    kRightWristPitch = 27
-    kRightWristYaw = 28
-
-
-class G1_29_JointIndex(IntEnum):
-    # Left leg
-    kLeftHipPitch = 0
-    kLeftHipRoll = 1
-    kLeftHipYaw = 2
-    kLeftKnee = 3
-    kLeftAnklePitch = 4
-    kLeftAnkleRoll = 5
-
-    # Right leg
-    kRightHipPitch = 6
-    kRightHipRoll = 7
-    kRightHipYaw = 8
-    kRightKnee = 9
-    kRightAnklePitch = 10
-    kRightAnkleRoll = 11
-
-    kWaistYaw = 12
-    kWaistRoll = 13
-    kWaistPitch = 14
-
-    # Left arm
-    kLeftShoulderPitch = 15
-    kLeftShoulderRoll = 16
-    kLeftShoulderYaw = 17
-    kLeftElbow = 18
-    kLeftWristRoll = 19
-    kLeftWristPitch = 20
-    kLeftWristyaw = 21
-
-    # Right arm
-    kRightShoulderPitch = 22
-    kRightShoulderRoll = 23
-    kRightShoulderYaw = 24
-    kRightElbow = 25
-    kRightWristRoll = 26
-    kRightWristPitch = 27
-    kRightWristYaw = 28
-
-    # not used
-    kNotUsedJoint0 = 29
-    kNotUsedJoint1 = 30
-    kNotUsedJoint2 = 31
-    kNotUsedJoint3 = 32
-    kNotUsedJoint4 = 33
-    kNotUsedJoint5 = 34

src/lerobot/robots/unitree_g1/run_g1_server.py

@@ -1,212 +0,0 @@
-#!/usr/bin/env python3
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-DDS-to-ZMQ bridge server for Unitree G1 robot.
-
-This server runs on the robot and forwards:
-- Robot state (LowState) from DDS to ZMQ (for remote clients)
-- Robot commands (LowCmd) from ZMQ to DDS (from remote clients)
-
-Uses JSON for secure serialization instead of pickle.
-"""
-
-import base64
-import contextlib
-import json
-import threading
-import time
-from typing import Any
-
-import zmq
-from unitree_sdk2py.comm.motion_switcher.motion_switcher_client import MotionSwitcherClient
-from unitree_sdk2py.core.channel import ChannelFactoryInitialize, ChannelPublisher, ChannelSubscriber
-from unitree_sdk2py.idl.default import unitree_hg_msg_dds__LowCmd_
-from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowCmd_ as hg_LowCmd, LowState_ as hg_LowState
-from unitree_sdk2py.utils.crc import CRC
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"  # action to robot
-kTopicLowState = "rt/lowstate"  # observation from robot
-
-LOWCMD_PORT = 6000
-LOWSTATE_PORT = 6001
-NUM_MOTORS = 35
-
-
-def lowstate_to_dict(msg: hg_LowState) -> dict[str, Any]:
-    """Convert LowState SDK message to a JSON-serializable dictionary."""
-    motor_states = []
-    for i in range(NUM_MOTORS):
-        temp = msg.motor_state[i].temperature
-        avg_temp = float(sum(temp) / len(temp)) if isinstance(temp, list) else float(temp)
-        motor_states.append(
-            {
-                "q": float(msg.motor_state[i].q),
-                "dq": float(msg.motor_state[i].dq),
-                "tau_est": float(msg.motor_state[i].tau_est),
-                "temperature": avg_temp,
-            }
-        )
-
-    return {
-        "motor_state": motor_states,
-        "imu_state": {
-            "quaternion": [float(x) for x in msg.imu_state.quaternion],
-            "gyroscope": [float(x) for x in msg.imu_state.gyroscope],
-            "accelerometer": [float(x) for x in msg.imu_state.accelerometer],
-            "rpy": [float(x) for x in msg.imu_state.rpy],
-            "temperature": float(msg.imu_state.temperature),
-        },
-        # Encode bytes as base64 for JSON compatibility
-        "wireless_remote": base64.b64encode(bytes(msg.wireless_remote)).decode("ascii"),
-        "mode_machine": int(msg.mode_machine),
-    }
-
-
-def dict_to_lowcmd(data: dict[str, Any]) -> hg_LowCmd:
-    """Convert dictionary back to LowCmd SDK message."""
-    cmd = unitree_hg_msg_dds__LowCmd_()
-    cmd.mode_pr = data.get("mode_pr", 0)
-    cmd.mode_machine = data.get("mode_machine", 0)
-
-    for i, motor_data in enumerate(data.get("motor_cmd", [])):
-        cmd.motor_cmd[i].mode = motor_data.get("mode", 0)
-        cmd.motor_cmd[i].q = motor_data.get("q", 0.0)
-        cmd.motor_cmd[i].dq = motor_data.get("dq", 0.0)
-        cmd.motor_cmd[i].kp = motor_data.get("kp", 0.0)
-        cmd.motor_cmd[i].kd = motor_data.get("kd", 0.0)
-        cmd.motor_cmd[i].tau = motor_data.get("tau", 0.0)
-
-    return cmd
-
-
-def state_forward_loop(
-    lowstate_sub: ChannelSubscriber,
-    lowstate_sock: zmq.Socket,
-    state_period: float,
-) -> None:
-    """Read observation from DDS and forward to ZMQ clients."""
-    last_state_time = 0.0
-
-    while True:
-        # read from DDS
-        msg = lowstate_sub.Read()
-        if msg is None:
-            continue
-
-        now = time.time()
-        # optional downsampling (if robot dds rate > state_period)
-        if now - last_state_time >= state_period:
-            # Convert to dict and serialize with JSON
-            state_dict = lowstate_to_dict(msg)
-            payload = json.dumps({"topic": kTopicLowState, "data": state_dict}).encode("utf-8")
-            # if no subscribers / tx buffer full, just drop
-            with contextlib.suppress(zmq.Again):
-                lowstate_sock.send(payload, zmq.NOBLOCK)
-            last_state_time = now
-
-
-def cmd_forward_loop(
-    lowcmd_sock: zmq.Socket,
-    lowcmd_pub_debug: ChannelPublisher,
-    crc: CRC,
-) -> None:
-    """Receive commands from ZMQ and forward to DDS."""
-    while True:
-        payload = lowcmd_sock.recv()
-        msg_dict = json.loads(payload.decode("utf-8"))
-
-        topic = msg_dict.get("topic", "")
-        cmd_data = msg_dict.get("data", {})
-
-        # Reconstruct LowCmd object from dict
-        cmd = dict_to_lowcmd(cmd_data)
-
-        # recompute crc
-        cmd.crc = crc.Crc(cmd)
-
-        if topic == kTopicLowCommand_Debug:
-            lowcmd_pub_debug.Write(cmd)
-
-
-def main() -> None:
-    """Main entry point for the robot server bridge."""
-    # initialize DDS
-    ChannelFactoryInitialize(0)
-
-    # stop all active publishers on the robot
-    msc = MotionSwitcherClient()
-    msc.SetTimeout(5.0)
-    msc.Init()
-
-    status, result = msc.CheckMode()
-    while result is not None and "name" in result and result["name"]:
-        msc.ReleaseMode()
-        status, result = msc.CheckMode()
-        time.sleep(1.0)
-
-    crc = CRC()
-
-    # initialize DDS publisher
-    lowcmd_pub_debug = ChannelPublisher(kTopicLowCommand_Debug, hg_LowCmd)
-    lowcmd_pub_debug.Init()
-
-    # initialize DDS subscriber
-    lowstate_sub = ChannelSubscriber(kTopicLowState, hg_LowState)
-    lowstate_sub.Init()
-
-    # initialize ZMQ
-    ctx = zmq.Context.instance()
-
-    # receive commands from remote client
-    lowcmd_sock = ctx.socket(zmq.PULL)
-    lowcmd_sock.bind(f"tcp://0.0.0.0:{LOWCMD_PORT}")
-
-    # publish state to remote clients
-    lowstate_sock = ctx.socket(zmq.PUB)
-    lowstate_sock.bind(f"tcp://0.0.0.0:{LOWSTATE_PORT}")
-
-    state_period = 0.002  # ~500 hz
-
-    # start observation forwarding thread
-    t_state = threading.Thread(
-        target=state_forward_loop,
-        args=(lowstate_sub, lowstate_sock, state_period),
-        daemon=True,
-    )
-    t_state.start()
-
-    # start action forwarding thread
-    t_cmd = threading.Thread(
-        target=cmd_forward_loop,
-        args=(lowcmd_sock, lowcmd_pub_debug, crc),
-        daemon=True,
-    )
-    t_cmd.start()
-
-    print("bridge running (lowstate -> zmq, lowcmd -> dds)")
-    # keep main thread alive so daemon threads don't exit
-    try:
-        while True:
-            time.sleep(1.0)
-    except KeyboardInterrupt:
-        print("shutting down bridge...")
-
-
-if __name__ == "__main__":
-    main()

src/lerobot/robots/unitree_g1/unitree_g1.py

@@ -1,267 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import struct
-import threading
-import time
-from dataclasses import dataclass, field
-from functools import cached_property
-from typing import Any
-
-import numpy as np
-from unitree_sdk2py.idl.default import unitree_hg_msg_dds__LowCmd_
-from unitree_sdk2py.idl.unitree_hg.msg.dds_ import (
-    LowCmd_ as hg_LowCmd,
-    LowState_ as hg_LowState,
-)
-from unitree_sdk2py.utils.crc import CRC
-
-from lerobot.robots.unitree_g1.g1_utils import G1_29_JointIndex
-from lerobot.robots.unitree_g1.unitree_sdk2_socket import (
-    ChannelFactoryInitialize,
-    ChannelPublisher,
-    ChannelSubscriber,
-)
-
-from ..robot import Robot
-from .config_unitree_g1 import UnitreeG1Config
-
-logger = logging.getLogger(__name__)
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"
-kTopicLowState = "rt/lowstate"
-
-G1_29_Num_Motors = 35
-G1_23_Num_Motors = 35
-H1_2_Num_Motors = 35
-H1_Num_Motors = 20
-
-
-@dataclass
-class MotorState:
-    q: float | None = None  # position
-    dq: float | None = None  # velocity
-    tau_est: float | None = None  # estimated torque
-    temperature: float | None = None  # motor temperature
-
-
-@dataclass
-class IMUState:
-    quaternion: np.ndarray | None = None  # [w, x, y, z]
-    gyroscope: np.ndarray | None = None  # [x, y, z] angular velocity (rad/s)
-    accelerometer: np.ndarray | None = None  # [x, y, z] linear acceleration (m/s²)
-    rpy: np.ndarray | None = None  # [roll, pitch, yaw] (rad)
-    temperature: float | None = None  # IMU temperature
-
-
-# g1 observation class
-@dataclass
-class G1_29_LowState:  # noqa: N801
-    motor_state: list[MotorState] = field(
-        default_factory=lambda: [MotorState() for _ in range(G1_29_Num_Motors)]
-    )
-    imu_state: IMUState = field(default_factory=IMUState)
-    wireless_remote: Any = None  # Raw wireless remote data
-    mode_machine: int = 0  # Robot mode
-
-
-class DataBuffer:
-    def __init__(self):
-        self.data = None
-        self.lock = threading.Lock()
-
-    def get_data(self):
-        with self.lock:
-            return self.data
-
-    def set_data(self, data):
-        with self.lock:
-            self.data = data
-
-
-class UnitreeG1(Robot):
-    config_class = UnitreeG1Config
-    name = "unitree_g1"
-
-    # unitree remote controller
-    class RemoteController:
-        def __init__(self):
-            self.lx = 0
-            self.ly = 0
-            self.rx = 0
-            self.ry = 0
-            self.button = [0] * 16
-
-        def set(self, data):
-            # wireless_remote
-            keys = struct.unpack("H", data[2:4])[0]
-            for i in range(16):
-                self.button[i] = (keys & (1 << i)) >> i
-            self.lx = struct.unpack("f", data[4:8])[0]
-            self.rx = struct.unpack("f", data[8:12])[0]
-            self.ry = struct.unpack("f", data[12:16])[0]
-            self.ly = struct.unpack("f", data[20:24])[0]
-
-    def __init__(self, config: UnitreeG1Config):
-        super().__init__(config)
-
-        logger.info("Initialize UnitreeG1...")
-
-        self.config = config
-
-        self.control_dt = config.control_dt
-
-        # connect robot
-        self.connect()
-
-        # initialize direct motor control interface
-        self.lowcmd_publisher = ChannelPublisher(kTopicLowCommand_Debug, hg_LowCmd)
-        self.lowcmd_publisher.Init()
-        self.lowstate_subscriber = ChannelSubscriber(kTopicLowState, hg_LowState)
-        self.lowstate_subscriber.Init()
-        self.lowstate_buffer = DataBuffer()
-
-        # initialize subscribe thread to read robot state
-        self.subscribe_thread = threading.Thread(target=self._subscribe_motor_state)
-        self.subscribe_thread.daemon = True
-        self.subscribe_thread.start()
-
-        while not self.is_connected:
-            time.sleep(0.1)
-
-        # initialize hg's lowcmd msg
-        self.crc = CRC()
-        self.msg = unitree_hg_msg_dds__LowCmd_()
-        self.msg.mode_pr = 0
-
-        # Wait for first state message to arrive
-        lowstate = None
-        while lowstate is None:
-            lowstate = self.lowstate_buffer.get_data()
-            if lowstate is None:
-                time.sleep(0.01)
-            logger.warning("[UnitreeG1] Waiting for robot state...")
-        logger.warning("[UnitreeG1] Connected to robot.")
-        self.msg.mode_machine = lowstate.mode_machine
-
-        # initialize all motors with unified kp/kd from config
-        self.kp = np.array(config.kp, dtype=np.float32)
-        self.kd = np.array(config.kd, dtype=np.float32)
-
-        for id in G1_29_JointIndex:
-            self.msg.motor_cmd[id].mode = 1
-            self.msg.motor_cmd[id].kp = self.kp[id.value]
-            self.msg.motor_cmd[id].kd = self.kd[id.value]
-            self.msg.motor_cmd[id].q = lowstate.motor_state[id.value].q
-
-        # Initialize remote controller
-        self.remote_controller = self.RemoteController()
-
-    def _subscribe_motor_state(self):  # polls robot state @ 250Hz
-        while True:
-            start_time = time.time()
-            msg = self.lowstate_subscriber.Read()
-            if msg is not None:
-                lowstate = G1_29_LowState()
-
-                # Capture motor states
-                for id in range(G1_29_Num_Motors):
-                    lowstate.motor_state[id].q = msg.motor_state[id].q
-                    lowstate.motor_state[id].dq = msg.motor_state[id].dq
-                    lowstate.motor_state[id].tau_est = msg.motor_state[id].tau_est
-                    lowstate.motor_state[id].temperature = msg.motor_state[id].temperature
-
-                # Capture IMU state
-                lowstate.imu_state.quaternion = list(msg.imu_state.quaternion)
-                lowstate.imu_state.gyroscope = list(msg.imu_state.gyroscope)
-                lowstate.imu_state.accelerometer = list(msg.imu_state.accelerometer)
-                lowstate.imu_state.rpy = list(msg.imu_state.rpy)
-                lowstate.imu_state.temperature = msg.imu_state.temperature
-
-                # Capture wireless remote data
-                lowstate.wireless_remote = msg.wireless_remote
-
-                # Capture mode_machine
-                lowstate.mode_machine = msg.mode_machine
-
-                self.lowstate_buffer.set_data(lowstate)
-
-            current_time = time.time()
-            all_t_elapsed = current_time - start_time
-            sleep_time = max(0, (self.control_dt - all_t_elapsed))  # maintain constant control dt
-            time.sleep(sleep_time)
-
-    @cached_property
-    def action_features(self) -> dict[str, type]:
-        return {f"{G1_29_JointIndex(motor).name}.pos": float for motor in G1_29_JointIndex}
-
-    def calibrate(self) -> None:  # robot is already calibrated
-        pass
-
-    def configure(self) -> None:
-        pass
-
-    def connect(self, calibrate: bool = True) -> None:  # connect to DDS
-        ChannelFactoryInitialize(0)
-
-    def disconnect(self):
-        pass
-
-    def get_observation(self) -> dict[str, Any]:
-        return self.lowstate_buffer.get_data()
-
-    @property
-    def is_calibrated(self) -> bool:
-        return True
-
-    @property
-    def is_connected(self) -> bool:
-        return self.lowstate_buffer.get_data() is not None
-
-    @property
-    def _motors_ft(self) -> dict[str, type]:
-        return {f"{G1_29_JointIndex(motor).name}.pos": float for motor in G1_29_JointIndex}
-
-    @property
-    def _cameras_ft(self) -> dict[str, tuple]:
-        return {
-            cam: (self.config.cameras[cam].height, self.config.cameras[cam].width, 3) for cam in self.cameras
-        }
-
-    @cached_property
-    def observation_features(self) -> dict[str, type | tuple]:
-        return {**self._motors_ft, **self._cameras_ft}
-
-    def send_action(self, action: dict[str, Any]) -> dict[str, Any]:
-        self.msg.crc = self.crc.Crc(action)
-        self.lowcmd_publisher.Write(action)
-        return action
-
-    def get_gravity_orientation(self, quaternion):  # get gravity orientation from quaternion
-        """Get gravity orientation from quaternion."""
-        qw = quaternion[0]
-        qx = quaternion[1]
-        qy = quaternion[2]
-        qz = quaternion[3]
-
-        gravity_orientation = np.zeros(3)
-        gravity_orientation[0] = 2 * (-qz * qx + qw * qy)
-        gravity_orientation[1] = -2 * (qz * qy + qw * qx)
-        gravity_orientation[2] = 1 - 2 * (qw * qw + qz * qz)
-        return gravity_orientation

src/lerobot/robots/unitree_g1/unitree_sdk2_socket.py

@@ -1,168 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import base64
-import json
-from typing import Any
-
-import zmq
-
-from lerobot.robots.unitree_g1.config_unitree_g1 import UnitreeG1Config
-
-_ctx: zmq.Context | None = None
-_lowcmd_sock: zmq.Socket | None = None
-_lowstate_sock: zmq.Socket | None = None
-
-LOWCMD_PORT = 6000
-LOWSTATE_PORT = 6001
-
-# DDS topic names follow Unitree SDK naming conventions
-# ruff: noqa: N816
-kTopicLowCommand_Debug = "rt/lowcmd"
-
-
-class LowStateMsg:
-    """
-    Wrapper class that mimics the Unitree SDK LowState_ message structure.
-
-    Reconstructs the message from deserialized JSON data to maintain
-    compatibility with existing code that expects SDK message objects.
-    """
-
-    class MotorState:
-        """Motor state data for a single joint."""
-
-        def __init__(self, data: dict[str, Any]) -> None:
-            self.q: float = data.get("q", 0.0)
-            self.dq: float = data.get("dq", 0.0)
-            self.tau_est: float = data.get("tau_est", 0.0)
-            self.temperature: float = data.get("temperature", 0.0)
-
-    class IMUState:
-        """IMU sensor data."""
-
-        def __init__(self, data: dict[str, Any]) -> None:
-            self.quaternion: list[float] = data.get("quaternion", [1.0, 0.0, 0.0, 0.0])
-            self.gyroscope: list[float] = data.get("gyroscope", [0.0, 0.0, 0.0])
-            self.accelerometer: list[float] = data.get("accelerometer", [0.0, 0.0, 0.0])
-            self.rpy: list[float] = data.get("rpy", [0.0, 0.0, 0.0])
-            self.temperature: float = data.get("temperature", 0.0)
-
-    def __init__(self, data: dict[str, Any]) -> None:
-        """Initialize from deserialized JSON data."""
-        self.motor_state = [self.MotorState(m) for m in data.get("motor_state", [])]
-        self.imu_state = self.IMUState(data.get("imu_state", {}))
-        # Decode base64-encoded wireless_remote bytes
-        wireless_b64 = data.get("wireless_remote", "")
-        self.wireless_remote: bytes = base64.b64decode(wireless_b64) if wireless_b64 else b""
-        self.mode_machine: int = data.get("mode_machine", 0)
-
-
-def lowcmd_to_dict(topic: str, msg: Any) -> dict[str, Any]:
-    """Convert LowCmd message to a JSON-serializable dictionary."""
-    motor_cmds = []
-    # Iterate over all motor commands in the message
-    for i in range(len(msg.motor_cmd)):
-        motor_cmds.append(
-            {
-                "mode": int(msg.motor_cmd[i].mode),
-                "q": float(msg.motor_cmd[i].q),
-                "dq": float(msg.motor_cmd[i].dq),
-                "kp": float(msg.motor_cmd[i].kp),
-                "kd": float(msg.motor_cmd[i].kd),
-                "tau": float(msg.motor_cmd[i].tau),
-            }
-        )
-
-    return {
-        "topic": topic,
-        "data": {
-            "mode_pr": int(msg.mode_pr),
-            "mode_machine": int(msg.mode_machine),
-            "motor_cmd": motor_cmds,
-        },
-    }
-
-
-def ChannelFactoryInitialize(*args: Any, **kwargs: Any) -> None:  # noqa: N802
-    """
-    Initialize ZMQ sockets for robot communication.
-
-    This function mimics the Unitree SDK's ChannelFactoryInitialize but uses
-    ZMQ sockets to connect to the robot server bridge instead of DDS.
-    """
-    global _ctx, _lowcmd_sock, _lowstate_sock
-
-    # read socket config
-    config = UnitreeG1Config()
-    robot_ip = config.robot_ip
-
-    ctx = zmq.Context.instance()
-    _ctx = ctx
-
-    # lowcmd: send robot commands
-    lowcmd_sock = ctx.socket(zmq.PUSH)
-    lowcmd_sock.setsockopt(zmq.CONFLATE, 1)  # keep only last message
-    lowcmd_sock.connect(f"tcp://{robot_ip}:{LOWCMD_PORT}")
-    _lowcmd_sock = lowcmd_sock
-
-    # lowstate: receive robot observations
-    lowstate_sock = ctx.socket(zmq.SUB)
-    lowstate_sock.setsockopt(zmq.CONFLATE, 1)  # keep only last message
-    lowstate_sock.connect(f"tcp://{robot_ip}:{LOWSTATE_PORT}")
-    lowstate_sock.setsockopt_string(zmq.SUBSCRIBE, "")
-    _lowstate_sock = lowstate_sock
-
-
-class ChannelPublisher:
-    """ZMQ-based publisher that sends commands to the robot server."""
-
-    def __init__(self, topic: str, msg_type: type) -> None:
-        self.topic = topic
-        self.msg_type = msg_type
-
-    def Init(self) -> None:  # noqa: N802
-        """Initialize the publisher (no-op for ZMQ)."""
-        pass
-
-    def Write(self, msg: Any) -> None:  # noqa: N802
-        """Serialize and send a command message to the robot."""
-        if _lowcmd_sock is None:
-            raise RuntimeError("ChannelFactoryInitialize must be called first")
-
-        payload = json.dumps(lowcmd_to_dict(self.topic, msg)).encode("utf-8")
-        _lowcmd_sock.send(payload)
-
-
-class ChannelSubscriber:
-    """ZMQ-based subscriber that receives state from the robot server."""
-
-    def __init__(self, topic: str, msg_type: type) -> None:
-        self.topic = topic
-        self.msg_type = msg_type
-
-    def Init(self) -> None:  # noqa: N802
-        """Initialize the subscriber (no-op for ZMQ)."""
-        pass
-
-    def Read(self) -> LowStateMsg:  # noqa: N802
-        """Receive and deserialize a state message from the robot."""
-        if _lowstate_sock is None:
-            raise RuntimeError("ChannelFactoryInitialize must be called first")
-
-        payload = _lowstate_sock.recv()
-        msg_dict = json.loads(payload.decode("utf-8"))
-        return LowStateMsg(msg_dict.get("data", {}))

src/lerobot/scripts/lerobot_calibrate.py

@@ -52,7 +52,7 @@ from lerobot.teleoperators import (  # noqa: F401
     so100_leader,
     so101_leader,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.utils import init_logging
 
 
@@ -84,7 +84,7 @@ def calibrate(cfg: CalibrateConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     calibrate()
 
 

src/lerobot/scripts/lerobot_eval.py

@@ -82,7 +82,6 @@ from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
 from lerobot.processor import PolicyAction, PolicyProcessorPipeline
 from lerobot.utils.constants import ACTION, DONE, OBS_STR, REWARD
-from lerobot.utils.import_utils import register_third_party_plugins
 from lerobot.utils.io_utils import write_video
 from lerobot.utils.random_utils import set_seed
 from lerobot.utils.utils import (
@@ -173,7 +172,6 @@ def rollout(
         observation = env_preprocessor(observation)
 
         observation = preprocessor(observation)
-
         with torch.inference_mode():
             action = policy.select_action(observation)
         action = postprocessor(action)
@@ -794,7 +792,6 @@ def eval_policy_all(
 
 def main():
     init_logging()
-    register_third_party_plugins()
     eval_main()
 
 

src/lerobot/scripts/lerobot_find_joint_limits.py

@@ -15,23 +15,18 @@
 # limitations under the License.
 
 """
-Script to find joint limits and end-effector bounds via teleoperation.
+Simple script to control a robot from teleoperation.
 
 Example:
 
 ```shell
 lerobot-find-joint-limits \
   --robot.type=so100_follower \
-  --robot.port=/dev/tty.usbmodem58760432981 \
+  --robot.port=/dev/tty.usbmodem58760431541 \
   --robot.id=black \
   --teleop.type=so100_leader \
-  --teleop.port=/dev/tty.usbmodem58760434471 \
-  --teleop.id=blue \
-  --urdf_path=<user>/SO-ARM100-main/Simulation/SO101/so101_new_calib.urdf \
-  --target_frame_name=gripper \
-  --teleop_time_s=30 \
-  --warmup_time_s=5 \
-  --control_loop_fps=30
+  --teleop.port=/dev/tty.usbmodem58760431551 \
+  --teleop.id=blue
 ```
 """
 
@@ -47,7 +42,6 @@ from lerobot.robots import (  # noqa: F401
     koch_follower,
     make_robot_from_config,
     so100_follower,
-    so101_follower,
 )
 from lerobot.teleoperators import (  # noqa: F401
     TeleoperatorConfig,
@@ -55,7 +49,6 @@ from lerobot.teleoperators import (  # noqa: F401
     koch_leader,
     make_teleoperator_from_config,
     so100_leader,
-    so101_leader,
 )
 from lerobot.utils.robot_utils import precise_sleep
 
@@ -64,19 +57,10 @@ from lerobot.utils.robot_utils import precise_sleep
 class FindJointLimitsConfig:
     teleop: TeleoperatorConfig
     robot: RobotConfig
-
-    # Path to URDF file for kinematics
-    # NOTE: It is highly recommended to use the urdf in the SO-ARM100 repo:
-    # https://github.com/TheRobotStudio/SO-ARM100/blob/main/Simulation/SO101/so101_new_calib.urdf
-    urdf_path: str
-    target_frame_name: str = "gripper"
-
-    # Duration of the recording phase in seconds
+    # Limit the maximum frames per second. By default, no limit.
     teleop_time_s: float = 30
-    # Duration of the warmup phase in seconds
-    warmup_time_s: float = 5
-    # Control loop frequency
-    control_loop_fps: int = 30
+    # Display all cameras on screen
+    display_data: bool = False
 
 
 @draccus.wrap()
@@ -84,127 +68,53 @@ def find_joint_and_ee_bounds(cfg: FindJointLimitsConfig):
     teleop = make_teleoperator_from_config(cfg.teleop)
     robot = make_robot_from_config(cfg.robot)
 
-    print(f"Connecting to robot: {cfg.robot.type}...")
     teleop.connect()
     robot.connect()
-    print("Devices connected.")
 
-    # Initialize Kinematics
-    try:
-        kinematics = RobotKinematics(cfg.urdf_path, cfg.target_frame_name)
-    except Exception as e:
-        print(f"Error initializing kinematics: {e}")
-        print("Ensure URDF path and target frame name are correct.")
-        robot.disconnect()
-        teleop.disconnect()
-        return
+    start_episode_t = time.perf_counter()
+    robot_type = getattr(robot.config, "robot_type", "so101")
+    if "so100" in robot_type or "so101" in robot_type:
+        # Note to be compatible with the rest of the codebase,
+        # we are using the new calibration method for so101 and so100
+        robot_type = "so_new_calibration"
+    kinematics = RobotKinematics(cfg.robot.urdf_path, cfg.robot.target_frame_name)
 
-    # Initialize variables
-    max_pos = None
-    min_pos = None
-    max_ee = None
-    min_ee = None
+    # Initialize min/max values
+    observation = robot.get_observation()
+    joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+    ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
 
-    start_t = time.perf_counter()
-    warmup_done = False
+    max_pos = joint_positions.copy()
+    min_pos = joint_positions.copy()
+    max_ee = ee_pos.copy()
+    min_ee = ee_pos.copy()
 
-    print("\n" + "=" * 40)
-    print(f"  WARMUP PHASE ({cfg.warmup_time_s}s)")
-    print("  Move the robot freely to ensure control works.")
-    print("  Data is NOT being recorded yet.")
-    print("=" * 40 + "\n")
+    while True:
+        action = teleop.get_action()
+        robot.send_action(action)
 
-    try:
-        while True:
-            t0 = time.perf_counter()
+        observation = robot.get_observation()
+        joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+        ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
 
-            # 1. Teleoperation Control Loop
-            action = teleop.get_action()
-            robot.send_action(action)
+        # Skip initial warmup period
+        if (time.perf_counter() - start_episode_t) < 5:
+            continue
 
-            # 2. Read Observations
-            observation = robot.get_observation()
-            joint_positions = np.array([observation[f"{key}.pos"] for key in robot.bus.motors])
+        # Update min/max values
+        max_ee = np.maximum(max_ee, ee_pos)
+        min_ee = np.minimum(min_ee, ee_pos)
+        max_pos = np.maximum(max_pos, joint_positions)
+        min_pos = np.minimum(min_pos, joint_positions)
 
-            # 3. Calculate Kinematics
-            # Forward kinematics to get (x, y, z) translation
-            ee_pos = kinematics.forward_kinematics(joint_positions)[:3, 3]
+        if time.perf_counter() - start_episode_t > cfg.teleop_time_s:
+            print(f"Max ee position {np.round(max_ee, 4).tolist()}")
+            print(f"Min ee position {np.round(min_ee, 4).tolist()}")
+            print(f"Max joint pos position {np.round(max_pos, 4).tolist()}")
+            print(f"Min joint pos position {np.round(min_pos, 4).tolist()}")
+            break
 
-            current_time = time.perf_counter()
-            elapsed = current_time - start_t
-
-            # 4. Handle Phases
-            if elapsed < cfg.warmup_time_s:
-                # Still in warmup
-                pass
-
-            else:
-                # Phase Transition: Warmup -> Recording
-                if not warmup_done:
-                    print("\n" + "=" * 40)
-                    print("  RECORDING STARTED")
-                    print("  Move robot to ALL joint limits.")
-                    print("  Press Ctrl+C to stop early and save results.")
-                    print("=" * 40 + "\n")
-
-                    # Initialize limits with current position at start of recording
-                    max_pos = joint_positions.copy()
-                    min_pos = joint_positions.copy()
-                    max_ee = ee_pos.copy()
-                    min_ee = ee_pos.copy()
-                    warmup_done = True
-
-                # Update Limits
-                max_ee = np.maximum(max_ee, ee_pos)
-                min_ee = np.minimum(min_ee, ee_pos)
-                max_pos = np.maximum(max_pos, joint_positions)
-                min_pos = np.minimum(min_pos, joint_positions)
-
-                # Time check
-                recording_time = elapsed - cfg.warmup_time_s
-                remaining = cfg.teleop_time_s - recording_time
-
-                # Simple throttle for print statements (every ~1 sec)
-                if int(recording_time * 100) % 100 == 0:
-                    print(f"Time remaining: {remaining:.1f}s", end="\r")
-
-                if recording_time > cfg.teleop_time_s:
-                    print("\nTime limit reached.")
-                    break
-
-            precise_sleep(max(1.0 / cfg.control_loop_fps - (time.perf_counter() - t0), 0.0))
-
-    except KeyboardInterrupt:
-        print("\n\nInterrupted by user. Stopping safely...")
-
-    finally:
-        # Safety: Disconnect devices
-        print("\nDisconnecting devices...")
-        robot.disconnect()
-        teleop.disconnect()
-
-    # Results Output
-    if max_pos is not None:
-        print("\n" + "=" * 40)
-        print("FINAL RESULTS")
-        print("=" * 40)
-
-        # Rounding for readability
-        r_max_ee = np.round(max_ee, 4).tolist()
-        r_min_ee = np.round(min_ee, 4).tolist()
-        r_max_pos = np.round(max_pos, 4).tolist()
-        r_min_pos = np.round(min_pos, 4).tolist()
-
-        print("\n# End Effector Bounds (x, y, z):")
-        print(f"max_ee = {r_max_ee}")
-        print(f"min_ee = {r_min_ee}")
-
-        print("\n# Joint Position Limits (radians):")
-        print(f"max_pos = {r_max_pos}")
-        print(f"min_pos = {r_min_pos}")
-
-    else:
-        print("No data recorded (exited during warmup).")
+        precise_sleep(0.01)
 
 
 def main():

src/lerobot/scripts/lerobot_record.py

@@ -93,7 +93,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -119,7 +118,7 @@ from lerobot.utils.control_utils import (
     sanity_check_dataset_name,
     sanity_check_dataset_robot_compatibility,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import (
     get_safe_torch_device,
@@ -513,7 +512,7 @@ def record(cfg: RecordConfig) -> LeRobotDataset:
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     record()
 
 

src/lerobot/scripts/lerobot_replay.py

@@ -54,7 +54,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -62,7 +61,7 @@ from lerobot.robots import (  # noqa: F401
     so101_follower,
 )
 from lerobot.utils.constants import ACTION
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import (
     init_logging,
@@ -128,7 +127,7 @@ def replay(cfg: ReplayConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     replay()
 
 

src/lerobot/scripts/lerobot_teleoperate.py

@@ -71,7 +71,6 @@ from lerobot.robots import (  # noqa: F401
     Robot,
     RobotConfig,
     bi_so100_follower,
-    earthrover_mini_plus,
     hope_jr,
     koch_follower,
     make_robot_from_config,
@@ -84,13 +83,12 @@ from lerobot.teleoperators import (  # noqa: F401
     bi_so100_leader,
     gamepad,
     homunculus,
-    keyboard,
     koch_leader,
     make_teleoperator_from_config,
     so100_leader,
     so101_leader,
 )
-from lerobot.utils.import_utils import register_third_party_plugins
+from lerobot.utils.import_utils import register_third_party_devices
 from lerobot.utils.robot_utils import precise_sleep
 from lerobot.utils.utils import init_logging, move_cursor_up
 from lerobot.utils.visualization_utils import init_rerun, log_rerun_data
@@ -219,7 +217,7 @@ def teleoperate(cfg: TeleoperateConfig):
 
 
 def main():
-    register_third_party_plugins()
+    register_third_party_devices()
     teleoperate()
 
 

src/lerobot/scripts/lerobot_train.py

@@ -36,7 +36,6 @@ from lerobot.policies.factory import make_policy, make_pre_post_processors
 from lerobot.policies.pretrained import PreTrainedPolicy
 from lerobot.rl.wandb_utils import WandBLogger
 from lerobot.scripts.lerobot_eval import eval_policy_all
-from lerobot.utils.import_utils import register_third_party_plugins
 from lerobot.utils.logging_utils import AverageMeter, MetricsTracker
 from lerobot.utils.random_utils import set_seed
 from lerobot.utils.train_utils import (
@@ -62,7 +61,6 @@ def update_policy(
     accelerator: Accelerator,
     lr_scheduler=None,
     lock=None,
-    postprocessor = None,
 ) -> tuple[MetricsTracker, dict]:
     """
     Performs a single training step to update the policy's weights.
@@ -91,10 +89,6 @@ def update_policy(
     # Let accelerator handle mixed precision
     with accelerator.autocast():
         loss, output_dict = policy.forward(batch)
-        # action = policy.predict_action_chunk(batch)
-        # if postprocessor is not None:
-        #     action = postprocessor(action)
-        # breakpoint()
         # TODO(rcadene): policy.unnormalize_outputs(out_dict)
 
     # Use accelerator's backward method
@@ -156,7 +150,7 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
         from accelerate.utils import DistributedDataParallelKwargs
 
         ddp_kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
-        accelerator = Accelerator(step_scheduler_with_optimizer=False, gradient_accumulation_steps=4, kwargs_handlers=[ddp_kwargs])
+        accelerator = Accelerator(step_scheduler_with_optimizer=False, kwargs_handlers=[ddp_kwargs])
 
     init_logging(accelerator=accelerator)
 
@@ -211,7 +205,6 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
         ds_meta=dataset.meta,
         rename_map=cfg.rename_map,
     )
-    
 
     # Wait for all processes to finish policy creation before continuing
     accelerator.wait_for_everyone()
@@ -250,7 +243,6 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
         **postprocessor_kwargs,
     )
 
-
     if is_main_process:
         logging.info("Creating optimizer and scheduler")
     optimizer, lr_scheduler = make_optimizer_and_scheduler(cfg, policy)
@@ -350,7 +342,6 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
             cfg.optimizer.grad_clip_norm,
             accelerator=accelerator,
             lr_scheduler=lr_scheduler,
-            postprocessor=postprocessor,
         )
 
         # Note: eval and checkpoint happens *after* the `step`th training update has completed, so we
@@ -457,7 +448,6 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
 
 
 def main():
-    register_third_party_plugins()
     train()
 
 

src/lerobot/teleoperators/keyboard/__init__.py

@@ -14,18 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from .configuration_keyboard import (
-    KeyboardEndEffectorTeleopConfig,
-    KeyboardRoverTeleopConfig,
-    KeyboardTeleopConfig,
-)
-from .teleop_keyboard import KeyboardEndEffectorTeleop, KeyboardRoverTeleop, KeyboardTeleop
+from .configuration_keyboard import KeyboardEndEffectorTeleopConfig, KeyboardTeleopConfig
+from .teleop_keyboard import KeyboardEndEffectorTeleop, KeyboardTeleop
 
 __all__ = [
     "KeyboardTeleopConfig",
     "KeyboardTeleop",
     "KeyboardEndEffectorTeleopConfig",
     "KeyboardEndEffectorTeleop",
-    "KeyboardRoverTeleopConfig",
-    "KeyboardRoverTeleop",
 ]

src/lerobot/teleoperators/keyboard/configuration_keyboard.py

@@ -13,7 +13,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""Configuration for keyboard teleoperators."""
 
 from dataclasses import dataclass
 
@@ -31,38 +30,4 @@ class KeyboardTeleopConfig(TeleoperatorConfig):
 @TeleoperatorConfig.register_subclass("keyboard_ee")
 @dataclass
 class KeyboardEndEffectorTeleopConfig(KeyboardTeleopConfig):
-    """Configuration for keyboard end-effector teleoperator.
-
-    Used for controlling robot end-effectors with keyboard inputs.
-
-    Attributes:
-        use_gripper: Whether to include gripper control in actions
-    """
-
     use_gripper: bool = True
-
-
-@TeleoperatorConfig.register_subclass("keyboard_rover")
-@dataclass
-class KeyboardRoverTeleopConfig(TeleoperatorConfig):
-    """Configuration for keyboard rover teleoperator.
-
-    Used for controlling mobile robots like EarthRover Mini Plus with WASD controls.
-
-    Attributes:
-        linear_speed: Default linear velocity magnitude (-1 to 1 range for SDK robots)
-        angular_speed: Default angular velocity magnitude (-1 to 1 range for SDK robots)
-        speed_increment: Amount to increase/decrease speed with +/- keys
-        turn_assist_ratio: Forward motion multiplier when turning with A/D keys (0.0-1.0)
-        angular_speed_ratio: Ratio of angular to linear speed for synchronized adjustments
-        min_linear_speed: Minimum linear speed when decreasing (prevents zero speed)
-        min_angular_speed: Minimum angular speed when decreasing (prevents zero speed)
-    """
-
-    linear_speed: float = 1.0
-    angular_speed: float = 1.0
-    speed_increment: float = 0.1
-    turn_assist_ratio: float = 0.3
-    angular_speed_ratio: float = 0.6
-    min_linear_speed: float = 0.1
-    min_angular_speed: float = 0.05

src/lerobot/teleoperators/keyboard/teleop_keyboard.py

@@ -25,11 +25,7 @@ from lerobot.utils.errors import DeviceAlreadyConnectedError, DeviceNotConnected
 
 from ..teleoperator import Teleoperator
 from ..utils import TeleopEvents
-from .configuration_keyboard import (
-    KeyboardEndEffectorTeleopConfig,
-    KeyboardRoverTeleopConfig,
-    KeyboardTeleopConfig,
-)
+from .configuration_keyboard import KeyboardEndEffectorTeleopConfig, KeyboardTeleopConfig
 
 PYNPUT_AVAILABLE = True
 try:
@@ -293,158 +289,3 @@ class KeyboardEndEffectorTeleop(KeyboardTeleop):
             TeleopEvents.SUCCESS: success,
             TeleopEvents.RERECORD_EPISODE: rerecord_episode,
         }
-
-
-class KeyboardRoverTeleop(KeyboardTeleop):
-    """
-    Keyboard teleoperator for mobile robots like EarthRover Mini Plus.
-
-    Provides intuitive WASD-style controls for driving a mobile robot:
-    - Linear movement (forward/backward)
-    - Angular movement (turning/rotation)
-    - Speed adjustment
-    - Emergency stop
-
-    Keyboard Controls:
-        Movement:
-            - W: Move forward
-            - S: Move backward
-            - A: Turn left (with forward motion)
-            - D: Turn right (with forward motion)
-            - Q: Rotate left in place
-            - E: Rotate right in place
-            - X: Emergency stop
-
-        Speed Control:
-            - +/=: Increase speed
-            - -: Decrease speed
-
-        System:
-            - ESC: Disconnect teleoperator
-
-    Attributes:
-        config: Teleoperator configuration
-        current_linear_speed: Current linear velocity magnitude
-        current_angular_speed: Current angular velocity magnitude
-
-    Example:
-        ```python
-        from lerobot.teleoperators.keyboard import KeyboardRoverTeleop, KeyboardRoverTeleopConfig
-
-        teleop = KeyboardRoverTeleop(
-            KeyboardRoverTeleopConfig(linear_speed=1.0, angular_speed=1.0, speed_increment=0.1)
-        )
-        teleop.connect()
-
-        while teleop.is_connected:
-            action = teleop.get_action()
-            robot.send_action(action)
-        ```
-    """
-
-    config_class = KeyboardRoverTeleopConfig
-    name = "keyboard_rover"
-
-    def __init__(self, config: KeyboardRoverTeleopConfig):
-        super().__init__(config)
-        # Add rover-specific speed settings
-        self.current_linear_speed = config.linear_speed
-        self.current_angular_speed = config.angular_speed
-
-    @property
-    def action_features(self) -> dict:
-        """Return action format for rover (linear and angular velocities)."""
-        return {
-            "linear.vel": float,
-            "angular.vel": float,
-        }
-
-    @property
-    def is_calibrated(self) -> bool:
-        """Rover teleop doesn't require calibration."""
-        return True
-
-    def _drain_pressed_keys(self):
-        """Update current_pressed state from event queue without clearing held keys"""
-        while not self.event_queue.empty():
-            key_char, is_pressed = self.event_queue.get_nowait()
-            if is_pressed:
-                self.current_pressed[key_char] = True
-            else:
-                # Only remove key if it's being released
-                self.current_pressed.pop(key_char, None)
-
-    def get_action(self) -> dict[str, Any]:
-        """
-        Get the current action based on pressed keys.
-
-        Returns:
-            dict with 'linear.vel' and 'angular.vel' keys
-        """
-        before_read_t = time.perf_counter()
-
-        if not self.is_connected:
-            raise DeviceNotConnectedError(
-                "KeyboardRoverTeleop is not connected. You need to run `connect()` before `get_action()`."
-            )
-
-        self._drain_pressed_keys()
-
-        linear_velocity = 0.0
-        angular_velocity = 0.0
-
-        # Check which keys are currently pressed (not released)
-        active_keys = {key for key, is_pressed in self.current_pressed.items() if is_pressed}
-
-        # Linear movement (W/S) - these take priority
-        if "w" in active_keys:
-            linear_velocity = self.current_linear_speed
-        elif "s" in active_keys:
-            linear_velocity = -self.current_linear_speed
-
-        # Turning (A/D/Q/E)
-        if "d" in active_keys:
-            angular_velocity = -self.current_angular_speed
-            if linear_velocity == 0:  # If not moving forward/back, add slight forward motion
-                linear_velocity = self.current_linear_speed * self.config.turn_assist_ratio
-        elif "a" in active_keys:
-            angular_velocity = self.current_angular_speed
-            if linear_velocity == 0:  # If not moving forward/back, add slight forward motion
-                linear_velocity = self.current_linear_speed * self.config.turn_assist_ratio
-        elif "q" in active_keys:
-            angular_velocity = self.current_angular_speed
-            linear_velocity = 0  # Rotate in place
-        elif "e" in active_keys:
-            angular_velocity = -self.current_angular_speed
-            linear_velocity = 0  # Rotate in place
-
-        # Stop (X) - overrides everything
-        if "x" in active_keys:
-            linear_velocity = 0
-            angular_velocity = 0
-
-        # Speed adjustment
-        if "+" in active_keys or "=" in active_keys:
-            self.current_linear_speed += self.config.speed_increment
-            self.current_angular_speed += self.config.speed_increment * self.config.angular_speed_ratio
-            logging.info(
-                f"Speed increased: linear={self.current_linear_speed:.2f}, angular={self.current_angular_speed:.2f}"
-            )
-        if "-" in active_keys:
-            self.current_linear_speed = max(
-                self.config.min_linear_speed, self.current_linear_speed - self.config.speed_increment
-            )
-            self.current_angular_speed = max(
-                self.config.min_angular_speed,
-                self.current_angular_speed - self.config.speed_increment * self.config.angular_speed_ratio,
-            )
-            logging.info(
-                f"Speed decreased: linear={self.current_linear_speed:.2f}, angular={self.current_angular_speed:.2f}"
-            )
-
-        self.logs["read_pos_dt_s"] = time.perf_counter() - before_read_t
-
-        return {
-            "linear.vel": linear_velocity,
-            "angular.vel": angular_velocity,
-        }

src/lerobot/transport/utils.py

@@ -19,7 +19,7 @@ import io
 import json
 import logging
 import pickle  # nosec B403: Safe usage for internal serialization only
-from multiprocessing.synchronize import Event as MpEvent
+from multiprocessing import Event
 from queue import Queue
 from typing import Any
 
@@ -28,9 +28,6 @@ import torch
 from lerobot.transport import services_pb2
 from lerobot.utils.transition import Transition
 
-# FIX for protobuf: Assign the enum to a variable and ignore the type error once
-TransferState = services_pb2.TransferState  # type: ignore[attr-defined]
-
 CHUNK_SIZE = 2 * 1024 * 1024  # 2 MB
 MAX_MESSAGE_SIZE = 4 * 1024 * 1024  # 4 MB
 
@@ -43,8 +40,8 @@ def bytes_buffer_size(buffer: io.BytesIO) -> int:
 
 
 def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = "", silent: bool = True):
-    bytes_buffer: io.BytesIO = io.BytesIO(buffer)
-    size_in_bytes = bytes_buffer_size(bytes_buffer)
+    buffer = io.BytesIO(buffer)
+    size_in_bytes = bytes_buffer_size(buffer)
 
     sent_bytes = 0
 
@@ -53,15 +50,15 @@ def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = ""
     logging_method(f"{log_prefix} Buffer size {size_in_bytes / 1024 / 1024} MB with")
 
     while sent_bytes < size_in_bytes:
-        transfer_state = TransferState.TRANSFER_MIDDLE
+        transfer_state = services_pb2.TransferState.TRANSFER_MIDDLE
 
         if sent_bytes + CHUNK_SIZE >= size_in_bytes:
-            transfer_state = TransferState.TRANSFER_END
+            transfer_state = services_pb2.TransferState.TRANSFER_END
         elif sent_bytes == 0:
-            transfer_state = TransferState.TRANSFER_BEGIN
+            transfer_state = services_pb2.TransferState.TRANSFER_BEGIN
 
         size_to_read = min(CHUNK_SIZE, size_in_bytes - sent_bytes)
-        chunk = bytes_buffer.read(size_to_read)
+        chunk = buffer.read(size_to_read)
 
         yield message_class(transfer_state=transfer_state, data=chunk)
         sent_bytes += size_to_read
@@ -70,7 +67,7 @@ def send_bytes_in_chunks(buffer: bytes, message_class: Any, log_prefix: str = ""
     logging_method(f"{log_prefix} Published {sent_bytes / 1024 / 1024} MB")
 
 
-def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEvent, log_prefix: str = ""):
+def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: Event, log_prefix: str = ""):
     bytes_buffer = io.BytesIO()
     step = 0
 
@@ -81,17 +78,17 @@ def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEve
             logging.info(f"{log_prefix} Shutting down receiver")
             return
 
-        if item.transfer_state == TransferState.TRANSFER_BEGIN:
+        if item.transfer_state == services_pb2.TransferState.TRANSFER_BEGIN:
             bytes_buffer.seek(0)
             bytes_buffer.truncate(0)
             bytes_buffer.write(item.data)
             logging.debug(f"{log_prefix} Received data at step 0")
             step = 0
-        elif item.transfer_state == TransferState.TRANSFER_MIDDLE:
+        elif item.transfer_state == services_pb2.TransferState.TRANSFER_MIDDLE:
             bytes_buffer.write(item.data)
             step += 1
             logging.debug(f"{log_prefix} Received data at step {step}")
-        elif item.transfer_state == TransferState.TRANSFER_END:
+        elif item.transfer_state == services_pb2.TransferState.TRANSFER_END:
             bytes_buffer.write(item.data)
             logging.debug(f"{log_prefix} Received data at step end size {bytes_buffer_size(bytes_buffer)}")
 
@@ -112,17 +109,17 @@ def receive_bytes_in_chunks(iterator, queue: Queue | None, shutdown_event: MpEve
 
 def state_to_bytes(state_dict: dict[str, torch.Tensor]) -> bytes:
     """Convert model state dict to flat array for transmission"""
-    bytes_buffer = io.BytesIO()
+    buffer = io.BytesIO()
 
-    torch.save(state_dict, bytes_buffer)
+    torch.save(state_dict, buffer)
 
-    return bytes_buffer.getvalue()
+    return buffer.getvalue()
 
 
 def bytes_to_state_dict(buffer: bytes) -> dict[str, torch.Tensor]:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    return torch.load(bytes_buffer, weights_only=True)
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    return torch.load(buffer, weights_only=True)
 
 
 def python_object_to_bytes(python_object: Any) -> bytes:
@@ -130,24 +127,24 @@ def python_object_to_bytes(python_object: Any) -> bytes:
 
 
 def bytes_to_python_object(buffer: bytes) -> Any:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    obj = pickle.load(bytes_buffer)  # nosec B301: Safe usage of pickle.load
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    obj = pickle.load(buffer)  # nosec B301: Safe usage of pickle.load
     # Add validation checks here
     return obj
 
 
 def bytes_to_transitions(buffer: bytes) -> list[Transition]:
-    bytes_buffer = io.BytesIO(buffer)
-    bytes_buffer.seek(0)
-    transitions = torch.load(bytes_buffer, weights_only=True)
+    buffer = io.BytesIO(buffer)
+    buffer.seek(0)
+    transitions = torch.load(buffer, weights_only=True)
     return transitions
 
 
 def transitions_to_bytes(transitions: list[Transition]) -> bytes:
-    bytes_buffer = io.BytesIO()
-    torch.save(transitions, bytes_buffer)
-    return bytes_buffer.getvalue()
+    buffer = io.BytesIO()
+    torch.save(transitions, buffer)
+    return buffer.getvalue()
 
 
 def grpc_channel_options(

src/lerobot/utils/constants.py

@@ -26,15 +26,8 @@ OBS_IMAGES = OBS_IMAGE + "s"
 OBS_LANGUAGE = OBS_STR + ".language"
 OBS_LANGUAGE_TOKENS = OBS_LANGUAGE + ".tokens"
 OBS_LANGUAGE_ATTENTION_MASK = OBS_LANGUAGE + ".attention_mask"
-OBS_LANGUAGE_HIGH_LEVEL_TASK = OBS_STR + ".user_prompt"
-OBS_LANGUAGE_HIGH_LEVEL_TASK_TOKENS = OBS_LANGUAGE_HIGH_LEVEL_TASK + ".tokens"
-OBS_LANGUAGE_HIGH_LEVEL_TASK_ATTENTION_MASK = OBS_LANGUAGE_HIGH_LEVEL_TASK + ".attention_mask"
-OBS_LANGUAGE_SUBTASK_ONLY = OBS_STR + ".subtask"
-OBS_LANGUAGE_SUBTASK_ONLY_TOKENS = OBS_LANGUAGE_SUBTASK_ONLY + ".tokens"
-OBS_LANGUAGE_SUBTASK_ONLY_ATTENTION_MASK = OBS_LANGUAGE_SUBTASK_ONLY + ".attention_mask"
+
 ACTION = "action"
-ACTION_TOKENS = ACTION + ".tokens"
-ACTION_TOKEN_MASK = ACTION + ".token_mask"
 REWARD = "next.reward"
 TRUNCATED = "next.truncated"
 DONE = "next.done"

src/lerobot/utils/import_utils.py

@@ -130,14 +130,14 @@ def make_device_from_device_class(config: ChoiceRegistry) -> Any:
     )
 
 
-def register_third_party_plugins() -> None:
+def register_third_party_devices() -> None:
     """
     Discover and import third-party lerobot_* plugins so they can register themselves.
 
     Scans top-level modules on sys.path for packages starting with
-    'lerobot_robot_', 'lerobot_camera_', 'lerobot_teleoperator_' or 'lerobot_policy_' and imports them.
+    'lerobot_robot_', 'lerobot_camera_' or 'lerobot_teleoperator_' and imports them.
     """
-    prefixes = ("lerobot_robot_", "lerobot_camera_", "lerobot_teleoperator_", "lerobot_policy_")
+    prefixes = ("lerobot_robot_", "lerobot_camera_", "lerobot_teleoperator_")
     imported: list[str] = []
     failed: list[str] = []
 

tests/policies/pi0_pi05/test_pi05_original_vs_lerobot.py

@@ -266,7 +266,7 @@ def create_original_observation_with_openpi_preprocessing(batch):
     elif len(tasks) == 1:
         tasks = tasks * batch_size
 
-    # Use pi05 state and input tokenizer logic (same as Pi05PrepareStateAndLanguageTokenizerProcessorStep)
+    # Use pi05 state and input tokenizer logic (same as Pi05PrepareStateTokenizerProcessorStep)
     state = batch["observation.state"]
     state = deepcopy(state)
 

tests/policies/xvla/test_xvla_original_vs_lerobot.py

@@ -17,6 +17,7 @@
 """Test script to verify XVLA policy integration with LeRobot vs the original implementation, only meant to be run locally!"""
 # ruff: noqa: E402
 
+import gc
 import random
 from copy import deepcopy
 from typing import Any
@@ -51,6 +52,18 @@ EXPECTED_ACTIONS_STD = 0.245411
 EXPECTED_ACTIONS_FIRST_5 = torch.tensor([0.2742, 0.4977, 0.0500, 0.7040, -0.2653])
 
 
+def cleanup_memory():
+    """Clean up GPU/MPS memory to prevent OOM errors between tests."""
+    print("\nCleaning up memory...")
+    gc.collect()
+    if torch.cuda.is_available():
+        torch.cuda.empty_cache()
+        torch.cuda.synchronize()
+    if torch.backends.mps.is_available():
+        torch.mps.empty_cache()
+    print("Memory cleanup complete.")
+
+
 def set_seed_all(seed: int):
     """Set random seed for all RNG sources to ensure reproducibility."""
     random.seed(seed)
@@ -136,6 +149,7 @@ def xvla_components():
     policy_obj, preprocessor_obj, postprocessor_obj = instantiate_lerobot_xvla(from_pretrained=True)
     print("✔️ Model loaded successfully")
     yield policy_obj, preprocessor_obj, postprocessor_obj
+    cleanup_memory()
 
 
 @pytest.fixture(scope="module")
@@ -316,3 +330,32 @@ def test_xvla_inference_reproducibility(policy, preprocessor):
     assert torch.allclose(actions_1, actions_2, atol=1e-6), "Inference should be reproducible!"
 
     print("\nInference is reproducible!")
+
+
+if __name__ == "__main__":
+    print("\n" + "=" * 80)
+    print("XVLA LeRobot Validation Test Suite")
+    print("=" * 80)
+
+    try:
+        # Initialize model once for all tests
+        print("\n[Setup] Instantiating LeRobot XVLA policy...")
+        policy, preprocessor, postprocessor = instantiate_lerobot_xvla(from_pretrained=True)
+        print("✔️ Model loaded successfully")
+
+        # Run all tests with the same model instance
+        test_xvla_preprocessor_alignment(policy, preprocessor)
+        test_xvla_action_generation(policy, preprocessor)
+        test_xvla_inference_reproducibility(policy, preprocessor)
+
+        print("\n" + "=" * 80)
+        print("All tests passed!")
+        print("=" * 80)
+
+        cleanup_memory()
+    except Exception as e:
+        print("\n" + "=" * 80)
+        print(f"Test failed with error: {e}")
+        print("=" * 80)
+        cleanup_memory()
+        raise