feat(envs): Add NVIDIA IsaacLab-Arena Lerobot (#2699)

* adding Isaaclab Arena from collab

* adding into lerobot-eval

* minor modification

* added bash script for env setup

* setups

* fix applauncher not getting the arguments

* data conversion, train and eval smolvla

* fixed imports

* clean-up

* added test suits & clean up - wip

* fixed video recording

* clean-up

* hub integration working

* clean-up

* added kwargs

* Revert "added kwargs"

This reverts commit 9b445356385d0707655cf04d02be058b25138119.

* added kwargs

* clean-up

* cleaned unused function

* added logging

* docs

* cleaned up IsaaclabArenaEnv

* clean-up

* clean-up

* clean up

* added tests

* minor clean-up

* fix: support for state based envs

* feat(envs): Add NVIDIA IsaacLab Arena integration with LeRobot for policy evaluation at scale

* feat(envs): Add IsaacLab Arena integration for policy evaluation

Integrate NVIDIA IsaacLab Arena with LeRobot to enable GPU-accelerated
simulation through the EnvHub infrastructure.

This enables:
- Training imitation learning policies (PI0, SmolVLA, etc.)
- Evaluating trained policies in with IsaacLab Arena

The implementation adds:
- IsaaclabArenaEnv config with Arena-specific parameters
- IsaaclabArenaProcessorStep for observation processing
- Hub loading from nvkartik/isaaclab-arena-envs repository
- Video recording support

Available environments include GR1 microwave manipulation, Galileo
pick-and-place, G1 loco-manipulation, and button pressing tasks.

Datasets: nvkartik/Arena-GR1-Manipulation-Task
Policies: nvkartik/pi05-arena-gr1-microwave,
          nvkartik/smolvla-arena-gr1-microwave

* added isaaclab arena wrapper and corresponding tests

* added error handling

* renamed wrapper file: isaaclab_arena to isaaclab

* added extra kwarg changes

* adjustments for hub envs

* correct class name in test file

* fixed parsing of env_kwargs

* tested end to end

* removed unused code

* refactor design

* shifted IsaacLab to hub

* removed IsaacLab tests

* docs: Add LW-BenchHub evaluation instructions

* docs: Add LW-BenchHub evaluation instructions

* docs diet

* minor edits to texts

* IL Arena commit hash

* update links

* minor edits

* fix numpy version after install of lerobot

* links update

* valideated on vanilla brev

* docs: Add LW-BenchHub evaluation instructions

* remove kwargs from all make_env calls

* remove kwargs from all make_env calls

* fix LW table and indentations

* remove environment list from docs

* docs: Update lw-benchhub eval config in envhub docs

* removing kwargs

* removed extra line

* ensure pinocchio install for lightwheel + add lightwheel website link

* remove env_kwargs

* no default empty value for hub_path

* not using assert method

* remove env_processor defaults

* revert and adding default "" value for hub_path

* pinning down packages versions

* explicit None value for hub_path

* Update src/lerobot/configs/eval.py

Co-authored-by: Jade Choghari <chogharijade@gmail.com>
Signed-off-by: Lior Ben Horin <liorbenhorin@gmail.com>

* corrected formatting

* corrected job_name var in config

* updated docs and namespace

* updated namespace

* updated docs

* updated docs

* added hardware requirements

* updated docs

---------

Signed-off-by: Lior Ben Horin <liorbenhorin@gmail.com>
Co-authored-by: lbenhorin <lbenhorin@nvidia.com>
Co-authored-by: Lior Ben Horin <liorbenhorin@gmail.com>
Co-authored-by: Jade Choghari <chogharijade@gmail.com>
Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
Co-authored-by: tianheng.wu <tianheng.wu@lightwheel.ai>

src/lerobot/processor/env_processor.py

@@ -18,7 +18,7 @@ from dataclasses import dataclass
 import torch
 
 from lerobot.configs.types import PipelineFeatureType, PolicyFeature
-from lerobot.utils.constants import OBS_IMAGES, OBS_STATE
+from lerobot.utils.constants import OBS_IMAGES, OBS_STATE, OBS_STR
 
 from .pipeline import ObservationProcessorStep, ProcessorStepRegistry
 
@@ -152,3 +152,78 @@ class LiberoProcessorStep(ObservationProcessorStep):
             result[mask] = axis * angle.unsqueeze(1)
 
         return result
+
+
+@dataclass
+@ProcessorStepRegistry.register(name="isaaclab_arena_processor")
+class IsaaclabArenaProcessorStep(ObservationProcessorStep):
+    """
+    Processes IsaacLab Arena observations into LeRobot format.
+
+    **State Processing:**
+    - Extracts state components from obs["policy"] based on `state_keys`.
+    - Concatenates into a flat vector mapped to "observation.state".
+
+    **Image Processing:**
+    - Extracts images from obs["camera_obs"] based on `camera_keys`.
+    - Converts from (B, H, W, C) uint8 to (B, C, H, W) float32 [0, 1].
+    - Maps to "observation.images.<camera_name>".
+    """
+
+    # Configurable from IsaacLabEnv config / cli args: --env.state_keys="robot_joint_pos,left_eef_pos"
+    state_keys: tuple[str, ...]
+
+    # Configurable from IsaacLabEnv config / cli args: --env.camera_keys="robot_pov_cam_rgb"
+    camera_keys: tuple[str, ...]
+
+    def _process_observation(self, observation):
+        """
+        Processes both image and policy state observations from IsaacLab Arena.
+        """
+        processed_obs = {}
+
+        if f"{OBS_STR}.camera_obs" in observation:
+            camera_obs = observation[f"{OBS_STR}.camera_obs"]
+
+            for cam_name, img in camera_obs.items():
+                if cam_name not in self.camera_keys:
+                    continue
+
+                img = img.permute(0, 3, 1, 2).contiguous()
+                if img.dtype == torch.uint8:
+                    img = img.float() / 255.0
+                elif img.dtype != torch.float32:
+                    img = img.float()
+
+                processed_obs[f"{OBS_IMAGES}.{cam_name}"] = img
+
+        # Process policy state -> observation.state
+        if f"{OBS_STR}.policy" in observation:
+            policy_obs = observation[f"{OBS_STR}.policy"]
+
+            # Collect state components in order
+            state_components = []
+            for key in self.state_keys:
+                if key in policy_obs:
+                    component = policy_obs[key]
+                    # Flatten extra dims: (B, N, M) -> (B, N*M)
+                    if component.dim() > 2:
+                        batch_size = component.shape[0]
+                        component = component.view(batch_size, -1)
+                    state_components.append(component)
+
+            if state_components:
+                state = torch.cat(state_components, dim=-1)
+                state = state.float()
+                processed_obs[OBS_STATE] = state
+
+        return processed_obs
+
+    def transform_features(
+        self, features: dict[PipelineFeatureType, dict[str, PolicyFeature]]
+    ) -> dict[PipelineFeatureType, dict[str, PolicyFeature]]:
+        """Not used for policy evaluation."""
+        return features
+
+    def observation(self, observation):
+        return self._process_observation(observation)