ydy0615/lerobot-clone
1
0
Fork 0
mirror of https://github.com/huggingface/lerobot.git synced 2026-05-31 02:41:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(train): add cudnn_deterministic option for reproducible training (#3102)

Browse Source 
Add a `cudnn_deterministic` flag to `TrainPipelineConfig` (default: False)
that sets `torch.backends.cudnn.deterministic = True` and disables benchmark
mode, eliminating CUDA floating-point non-determinism at the cost of ~10-20%
training speed. When False (default) the existing benchmark=True behaviour
is preserved.
This commit is contained in:
Ignat Georgiev
2026-03-08 13:29:33 +02:00
committed by GitHub
parent 4f2ef024d8
commit 2fb5c7add0
2 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
src/lerobot/scripts/lerobot_train.py
View File

@@ -209,7 +209,11 @@ def train(cfg: TrainPipelineConfig, accelerator: Accelerator | None = None):
# Use accelerator's device
device = accelerator.device
torch.backends.cudnn.benchmark = True
if cfg.cudnn_deterministic:
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
else:
torch.backends.cudnn.benchmark = True
torch.backends.cuda.matmul.allow_tf32 = True
# Dataset loading synchronization: main process downloads first to avoid race conditions