Tune it your way: configurable
video and depth encoding in LeRobot

A robot dataset is mostly storage-heavy camera footage, and the way you store it can make a real difference.

It’s why LeRobot stores camera streams as compressed video rather than folders of images, something we unpacked in our first post on video encoding. Along the way, we settled on a set of defaults that work well across the board. Those defaults are still a solid starting point, but the right balance between file size, quality, and speed usually depends on your data and your hardware, so the ability to tune them yourself can be a real asset.

This follow-up is about putting that control in your hands. Starting from LeRobot v0.6.0, you can tune any encoding parameter to fit your setup, and even record depth alongside RGB, end to end. These same tools are behind the BEHAVIOR-1K 2026 challenge demos dataset released by the Stanford Vision & Learning Lab, a glimpse of where this can lead: a ~3 TB dataset of 20,000 simulated manipulation episodes where depth is recorded as a first-class modality alongside RGB.

We’ll walk you through these new features in three steps: first, configurable video encoding; then a benchmark to help you pick the settings that suit your data; and finally, end-to-end depth support.

A quick refresher on video encoding

Video encoding is never a single dial. It’s a careful mix of several optimizations pulling in different directions, and where each one lands ripples through the whole pipeline. Rerun recently mapped the cost robotics teams pay for immature data infrastructure, what they call the data layer tax, and video sits at the centre of it, routinely making up 90% or more of a dataset’s size.

How videos are encoded shapes more than how much room they take: it also affects how faithfully the pixels survive, how long recording takes, how much CPU is spent decoding, and how quickly frames reach the GPU at train time - and ultimately how fast you can iterate.

An episode of a robotics dataset is mostly camera data. The proprioceptive streams (joint angles, gripper state, actions) are a few floats per timestep, but a single camera at 30 fps pours out millions of pixels every second, and most setups run several of them at once (wrist, front, overhead).

And that camera data is extraordinarily redundant. A manipulation episode is essentially a static scene, with one arm and maybe an object shifting slightly between frames, so consecutive frames are nearly identical. Video codecs exploit exactly that: they compress each frame internally, throwing away detail the eye won’t miss (spatial compression), and they store most frames not as full images but as differences from a nearby keyframe (temporal compression).

A static background costs almost nothing to encode once only what moves is being recorded. That is how a 20 GB dataset becomes roughly 1 GB while staying fully trainable.

We walked through the full video encoding pipeline last time, along with the costs it has to juggle. The catch is that these costs pull against one another: compression, precision, speed, you rarely get all three at once. Back then we weighed them for you, benchmarking file size, decoding speed, quality, and compatibility to settle on a single default that struck the balance on your behalf: fast decoding for training, files small enough to store and share, quality good enough not to hurt policies, and videos that play everywhere.

But that benchmark left one cost out entirely: encoding time. It climbs the harder the others are pushed, and unlike them it is paid up front, at recording time. When frames are encoded live as they are captured, the encoder has to keep up with the camera in real time, and that burden usually lands on the same machine already busy running the robot. A setting that’s too demanding may fall behind, dropping frames or stalling the recording outright.

Eventually, no single setting is best everywhere, and where you land on the trade-off should be the user’s call.

1. Configurable encoding parameters

For a long time that choice wasn’t yours to make. Our first post settled on one good default — AV1 (libsvtav1), yuv420p, g=2, crf=30 — and baked it in. LeRobot v0.6.0 unbakes it. The defaults still hold, but the best compromise is now yours to decide, depending on your data and on the metric you care about: minimizing file size, maximizing decode speed at train time, preserving top quality for a sensitive task, or keeping encoding time short while recording. Every video encoding parameter is now configurable, through a single VideoEncoderConfig:

lerobot-record \
  --robot.type=so100_follower \
  # ... existing parameters ...
  --dataset.rgb_encoder.vcodec=h264 \
  --dataset.rgb_encoder.pix_fmt=yuv420p \
  --dataset.rgb_encoder.g=30 \
  --dataset.rgb_encoder.crf=23 \
  --dataset.rgb_encoder.preset=medium \
  --dataset.rgb_encoder.fast_decode=0 \
  --dataset.rgb_encoder.extra_options='{profile: high, bf: 2, refs: 3, tune: film}'

Opening up every parameter could easily turn into a footgun, so it rests on two safeguards that keep the added flexibility from causing trouble:

1. Up-front validation (via PyAV), before recording starts. An unsupported codec, or a combination that FFmpeg accepts but a browser rejects, is caught right away with a clear, actionable message, so you can adjust before recording starts rather than an hour in.
2. Per-camera metadata in the dataset, so every dataset is self-describing and reproducible. When merging datasets, only the codec and pixel format have to match, the tuning parameters are free to differ.

2. A benchmark to find the right settings

Naturally, it’s hard to tune what you can’t measure. So on top of exposing the knobs, we built a benchmarking platform, published as a live leaderboard you can filter, sort, and contribute to.

It sweeps the main video encoding parameters - codec, pixel format, GOP size, and CRF - across:

• Representative datasets: a fixed-camera simulation, an indoor mobile-manipulator scene, an outdoor street, and a high-resolution kitchen.
• Realistic decode patterns: a single frame, two or six consecutive frames, and two frames spaced apart (the worst case for large GOPs).

For each configuration the benchmark reports the core metrics: size compression ratio (how much smaller the encoded video is than the raw frames), load-time ratio (how fast frames decode relative to a baseline), and quality, captured by MSE, PSNR, and SSIM (how faithfully the decoded pixels match the originals). It also adds encoding metrics (encoding time and throughput in fps). All of this is measured under both available video backends, PyAV and torchcodec.

A ranked leaderboard scores video encoding configurations under a configurable weighting (overall, quality, encoding, or decoding), and anyone can queue new sweeps, so the results keep expanding as more people contribute. It runs alongside a companion depth leaderboard dedicated to this newly supported modality. There’s no universal winner here, the goal is rather to show the trade-off each setting makes, so the right point for your use case becomes clear.

3. End-to-end depth support

Depth maps were already riding along in a few of the datasets we’d ported, but the modality sat unused, flagged as future work. LeRobot v0.6.0 takes it on: depth cameras now feed depth frames into the robot’s observations, and those frames ride on the very same VideoEncoderConfig machinery, with the extra pieces depth specifically needs.

Depth is not just a regular RGB image, and treating it like one leaves its most useful information behind. The two formats are indeed built for fundamentally different purposes:

That difference leads to two consequences:

1. Bit depth needs headroom. Push depth through an ordinary RGB pipeline and it gets flattened to 8 bits, throwing away most of the precision the sensor captured.
2. Codecs are tuned for eyes, not numbers. Even if every bit were kept, ordinary codecs are built for human vision rather than numerical fidelity, so lossy compression the eye would never notice can still corrupt a distance the policy relies on.

Depth therefore leans much harder toward the precision corner than colour ever does.

When precision comes first, you can skip video encoding entirely. Depth maps can be kept raw, saved frame by frame as lossless TIFF images, never encoded into a video. Nothing is quantized and nothing is discarded, so every value the policy reads is exactly what the sensor captured. It is the most uncompromising point on the trade-off: size and speed traded away for full fidelity. The framework makes that an explicit, up-front choice, so you stay in control of exactly what your data preserves.

Raw is exact but large, and often the compression that video buys is still worth having. Making depth MP4-compatible means overcoming the two problems above, with two adaptations.

Adaptation 1: preserve the bit depth. The fix for the first problem (bit depth reduction) is a pixel format and a codec that keeps it: gray12le (12-bit single channel) paired with HEVC. Twelve bits gives 4096 depth levels, a deliberate middle ground between the 8 bits a standard RGB pipeline allows and the full bit depth the sensor produces. The format and codec also have to be ones the MP4 container actually accepts: plenty of high-bit-depth pixel formats exist, but not all of them encode into a standard, playable .mp4 file, gray12le on HEVC does.

Adaptation 2: quantize deliberately. Mapping metric depth into a 12-bit range is a precision decision, and it should be parametric rather than fixed: the working range is set with depth_min and depth_max, and a logarithmic mapping can optionally be applied. Either way, a depth value $d$ becomes a code $q$:

• Linearly, spreading all 4096 codes evenly across the range:

  $$q = \operatorname{round}\!\left(4095 \cdot \frac{d - d_{\min}}{d_{\max} - d_{\min}}\right)$$

• Logarithmically with use_log, packing more codes into nearby distances than far ones, via a small offset $s$ (shift) that keeps it finite near zero:

  $$q = \operatorname{round}\!\left(4095 \cdot \frac{\ln(d + s) - \ln(d_{\min} + s)}{\ln(d_{\max} + s) - \ln(d_{\min} + s)}\right)$$

  This is the default curve: it matches the way sensor error grows with distance and suits manipulation, where a centimetre of error up close matters more than a centimetre ten metres away.

At decoding time, the same mapping runs in reverse, turning the codes back into metric depth values. Metadata makes that possible: the four quantization parameters (depth_min, depth_max, shift, use_log) are stored in the dataset alongside the codec settings, so a depth video carries everything a reader needs to dequantize it.

In practice depth builds on the same encoder configuration as RGB, adding only the quantization parameters and depth-appropriate defaults, so it reuses the same CLI but under depth_encoder:

lerobot-record \
  --robot.type=so100_follower \
  # ... existing parameters ...
  --dataset.depth_encoder.vcodec=hevc \
  --dataset.depth_encoder.pix_fmt=gray12le \
  --dataset.depth_encoder.g=2 \
  --dataset.depth_encoder.crf=15 \
  --dataset.depth_encoder.depth_min=0.3 \
  --dataset.depth_encoder.depth_max=8.0 \
  --dataset.depth_encoder.use_log=true

One more piece of metadata matters in the same self-describing spirit: the depth unit. Raw TIFF depth maps and the dataset’s depth statistics are stored as un-normalized values in whatever unit the sensor reports, metres or millimetres. Recording that unit keeps the numbers unambiguous, so a distance is never accidentally off by a factor of a thousand. This gap is closed by adding a depth_unit field to the dataset metadata (right alongside is_depth_map), inferred from the frame type:

The depth unit you actually get is a read-time choice: set depth_output_unit at load time, and both the frames and the stored statistics are converted to match. The same dataset can therefore serve a policy that expects metres and another that expects millimetres:

from lerobot.datasets.lerobot_dataset import LeRobotDataset

# Stored depth keeps its native unit; ask for the unit your policy expects.
# Frames and the dataset's depth statistics are both converted to that unit.
dataset = LeRobotDataset("your-org/depth-dataset", depth_output_unit="m")

depth_key = dataset.meta.depth_keys[0]  # e.g. "observation.depth.front"

# A single depth frame, already dequantized/converted to metres.
depth_frame_meters = dataset[0][depth_key]

# The stored statistics for that key, rescaled to the same unit.
depth_stats_meters = dataset.meta.stats[depth_key]  # {"mean", "std", "min", "max"}

Remember the BEHAVIOR-1K 2026 challenge demos from the introduction? This is exactly the machinery behind them. You can browse the dataset episode by episode, depth streams included, right in the LeRobot dataset visualizer. A direct look at depth as a first-class feature across 20,000 simulated manipulation episodes.

Takeaways and what’s next

Our first blog post answered “should robotics datasets use video?” with a clear yes and one good default. This follow-up answers a different question: who decides where on the trade-off curve a dataset sits? And the answer is now you, because that point really does depend on your data and your task.

Configurable encoding turns the implicit defaults into explicit, validated, self-documenting choices. The benchmark helps you make them with numbers rather than guesswork, and depth support reuses the whole framework for the modality where the trade-off matters most. Both ship in LeRobot v0.6.0.

Of course, there’s more to come. We’re working on full torchcodec support for depth, along with GPU-accelerated encoding and decoding. Further out, the LeRobot dataset format could grow beyond RGB and depth to other sensor modalities. Which ones would make the biggest difference for your work? Feel free to open an issue on GitHub or join us on Discord. LeRobot is shaped by its community, and we’d love for you to help steer where it goes next 🤗

Caroline Pascal @CarolinePascal

Pepijn Kooijmans @pepijn223

Steven Palma @imstevenpmwork

Jade Choghari @jadechoghari

Khalil Meftah @lilkm

Martino Russi @nepyope

Nicolas Rabault @Nico-robot

Nikodem Bartnik @nikodembartnik

Maxime Ellerbach @maximellerbach

Leandro von Werra @lvwerra

Thomas Wolf @thomwolf

Code and docs

• Video encoding parameters docs
• Depth cameras docs
• Video benchmark Space

Thanks to everyone who reviewed and contributed to this release, in particular Wensi Ai from the Stanford Vision & Learning Lab, and the LeRobot team and wider LeRobot community for their insightful ideas.