Schlepp

A multi-view synthetic dataset of humans carrying objects through space

Paul Engstler1,2 Yihan Wang1,3 Amaury Aubel1 Priyamvad Deshmukh1 Shai Segev1 Zhengqin Li1 Dan Barnes1 Christian Rupprecht2 Yawar Siddiqui1 Lingni Ma1 Julian Straub1 Zhao Dong1 Richard Newcombe1 Andrea Vedaldi1,2 Jakob Engel1 Adam W. Harley1

1Meta Reality Labs 2Visual Geometry Group, University of Oxford 3Princeton University

Data & code coming soon
  • 13.3M views2.2M frames
    × 6 cameras
  • 6 cameras3 exocentric,
    3 egocentric
  • 7 modalitiesdense ground truth
  • Full HD1920 × 1080 exo,
    2016 × 1512 Aria RGB
  • MHR + OBBsbody parameters and 3D object boxes

Explore the data

For object interaction, details matter. That is why we provide high-resolution images with dense annotations across cameras.

Camera
Modality
body_follow · Overview

Five modalities next to the RGB frame, all pixel-aligned: depth, segmentation, forward flow, point tracks, and 3D boxes. Select one to see it on its own. Camera: body_follow, 1920 × 1080, pinhole.

Comparison

Prior synthetic datasets tend to specialize either on human body motion or on rigid object dynamics. Schlepp covers the intersection, with sequences of people grasping, carrying, and placing objects in furnished indoor scenes.

TypeFramesCamerasEgo + ExoHOI Multi-actorSim. clothDense flowPoint tracks Dense depth3D body + obj poses
Spring Sim 6K 2 –✓✓✓✓–✓–
Hypersim Sim 77K 1 ––––––✓Obj only
PointOdyssey Sim 208K 1 ––✓––✓✓Body only
Kubric (MOVi) Sim 1.2M 1 ––––✓✓✓Obj only
Syn4D Sim 1.4M 8 –––✓✓✓✓Body only
BEDLAM 2.0 Sim 8M 1 ––✓✓––PartialBody only
NymeriaPlus Real >30M Multi ✓✓✓Real–––MHR + OBB
Ego-Exo4D Real >100M5–6 ✓✓✓Real–––Body only
SchleppSim2.2M6✓✓✓✓✓✓✓MHR + OBB

Downstream tasks

Motion

Point tracking

~20K 3D tracks per sequence

Labelled as body, clothing, carried object, or scene, with visibility per camera.

Optical flow

Forward and backward

At full RGB resolution, with occlusion masks from depth.

Geometry

Monocular depth

Metric depth in metres

Furnished indoor scenes with people carrying objects; invalid pixels such as windows are masked.

Camera pose estimation

Six synchronized cameras

Three exocentric and three egocentric, with full intrinsics and extrinsics at metric scale.

Bodies and objects

Human pose and shape

MHR parameters per actor

Several actors per scene, often occluding each other and the furniture.

3D object detection

Bounding boxes with categories

Fully annotated scenes, where carried objects move every frame.

How it was built

Schlepp was created by distilling motion from video models. The generated videos are set in existing 3D scenes where we replicate and align the motion.

Goal

A person grips a coffee table; a dashed outline marks where the table should end up

The goal

We want to generate a person picking up an object, moving with it inside the room, and placing it down somewhere else. As we want to use arbitrary objects and scenes, we do not want to rely on motion capture data.

The motion split into a grasp, a carry, and the same grasp again

Moving an object: grasp, carry, grasp

We query a video model for the two components that make up a carry sequence separately. This makes 2D-3D alignment more feasible, reduces prompt complexity, and limits motion reconstruction errors.

Contact pose

Empty room with a level camera C1 facing the coffee table

Defining the frame

We place a camera C1 that frames the desired object in an artist-made 3D room, which will stage our motion sequence.

The room with camera C1 and a person standing in a neutral pose at the table

Placing an actor

We add a randomized person in a neutral pose close to the target object. This fixes the person's scale, simplifying the pose and shape recovery.

Camera view t0: the actor stands at the table, which is highlightedCamera view tN: the room without the table

Conditioning the video model to touch the object

By removing the object, we nudge the model to have the human touch the object and carry it outside the frame, a sequence that naturally includes a grasp.

t0: the actor stands at the tabletg: the generated person grips the tablet_g+k: the generated person carries the table outtN: room without the table

Generating the grasp motion

The video model completes the sequence, including a grasp frame tg, where the object is about to move. We discard the remaining frames.

Grasp frame tgThe extracted body, shown on its own

Extracting the contact pose

We estimate the body m in the grasp frame and refine it into the contact pose m⋆. We ensure that the person and grasp are in alignment with the 3D world.

Carry motion

Overview with camera C2, the start pose and the same pose moved and turned to the end position

Setting up the carry sequence conditioning

To tightly condition the sequence the video model needs to hallucinate, we define an end pose that reuses m⋆, translated by Δ and rotated by ψ (the transformation T). For better framing, we choose a new camera C2.

Camera view s0: start stateCamera view sN: end state

Conditioning for carry

We render the start and end states from the new camera C2 as the first and last frames for the carry motion sequence.

s0: start states_i: the generated person lifts the tables_j: the generated person carries the tablesN: end state

Generating the carry motion

The video model produces a sequence of frames showing the lift, the walk, and the set-down.

The extracted bodies of all four frames along the carry path, with the table

Recovering the carry motion

We extract the pose in every frame, smooth the motion, and "glue" the object to the hands. This eliminates tracking errors and plausibly matches real carrying motion.

Human-verified sequences

Annotators reviewed every sequence that passed our automatic checks and judged whether the motion and the grasp look plausible. They accepted about 15.9K of 57.2K.

Generated motion sequences 162K Passed checks automatic filters 57.2K Accepted human review 15.9K

Diverse human actors

Every actor wears physically simulated garments and shows fabric motion under real body dynamics. Skin textures, body shapes, and clothing combinations are drawn from a diverse pool.

Synthetic actor in a restaurant scene Synthetic actor in a bar scene Synthetic actor in a playroom scene Synthetic actor in a coffee shop scene Synthetic actor in a restaurant scene Synthetic actor in a bar scene Synthetic actor in a playroom scene Synthetic actor in a restaurant scene Synthetic actor in a restaurant scene

Details

Cameras

NameResolutionDistortion
static 1920 × 1080pinhole
body_follow 1920 × 1080pinhole
object_orbit1920 × 1080pinhole
aria_rgb 2016 × 1512KB4
aria_slamL 512 × 512 KB4
aria_slamR 512 × 512 KB4

Sequences

  • 15,850 sequences.
  • Mostly 130 to 144 frames at 30 fps (mean 140).
  • 20.6 hours in total.

Conventions

  • World: Z-up, right-handed, metres.
  • Camera: OpenCV; X right, Y down, Z forward.
  • Quaternions: [x, y, z, w].

Get the data

Download

Hugging Face · facebook/schlepp
Cameras
Per camera
Per sequence 
hf download facebook/schlepp \
    --repo-type dataset \
    --local-dir /data/schlepp \
    --include "*_static_rgb.tar" \
              "*_bodyfollow_rgb.tar" \
              "*_objectorbit_rgb.tar" \
              "*_pointtracks.tar" \
              "*_metadata.tar" \
              "*.parquet"

git clone https://github.com/facebookresearch/schlepp.git
pip install -e schlepp

cd schlepp && python -m scripts.unpack_chunks /data/schlepp

Only download the cameras and modalities you need. Chunks are stratified by carried-object category, so a partial download is still representative. Always include *_metadata.tar and *.parquet.

Then, pull the code and install the dependencies. To interface with the dataset, you'll need to unpack the chunks first.

Use the dataset

GitHub · facebookresearch/schlepp
from torch.utils.data import DataLoader
from schlepp import SchleppDataset
from schlepp.utils import resize_sample, subsample_tracks

def to_point_tracker(sample):
    sample = resize_sample(sample, short_side=384)
    pt = subsample_tracks(
        sample["point_tracks"], 512, mode="stratified",
    )
    cam = sample["cameras"]["body_follow"]
    return (
        cam["rgb"].permute(0, 2, 3, 1),       # (S, H, W, 3)
        pt["trajs_2d_pix"]["body_follow"],    # (S, 512, 2)
        pt["visible"]["body_follow"],         # (S, 512)
    )

ds = SchleppDataset(
    "/data/schlepp",
    modalities=["rgb", "point_tracks"],
    cameras="body_follow",
    seq_len=24,
    transform=to_point_tracker,
)

loader = DataLoader(ds, batch_size=4, num_workers=8)
for rgbs, trajs, visibs in loader:
    ...   # train your point tracker

Example code to train a point tracker. We provide a Dataset class for Schlepp, which gets you quickly up and running. See the README for full recipes for flow, depth, pose, MHR, and OBB tasks, plus the bundled evaluation and visualisation tools.

Citation

@inproceedings{engstler2026schlepp,
  title     = {Schlepp: A Multi-View Synthetic Dataset of Humans Carrying Objects},
  author    = {Engstler, Paul and Wang, Yihan and Aubel, Amaury and Deshmukh, Priyamvad and
               Segev, Shai and Li, Zhengqin and Barnes, Dan and Rupprecht, Christian and
               Siddiqui, Yawar and Ma, Lingni and Straub, Julian and Dong, Zhao and
               Newcombe, Richard and Vedaldi, Andrea and Engel, Jakob and Harley, Adam W.},
  booktitle = {TBD},
  year      = {2026},
}