Skip to main content

DTC Data Format

The Digital Twin Catalog Dataset (DTC) provides real world raw Project Aria and digital single-lens reflex (DSLR) camera data, estimated object pose as well as derived data generated by Project Aria's Machine Perception Services (MPS).

Aria Sequence Structure​

|Sequence1Name|
    ├──video.vrs  # Aria recording data
    ├──CC_BY-SA.txt  # license file
    ├──object_pose.json  # aligned object pose file
    ├──mps  # derived data generated by Project Aria’s MPS
    ├──metadata.json

The object_pose.json file in each sequence directory contains the object pose aligned with the MPS data including the camera trajectories as well as the semi-dense point clouds.

DSLR Sequence Structure​

|Sequence1Name|
    ├──exrs  # DSLR captured high dynamic range frames
    ├──pngs  # converted low dynamic range frames
    ├──CC_BY-SA.txt  # license file
    ├──camera_poses.json  # DSLR camera poses file
    ├──object_pose.json  # aligned object pose file
    ├──metadata.json
    ├──env.exr # scene environment file

The camera_poses.json file in each sequence directory contains DSLR camera poses. The object_pose.json file in each sequence directory contains the object pose aligned with the camera poses.

Data Format​

video.vrs​

video.vrs contains the raw sensor recording from the Aria device.

  • Aria Hardware Specifications shows the sensors used to make recordings
    • RGB images were all recorded at 10 fps with fixed exposure (2ms) and gain (3)

mps​

mps folder contains the derived data from Machine Perception Service (MPS) including

  • open_loop_trajectory.csv records the camera trajectory estimated using the online estimation
  • closed_loop_trajectory.csv records the camera trajectory estimated using the offline optimization
  • semidense_observations.csv.gz records the per frame observability for each point in the semi-dense point cloud
  • semidense_points.csv.gz records the full semi-dense point cloud estimated from the recorded sequence
  • online_calibration.jsonl records the online calibration results for SLAM cameras, RGB camera and IMU

object_pose.json​

object_pose.json contains the pose (i.e. transformation from object space to world space) of the captured object’s digital copy (defined in object space) aligned with the MPS derived data or the DSLR camera poses.

{
    "mesh": {
        "T_world_object": [
            …
        ]
    }
}

exrs​

exrs folder contains the high dynamic range images captured using three DSLR cameras.

pngs​

pngs folder contains the converted png files of the images in the exrs folder.

camera_poses.json​

camera_poses.json contains the estimated camera poses (i.e. transformation from world to camera), camera intrinsics and corresponding image files of DSLR cameras.

{
    "cameras": [
        {
            "T_camera_world": [
               …
            ],
            "focal": …,
            "height": …,
            "image": "...",
            "intrinsic": {
                …
            },
            "width": …
        },
        …
     ]
}

env.exr​

env.exr contains the high dynamic range environment map of DSLR captures aligned with the camera poses.