Skip to main content

Creating pointclouds from depth maps & RGB images

This tutorial will walk you through how to take the depth maps and RGB images from Aria Digital Twin (ADT) and convert to a colored 3D pointcloud in the Scene coordinate frame.

For more information about the depth maps, go to ADT Data Formats

Methodology​

ADT provides a depth map corresponding to each Aria image for each of the 3 cameras (1 RGB and 2 SLAM).

To create a combined 3D pointcloud with color, the tutorial will:

  1. Get RGB image timestamps
  2. iterate through the timestamps and sample the RGB image and depth image with that timestamp
  3. get the camera pose at the query timestamp
  4. iterate through all pixels and get the RGB & XYZ data:
    • back project pixel coordinate using the camera model to get the ray direction
    • get the ray depth from depth map using the current pixel coordinates
    • get the 3D point in the camera frame by multiplying the ray by its depth
    • transform the point from the camera frame to scene frame using the camera pose
    • assign RGB values using the RGB image with the current pixel coordinates

Run tutorial notebook​

You can run this tutorial on Google Colab via the ADT Depth Maps to Pointcloud Tutorial.

To run the notebook locally using Jupyter, follow these steps:

Step 0 : Install project_aria_tools package and create venv if not already done​

Follow Step 0 to Step 3 in Getting Started.

Step 1 : Download codebase​

Follow the instructions in Download codebase to clone the code

Step 2 : Run the tutorial notebook​

From your Python virtual environment, run:

cd $HOME/Documents/projectaria_sandbox

jupyter notebook projectaria_tools/projects/AriaDigitalTwinDatasetTools/examples/adt_depth_maps_to_pointcloud_tutorial.ipynb

This notebook automatically downloads the 10s sample sequence. Feel free to use the ADT downloader to try on other sequences!