Advanced Image Utilities

Overview

This page provides advanced image utilities code snippets for Project Aria Tools, see also Image Code Snippets.

Image debayer

Some recording profiles outputs raw RGB images (Profile 7 in Recording Profile). We provide functionalities to debayer them and perform white-balancing to get RGB images.

Python

from projectaria_tools.core import data_provider, image

stream_id = provider.get_stream_id_from_label("camera-rgb")
image_data = provider.get_image_data_by_index(stream_id, 0)
image_data_array = image_data[0].to_numpy_array()
debayered_array = image.debayer(image_data_array)

C++

#include <dataprovider/VrsDataProvider.h>
#include <image/utility/Debayer.h>

auto streamId = provider.getStreamIdFromLabel("camera-rgb");
auto imageData = provider.getImageDataByIndex(streamId, 0);
auto debayeredImage = debayer(imageData.first.imageVariant());

See projectaria_tools/core/image/utility/Debayer.cpp for implementation

Image undistortion

In this example, we remove distortions in raw sensor data so that straight 3D lines appear straight in the undistorted images. There is existing C++ implementation and Python wrapper of this helper function in the data utilities.

Python

from projectaria_tools.core import data_provider, calibration

camera_label = "camera-slam-left"
stream_id = provider.get_stream_id_from_label(camera_label)

calib = provider.get_device_calibration().get_camera_calib(camera_label)
pinhole = calibration.get_linear_camera_calibration(512, 512, 150)

raw_image = provider.get_image_data_by_index(stream_id, 0)[0].to_numpy_array()
undistorted_image = calibration.distort_by_calibration(raw_image, pinhole, calib)

C++

#include <dataprovider/VrsDataProvider.h>
#include <calibration/utility/Distort.h>

std::string cameraLabel = "camera-rgb";
vrs::StreamId streamId = provider->getStreamIdFromLabel(cameraLabel);

CameraCalibration calib = provider->getCameraCalibration(streamId);
CameraCalibration pinhole = getLinearCameraCalibration(512, 512, 150);

ImageData rawImage = provider->getImageDataByIndex(streamId, 0);
auto undistortedImage = distortByCalibration(rawImage.imageVariant(), pinhole, calib);

Go to projectaria_tools/core/calibration/utility/Distort.cpp for implementation.

Go to the Project Aria FAQ for more calibration information and resources.

Rotated image clockwise 90 degrees

In this example, we rotated the RGB image 90 degrees and provide the new calibration object.

• Calibration rotation only applies to pinhole camera model
• Pinhole camera calibration object needs to be initialized as pinhole = calibration.get_linear_camera_calibration(512, 512, 150, camera_label, calib.get_transform_device_camera()) with camera_label and the pose calib.get_transform_device_camera() so that pinhole_cw90 can have the correct transformation matrix when unprojecting a pixel to get ray_in_device_frame.

Python

camera_label = "camera-rgb"
stream_id = provider.get_stream_id_from_label(camera_label)

calib = provider.get_device_calibration().get_camera_calib(camera_label)
pinhole = calibration.get_linear_camera_calibration(512, 512, 150, camera_label,
calib.get_transform_device_camera())

raw_image = provider.get_image_data_by_index(stream_id, 0)[0].to_numpy_array()
undistorted_image = calibration.distort_by_calibration(raw_image, pinhole, calib)

# Rotated image by CW90 degrees
rotated_image = np.rot90(undistorted_image, k=3)

# Get rotated image calibration
pinhole_cw90 = calibration.rotate_camera_calib_cw90deg(pinhole)

# Unproject a pixel and get a ray from device coordinate frame
test_pixel_in_rotated_image = [10,0]
ray_in_device_frame =  pinhole_cw90.get_transform_device_camera() @ pinhole_cw90.unproject_no_checks(test_pixel_in_rotated_image)

C++

#include <dataprovider/VrsDataProvider.h>
#include <calibration/utility/Distort.h>

std::string cameraLabel = "camera-rgb";
vrs::StreamId streamId = provider->getStreamIdFromLabel(cameraLabel);

CameraCalibration calib = provider->getCameraCalibration(streamId);
CameraCalibration pinhole = getLinearCameraCalibration(512, 512, 150, cameraLabel, calib.getT_Device_Camera());

// Get rotated image calibration
CameraCalibration pinholeCW90 = rotatedCameraCalibCW90Deg(pinhole);

// Unproject a pixel and get a ray from device coordinate frame
Eigen::Vector2d textPixelInRotatedImage{10,0};
auto ray = pinholeCW90.getT_Device_Camera() * pinholeCW90.projectNoChecks(textPixelInRotatedImage);