Advanced Image Utilities

Overview

This page provides advanced image utilities code snippets for Project Aria Tools, see also Image Code Snippets.

Image debayer

Some recording profiles outputs raw RGB images (Profile 7 in Recording Profile). We provide functionalities to debayer them and perform white-balancing to get RGB images.

Python

from projectaria_tools.core import data_provider, image

stream_id = provider.get_stream_id_from_label("camera-rgb")
image_data = provider.get_image_data_by_index(stream_id, 0)
image_data_array = image_data[0].to_numpy_array()
debayered_array = image.debayer(image_data_array)

C++

#include <dataprovider/VrsDataProvider.h>
#include <image/utility/Debayer.h>

auto streamId = provider.getStreamIdFromLabel("camera-rgb");
auto imageData = provider.getImageDataByIndex(streamId, 0);
auto debayeredImage = debayer(imageData.first.imageVariant());

See projectaria_tools/core/image/utility/Debayer.cpp for implementation

Image undistortion

In this example, we remove distortions in raw sensor data so that straight 3D lines appear straight in the undistorted images. There is existing C++ implementation and Python wrapper of this helper function in the data utilities.

Python

from projectaria_tools.core import data_provider, calibration

camera_label = "camera-slam-left"
stream_id = provider.get_stream_id_from_label(camera_label)

calib = provider.get_device_calibration().get_camera_calib(camera_label)
pinhole = calibration.get_linear_camera_calibration(512, 512, 150)

raw_image = provider.get_image_data_by_index(stream_id, 0)[0].to_numpy_array()
undistorted_image = calibration.distort_by_calibration(raw_image, pinhole, calib)

C++

#include <dataprovider/VrsDataProvider.h>
#include <calibration/utility/Distort.h>

std::string cameraLabel = "camera-rgb";
vrs::StreamId streamId = provider->getStreamIdFromLabel(cameraLabel);

CameraCalibration calib = provider->getCameraCalibration(streamId);
CameraCalibration pinhole = getLinearCameraCalibration(512, 512, 150);

ImageData rawImage = provider->getImageDataByIndex(streamId, 0);
auto undistortedImage = distortByCalibration(rawImage.imageVariant(), pinhole, calib);

Go to projectaria_tools/core/calibration/utility/Distort.cpp for implementation.

Go to the Project Aria FAQ for more calibration information and resources.

Rotated image clockwise 90 degrees

In this example, we rotated the RGB image 90 degrees and provide the new calibration object.

• Calibration rotation only applies to pinhole camera model
• Pinhole camera calibration object needs to be initialized as pinhole = calibration.get_linear_camera_calibration(512, 512, 150, camera_label, calib.get_transform_device_camera()) with camera_label and the pose calib.get_transform_device_camera() so that pinhole_cw90 can have the correct transformation matrix when unprojecting a pixel to get ray_in_device_frame.

Python

camera_label = "camera-rgb"
stream_id = provider.get_stream_id_from_label(camera_label)

calib = provider.get_device_calibration().get_camera_calib(camera_label)
pinhole = calibration.get_linear_camera_calibration(512, 512, 150, camera_label,
calib.get_transform_device_camera())

raw_image = provider.get_image_data_by_index(stream_id, 0)[0].to_numpy_array()
undistorted_image = calibration.distort_by_calibration(raw_image, pinhole, calib)

# Rotated image by CW90 degrees
rotated_image = np.rot90(undistorted_image, k=3)

# Get rotated image calibration
pinhole_cw90 = calibration.rotate_camera_calib_cw90deg(pinhole)

# Unproject a pixel and get a ray from device coordinate frame
test_pixel_in_rotated_image = [10,0]
ray_in_device_frame =  pinhole_cw90.get_transform_device_camera() @ pinhole_cw90.unproject_no_checks(test_pixel_in_rotated_image)

C++

#include <dataprovider/VrsDataProvider.h>
#include <calibration/utility/Distort.h>

std::string cameraLabel = "camera-rgb";
vrs::StreamId streamId = provider->getStreamIdFromLabel(cameraLabel);

CameraCalibration calib = provider->getCameraCalibration(streamId);
CameraCalibration pinhole = getLinearCameraCalibration(512, 512, 150, cameraLabel, calib.getT_Device_Camera());

// Get rotated image calibration
CameraCalibration pinholeCW90 = rotatedCameraCalibCW90Deg(pinhole);

// Unproject a pixel and get a ray from device coordinate frame
Eigen::Vector2d textPixelInRotatedImage{10,0};
auto ray = pinholeCW90.getT_Device_Camera() * pinholeCW90.projectNoChecks(textPixelInRotatedImage);

Image devignetting

Devignetting corrects uneven lighting, enhancing image uniformity and clarity. We provide devignetting for camera-rgb full size image [2880, 2880], camera-rgb half size image[1408, 1408] and slam iamge [640, 480].

1. Download devignetting masks from Link
2. Unzip to local folder. The folder should follow the structure below:

aria_camera_devignetting_masks
|- slam_devignetting_mask.bin
|- rgb_half_devignetting_mask.bin
|- rgb_full_devignetting_mask.bin

3. set devignetting mask folder path with local folder.
4. load mask by passing camera label, now supporting "camera-rgb", "camera-slam-left" and "camera-slam-right"
5. apply devignetting to source image.

Python

from projectaria_tools.core import data_provider, calibration, image
camera_label = "camera-rgb"
devignetting_mask_folder_path = <FOLDER_PATH_CONTAINING_DEVIGNETTING_MASK>
device_calib = provider.get_device_calibration()
stream_id = provider.get_stream_id_from_label(camera_label)
raw_image = provider.get_image_data_by_index(stream_id, 0)[0].to_numpy_array()

# set devignetting mask with local folder path
device_calib.set_devignetting_mask_folder_path(devignetting_mask_folder_path)

# load devignetting_mask by camera label
devignetting_mask = device_calib.load_devignetting_mask(camera_label)

# apply devignetting to source image
devigneted_image = calibration.devignetting(raw_image, devignetting_mask)

C++

#include <dataprovider/VrsDataProvider.h>
#include <image/utility/Devignetting.h>

std::string cameraLabel = "camera-rgb";
std::string devignettingMaskFolderPath = <FOLDER_PATH_CONTAINING_DEVIGNETTING_MASK>;
auto deviceCalib = provider->getDeviceCalibration();
vrs::StreamId streamId = provider->getStreamIdFromLabel(cameraLabel);
auto imageData = provider->getImageDataByIndex(streamId, 0);

// set devignetting mask with local folder path
deviceCalib.SetDevignettingMaskFolderPath(devignettingMaskFolderPath);

// load devignetting_mask by camera label
Eigen::MatrixXf devignettingMask = deviceCalib.loadDevignettingMask(cameraLabel);

// apply devignetting to source image
auto devignettedImage = devignetting(imageData.first.imageVariant(), devignettingMask);

Go to projectaria_tools/core/image/utility/Devignetting.cpp for implementation.