Image Code Snippets

In this section, we introduce the Python/C++ API to access and manipulate Project Aria images (projectaria_tools/main/core/image). Raw Aria data is stored in VRS files.

Raw sensor data

Raw image data is stored in ImageData. ImageData is a type alias of an std::pair. The two components of that pair are:

1. The image frame stored in vrs::PixelFrame class (potentially compressed)
   • We recommend that users do not directly use PixelFrame
2. Image data records
   • Image acquisition information such as timestamps, exposure and gain

Python

from projectaria_tools.core import data_provider, image
from projectaria_tools.core.stream_id import StreamId

vrsfile = "example.vrs"
provider = data_provider.create_vrs_data_provider(vrsfile)

stream_id = provider.get_stream_id_from_label("camera-slam-left")
image_data =  provider.get_image_data_by_index(stream_id, 0)
pixel_frame = image_data[0].pixel_frame

C++

auto streamId = provider.getStreamIdFromLabel("camera-slam-left");
auto imageData =  provider.getImageDataByIndex(streamId, i);
auto pixelFrame = imageData->pixelFrame();

Since PixelFrame may contain compressed data, the class does not provide an interface for accessing pixel values.

ImageData provides an interface to get an ImageVariant interface for the data described below:

auto maybeImageVariant = imageData.imageVariant();
XR_CHECK(maybeImageVariant, "Image is invalid");
auto& imageVariant = *maybeImageVariant();

We recommend that C++ users to manipulate images using the Image and ManagedImage and their variant classes.

Manipulating images

Python

In Python, we provide an interface for converting from ImageData into numpy arrays.

image_array = image_data[0].to_numpy_array()

C++

Image and ImageVariants (C++)

The Image class provides an interface to access image information and pixels. The class is templated, with different specializations varying by number of channels and scalar data type.

ImageU8 image = std::get<ImageU8>(imageVariant); // get grayscale image
int width = image.width(); // image width
int height = image.height(); // image height
int channel = image.channel(); // number of channels
int stride = image.stride(); // number of bytes per row
uint8_t* data = image.data(); // weak pointer to data
uint8_t pixel_value = image(0, 0); // access to pixel value if coordinate is of integral type
uint8_t pixel_value = image(0,5, 0.5); // bilinear interpolate pixel value if coordinate is of float type

You can iterate through an image by using:

for (const uint8_t& pixel : image) {
  // process pixel
}

Note that the Image class is non-owning. It is a wrapper of a chunk of data, which might be managed by PixelFrame or ManagedImage or even a raw data pointer.

The ImageVariant class represents uncompressed image frames in matrix form. Under the hood, it is a std::variant of Image classes of different specializations.

We provide similar APIs to access data from image variants.

int width = getWidth(imageVariant); // image width
int height = getHeight(imageVariant); // image width
int channel = getChannel(imageVariant); // number of channels
uint8_t* data = getDataPtr(imageVariant); // pointer to data
uint8_t pixel_value = at(imageVariant, 0, 0); // access to pixel value if coordinate is of integral type
// bilinear interpolation not available yet, but you can do the following
uint8_t pixel_value = std::visit([](auto& image) {return PixelValueVariant(image(0.5, 0.5, 0))}, imageVariant);

The image variant types used in Aria raw sensor data are listed in the table below.

Sensor	Number of Channels	Scalar Data Type	Image Type	ManagedImage Type
Eyetracking	1	uint8_t	ImageU8	ManagedImageU8
Mono Scene (SLAM)	1	uint8_t	ImageU8	ManagedImageU8
RGB	3	uint8_t	Image3U8	ManagedImage3U8
Depth	1	uint16_t	ImageU16	ManagedImageU16
Segmentation	1	uint64_t	ImageU64	ManagedImageU64

ManagedImage and ManagedImageVariant (C++)

The templated ManagedImage class manages the data of an image. Most importantly, you can initialize a ManagedImage via:

ManagedImageU8 managedImage(width, height); // grayscale image

You can change the size of an existing ManagedImage via:

managedImage.reinitialize(newWidth, newHeight); // grayscale image

The class is a derived class of the corresponding class, and therefore inherits all the Image interface. All functions taking Image as input can also take ManagedImages.

The ManagedImageVariant class is the std::variant of all supported ManagedImage specializations. Notably, ManagedImageVariant is not a derived class of ImageVariant. However, assume you have a function fn that takes ImageVariant, you can pass a ManagedImageVariant object by using:

ImageVariant imageVariant = toImageVariant(managedImageVariant);
fn(imageVariant);