Streaming Undistorted RGB Image Using Calibration

Overview

This page shows how to run the code sample undistort_rgb_image to:

• Access a Project Aria Tools type device calibration object
• Use core data utilities in projectaria_tools to undistort streamed camera data

Run undistort_rgb_image

1. Plug your Aria glasses into your computer, using the provided USB cable
2. From the samples directory in Terminal, run:

python -m undistort_rgb_image --interface usb --update_iptables

info

Use --interface wifi to stream over Wi-FI

Code walkthrough

The code walkthrough for undistort_rgb_image.py is similar to device_stream.py, but has 2 key differences:

1. Access sensor calibration

Once the sensors have been configured, the recording manager can provide the sensor calibration data for those settings.

sensors_calib_json = streaming_manager.sensors_calibration()

2. Use Project Aria Tools for calibration operations

A Project Aria Tools type device calibration object can then be retrieved by using the device_calibration_from_json_string function:

from projectaria_tools.core.calibration import (
    device_calibration_from_json_string,
    distort_by_calibration,
    get_linear_camera_calibration,
)
sensors_calib = device_calibration_from_json_string(sensors_calib_json)
rgb_calib = sensors_calib.get_camera_calib("camera-rgb")

Get a linear camera calibration object to be used in RGB image undistortion:

dst_calib = get_linear_camera_calibration(512, 512, 150, "camera-rgb")

To find out more about how to use sensor calibration, go to the Accessing Sensor Calibration page.

3. Undistort and visualize the live RGB image stream

Unlike device_stream.py that uses custom streaming client observer class, undistort_rgb_image.py uses a simple streaming client observer class to define callbacks:

class StreamingClientObserver:
  def __init__(self):
      self.rgb_image = None

  def on_image_received(self, image: np.array, record: ImageDataRecord):
      self.rgb_image = image

Undistort the RGB image using distort_by_calibration and visualize it in a while loop. The camera RGB image and the undistorted RGB image are visualized in separate windows using OpenCV. The images are processed and displayed the streaming stops or the application quit.

while not (quit_keypress() or ctrl_c):
    if observer.rgb_image is not None:
        rgb_image = cv2.cvtColor(observer.rgb_image, cv2.COLOR_BGR2RGB)
        cv2.imshow(rgb_window, np.rot90(rgb_image, -1))

        # Apply the undistortion correction
        undistorted_rgb_image = distort_by_calibration(
            rgb_image, dst_calib, rgb_calib
        )
        # Show the undistorted image
        cv2.imshow(undistorted_window, np.rot90(undistorted_rgb_image, -1))

        observer.rgb_image = None

note

Cameras on Aria glasses are installed sideways. The visualizer rotates the raw image data for a more natural view.