Tutorial 5: On-Device VIO data streams

Introduction

In Aria-Gen2 glasses, one of the key upgrade from Aria-Gen1 is the capability to run Machine Perception (MP) algorithms on the device during streaming / recording. Currently supported on-device MP algorithms include Eye-tracking, Hand-tracking, and VIO. These algorithm results are stored as separate data streams in the VRS file.

VIO (Visual Inertial Odometry) combines camera images and IMU (Inertial Measurement Unit) data to estimate device pose and motion in real-time. VIO tracks the device's position, orientation, and velocity by performing visual tracking, IMU integration, sensor fusion, etc, making it the foundation for spatial tracking and understanding.

In Aria-Gen2 devices, the VIO algorithm are run on device to produce 2 types of tracking results as part of the VRS file: VIO and VIO High Frequency. This tutorial focuses on demonstration of how to use the on-device VIO and VIO_high_frequency results.

What you'll learn:

How to access on-device VIO and VIO_high_frequency data from VRS files
How to visualize 3D trajectory from on-device VIO data.

Prerequisites

Complete Tutorial 1 (VrsDataProvider Basics) to understand basic data provider concepts
Complete Tutorial 2 (Device Calibration) to understand how to properly use calibration in Aria data.
Download Aria Gen2 sample data from link

Note on visualization: If visualization window is not showing up, this is due to Rerun lib's caching issue. Just rerun the specific code cell.

from projectaria_tools.core import data_provider

# Load local VRS file
vrs_file_path = "path/to/your/recording.vrs"
vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file_path)

# Query VIO data streams
vio_label = "vio"
vio_stream_id = vrs_data_provider.get_stream_id_from_label(vio_label)
if vio_stream_id is None:
    raise RuntimeError(
        f"{vio_label} data stream does not exist! Please use a VRS that contains valid VIO data for this tutorial."
    )

# Query VIO_high_frequency data streams
vio_high_freq_label = "vio_high_frequency"
vio_high_freq_stream_id = vrs_data_provider.get_stream_id_from_label(vio_high_freq_label)
if vio_high_freq_stream_id is None:
    raise RuntimeError(
        f"{vio_high_freq_label} data stream does not exist! Please use a VRS that contains valid VIO high frequency data for this tutorial."
    )

On-Device VIO Data Stream

Data Type: `FrontendOutput`

This a new data type introduced to store the results from the VIO system, containing the following fields:

Field Name	Description
`frontend_session_uid`	Session identifier (resets on VIO restart)
`frame_id`	Frame set identifier
`capture_timestamp_ns`	Center capture time in nanoseconds
`unix_timestamp_ns`	Unix timestamp in nanoseconds
`status`	VIO status (VALID/INVALID)
`pose_quality`	Pose quality (GOOD/BAD/UNKNOWN)
`visual_tracking_quality`	Visual-only tracking quality
`online_calib`	Online calibration estimates for SLAM cameras and IMUs
`gravity_in_odometry`	Gravity vector in odometry frame
`transform_odometry_bodyimu`	Body IMU's pose in odometry reference frame
`transform_bodyimu_device`	Transform from body IMU to device frame
`linear_velocity_in_odometry`	Linear velocity in odometry frame in m/s
`angular_velocity_in_bodyimu`	Angular velocity in body IMU frame in rad/s

Here, body IMU is the IMU that is picked as the reference for motion tracking. For Aria-Gen2' on-device VIO algorithm, this is often imu-left.

Important Note: Always check status == VioStatus.VALID and pose_quality == TrackingQuality.GOOD for VIO data validity!

Data Access API

from projectaria_tools.core.sensor_data import VioStatus, TrackingQuality

print("=== VIO Data Sample ===")

# Find the first valid VIO data sample
num_vio_samples = vrs_data_provider.get_num_data(vio_stream_id)
first_valid_index = None
for idx in range(num_vio_samples):
    vio_data = vrs_data_provider.get_vio_data_by_index(vio_stream_id, idx)
    if (
        vio_data.status == VioStatus.VALID
        and vio_data.pose_quality == TrackingQuality.GOOD
    ):
        first_valid_index = idx
        break

if first_valid_index is not None:
    print("=" * 50)
    print(f"First VALID VIO Data Sample (Index: {first_valid_index})")
    print("=" * 50)

    # Session Information
    print(f"Session UID: {vio_data.frontend_session_uid}")
    print(f"Frame ID: {vio_data.frame_id}")

    # Timestamps
    print(f"Capture Time: {vio_data.capture_timestamp_ns} ns")
    print(f"Unix Time: {vio_data.unix_timestamp_ns} ns")

    # Quality Status
    print(f"Status: {vio_data.status}")
    print(f"Pose Quality: {vio_data.pose_quality}")
    print(f"Visual Quality: {vio_data.visual_tracking_quality}")

    # Transforms
    print(f"Transform Odometry → Body IMU:\n{vio_data.transform_odometry_bodyimu.to_matrix()}")
    print(f"Transform Body IMU → Device:\n{vio_data.transform_bodyimu_device.to_matrix()}")

    # Motion
    print(f"Linear Velocity: {vio_data.linear_velocity_in_odometry}")
    print(f"Angular Velocity: {vio_data.angular_velocity_in_bodyimu}")
    print(f"Gravity Vector: {vio_data.gravity_in_odometry}")
else:
    print("⚠️  No valid VIO sample found")

On-Device VIO High Frequency Data Stream

VIO High Frequency results are generated directly from the on-device VIO results by performing IMU integration between VIO poses, hence provides a much higher data rate at approximately 800Hz.

Data Type: `OpenLoopTrajectoryPose`

The VioHighFrequency stream re-uses the OpenLoopTrajectoryPose data structure defined in MPS.

Field Name	Description
`tracking_timestamp`	Timestamp in device time domain, in microseconds
`transform_odometry_device`	Transformation from device to odometry coordinate frame, represented as a SE3 instance.
`device_linear_velocity_odometry`	Translational velocity of device in odometry frame, in m/s
`angular_velocity_device`	Angular velocity of device in device frame, in rad/s
`quality_score`	Quality of pose estimation (higher = better)
`gravity_odometry`	Earth gravity vector in odometry frame
`session_uid`	Unique identifier for VIO tracking session

Important Note: Due to the high frequency nature of this data (~800Hz), consider subsampling for visualization to maintain performance.

print("=== VIO High-Frequency Data Sample ===")

# Find the first VIO high_frequency data sample with high quality value
num_vio_high_freq_samples = vrs_data_provider.get_num_data(vio_high_freq_stream_id)
first_valid_index = None
for idx in range(num_vio_samples):
    vio_high_freq_data = vrs_data_provider.get_vio_high_freq_data_by_index(vio_high_freq_stream_id, idx)
    if (
        vio_high_freq_data.quality_score > 0.5
    ):
        first_valid_index = idx
        break

if first_valid_index is not None:
    print("=" * 50)
    print(f"First VIO High Freq Data Sample with good quality score (Index: {first_valid_index})")
    print("=" * 50)

    # Timestamps, convert timedelta to nanoseconds
    capture_timestamp_ns = int(vio_high_freq_data.tracking_timestamp.total_seconds() * 1e9)

    # Session Information
    print(f"Session UID: {vio_high_freq_data.session_uid}")

    # Timestamps
    print(f"Tracking Time: {capture_timestamp_ns} ns")

    # Quality
    print(f"Quality Score: {vio_high_freq_data.quality_score:.3f}")

    # Transform
    print(f"Transform Odometry → Device:\n{vio_high_freq_data.transform_odometry_device.to_matrix()}")

    # Motion
    print(f"Linear Velocity: {vio_high_freq_data.device_linear_velocity_odometry}")
    print(f"Angular Velocity: {vio_high_freq_data.angular_velocity_device}")
    print(f"Gravity Vector: {vio_high_freq_data.gravity_odometry}")

Visualizing On-Device VIO trajectory

The following code snippets demonstrate how to visualize a VIO trajectory, along with glass frame + hand tracking results, in a 3D view.

def plot_single_hand_3d(
    hand_joints_in_device, hand_label
):
    """
    A helper function to plot single hand data in 3D view
    """
    marker_color = [255,64,0] if hand_label == "left" else [255, 255, 0]

    hand_skeleton_3d = create_hand_skeleton_from_landmarks(hand_joints_in_device)
    rr.log(
        f"world/device/handtracking/{hand_label}/landmarks",
        rr.Points3D(
            positions=hand_joints_in_device,
            colors= marker_color,
            radii=5e-3,
        ),
    )
    rr.log(
        f"world/device/handtracking/{hand_label}/hand_skeleton",
        rr.LineStrips3D(
            hand_skeleton_3d,
            colors=[0, 255, 0],
            radii=3e-3,
        ),
    )


def plot_hand_pose_data_3d(hand_pose_data):
    """
    A helper function to plot hand pose data in 3D world view
    """
    # Clear the canvas (only if hand_tracking_label exists for this device version)
    rr.log(
        f"world/device/handtracking",
        rr.Clear.recursive(),
    )

    # Plot both hands
    if hand_pose_data.left_hand is not None:
        plot_single_hand_3d(
            hand_joints_in_device=hand_pose_data.left_hand.landmark_positions_device,
            hand_label="left",
        )
    if hand_pose_data.right_hand is not None:
        plot_single_hand_3d(
            hand_joints_in_device=hand_pose_data.right_hand.landmark_positions_device,
            hand_label="right",
        )

3D Trajectory Visualization

import rerun as rr
from projectaria_tools.core.sensor_data import SensorDataType, TimeDomain, TimeQueryOptions
from projectaria_tools.utils.rerun_helpers import (
    create_hand_skeleton_from_landmarks,
    AriaGlassesOutline,
    ToTransform3D
)

print("\n=== Visualizing on-device VIO trajectory + HandTracking in 3D view ===")

rr.init("rerun_viz_vio_trajectory")

device_calib = vrs_data_provider.get_device_calibration()
handtracking_stream_id = vrs_data_provider.get_stream_id_from_label("handtracking")

# Set up a data queue
deliver_options = vrs_data_provider.get_default_deliver_queued_options()
deliver_options.deactivate_stream_all()
deliver_options.activate_stream(vio_stream_id)

# Play for only 3 seconds
total_length_ns = vrs_data_provider.get_last_time_ns_all_streams(TimeDomain.DEVICE_TIME) - vrs_data_provider.get_first_time_ns_all_streams(TimeDomain.DEVICE_TIME)
skip_begin_ns = int(15 * 1e9) # Skip 15 seconds
duration_ns = int(3 * 1e9) # 3 seconds
skip_end_ns = max(total_length_ns - skip_begin_ns - duration_ns, 0)
deliver_options.set_truncate_first_device_time_ns(skip_begin_ns)
deliver_options.set_truncate_last_device_time_ns(skip_end_ns)

# Plot VIO trajectory in 3D view.
# Need to keep a cache to store already-loaded trajectory
vio_traj_cached_full = []
for sensor_data in vrs_data_provider.deliver_queued_sensor_data(deliver_options):
    # Convert sensor data to VIO data
    vio_data = sensor_data.vio_data()

    # Check VIO data validity, only plot for valid data
    if ( vio_data.status != VioStatus.VALID or vio_data.pose_quality != TrackingQuality.GOOD):
        print(f"VIO data is invalid for timestamp {sensor_data.get_time_ns(TimeDomain.DEVICE_TIME)}")
        continue

    # Set timestamp
    rr.set_time_nanos("device_time", vio_data.capture_timestamp_ns)

    # Set and plot the Device pose for the current timestamp, as a RGB axis
    T_World_Device = (
        vio_data.transform_odometry_bodyimu @ vio_data.transform_bodyimu_device
    )
    rr.log(
        "world/device",
        ToTransform3D(
            T_World_Device,
            axis_length=0.05,
        ),
    )

    # Also plot Aria glass outline for visualization
    aria_glasses_point_outline = AriaGlassesOutline(
        device_calib, use_cad_calib=True
    )
    rr.log(
        "world/device/glasses_outline",
        rr.LineStrips3D(
            aria_glasses_point_outline,
            colors=[200,200,200],
            radii=5e-4,
        ),
    )

    # Plot gravity direction vector
    rr.log(
        "world/vio_gravity",
        rr.Arrows3D(
            origins=[T_World_Device.translation()[0]],
            vectors=[
                vio_data.gravity_in_odometry * 1e-2
            ],  # length converted from 9.8 meter -> 10 cm
            colors=[101,67,33],
            radii=1.5e-3,
        ),
        static=False,
    )

    # Plot VIO trajectory that are cached so far
    vio_traj_cached_full.append(T_World_Device.translation()[0])
    rr.log(
        "world/vio_trajectory",
        rr.LineStrips3D(
            vio_traj_cached_full,
            colors=[173, 216, 255],
            radii=1.5e-3,
        ),
        static=False,
    )

    # For visualization purpose, also plot the hand tracking results
    interpolated_hand_pose = vrs_data_provider.get_interpolated_hand_pose_data(handtracking_stream_id, vio_data.capture_timestamp_ns, TimeDomain.DEVICE_TIME)
    if interpolated_hand_pose is not None:
        plot_hand_pose_data_3d(hand_pose_data = interpolated_hand_pose)

rr.notebook_show()

Introduction​

On-Device VIO Data Stream​

Data Type: FrontendOutput​

Data Access API​

On-Device VIO High Frequency Data Stream​

Data Type: OpenLoopTrajectoryPose​

Visualizing On-Device VIO trajectory​

3D Trajectory Visualization​