Skip to main content

MPS Code Snippets

Project Aria Machine Perception Services (MPS) enables Aria users with access to the Aria Research Kit to request derived data on Aria VRS files.

Open Datasets also contain MPS outputs and may have specific tools to use and visualize their data.

This page covers how to consume standard MPS outputs.

Load MPS output​

The loaders for MPS outputs (projectaria_tools/main/core/mps) make it easer to use the data downstream. As part of this, the loaders put the outputs into data structures that are easier for other tools to consume.

MPS Data Formats provides details about output schemas and the specifics of each MPS output. This page focuses loading APIs in Python and C++, where there isn't a standalone code samples page:

Open loop/Closed loop trajectory​

import projectaria_tools.core.mps as mps

open_loop_path = "/path/to/mps/output/trajectory/open_loop_trajectory.csv"
open_loop_traj = mps.read_open_loop_trajectory(open_loop_path)

closed_loop_path = "/path/to/mps/output/trajectory/closed_loop_trajectory.csv"
closed_loop_traj = mps.read_closed_loop_trajectory(closed_loop_path)

# example: get transformation from this device to a world coordinate frame
for closed_loop_pose in closed_loop_traj:
    transform_world_device = closed_loop_pose.transform_world_device

# example: query to find the closest Timestamp device pose and move it to the Aria RGB camera pose
from projectaria_tools.core import data_provider
from projectaria_tools.core.mps.utils import get_nearest_pose
from projectaria_tools.core.stream_id import StreamId

query_timestamp_ns = int(closed_loop_traj[1].tracking_timestamp.total_seconds() * 1e9) # to be updated with your VRS timestamps

pose_info = get_nearest_pose(closed_loop_traj, query_timestamp_ns)
if pose_info:
    T_world_device = pose_info.transform_world_device

    # Move this pose to the Project Aria RGB camera
    vrs_file = "example.vrs"
    vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file)
    rgb_stream_id = StreamId("214-1")
    rgb_stream_label = vrs_data_provider.get_label_from_stream_id(rgb_stream_id)
    device_calibration = vrs_data_provider.get_device_calibration()
    rgb_camera_calibration = device_calibration.get_camera_calib(rgb_stream_label)

    T_device_rgb_camera = rgb_camera_calibration.get_transform_device_camera()
    T_world_rgb_camera = T_world_device @ T_device_rgb_camera

    print(T_world_rgb_camera)

Point cloud​

Always filter global point clouds in 3D

Post-filtering the point cloud using inverse distance and distance certainty is required to get point cloud accurate in 3D space. There are points cannot be accurately estimated in 3D space due to low parallax, but those points are well tracked in 2D images, and produce valid 2D observations. We choose to output all the points, include those have poor 3D estimations, in case researchers need them. Go to the Semi-Dense Point Cloud page for more information.

Loading observations could be slow

When the Aria recording is long, loading point observations could be memory and time consuming (> 1 minute). A typical 20 minutes long Aria recording will have roughly total 10+ millions of 3D points with total 100+ millions of 2D observations.

import projectaria_tools.core.mps as mps
from projectaria_tools.core.mps.utils import filter_points_from_confidence

global_points_path = "/path/to/mps/output/trajectory/semidense_points.csv.gz"
points = mps.read_global_point_cloud(global_points_path)

# filter the point cloud using thresholds on the inverse depth and distance standard deviation
inverse_distance_std_threshold = 0.001
distance_std_threshold = 0.15

filtered_points = filter_points_from_confidence(points, inverse_distance_std_threshold, distance_std_threshold)

# example: get position of this point in the world coordinate frame
for point in filtered_points:
    position_world = point.position_world

observations_path = "/path/to/mps/output/trajectory/semidense_observations.csv.gz"
observations = mps.read_point_observations(observations_path)

Online calibration​

import projectaria_tools.core.mps as mps

online_calib_path = "/path/to/mps/output/trajectory/online_calibration.jsonl"
online_calibs = mps.read_online_calibration(online_calib_path)

for calib in online_calibs:
    # example: get left IMU's online calibration
    for imuCalib in calib.imu_calibs:
        if imuCalib.get_label() == "imu-left":
            leftImuCalib = imuCalib
    # example: get left SLAM camera's online calibration
    for camCalib in calib.camera_calibs:
        if camCalib.get_label() == "camera-slam-left":
            leftCamCalib = camCalib

Hand Tracking​

import projectaria_tools.core.mps as mps
from projectaria_tools.core.mps.utils import get_nearest_wrist_and_palm_pose


## Load wrist and palm pose data
wrist_and_palm_poses_path = "/path/to/mps/output/hand_tracking/wrist_and_palm_poses.csv"
wrist_and_palm_poses = mps.hand_tracking.read_wrist_and_palm_poses(
   wrist_and_palm_poses_path
)


# Example query: find the nearest hand tracking data outputs in relation to a specific timestamp
wrist_and_palm_pose = get_nearest_wrist_and_palm_pose(
   wrist_and_palm_poses, query_timestamp_ns
)


# Get left- and right-hand confidences and wrist and palm positions in the device frame
# Left-hand side
left_pose_confidence = wrist_and_palm_pose.left_hand.confidence
left_wrist_position_device = wrist_and_palm_pose.left_hand.wrist_position_device
left_palm_position_device = wrist_and_palm_pose.left_hand.palm_position_device
# Right-hand side
right_pose_confidence = wrist_and_palm_pose.right_hand.confidence
right_wrist_position_device = wrist_and_palm_pose.right_hand.wrist_position_device
right_palm_position_device = wrist_and_palm_pose.right_hand.palm_position_device