MPS Code Snippets

Project Aria Machine Perception Services (MPS) enables Aria users with access to the Aria Research Kit to request derived data on Aria VRS files.

Open Datasets also contain MPS outputs and may have specific tools to use and visualize their data.

This page covers how to consume standard MPS outputs.

Load MPS output

The loaders for MPS outputs (projectaria_tools/main/core/mps) make it easer to use the data downstream. As part of this, the loaders put the outputs into data structures that are easier for other tools to consume.

MPS Data Formats provides details about output schemas and the specifics of each MPS output. This page focuses loading APIs in Python and C++, where there isn't a standalone code samples page:

Eye Gaze Code Samples

Open loop/Closed loop trajectory

Python
C++

import projectaria_tools.core.mps as mps

open_loop_path = "/path/to/mps/output/trajectory/open_loop_trajectory.csv"
open_loop_traj = mps.read_open_loop_trajectory(open_loop_path)

closed_loop_path = "/path/to/mps/output/trajectory/closed_loop_trajectory.csv"
closed_loop_traj = mps.read_closed_loop_trajectory(closed_loop_path)

# example: get transformation from this device to a world coordinate frame
for closed_loop_pose in closed_loop_traj:
    transform_world_device = closed_loop_pose.transform_world_device

# example: query to find the closest Timestamp device pose and move it to the Aria RGB camera pose
from projectaria_tools.core import data_provider
from projectaria_tools.core.mps.utils import get_nearest_pose
from projectaria_tools.core.stream_id import StreamId

query_timestamp_ns = int(closed_loop_traj[1].tracking_timestamp.total_seconds() * 1e9) # to be updated with your VRS timestamps

pose_info = get_nearest_pose(closed_loop_traj, query_timestamp_ns)
if pose_info:
    T_world_device = pose_info.transform_world_device

    # Move this pose to the Project Aria RGB camera
    vrs_file = "example.vrs"
    vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file)
    rgb_stream_id = StreamId("214-1")
    rgb_stream_label = vrs_data_provider.get_label_from_stream_id(rgb_stream_id)
    device_calibration = vrs_data_provider.get_device_calibration()
    rgb_camera_calibration = device_calibration.get_camera_calib(rgb_stream_label)

    T_device_rgb_camera = rgb_camera_calibration.get_transform_device_camera()
    T_world_rgb_camera = T_world_device @ T_device_rgb_camera

    print(T_world_rgb_camera)

#include <TrajectoryReaders.h>
using namespace projectaria::tools::mps;

std::string openLoopTrajPath = "/path/to/mps/output/trajectory/open_loop_trajectory.csv";
OpenLoopTrajectory openLoopTraj = readOpenLoopTrajectory(openLoopTrajPath);

std::string closedLoopTrajPath = "/path/to/mps/output/trajectory/closed_loop_trajectory.csv";
ClosedLoopTrajectory closedLoopTraj = readClosedLoopTrajectory(closedLoopTrajPath);

// example: get transformation from this device to world coordinate frame
for (const ClosedLoopTrajectoryPose& closedLoopPose : closedLoopTraj) {
    const Sophus::SE3d& T_world_device = closedLoopPose.T_world_device;
}

Point cloud

Always filter global point clouds in 3D

Post-filtering the point cloud using inverse distance and distance certainty is required to get point cloud accurate in 3D space. There are points cannot be accurately estimated in 3D space due to low parallax, but those points are well tracked in 2D images, and produce valid 2D observations. We choose to output all the points, include those have poor 3D estimations, in case researchers need them. Go to the Semi-Dense Point Cloud page for more information.

Loading observations could be slow

When the Aria recording is long, loading point observations could be memory and time consuming (> 1 minute). A typical 20 minutes long Aria recording will have roughly total 10+ millions of 3D points with total 100+ millions of 2D observations.

Python
C++

import projectaria_tools.core.mps as mps
from projectaria_tools.core.mps.utils import filter_points_from_confidence

global_points_path = "/path/to/mps/output/trajectory/semidense_points.csv.gz"
points = mps.read_global_point_cloud(global_points_path)

# filter the point cloud using thresholds on the inverse depth and distance standard deviation
inverse_distance_std_threshold = 0.001
distance_std_threshold = 0.15

filtered_points = filter_points_from_confidence(points, inverse_distance_std_threshold, distance_std_threshold)

# example: get position of this point in the world coordinate frame
for point in filtered_points:
    position_world = point.position_world

observations_path = "/path/to/mps/output/trajectory/semidense_observations.csv.gz"
observations = mps.read_point_observations(observations_path)

#include <GlobalPointCloudFilter.h>
#include <GlobalPointCloudReader.h>
#include <PointObservationReader.h>

using namespace projectaria::tools::mps;

std::string globalPointsPath = "/path/to/mps/output/trajectory/open_loop_trajectory.csv";
GlobalPointCloud points = readGlobalPointCloud(globalPointsPath);

// filter the point cloud by inverse depth and depth
const float inverseDistanceStdThreshold = 0.001;
const float distanceStdThreshold = 0.15;
GlobalPointCloud filteredPoints = filterPointsFromConfidence(points, inverseDistanceStdThreshold, distanceStdThreshold);

// example: get the position of this point in the world coordinate frame
for (const GlobalPointPosition& point : filteredPoints) {
    const Eigen::Vector3d& position_world = point.position_world;
}

std::string observationsPath = "/path/to/mps/output/trajectory/semidense_observations.csv.gz";
PointObservations observations = readPointObservations(observationsPath);

Online calibration

Python
C++

import projectaria_tools.core.mps as mps

online_calib_path = "/path/to/mps/output/trajectory/online_calibration.jsonl"
online_calibs = mps.read_online_calibration(online_calib_path)

for calib in online_calibs:
    # example: get left IMU's online calibration
    for imuCalib in calib.imu_calibs:
        if imuCalib.get_label() == "imu-left":
            leftImuCalib = imuCalib
    # example: get left SLAM camera's online calibration
    for camCalib in calib.camera_calibs:
        if camCalib.get_label() == "camera-slam-left":
            leftCamCalib = camCalib

#include <OnlineCalibrationsReader.h>
using namespace projectaria::tools::calibration;
using namespace projectaria::tools::mps;

std::string onlineCalibPath = "/path/to/mps/output/trajectory/online_calibration.jsonl";
OnlineCalibrations onlineCalibs = readOnlineCalibration(onlineCalibPath);

for (const OnlineCalibration& calib : onlineCalibs) {
    // example: get left IMU's online calibration
    for (const ImuCalibration& imuCalib : calib.imuCalibs) {
        if (imuCalib.getLabel() == "imu-left") {
            const ImuCalibration& leftImuCalib = imuCalib;
        }
    }
    // example: get left SLAM camera's online calibration
    for (const CameraCalibration& camCalib : calib.cameraCalibs) {
        if (camCalib.getLabel() == "camera-slam-left") {
            const CameraCalibration& leftCamCalib = camCalib;
        }
    }
}

Hand Tracking

Python
C++

import projectaria_tools.core.mps as mps
from projectaria_tools.core.mps.utils import get_nearest_wrist_and_palm_pose


## Load wrist and palm pose data
wrist_and_palm_poses_path = "/path/to/mps/output/hand_tracking/wrist_and_palm_poses.csv"
wrist_and_palm_poses = mps.hand_tracking.read_wrist_and_palm_poses(
   wrist_and_palm_poses_path
)


# Example query: find the nearest hand tracking data outputs in relation to a specific timestamp
wrist_and_palm_pose = get_nearest_wrist_and_palm_pose(
   wrist_and_palm_poses, query_timestamp_ns
)


# Get left and right-hand confidences and wrist and palm positions and normals in the device frame
# Left-hand side
left_pose_confidence = wrist_and_palm_pose.left_hand.confidence
left_wrist_position_device = wrist_and_palm_pose.left_hand.wrist_position_device
left_palm_position_device = wrist_and_palm_pose.left_hand.palm_position_device
left_wrist_normal_device = wrist_and_palm_pose.left_hand.wrist_and_palm_normal_device.wrist_normal_device
left_palm_normal_device = wrist_and_palm_pose.left_hand.wrist_and_palm_normal_device.palm_normal_device
# Right-hand side
right_pose_confidence = wrist_and_palm_pose.right_hand.confidence
right_wrist_position_device = wrist_and_palm_pose.right_hand.wrist_position_device
right_palm_position_device = wrist_and_palm_pose.right_hand.palm_position_device
right_wrist_normal_device = wrist_and_palm_pose.right_hand.wrist_and_palm_normal_device.wrist_normal_device
right_palm_normal_device = wrist_and_palm_pose.right_hand.wrist_and_palm_normal_device.palm_normal_device

#include <mps/HandTrackingReader.h>

using namespace projectaria::tools::mps;

const auto wristAndPalmPoses = readWristAndPalmPoses("/path/to/mps/output/hand_tracking/wrist_and_palm_poses.csv");

for (const auto& wristAndPalmPose: wristAndPalmPoses) {
  for (HANDEDNESS handedness : {HANDEDNESS::LEFT, HANDEDNESS::RIGHT}) {
    const auto& oneHandWristAndPalmPose = wristAndPalmPose.value()[handedness];
    if (!oneHandWristAndPalmPose ||
        oneHandWristAndPalmPose.value().confidence < MIN_CONFIDENCE_) {
      continue;
    }
    const auto& wristPose = oneHandWristAndPalmPose.value().wristPosition_device;
    const auto& palmPose = oneHandWristAndPalmPose.value().palmPosition_device;
    const auto& wristPose_world = T_World_Device.value() * wristPose;
    const auto& palmPose_world = T_World_Device.value() * palmPose;

    if (oneHandWristAndPalmPose.wristAndPalmNormal_device.has_value()) {
        const auto& wristNormal_device = oneSideWristAndPalmPose.wristAndPalmNormal_device.value().wristNormal_device;
        const auto& palmNormal_device = oneSideWristAndPalmPose.wristAndPalmNormal_device.value().palmNormal_device;
        const auto wristNormal_world = T_World_Device.value().so3() * wristNormal_device;
        const auto palmNormal_world = T_World_Device.value().so3() * palmNormal_device;
    }
  }
}

MPS Code Snippets

Load MPS output​

Open loop/Closed loop trajectory​

Point cloud​

Online calibration​

Hand Tracking​

Load MPS output

Open loop/Closed loop trajectory

Point cloud

Online calibration

Hand Tracking