Skip to main content

Eye Gaze Code Snippets

This page provides a number of Eye Gaze code snippets that can be useful when working with Eye Gaze data provided in Open Datasets or generated by Machine Perception Services.

Further resources​

General​

Load Eye Gaze in Python and C++​

Data loaders for MPS outputs are provided as part of Project Aria Tools (projectaria_tools/main/core/mps). As part of this, the loaders put the outputs into data structures that are easier for other tools to consume.

In this example, we set the default eye gaze depth to 1 meter. We use 1m because it is close to what people could reach with their arms and look at if they were grabbing objects. You can also use .depth to define depth_m when using data generated using the new eye gaze model.

import projectaria_tools.core.mps as mps

gaze_path = "/path/to/mps/output/eye_gaze/general_eye_gaze.csv"
gaze_cpf = mps.read_eyegaze(eye_gaze_path)

# Set default eye gaze depth for 3D points to 1 meter
depth_m = 1.0
gaze_point_cpf = mps.get_eyegaze_point_at_depth(gaze_cpf[1].yaw, gaze_cpf[1].pitch, depth_m)

# Example query: find the nearest eye gaze data outputs in relation to a specific timestamp
from projectaria_tools.core import data_provider
from projectaria_tools.core.stream_id import StreamId
from projectaria_tools.core.mps.utils import (
    get_gaze_vector_reprojection,
    get_nearest_eye_gaze
)

# Query Eye Gaze data at a desired timestamp
# For this example we use an eyegaze data timestamp
# You can also use a VRS timestamp (i.e timestamp from a loop reading all the images)

query_timestamp_ns = int(gaze_cpf[1].tracking_timestamp.total_seconds() * 1e9)

eye_gaze_info = get_nearest_eye_gaze(gaze_cpf, query_timestamp_ns)

if eye_gaze_info:
    # Re-project the eye gaze point onto the RGB camera data
    vrs_file = "example.vrs"
    vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file)

    rgb_stream_id = StreamId("214-1")
    rgb_stream_label = vrs_data_provider.get_label_from_stream_id(rgb_stream_id)
    device_calibration = vrs_data_provider.get_device_calibration()
    rgb_camera_calibration = device_calibration.get_camera_calib(rgb_stream_label)

    gaze_projection = get_gaze_vector_reprojection(
    eye_gaze_info,
    rgb_stream_label,
    device_calibration,
    rgb_camera_calibration,
    depth_m,
)
    print(gaze_projection)

Yaw/Pitch to 3D vector conversion​​

Convert the gaze angles into 3D vectors. To convert a gaze measurement (yaw/pitch) into a 3D gaze vector originating at the origin of CPF (or 3D gaze point) use the Eigen::Vector3d getEyeGazePointAtDepth operation in EyeGazeReader.h. This function can be used with the new model output and old model output (go to Eye Gaze Data Format for more information about Eye Gaze changes).

Get Central Pupil Frame(CPF) to Device Transforms (Python)​

vrs_file = "example.vrs"
vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file)
rgb_stream_id = StreamId("214-1")
rgb_stream_label = vrs_data_provider.get_label_from_stream_id(rgb_stream_id)
device_calibration = vrs_data_provider.get_device_calibration()
rgb_camera_calibration = device_calibration.get_camera_calib(rgb_stream_label)
# transform_cpf_sensor is the transform of sensor to cpf frame
transform_cpf_sensor = device_calibration.get_transform_cpf_sensor(stream_id_label)

Helper functions to support the new Eye Gaze model​

The following commands can be used to generate yaw_rads_cpf values.

Get combined gaze direction and depth from left and right gaze direction angles​

gaze_path = "/path/to/mps/output/eye_gaze/general_eye_gaze.csv"
gaze_cpf = mps.read_eyegaze(eye_gaze_path)
depth, combined_yaw, combined_pitch = (
    mps.compute_depth_and_combined_gaze_direction(
        gaze_cpf[1].vergence.left_yaw, gaze_cpf[1].vergence.right_yaw, gaze_cpf[1].pitch
    )
)

Get gaze intersection in 3D coordinates in CPF frame from left and right gaze direction angles​

gaze_x, gaze_y, gaze_z = mps.get_gaze_intersection_point(
    gaze_cpf[1].vergence.left_yaw,
    gaze_cpf[1].vergence.right_yaw,
    gaze_cpf[1].pitch
)

Get left and right gaze directions in CPF frame​

Get left_gaze and right_gaze direction vectors (numpy arrays) given left and right yaws and common pitch. The vectors are from left and right eye positions to the point of intersection. Currently left eye and right eye origins are [0.0315, 0, 0] and [-0.0315, 0, 0] in CPF coordinates respectively.

left_gaze, right_gaze = mps.get_gaze_vectors(
    gaze_cpf[1].vergence.left_yaw,
    gaze_cpf[1].vergence.right_yaw,
    gaze_cpf[1].pitch
)