MPS Output - Hand Tracking
Project Aria's Machine Perception Services (MPS) uses Project Aria's SLAM (mono scene) camera images to estimate the hand movement of the wearer.
Hand Tracking outputs are available for all recordings made with SLAM cameras and can be requested using the MPS CLI or Desktop Companion app. When research partners submit data for processing, the data is only used to serve MPS requests. Partner data is not made available to Meta researchers or Meta’s affiliates. Go to MPS Data Lifecycle for more details about how partner data is processed and stored.
Go to MPS Code Snippets for how to:
- Load the hand tracking data (21 landmarks, wrist to device transformation, wrist and palm positions and normals)
- Transform them to the world coordinate frame
- Project them onto the SLAM or RGB images
Hand Tracking data can be visualized in Python or C++.
Hand Tracking Outputs
We currently provide hand tracking data. The file outputs are:
hand_tracking_results.csv- the coordinates of 21 hand landmark positions, wrist and palm normals in the device frame, and the full 6DoF transform from the hand frame (origin at wrist position) to the device framewrist_and_palm_poses.csv- the coordinates of the wrist and palm positions, and normal vectors indicating orientation, in the device framesummary.json- high-level report on MPS hand tracking
hand_tracking_results.csv
hand_tracking_results.csv contains the following fields:
The landmark positions, and wrist and palm normal vectors are given in the device frame in meters.
The translation from the hand frame to the device frame is given in meters.
The 21 landmarks are indexed from 0 to 20, representing key points on the hand. These landmarks correspond to specific joints and locations as follows:
| Landmark ID | Description |
|---|---|
| 0 | Thumb Fingertip |
| 1 | Index Finger Fingertip |
| 2 | Middle Finger Fingertip |
| 3 | Ring Finger Fingertip |
| 4 | Pinky Finger Fingertip |
| 5 | Wrist Joint |
| 6 | Thumb Intermediate |
| 7 | Thumb Distal |
| 8 | Index Finger Proximal |
| 9 | Index Finger Intermediate |
| 10 | Index Finger Distal |
| 11 | Middle Finger Proximal |
| 12 | Middle Finger Intermediate |
| 13 | Middle Finger Distal |
| 14 | Ring Finger Proximal |
| 15 | Ring Finger Intermediate |
| 16 | Ring Finger Distal |
| 17 | Pinky Finger Proximal |
| 18 | Pinky Finger Intermediate |
| 19 | Pinky Finger Distal |
| 20 | Palm Center |
| Column | Type | Description |
|---|---|---|
| tracking_timestamp_us | int | Timestamp, in microseconds, of the SLAM camera frame in device time domain. This is the same time domain in which the MPS trajectory outputs are reported, so these timestamps can be directly used to infer the device pose from the MPS trajectory output. |
| left_tracking_confidence | float | A value between 0.0 and 1.0 indicating the confidence in the detected left hand. A value of -1.0 indicates that the left hand data is missing for the frame, and the coordinates *_left_landmark_i_device, _left_device_wrist, _left_wrist_device and _left_palm_device should not be used. |
| tx_left_landmark_0_device | float | X-coordinate of the left landmark 0 position given in the device frame in meters. |
| ty_left_landmark_0_device | float | Y-coordinate of the left landmark 0 position given in the device frame in meters. |
| tz_left_landmark_0_device | float | Z-coordinate of the left landmark 0 position given in the device frame in meters. |
| tx_left_landmark_1_device | float | X-coordinate of the left landmark 1 position given in the device frame in meters. |
| ty_left_landmark_1_device | float | Y-coordinate of the left landmark 1 position given in the device frame in meters. |
| tz_left_landmark_1_device | float | Z-coordinate of the left landmark 1 position given in the device frame in meters. |
| tx_left_landmark_2_device | float | X-coordinate of the left landmark 2 position given in the device frame in meters. |
| ty_left_landmark_2_device | float | Y-coordinate of the left landmark 2 position given in the device frame in meters. |
| tz_left_landmark_2_device | float | Z-coordinate of the left landmark 2 position given in the device frame in meters. |
| tx_left_landmark_3_device | float | X-coordinate of the left landmark 3 position given in the device frame in meters. |
| ty_left_landmark_3_device | float | Y-coordinate of the left landmark 3 position given in the device frame in meters. |
| tz_left_landmark_3_device | float | Z-coordinate of the left landmark 3 position given in the device frame in meters. |
| tx_left_landmark_4_device | float | X-coordinate of the left landmark 4 position given in the device frame in meters. |
| ty_left_landmark_4_device | float | Y-coordinate of the left landmark 4 position given in the device frame in meters. |
| tz_left_landmark_4_device | float | Z-coordinate of the left landmark 4 position given in the device frame in meters. |
| tx_left_landmark_5_device | float | X-coordinate of the left landmark 5 position given in the device frame in meters. |
| ty_left_landmark_5_device | float | Y-coordinate of the left landmark 5 position given in the device frame in meters. |
| tz_left_landmark_5_device | float | Z-coordinate of the left landmark 5 position given in the device frame in meters. |
| tx_left_landmark_6_device | float | X-coordinate of the left landmark 6 position given in the device frame in meters. |
| ty_left_landmark_6_device | float | Y-coordinate of the left landmark 6 position given in the device frame in meters. |
| tz_left_landmark_6_device | float | Z-coordinate of the left landmark 6 position given in the device frame in meters. |
| tx_left_landmark_7_device | float | X-coordinate of the left landmark 7 position given in the device frame in meters. |
| ty_left_landmark_7_device | float | Y-coordinate of the left landmark 7 position given in the device frame in meters. |
| tz_left_landmark_7_device | float | Z-coordinate of the left landmark 7 position given in the device frame in meters. |
| tx_left_landmark_8_device | float | X-coordinate of the left landmark 8 position given in the device frame in meters. |
| ty_left_landmark_8_device | float | Y-coordinate of the left landmark 8 position given in the device frame in meters. |
| tz_left_landmark_8_device | float | Z-coordinate of the left landmark 8 position given in the device frame in meters. |
| tx_left_landmark_9_device | float | X-coordinate of the left landmark 9 position given in the device frame in meters. |
| ty_left_landmark_9_device | float | Y-coordinate of the left landmark 9 position given in the device frame in meters. |
| tz_left_landmark_9_device | float | Z-coordinate of the left landmark 9 position given in the device frame in meters. |
| tx_left_landmark_10_device | float | X-coordinate of the left landmark 10 position given in the device frame in meters. |
| ty_left_landmark_10_device | float | Y-coordinate of the left landmark 10 position given in the device frame in meters. |
| tz_left_landmark_10_device | float | Z-coordinate of the left landmark 10 position given in the device frame in meters. |
| tx_left_landmark_11_device | float | X-coordinate of the left landmark 11 position given in the device frame in meters. |
| ty_left_landmark_11_device | float | Y-coordinate of the left landmark 11 position given in the device frame in meters. |
| tz_left_landmark_11_device | float | Z-coordinate of the left landmark 11 position given in the device frame in meters. |
| tx_left_landmark_12_device | float | X-coordinate of the left landmark 12 position given in the device frame in meters. |
| ty_left_landmark_12_device | float | Y-coordinate of the left landmark 12 position given in the device frame in meters. |
| tz_left_landmark_12_device | float | Z-coordinate of the left landmark 12 position given in the device frame in meters. |
| tx_left_landmark_13_device | float | X-coordinate of the left landmark 13 position given in the device frame in meters. |
| ty_left_landmark_13_device | float | Y-coordinate of the left landmark 13 position given in the device frame in meters. |
| tz_left_landmark_13_device | float | Z-coordinate of the left landmark 13 position given in the device frame in meters. |
| tx_left_landmark_14_device | float | X-coordinate of the left landmark 14 position given in the device frame in meters. |
| ty_left_landmark_14_device | float | Y-coordinate of the left landmark 14 position given in the device frame in meters. |
| tz_left_landmark_14_device | float | Z-coordinate of the left landmark 14 position given in the device frame in meters. |
| tx_left_landmark_15_device | float | X-coordinate of the left landmark 15 position given in the device frame in meters. |
| ty_left_landmark_15_device | float | Y-coordinate of the left landmark 15 position given in the device frame in meters. |
| tz_left_landmark_15_device | float | Z-coordinate of the left landmark 15 position given in the device frame in meters. |
| tx_left_landmark_16_device | float | X-coordinate of the left landmark 16 position given in the device frame in meters. |
| ty_left_landmark_16_device | float | Y-coordinate of the left landmark 16 position given in the device frame in meters. |
| tz_left_landmark_16_device | float | Z-coordinate of the left landmark 16 position given in the device frame in meters. |
| tx_left_landmark_17_device | float | X-coordinate of the left landmark 17 position given in the device frame in meters. |
| ty_left_landmark_17_device | float | Y-coordinate of the left landmark 17 position given in the device frame in meters. |
| tz_left_landmark_17_device | float | Z-coordinate of the left landmark 17 position given in the device frame in meters. |
| tx_left_landmark_18_device | float | X-coordinate of the left landmark 18 position given in the device frame in meters. |
| ty_left_landmark_18_device | float | Y-coordinate of the left landmark 18 position given in the device frame in meters. |
| tz_left_landmark_18_device | float | Z-coordinate of the left landmark 18 position given in the device frame in meters. |
| tx_left_landmark_19_device | float | X-coordinate of the left landmark 19 position given in the device frame in meters. |
| ty_left_landmark_19_device | float | Y-coordinate of the left landmark 19 position given in the device frame in meters. |
| tz_left_landmark_19_device | float | Z-coordinate of the left landmark 19 position given in the device frame in meters. |
| tx_left_landmark_20_device | float | X-coordinate of the left landmark 20 position given in the device frame in meters. |
| ty_left_landmark_20_device | float | Y-coordinate of the left landmark 20 position given in the device frame in meters. |
| tz_left_landmark_20_device | float | Z-coordinate of the left landmark 20 position given in the device frame in meters. |
| right_tracking_confidence | float | A value between 0.0 and 1.0 indicating the confidence in the detected right hand. A value of -1.0 indicates that the right hand data is missing for the frame, and the coordinates *_right_landmark_i_device, _right_device_wrist, _right_wrist_device and _right_palm_device should not be used. |
| tx_right_landmark_0_device | float | X-coordinate of the right landmark 0 position given in the device frame in meters. |
| ty_right_landmark_0_device | float | Y-coordinate of the right landmark 0 position given in the device frame in meters. |
| tz_right_landmark_0_device | float | Z-coordinate of the right landmark 0 position given in the device frame in meters. |
| tx_right_landmark_1_device | float | X-coordinate of the right landmark 1 position given in the device frame in meters. |
| ty_right_landmark_1_device | float | Y-coordinate of the right landmark 1 position given in the device frame in meters. |
| tz_right_landmark_1_device | float | Z-coordinate of the right landmark 1 position given in the device frame in meters. |
| tx_right_landmark_2_device | float | X-coordinate of the right landmark 2 position given in the device frame in meters. |
| ty_right_landmark_2_device | float | Y-coordinate of the right landmark 2 position given in the device frame in meters. |
| tz_right_landmark_2_device | float | Z-coordinate of the right landmark 2 position given in the device frame in meters. |
| tx_right_landmark_3_device | float | X-coordinate of the right landmark 3 position given in the device frame in meters. |
| ty_right_landmark_3_device | float | Y-coordinate of the right landmark 3 position given in the device frame in meters. |
| tz_right_landmark_3_device | float | Z-coordinate of the right landmark 3 position given in the device frame in meters. |
| tx_right_landmark_4_device | float | X-coordinate of the right landmark 4 position given in the device frame in meters. |
| ty_right_landmark_4_device | float | Y-coordinate of the right landmark 4 position given in the device frame in meters. |
| tz_right_landmark_4_device | float | Z-coordinate of the right landmark 4 position given in the device frame in meters. |
| tx_right_landmark_5_device | float | X-coordinate of the right landmark 5 position given in the device frame in meters. |
| ty_right_landmark_5_device | float | Y-coordinate of the right landmark 5 position given in the device frame in meters. |
| tz_right_landmark_5_device | float | Z-coordinate of the right landmark 5 position given in the device frame in meters. |
| tx_right_landmark_6_device | float | X-coordinate of the right landmark 6 position given in the device frame in meters. |
| ty_right_landmark_6_device | float | Y-coordinate of the right landmark 6 position given in the device frame in meters. |
| tz_right_landmark_6_device | float | Z-coordinate of the right landmark 6 position given in the device frame in meters. |
| tx_right_landmark_7_device | float | X-coordinate of the right landmark 7 position given in the device frame in meters. |
| ty_right_landmark_7_device | float | Y-coordinate of the right landmark 7 position given in the device frame in meters. |
| tz_right_landmark_7_device | float | Z-coordinate of the right landmark 7 position given in the device frame in meters. |
| tx_right_landmark_8_device | float | X-coordinate of the right landmark 8 position given in the device frame in meters. |
| ty_right_landmark_8_device | float | Y-coordinate of the right landmark 8 position given in the device frame in meters. |
| tz_right_landmark_8_device | float | Z-coordinate of the right landmark 8 position given in the device frame in meters. |
| tx_right_landmark_9_device | float | X-coordinate of the right landmark 9 position given in the device frame in meters. |
| ty_right_landmark_9_device | float | Y-coordinate of the right landmark 9 position given in the device frame in meters. |
| tz_right_landmark_9_device | float | Z-coordinate of the right landmark 9 position given in the device frame in meters. |
| tx_right_landmark_10_device | float | X-coordinate of the right landmark 10 position given in the device frame in meters. |
| ty_right_landmark_10_device | float | Y-coordinate of the right landmark 10 position given in the device frame in meters. |
| tz_right_landmark_10_device | float | Z-coordinate of the right landmark 10 position given in the device frame in meters. |
| tx_right_landmark_11_device | float | X-coordinate of the right landmark 11 position given in the device frame in meters. |
| ty_right_landmark_11_device | float | Y-coordinate of the right landmark 11 position given in the device frame in meters. |
| tz_right_landmark_11_device | float | Z-coordinate of the right landmark 11 position given in the device frame in meters. |
| tx_right_landmark_12_device | float | X-coordinate of the right landmark 12 position given in the device frame in meters. |
| ty_right_landmark_12_device | float | Y-coordinate of the right landmark 12 position given in the device frame in meters. |
| tz_right_landmark_12_device | float | Z-coordinate of the right landmark 12 position given in the device frame in meters. |
| tx_right_landmark_13_device | float | X-coordinate of the right landmark 13 position given in the device frame in meters. |
| ty_right_landmark_13_device | float | Y-coordinate of the right landmark 13 position given in the device frame in meters. |
| tz_right_landmark_13_device | float | Z-coordinate of the right landmark 13 position given in the device frame in meters. |
| tx_right_landmark_14_device | float | X-coordinate of the right landmark 14 position given in the device frame in meters. |
| ty_right_landmark_14_device | float | Y-coordinate of the right landmark 14 position given in the device frame in meters. |
| tz_right_landmark_14_device | float | Z-coordinate of the right landmark 14 position given in the device frame in meters. |
| tx_right_landmark_15_device | float | X-coordinate of the right landmark 15 position given in the device frame in meters. |
| ty_right_landmark_15_device | float | Y-coordinate of the right landmark 15 position given in the device frame in meters. |
| tz_right_landmark_15_device | float | Z-coordinate of the right landmark 15 position given in the device frame in meters. |
| tx_right_landmark_16_device | float | X-coordinate of the right landmark 16 position given in the device frame in meters. |
| ty_right_landmark_16_device | float | Y-coordinate of the right landmark 16 position given in the device frame in meters. |
| tz_right_landmark_16_device | float | Z-coordinate of the right landmark 16 position given in the device frame in meters. |
| tx_right_landmark_17_device | float | X-coordinate of the right landmark 17 position given in the device frame in meters. |
| ty_right_landmark_17_device | float | Y-coordinate of the right landmark 17 position given in the device frame in meters. |
| tz_right_landmark_17_device | float | Z-coordinate of the right landmark 17 position given in the device frame in meters. |
| tx_right_landmark_18_device | float | X-coordinate of the right landmark 18 position given in the device frame in meters. |
| ty_right_landmark_18_device | float | Y-coordinate of the right landmark 18 position given in the device frame in meters. |
| tz_right_landmark_18_device | float | Z-coordinate of the right landmark 18 position given in the device frame in meters. |
| tx_right_landmark_19_device | float | X-coordinate of the right landmark 19 position given in the device frame in meters. |
| ty_right_landmark_19_device | float | Y-coordinate of the right landmark 19 position given in the device frame in meters. |
| tz_right_landmark_19_device | float | Z-coordinate of the right landmark 19 position given in the device frame in meters. |
| tx_right_landmark_20_device | float | X-coordinate of the right landmark 20 position given in the device frame in meters. |
| ty_right_landmark_20_device | float | Y-coordinate of the right landmark 20 position given in the device frame in meters. |
| tz_right_landmark_20_device | float | Z-coordinate of the left landmark 20 position given in the device frame in meters. |
| tx_left_device_wrist | float | X-coordinate of the translation from the left hand frame (origin at wrist position) to the device frame in meters. |
| ty_left_device_wrist | float | Y-coordinate of the translation from the left hand frame (origin at wrist position) to the device frame in meters. |
| tz_left_device_wrist | float | Z-coordinate of the translation from the left hand frame (origin at wrist position) to the device frame in meters. |
| qx_left_device_wrist | float | X-component of the quaternion representing the rotation from the left hand frame (origin at wrist position) to the device frame. |
| qy_left_device_wrist | float | Y-component of the quaternion representing the rotation from the left hand frame (origin at wrist position) to the device frame. |
| qz_left_device_wrist | float | Z-component of the quaternion representing the rotation from the left hand frame (origin at wrist position) to the device frame. |
| qw_left_device_wrist | float | W-component of the quaternion representing the rotation from the left hand frame (origin at wrist position) to the device frame. |
| tx_right_device_wrist | float | X-coordinate of the translation from the right hand frame (origin at wrist position) to the device frame in meters. |
| ty_right_device_wrist | float | Y-coordinate of the translation from the right hand frame (origin at wrist position) to the device frame in meters. |
| tz_right_device_wrist | float | Z-coordinate of the translation from the right hand frame (origin at wrist position) to the device frame in meters. |
| qx_right_device_wrist | float | X-component of the quaternion representing the rotation from the right hand frame (origin at wrist position) to the device frame. |
| qy_right_device_wrist | float | Y-component of the quaternion representing the rotation from the right hand frame (origin at wrist position) to the device frame. |
| qz_right_device_wrist | float | Z-component of the quaternion representing the rotation from the right hand frame (origin at wrist position) to the device frame. |
| qw_right_device_wrist | float | W-component of the quaternion representing the rotation from the right hand frame (origin at wrist position) to the device frame. |
| nx_left_palm_device | float | X-coordinate of the left palm normal given in the device frame in meters. |
| ny_left_palm_device | float | Y-coordinate of the left palm normal given in the device frame in meters. |
| nz_left_palm_device | float | Z-coordinate of the left palm normal given in the device frame in meters. |
| nx_left_wrist_device | float | X-coordinate of the left wrist normal given in the device frame in meters. |
| ny_left_wrist_device | float | Y-coordinate of the left wrist normal given in the device frame in meters. |
| nz_left_wrist_device | float | Z-coordinate of the left wrist normal given in the device frame in meters. |
| nx_right_palm_device | float | X-coordinate of the right palm normal given in the device frame in meters. |
| ny_right_palm_device | float | Y-coordinate of the right palm normal given in the device frame in meters. |
| nz_right_palm_device | float | Z-coordinate of the right palm normal given in the device frame in meters. |
| nx_right_wrist_device | float | X-coordinate of the right wrist normal given in the device frame in meters. |
| ny_right_wrist_device | float | Y-coordinate of the right wrist normal given in the device frame in meters. |
| nz_right_wrist_device | float | Z-coordinate of the right wrist normal given in the device frame in meters. |
wrist_and_palm_poses.csv
wrist_and_palm_poses.csv contains the following fields:
The wrist and palm poses and normal vectors are given in the device frame in meters.
| Column | Type | Description |
|---|---|---|
| tracking_timestamp_us | int | Timestamp, in microseconds, of the SLAM camera frame in device time domain. This is the same time domain in which the MPS trajectory outputs are reported, so these timestamps can be directly used to infer the device pose from the MPS trajectory output. |
| left_tracking_confidence | float | A value between 0.0 and 1.0 indicating the confidence in the reported left wrist and palm positions. A value of -1.0 indicates that the left wrist and palm tracking data is missing for the frame, and the coordinates left_wrist_device_* and left_palm_device_* should not be used. |
| tx_left_wrist_device | float | X-coordinate of the left wrist position given in the device frame in meters. |
| ty_left_wrist_device | float | Y-coordinate of the left wrist position given in the device frame in meters. |
| tz_left_wrist_device | float | Z-coordinate of the left wrist position given in the device frame in meters. |
| tx_left_palm_device | float | X-coordinate of the left palm position given in the device frame in meters. |
| ty_left_palm_device | float | Y-coordinate of the left palm position given in the device frame in meters. |
| tz_left_palm_device | float | Z-coordinate of the left palm position given in the device frame in meters. |
| right_tracking_confidence | float | A value between 0.0 and 1.0 indicating the confidence in the reported right wrist and palm positions. A value of -1.0 indicates that the right wrist and palm tracking data is missing for the frame, and the coordinates right_wrist_device_* and right_palm_device_* should not be used. |
| tx_right_wrist_device | float | X-coordinate of the right wrist position given in the device frame in meters. |
| ty_right_wrist_device | float | Y-coordinate of the right wrist position given in the device frame in meters. |
| tz_right_wrist_device | float | Z-coordinate of the right wrist position given in the device frame in meters. |
| tx_right_palm_device | float | X-coordinate of the right palm position given in the device frame in meters. |
| ty_right_palm_device | float | Y-coordinate of the right palm position given in the device frame in meters. |
| tz_right_palm_device | float | Z-coordinate of the right palm position given in the device frame in meters. |
| nx_left_palm_device | float | X-coordinate of the left palm normal given in the device frame in meters. |
| ny_left_palm_device | float | Y-coordinate of the left palm normal given in the device frame in meters. |
| nz_left_palm_device | float | Z-coordinate of the left palm normal given in the device frame in meters. |
| nx_left_wrist_device | float | X-coordinate of the left wrist normal given in the device frame in meters. |
| ny_left_wrist_device | float | Y-coordinate of the left wrist normal given in the device frame in meters. |
| nz_left_wrist_device | float | Z-coordinate of the left wrist normal given in the device frame in meters. |
| nx_right_palm_device | float | X-coordinate of the right palm normal given in the device frame in meters. |
| ny_right_palm_device | float | Y-coordinate of the right palm normal given in the device frame in meters. |
| nz_right_palm_device | float | Z-coordinate of the right palm normal given in the device frame in meters. |
| nx_right_wrist_device | float | X-coordinate of the right wrist normal given in the device frame in meters. |
| ny_right_wrist_device | float | Y-coordinate of the right wrist normal given in the device frame in meters. |
| nz_right_wrist_device | float | Z-coordinate of the right wrist normal given in the device frame in meters. |
summary.json
summary.json contains the Operator Summary, listed in MPS Basics, as well as the following fields:
| Field | Type | Description |
|---|---|---|
| mean_confidence | float | Average left_tracking_confidence and right_tracking_confidence value for frames with valid results |
| total_frames | int | Total number of frames |
| valid_frame_fraction | float | Fraction of frames that have reported a valid tracking result |