XR_ANDROID_hand_mesh OpenXR extension

Name String

XR_ANDROID_hand_mesh

Extension Type

Instance extension

Registered Extension Number

704

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2024-09-10

IP Status

No known IP claims.

Contributors

Nihav Jain, Google

Cairn Overturf, Google

Spencer Quin, Google

Levana Chen, Google

Overview

This extension enables hand tracking represented as a dynamic hand mesh.

This extension is intended to provide vertex and index buffers for the mesh of a personalized representation of the user's hands. It can be used for occlusion and visualization.

This extension should not be used for other hand tracking purposes.

Hand tracking data can be sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications that store or transfer hand tracking data always ask the user for active and specific acceptance to do so.

Inspect system capability

An application can inspect whether the system is capable of hand tracking meshes by chaining an XrSystemHandMeshTrackingPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties.

typedef struct XrSystemHandMeshTrackingPropertiesANDROID {
  XrStructureType    type;
  void*              next;
  XrBool32           supportsHandMeshTracking;
  XrBool32           supportsTextureUV;
  XrBool32           supportsVertexNormal;
} XrSystemHandMeshTrackingPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsHandMeshTracking is an XrBool32, indicating if the selected XrSystemId supports hand mesh tracking.
  • supportsTextureUV is an XrBool32, indicating if the selected XrSystemId supports texture UVs for the mesh vertices.
  • supportsVertexNormal is an XrBool32, indicating if the selected XrSystemId supports vertex normals for the mesh vertices.

The application should avoid using hand mesh capabilities when supportsHandMeshTracking is XR_FALSE, since this means the system does not support hand mesh tracking. In this case, xrCreateHandMeshTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED.

If supportsHandMeshTracking returns XR_TRUE, the system supports hand mesh tracking. An application should use XrHandMeshANDROID::indexCount and XrHandMeshANDROID::vertexCount to access hand mesh buffers and reuse them in their render loop when calling xrGetHandMeshANDROID every frame.

If supportsTextureUV returns XR_FALSE, the system does not support texture UVs for the mesh vertices, and therefore an application would receive XrHandMeshANDROID::textureUVs NULL when calling xrGetHandMeshANDROID.

If supportsVertexNormal returns XR_FALSE, the system does not support vertex normals for the mesh vertices, and therefore an application would receive XrHandMeshANDROID::normals NULL when calling xrGetHandMeshANDROID.

Valid Usage (Implicit)

Create a hand mesh tracker handle

XR_DEFINE_HANDLE(XrHandMeshTrackerANDROID)

The XrHandMeshTrackerANDROID handle represents a hand mesh tracker for hand mesh tracking and managing the related resources.

This handle can be used to access hand mesh buffers using other functions in this extension.

An application can create an XrHandMeshTrackerANDROID handle using xrCreateHandMeshTrackerANDROID function. 

XrResult xrCreateHandMeshTrackerANDROID(
    XrSession                                   session,
    const XrHandMeshTrackerCreateInfoANDROID*   createInfo,
    XrHandMeshTrackerANDROID*                   handMeshTracker);

Parameter Descriptions

If the system does not support hand mesh tracking, xrCreateHandMeshTrackerANDROID will return XR_ERROR_FEATURE_UNSUPPORTED.

The XrHandMeshTrackerANDROID handle owns all the resources for hand mesh tracking. After finishing with hand mesh tracking experiences, the application must destroy the handle using the xrDestroyHandMeshTrackerANDROID function.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FEATURE_UNSUPPORTED
  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_LIMIT_REACHED

The XrHandMeshTrackerCreateInfoANDROID structure describes the information to create an XrHandMeshTrackerANDROID handle. 

typedef struct XrHandMeshTrackerCreateInfoANDROID {
    XrStructureType    type;
    const void*        next;
} XrHandMeshTrackerCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

Valid Usage (Implicit)

xrDestroyHandMeshTrackerANDROID function releases the handMeshTracker and the underlying resources when finished with hand mesh tracking experiences. 

XrResult xrDestroyHandMeshTrackerANDROID(
    XrHandMeshTrackerANDROID handMeshTracker);

Parameter Descriptions

Valid Usage (Implicit)

Thread Safety

  • Access to handMeshTracker, and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID

Locate hand meshes

The application can use the xrGetHandMeshANDROID function to retrieve the hand mesh at a given timestamp. The hand mesh's vertices' position and normal are represented in the space specified by XrHandMeshGetInfoANDROID::baseSpace when calling xrGetHandMeshANDROID.

XrResult xrGetHandMeshANDROID(
    XrHandMeshTrackerANDROID                    handMeshTracker,
    const XrHandMeshGetInfoANDROID*             getInfo,
    XrHandTrackingMeshesANDROID*                handMeshes);

Parameter Descriptions

The application can use the xrGetHandMeshANDROID function to access the hand mesh buffers generated by the runtime.

The application should call xrBeginFrame at least once during the session before the first call to xrGetHandMeshANDROID.

An application should use XrHandMeshANDROID::indexCount and XrHandMeshANDROID::vertexCount to access hand mesh buffers and reuse them in their render loop when calling xrGetHandMeshANDROID every frame.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_TIME_INVALID

A XrHandMeshGetInfoANDROID describes the information required to get hand mesh data. 

typedef struct XrHandMeshGetInfoANDROID {
    XrStructureType    type;
    const void*        next;
    XrSpace            baseSpace;
    XrTime             time;
} XrHandMeshGetInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • baseSpace is an XrSpace defining the reference space in which to locate the transform for the vertices at time.
  • time is the XrTime that describes the time at which the application wishes to query the hand mesh.

Valid Usage (Implicit)

XrHandTrackingMeshesANDROID structure contains mesh data for both hands. 

typedef struct XrHandTrackingMeshesANDROID {
    XrStructureType      type;
    void*                next;
    XrHandMeshANDROID    leftHandMesh;
    XrHandMeshANDROID    rightHandMesh;
} XrHandTrackingMeshesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • leftHandMesh is the XrHandMeshANDROID for the left hand.
  • rightHandMesh is the XrHandMeshANDROID for the right hand.

Valid Usage (Implicit)

A XrHandMeshANDROID structure contains data and buffers to receive hand mesh tracking data from xrGetHandMeshANDROID function for one hand.

typedef struct XrHandMeshANDROID {
    XrBool32             isActive;
    XrTime               dynamicLastUpdateTime;
    uint32_t             indexCount;
    uint32_t             vertexCount;
    const uint32_t*      indices;
    const XrVector2f*    textureUVs;
    const XrVector3f*    positions;
    const XrVector3f*    normals;
    XrPosef              baseSpaceFromVertexSpace;
} XrHandMeshANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • isActive is an XrBool32 indicating if the current hand mesh tracker is active and the mesh data is valid.
  • dynamicLastUpdateTime is the XrTime specifying the time at which the dynamic buffers were last updated.
  • indexCount is a uint32_t serving as the number of indices of the hand mesh.
  • vertexCount is a uint32_t serving as the number of positions of the hand mesh. It can also be used for textureUVs or normals when they are supported by the system.
  • indices is an array of uint32_t representing the mesh indices for triangles in counter-clockwise winder order. The number of values pointed to is indexCount.
  • textureUVs is NULL or an array of XrVector2f representing the vertex texture coordinates. The number of values pointed to is vertexCount.
  • positions is an array of XrVector3f representing the vertex positions in baseSpaceFromVertexSpace. The number of values pointed to is vertexCount.
  • normals is an NULL or an array of XrVector3f representing the vertex normals in baseSpaceFromVertexSpace. The number of values pointed to is vertexCount.
  • baseSpaceFromVertexSpace is the vertex XrSpace located in the XrHandMeshGetInfoANDROID::baseSpace when calling xrGetHandMeshANDROID. Applications can use this to transform the coordinate space of the mesh vertices and normals during rendering.

The hand mesh is represented in triangle lists and each triangle's vertices are in counter-clockwise order when looking from outside of the hand.

When the returned isActive value is XR_FALSE, this indicates the hand is not actively tracked; for example, the hand is outside of sensor's range, the input focus is taken away from the application, or the application does not have the permissions to access hand tracking data.

When the returned isActive value is XR_TRUE, the hand tracking mesh represented in indices and positions, including textureUVs and normals if they are supported by the system, are updated to the latest data of the XrHandMeshGetInfoANDROID::time given to the xrGetHandMeshANDROID function.

The memory pointed to by hand mesh buffers returned in XrHandMeshANDROID is owned by the runtime and shared with the application. The memory is safe to access from any thread until the next call to xrBeginFrame while the XrHandMeshTrackerANDROID handle is valid.

  • The values pointed to by indices and textureUVs are not dynamic
  • The pointer and the values pointed to by positions and normals are dynamic that both may change between calls to xrBeginFrame. The application can use dynamicLastUpdateTime to check if the values have changed since the last frame and avoid unnecessary data processing when there is no changes.

Valid Usage (Implicit)

  • The XR_ANDROID_hand_mesh extension must be enabled prior to using XrHandMeshANDROID
  • indices must be a pointer to a valid uint32_t value
  • textureUVs must be a pointer to a valid XrVector2f structure
  • positions must be a pointer to a valid XrVector3f structure
  • normals must be a pointer to a valid XrVector3f structure

Example code for hand mesh tracking

Following example code demonstrates how to access hand mesh buffers for rendering.

XrInstance instance;  // Created at app startup
XrSystemId systemId;  // Received from xrGetSystem() at app startup
XrSession session;    // Created at app startup.
XrSpace appPlaySpace; // Created at app startup.

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateHandMeshTrackerANDROID xrCreateHandMeshTrackerANDROID; // previously initialized
PFN_xrDestroyHandMeshTrackerANDROID xrDestroyHandMeshTrackerANDROID; // previously initialized
PFN_xrGetHandMeshANDROID xrGetHandMeshANDROID; // previously initialized

// Inspect system capability
XrSystemHandMeshTrackingPropertiesANDROID handMeshTrackingProps = {
  .type = XR_TYPE_SYSTEM_HAND_MESH_TRACKING_PROPERTIES_ANDROID,
};
XrSystemProperties sysProps = {
  .type = XR_TYPE_SYSTEM_PROPERTIES,
  .next = &handMeshTrackingProps
};
CHK_XR(xrGetSystemProperties(instance, systemId, &sysProps));
if (!handMeshTrackingProps.supportsHandMeshTracking) {
  // hand mesh tracking is not supported.
  return;
}

XrHandMeshTrackerCreateInfoANDROID trackerCreateInfo = {
  .type = XR_TYPE_HAND_MESH_TRACKER_CREATE_INFO_ANDROID
};
XrHandMeshTrackerANDROID handMeshTracker = XR_NULL_HANDLE;
CHK_XR(xrCreateHandMeshTrackerANDROID(
    session, &trackerCreateInfo, &handMeshTracker));
// app update loop
while (true) {
    // ...
    // For every frame in frame loop
    // ...

    XrFrameState frameState;  // previously returned from xrWaitFrame
    const XrTime time = frameState.predictedDisplayTime;

    // ...
    XrHandMeshGetInfoANDROID getInfo = {
        .type = XR_TYPE_HAND_MESH_GET_INFO_ANDROID,
        .baseSpace = appPlaySpace,
        .time = time,
    };
    XrHandTrackingMeshesANDROID handMeshes = {
        .type = XR_TYPE_HAND_TRACKING_MESHES_ANDROID
    };
    CHK_XR(xrGetHandMeshANDROID(handMeshTracker, &getInfo, &handMeshes));

    if (handMeshes.leftHandMesh.isActive) {
        // access vertex/index buffers for rendering.
    }

    // ...
    // Finish frame loop
    // ...
}

CHECK_XR(xrDestroyHandMeshTracker(handMeshTracker));

New Object Types

New Enum Constants

XrObjectType enumeration is extended with:

  • XR_OBJECT_TYPE_HAND_MESH_TRACKER_ANDROID

XrStructureType enumeration is extended with:

  • XR_TYPE_SYSTEM_HAND_MESH_TRACKING_PROPERTIES_ANDROID
  • XR_TYPE_HAND_MESH_TRACKER_CREATE_INFO_ANDROID
  • XR_TYPE_HAND_MESH_GET_INFO_ANDROID
  • XR_TYPE_HAND_TRACKING_MESHES_ANDROID

New Enums

New Structures

New Functions

Issues

Version History

  • Revision 1, 2024-09-10 (Levana Chen)
    • Initial extension description

OpenXR™ and the OpenXR logo are trademarks owned by The Khronos Group Inc. and are registered as a trademark in China, the European Union, Japan and the United Kingdom.