XR_ANDROID_scene_meshing

Name String

XR_ANDROID_scene_meshing

Extension Type

Instance extension

Registered Extension Number

719

Revision

3

Ratification Status

Not ratified

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2025-05-15

IP Status

No known IP claims.

Contributors

Spencer Quin, Google
Jared Finder, Google
Antonio Fontan, Google
Cairn Overturf, Google
Nihav Jain, Google
Salar Khan, Google
Sebastian Klose, Google
Jürgen Sturm, Google
Vinny DaSilva, Google
Ricardo Campbell, Google

Overview

This extension is intended to provide meshes that approximately represent the physical objects in the user’s environment.

Applications can use this extension to visualize the scene and to let virtual objects interact with physical objects such as collisions.

The scene mesh data is generated by the runtime and shared with the application using XrSceneMeshingTrackerANDROID and XrSceneMeshSnapshotANDROID . Applications can access the values in scene submesh information through XrSceneSubmeshStateANDROID returned from xrGetAllSubmeshStatesANDROID . Applications can access the values in scene submesh data through XrSceneSubmeshDataANDROID returned from xrGetSubmeshDataANDROID .

Scene meshing data is sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications that store or transfer scene meshing data always ask the user for active and specific acceptance to do so.

Permissions

Android applications must have the android.permission.SCENE_UNDERSTANDING_FINE permission listed in their manifest. The android.permission.SCENE_UNDERSTANDING_FINE permission is considered a sensitive permission as it allows the runtime to study the user environment.

The application must request the permission at runtime to use these functions:

(protection level: dangerous)

Inspect system capability

The XrSystemSceneMeshingPropertiesANDROID structure is defined as:

typedef struct XrSystemSceneMeshingPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsSceneMeshing;
} XrSystemSceneMeshingPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsSceneMeshing is an XrBool32 , indicating if the system supports scene meshing.

An application can inspect whether the system is capable of scene meshing by chaining an XrSystemSceneMeshingPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties .

If a runtime returns XR_FALSE for supportsSceneMeshing , the system does not support scene meshing, and therefore must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateSceneMeshingTrackerANDROID . The application should avoid using scene meshing functionality when supportsSceneMeshing is XR_FALSE .

If a runtime returns XR_TRUE for supportsSceneMeshing , the system supports scene meshing.

Valid Usage (Implicit)

The XrSceneMeshSemanticLabelSetANDROID enumeration describes the semantic label sets for scene meshing. Each value in this enum represents another enum that contains the semantic labels. For example, the XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID value represents the set XrSceneMeshSemanticLabelANDROID .

typedef enum XrSceneMeshSemanticLabelSetANDROID {
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID = 0,
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID = 1,
    XR_SCENE_MESH_SEMANTIC_LABEL_SET_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshSemanticLabelSetANDROID;

The enumeration values have the following meanings:

Enum Description

XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID

This semantic label set represents absence of semantic labels.

XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID

This semantic label set represents XrSceneMeshSemanticLabelANDROID.

The xrEnumerateSupportedSemanticLabelSetsANDROID function is defined as:

XrResult xrEnumerateSupportedSemanticLabelSetsANDROID(
    XrInstance                                  instance,
    XrSystemId                                  systemId,
    uint32_t                                    supportedSemanticLabelSetsInputCapacity,
    uint32_t*                                   supportedSemanticLabelSetsOutputCount,
    XrSceneMeshSemanticLabelSetANDROID*         supportedSemanticLabelSets);

Parameter Descriptions

  • instance is an XrInstance created by the application.
  • systemId is an XrSystemId retried from xrGetSystem .
  • supportedSemanticLabelSetsInputCapacity is the capacity of supportedSemanticLabelSets , or 0 to retrieve the required capacity.
  • supportedSemanticLabelSetsOutputCount is a pointer to the count of supportedSemanticLabelSets written, or a pointer to the required capacity in the case that supportedSemanticLabelSetsInputCapacity is insufficient.
  • supportedSemanticLabelSets is a pointer to an array of XrSceneMeshSemanticLabelSetANDROID . It can be NULL if supportedSemanticLabelSetsInputCapacity is 0.
  • See Buffer Size Parameters chapter for a detailed description of retrieving the required supportedSemanticLabelSets size.

The application can enumerate semantic label sets supported by the system using xrEnumerateSupportedSemanticLabelSetsANDROID function.

Every system must at least support XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID because this enum value represents no semantic label set and can be used in the case the application does not need vertex semantics.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_SYSTEM_INVALID
  • XR_ERROR_VALIDATION_FAILURE

Create a scene meshing tracker handle

XR_DEFINE_HANDLE(XrSceneMeshingTrackerANDROID)

The XrSceneMeshingTrackerANDROID handle represents a scene meshing tracker for scene meshing and managing the related resources.

This handle can be used to create a scene mesh snapshot using xrCreateSceneMeshSnapshotANDROID in this extension.

The xrCreateSceneMeshingTrackerANDROID function is defined as:

XrResult xrCreateSceneMeshingTrackerANDROID(
    XrSession                                   session,
    const XrSceneMeshingTrackerCreateInfoANDROID* createInfo,
    XrSceneMeshingTrackerANDROID*               tracker);

Parameter Descriptions

An application can create an XrSceneMeshingTrackerANDROID handle using the xrCreateSceneMeshingTrackerANDROID function.

If the system does not support scene meshing, the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateSceneMeshingTrackerANDROID . In this case, the runtime must return XR_FALSE for XrSystemSceneMeshingPropertiesANDROID :: supportsSceneMeshing when the xrGetSystemProperties function is called, so that the application can avoid creating a scene meshing tracker.

The application must only use the semantic label sets that are supported by the system in XrSceneMeshingTrackerCreateInfoANDROID :: semanticLabelSet when creating a scene meshing tracker. If the application uses an unsupported scene mesh vertex semantic label set, the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateSceneMeshingTrackerANDROID .

The application can enumerate the list of supported semantic label sets with the xrEnumerateSupportedSemanticLabelSetsANDROID function.

The xrCreateSceneMeshingTrackerANDROID handle owns all the resources for scene meshing. Once finished with scene meshing experience, the application must destroy the handle via the xrDestroySceneMeshingTrackerANDROID function.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FEATURE_UNSUPPORTED
  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_VALIDATION_FAILURE

The XrSceneMeshingTrackerCreateInfoANDROID structure is defined as:

typedef struct XrSceneMeshingTrackerCreateInfoANDROID {
    XrStructureType                       type;
    const void*                           next;
    XrSceneMeshSemanticLabelSetANDROID    semanticLabelSet;
    XrBool32                              enableNormals;
} XrSceneMeshingTrackerCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • semanticLabelSet is an XrSceneMeshSemanticLabelSetANDROID used to specify the semantic label set to be used for the scene meshing. If this is set to XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID , the runtime will ignore the XrSceneSubmeshDataANDROID :: vertexSemantics buffer field.
  • enableNormals is an XrBool32 used to specify whether to include vertex normals for the mesh vertices in the scene mesh buffer when getting the scene mesh data.

The XrSceneMeshingTrackerCreateInfoANDROID structure describes the information to create an XrSceneMeshingTrackerANDROID handle.

Valid Usage (Implicit)

The xrDestroySceneMeshingTrackerANDROID function is defined as:

XrResult xrDestroySceneMeshingTrackerANDROID(
    XrSceneMeshingTrackerANDROID                tracker);

Parameter Descriptions

The xrDestroySceneMeshingTrackerANDROID function releases the tracker and the underlying resources when finished with scene meshing experiences.

Valid Usage (Implicit)

Thread Safety

  • Access to tracker , and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_RUNTIME_FAILURE

Create a scene mesh snapshot handle

XR_DEFINE_HANDLE(XrSceneMeshSnapshotANDROID)

The XrSceneMeshSnapshotANDROID handle represents a scene mesh snapshot. This snapshot is created by using xrCreateSceneMeshSnapshotANDROID which takes a snapshot of the scene mesh data at the time of the function call.

This handle can be used to retrieve submesh information and data using xrGetAllSubmeshStatesANDROID and xrGetSubmeshDataANDROID .

The xrCreateSceneMeshSnapshotANDROID function is defined as:

XrResult xrCreateSceneMeshSnapshotANDROID(
    XrSceneMeshingTrackerANDROID                tracker,
    const XrSceneMeshSnapshotCreateInfoANDROID* createInfo,
    XrSceneMeshSnapshotCreationResultANDROID*   outSnapshotCreationResult);

Parameter Descriptions

The application can use the xrCreateSceneMeshSnapshotANDROID function to create a scene mesh snapshot from the scene meshing tracker. This function returns an XrSceneMeshSnapshotANDROID handle along with XrSceneMeshTrackingStateANDROID wrapped in an XrSceneMeshSnapshotCreationResultANDROID structure. The handle is technically a snapshot of the scene mesh data at the time of creation. This handle can be used to query the scene mesh info and data using xrGetAllSubmeshStatesANDROID and xrGetSubmeshDataANDROID respectively.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_LIMIT_REACHED
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_POSE_INVALID
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_TIME_INVALID

The XrSceneMeshSnapshotCreateInfoANDROID structure is defined as:

typedef struct XrSceneMeshSnapshotCreateInfoANDROID {
    XrStructureType    type;
    const void*        next;
    XrSpace            baseSpace;
    XrTime             time;
    XrBoxf             boundingBox;
} XrSceneMeshSnapshotCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • baseSpace is an XrSpace within which the scene submesh pose info is represented.
  • time is the XrTime that describes the time when the scene mesh will be processed or displayed, typically XrFrameState :: predictedDisplayTime .
  • boundingBox is an XrBoxf describing the bounding box within which to acquire the scene mesh.

The XrSceneMeshSnapshotCreateInfoANDROID structure describes the information to create an XrSceneMeshSnapshotANDROID handle

Valid Usage (Implicit)

The XrSceneMeshSnapshotCreationResultANDROID structure is defined as:

typedef struct XrSceneMeshSnapshotCreationResultANDROID {
    XrStructureType                    type;
    const void*                        next;
    XrSceneMeshSnapshotANDROID         snapshot;
    XrSceneMeshTrackingStateANDROID    trackingState;
} XrSceneMeshSnapshotCreationResultANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • snapshot is an XrSceneMeshSnapshotANDROID handle created by the runtime.
  • trackingState is an XrSceneMeshTrackingStateANDROID that describes the state of the scene meshing tracker at the time of the snapshot creation.

The XrSceneMeshSnapshotCreationResultANDROID structure stores the result of a scene mesh snapshot creation returned from xrCreateSceneMeshSnapshotANDROID .

Valid Usage (Implicit)

The XrSceneMeshTrackingStateANDROID enumeration describes the tracking state for a scene meshing tracker. Each value in this enum represents a state of the scene meshing tracker.

typedef enum XrSceneMeshTrackingStateANDROID {
    XR_SCENE_MESH_TRACKING_STATE_INITIALIZING_ANDROID = 0,
    XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID = 1,
    XR_SCENE_MESH_TRACKING_STATE_WAITING_ANDROID = 2,
    XR_SCENE_MESH_TRACKING_STATE_ERROR_ANDROID = 3,
    XR_SCENE_MESH_TRACKING_STATE_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshTrackingStateANDROID;

The enumeration values have the following meanings:

Enum Description

XR_SCENE_MESH_TRACKING_STATE_INITIALIZING_ANDROID

The tracker is not yet ready to provide mesh data.

XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID

The tracker is actively tracking.

XR_SCENE_MESH_TRACKING_STATE_WAITING_ANDROID

The tracker is waiting for valid measurements to integrate since the last mesh update.

XR_SCENE_MESH_TRACKING_STATE_ERROR_ANDROID

The tracker has not received valid measurements for multiple cycles and is in an error state.

This state serves only as additional information to the application. In the absence of other sources of errors, the runtime must continue to allow the application to create snapshots and retrieve data from them regardless of the current XrSceneMeshTrackingStateANDROID of the tracker. If the state is not XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID , the snapshots may not be up to date or empty.

Acquiring scene mesh data from the scene mesh snapshot

The xrGetAllSubmeshStatesANDROID function is defined as:

XrResult xrGetAllSubmeshStatesANDROID(
    XrSceneMeshSnapshotANDROID                  snapshot,
    uint32_t                                    submeshStateCapacityInput,
    uint32_t*                                   submeshStateCountOutput,
    XrSceneSubmeshStateANDROID*                 submeshStates);

Parameter Descriptions

  • snapshot is an XrSceneMeshSnapshotANDROID handle previously created with xrCreateSceneMeshSnapshotANDROID .
  • submeshStateCapacityInput is the capacity of submeshStates , or 0 to retrieve the required capacity.
  • submeshStateCountOutput is a pointer to the count of submeshStates written, or a pointer to the required capacity in the case that submeshStateCapacityInput is insufficient.
  • submeshStates is a pointer to an array of XrSceneSubmeshStateANDROID structures to which the runtime will output the state of all submeshes in the scene mesh. It can be NULL if submeshStateCapacityInput is 0.
  • See Buffer Size Parameters chapter for a detailed description of retrieving the required submeshStates size.

The application can use the xrGetAllSubmeshStatesANDROID function to get the state of all submeshes in the scene mesh. This contains basic information information to allow the application to select submeshes for which the application wants data for.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_VALIDATION_FAILURE

The XrSceneSubmeshStateANDROID structure is defined as:

typedef struct XrSceneSubmeshStateANDROID {
    XrStructureType    type;
    void*              next;
    XrUuid             submeshId;
    XrTime             lastUpdatedTime;
    XrPosef            submeshPoseInBaseSpace;
    XrExtent3Df        bounds;
} XrSceneSubmeshStateANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • submeshId is an XrUuid identifying the submesh.
  • lastUpdatedTime is an XrTime representing the time when the submesh was last updated.
  • submeshPoseInBaseSpace is an XrPosef representing the pose of the submesh in the base space used to create the scene mesh snapshot. The pose is also the center of the submesh bounding box.
  • bounds is an XrExtent3Df describing the dimensions of the bounding box that encloses the submesh. This bounds is defined in the base space that was used to create the scene mesh handle.

The runtime makes best effort to map submeshId with the region of the physical space occupied by the submesh. The Runtimes may at any time lose this mapping and assign a new UUID to any subset of the detected submeshes. In particular runtimes are not required to reuse the same UUIDs for the same meshes in subsequent application instances.

Valid Usage (Implicit)

The xrGetSubmeshDataANDROID function is defined as:

XrResult xrGetSubmeshDataANDROID(
    XrSceneMeshSnapshotANDROID                  snapshot,
    uint32_t                                    submeshDataCount,
    XrSceneSubmeshDataANDROID*                  inoutSubmeshData);

Parameter Descriptions

The application can use the xrGetSubmeshDataANDROID function to get batches of submeshes, or all submeshes at once, by passing a list of XrSceneSubmeshDataANDROID with each XrSceneSubmeshDataANDROID :: submeshId set to the respective id of the submesh to retrieve.

The XrSceneSubmeshDataANDROID :: submeshId must be unique for every element in inoutSubmeshData and retrievable by xrGetAllSubmeshStatesANDROID for the same snapshot . If the submesh Id is invalid or more than one elements in inoutSubmeshData have the same submesh id, then the runtime must return XR_ERROR_VALIDATION_FAILURE .

submeshDataCount must be equal or less than the value returned by xrGetAllSubmeshStatesANDROID :: submeshStateCountOutput , otherwise the runtime must return XR_ERROR_VALIDATION_FAILURE .

This can be used in a 2-call idiom to query the vertex and index count for all of the selected submeshes.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_VALIDATION_FAILURE

The XrSceneSubmeshDataANDROID structure is defined as:

typedef struct XrSceneSubmeshDataANDROID {
    XrStructureType    type;
    const void*        next;
    XrUuid             submeshId;
    uint32_t           vertexCapacityInput;
    uint32_t           vertexCountOutput;
    XrVector3f*        vertexPositions;
    XrVector3f*        vertexNormals;
    uint8_t*           vertexSemantics;
    uint32_t           indexCapacityInput;
    uint32_t           indexCountOutput;
    uint32_t*          indices;
} XrSceneSubmeshDataANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • submeshId is an XrUuid identifying the submesh.
  • vertexCapacityInput is the capacity of vertexPositions , vertexNormals and vertexSemantics , or 0 to retrieve the required capacity.
  • vertexCountOutput is a pointer to the count of vertexPositions , vertexNormals , and vertexSemantics written, or a pointer to the required capacity in the case that vertexCapacityInput is insufficient.
  • vertexPositions is an array of XrVector3f to which the runtime will output the vertex positions of the submesh.
  • vertexNormals is an array of XrVector3f to which the runtime will output the vertex normals of the submesh. This field can be left NULL if the application created a tracker with normals disabled.
  • vertexSemantics is an array of uint8_t to which the runtime will output the vertex semantics of the submesh. This field can be left NULL if the application created a tracker with XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID semantic label set.
  • indexCapacityInput is the capacity of indices , or 0 to retrieve the required capacity.
  • indexCountOutput is a pointer to the count of indices written, or a pointer to the required capacity in the case that vertexCapacityInput is insufficient.
  • indices is an array of uint32_t to which the runtime will output the indices of the submesh.
  • See Buffer Size Parameters chapter for a detailed description of retrieving the required vertexPositions , vertexNormals , vertexSemantics , and indices size.

Valid Usage (Implicit)

  • The XR_ANDROID_scene_meshing extension must be enabled prior to using XrSceneSubmeshDataANDROID
  • type must be XR_TYPE_SCENE_SUBMESH_DATA_ANDROID
  • next must be NULL or a valid pointer to the next structure in a structure chain
  • If vertexCapacityInput is not 0 , vertexPositions must be a pointer to an array of vertexCapacityInput XrVector3f structures
  • If vertexCapacityInput is not 0 , vertexNormals must be a pointer to an array of vertexCapacityInput XrVector3f structures
  • If vertexCapacityInput is not 0 , vertexSemantics must be a pointer to an array of vertexCapacityInput uint8_t values
  • If indexCapacityInput is not 0 , indices must be a pointer to an array of indexCapacityInput uint32_t values

The XrSceneMeshSemanticLabelANDROID enumeration is the default set of semantic labels that can be used to label mesh vertices to describe which physical environment surface the mesh vertex is on. This enum set is represented by the XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID value in XrSceneMeshSemanticLabelSetANDROID .

typedef enum XrSceneMeshSemanticLabelANDROID {
    XR_SCENE_MESH_SEMANTIC_LABEL_OTHER_ANDROID = 0,
    XR_SCENE_MESH_SEMANTIC_LABEL_FLOOR_ANDROID = 1,
    XR_SCENE_MESH_SEMANTIC_LABEL_CEILING_ANDROID = 2,
    XR_SCENE_MESH_SEMANTIC_LABEL_WALL_ANDROID = 3,
    XR_SCENE_MESH_SEMANTIC_LABEL_TABLE_ANDROID = 4,
    XR_SCENE_MESH_SEMANTIC_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrSceneMeshSemanticLabelANDROID;

The enums have the following meanings:

Enum Description

XR_SCENE_MESH_SEMANTIC_LABEL_OTHER_ANDROID

This semantic label indicates that the corresponding mesh element represents an unknown object.

XR_SCENE_MESH_SEMANTIC_LABEL_FLOOR_ANDROID

This semantic label indicates that the corresponding mesh element represents a floor.

XR_SCENE_MESH_SEMANTIC_LABEL_CEILING_ANDROID

This semantic label indicates that the corresponding mesh element represents a ceiling.

XR_SCENE_MESH_SEMANTIC_LABEL_WALL_ANDROID

This semantic label indicates that the corresponding mesh element represents a wall.

XR_SCENE_MESH_SEMANTIC_LABEL_TABLE_ANDROID

This semantic label indicates that the corresponding mesh element represents a table.

Destroy a scene mesh snapshot

The xrDestroySceneMeshSnapshotANDROID function is defined as:

XrResult xrDestroySceneMeshSnapshotANDROID(
    XrSceneMeshSnapshotANDROID                  snapshot);

Parameter Descriptions

The application can use the xrDestroySceneMeshSnapshotANDROID function to destroy a scene mesh snapshot. Once the handle is destroyed, the application can no longer use it to get the submesh info or data. The handle is automatically destroyed when the tracker is destroyed because the tracker handle is the parent of the scene mesh snapshot handle.

Valid Usage (Implicit)

Thread Safety

  • Access to snapshot , and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_RUNTIME_FAILURE

Example code for scene meshing

Following example code demonstrates how to access scene mesh data for rendering.

XrInstance instance;  // Created at app startup
XrSystemId systemId;  // Received from xrGetSystem() at app startup
XrSession session;    // Created at app startup
XrSpace appPlaySpace; // Created at app startup

// The function pointers are previously initialized using xrGetInstanceProcAddr.
PFN_xrCreateSceneMeshingTrackerANDROID xrCreateSceneMeshingTrackerANDROID;
PFN_xrDestroySceneMeshingTrackerANDROID xrDestroySceneMeshingTrackerANDROID;
PFN_xrEnumerateSupportedSemanticLabelSetsANDROID xrEnumerateSupportedSemanticLabelSetsANDROID;
PFN_xrCreateSceneMeshSnapshotANDROID xrCreateSceneMeshSnapshotANDROID;
PFN_xrDestroySceneMeshSnapshotANDROID xrDestroySceneMeshSnapshotANDROID;
PFN_xrGetAllSubmeshStatesANDROID xrGetAllSubmeshStatesANDROID;
PFN_xrGetSubmeshDataANDROID xrGetSubmeshDataANDROID;

// Inspect system capability
XrSystemSceneMeshingPropertiesANDROID sceneMeshingProps = {
  .type = XR_TYPE_SYSTEM_SCENE_MESHING_PROPERTIES_ANDROID,
};
XrSystemProperties sysProps = {
  .type = XR_TYPE_SYSTEM_PROPERTIES,
  .next = &sceneMeshingProps
};
CHK_XR(xrGetSystemProperties(instance, systemId, &sysProps));
if (!sceneMeshingProps.supportsSceneMeshing) {
    // scene meshing is not supported.
    return;
}

uint32_t supportedsemanticLabelSetsCount = 0;
xrEnumerateSupportedSemanticLabelSetsANDROID(
  instance, systemId, 0, &supportedsemanticLabelSetsCount, nullptr);
std::vector<XrSceneMeshSemanticLabelSetANDROID> supportedSemanticLabelSets(
  supportedsemanticLabelSetsCount);
xrEnumerateSupportedSemanticLabelSetsANDROID(
  instance, systemId, supportedsemanticLabelSetsCount,
  &supportedsemanticLabelSetsCount, supportedSemanticLabelSets.data());

XrSceneMeshSemanticLabelSetANDROID semanticLabelSet = XR_SCENE_MESH_SEMANTIC_LABEL_SET_NONE_ANDROID;
// Check if system supports XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID
if (std::find(supportedSemanticLabelSets.begin(), supportedSemanticLabelSets.end(),
              XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID) !=
    supportedSemanticLabelSets.end()) {
  semanticLabelSet = XR_SCENE_MESH_SEMANTIC_LABEL_SET_DEFAULT_ANDROID;
}

XrSceneMeshingTrackerCreateInfoANDROID trackerCreateInfo = {
  .type = XR_TYPE_SCENE_MESHING_TRACKER_CREATE_INFO_ANDROID,
  .semanticLabelSet = semanticLabelSet,
  .enableNormals = XR_TRUE
};

XrSceneMeshingTrackerANDROID tracker = XR_NULL_HANDLE;
CHK_XR(xrCreateSceneMeshingTrackerANDROID(session, &trackerCreateInfo, &tracker));

XrSceneMeshSnapshotCreationResultANDROID snapshotResult {
  .type = XR_TYPE_SCENE_MESH_SNAPSHOT_CREATION_RESULT_ANDROID,
  .snapshot = XR_NULL_HANDLE
};
XrSceneMeshSnapshotANDROID& snapshot = snapshotResult.snapshot;
XrSceneMeshSnapshotCreateInfoANDROID createInfo {
  .type = XR_TYPE_SCENE_MESH_SNAPSHOT_CREATE_INFO_ANDROID
};

// app update loop
while (true) {
  // ...
  // For every frame in frame loop
  // ...

  XrFrameState frameState; // previously returned from xrWaitFrame
  const XrTime time = frameState.predictedDisplayTime;
  XrBoxf box; // populated with the bounding box of the camera frustum

  // ...

  createInfo.baseSpace = appPlaySpace;
  createInfo.time = time;
  createInfo.boundingBox = box;

  // Grab the scene mesh snapshot. This way the app can: know all of the submesh infos,
  // choose the appropriate submeshes for which to get data, allocate the necessary
  // buffer for those submeshes, and then get the data.
  CHK_XR(xrCreateSceneMeshSnapshotANDROID(tracker, &createInfo, &snapshotResult));

  // check the tracking state
  if (snapshotResult.trackingState == XR_SCENE_MESH_TRACKING_STATE_ERROR_ANDROID) {
    // unrecoverable error. Exit the app.
    if (snapshot != XR_NULL_HANDLE) {
      CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
      snapshot = XR_NULL_HANDLE;
    }
    break;
  } else if (snapshotResult.trackingState != XR_SCENE_MESH_TRACKING_STATE_TRACKING_ANDROID) {
    // The tracker is not tracking. Try again later.
    if (snapshot != XR_NULL_HANDLE) {
      CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
      snapshot = XR_NULL_HANDLE;
    }
    continue;
  }

  // 2-call idiom for getting submesh states
  uint32_t submeshCount = 0;
  CHK_XR(xrGetAllSubmeshStatesANDROID(snapshot, 0, &submeshCount, nullptr));
  std::vector<XrSceneSubmeshStateANDROID> states(submeshCount, {
    .type = XR_TYPE_SCENE_SUBMESH_STATE_ANDROID,
  });
  CHK_XR(xrGetAllSubmeshStatesANDROID(snapshot, submeshCount, &submeshCount, states.data()));

  // To store the poses for the selected submeshes
  std::vector<XrPosef> submeshesPoses;
  submeshesPoses.reserve(submeshCount);

  std::vector<XrSceneSubmeshDataANDROID> submeshes;
  submeshes.reserve(submeshCount);
  // Iterate the states and determine which submeshes the app wants data for
  for (XrSceneSubmeshStateANDROID& state : states) {
    // Modify this bool as necessary by looking at states. Maybe state.lastUpdatedTime
    // is below a threshold or maybe the bounding box does not intersect with camera view
    // frustum (The submesh will be culled).
    bool needed;
    if (needed) {
      // Add the selected submesh to the submeshes list. Set the capacity
      // zero for now since the size of the buffer will be determined later.
      XrSceneSubmeshDataANDROID submesh {
        .type = XR_TYPE_SCENE_SUBMESH_DATA_ANDROID,
        .submeshId = state.submeshId,
        .vertexCapacityInput = 0,
        .vertexCountOutput = 0,
        .vertexPositions = nullptr,
        .vertexNormals = nullptr,
        .vertexSemantics = nullptr,
        .indexCapacityInput = 0,
        .indexCountOutput = 0,
        .indices = nullptr,
      };
      submeshes.push_back(submesh);
      submeshesPoses.push_back(state.submeshPoseInBaseSpace);
    }
  }

  // Grab the data for the selected submeshes using the 2-call idiom.
  CHK_XR(xrGetSubmeshDataANDROID(snapshot, submeshes.size(), submeshes.data()));
  for (XrSceneSubmeshDataANDROID& submesh : submeshes) {
    submesh.vertexCapacityInput = submesh.vertexCountOutput;
    submesh.vertexCountOutput = 0;
    submesh.vertexPositions = new XrVector3f[submesh.vertexCapacityInput];
    submesh.vertexNormals = new XrVector3f[submesh.vertexCapacityInput];
    submesh.vertexSemantics = new uint8_t[submesh.vertexCapacityInput];
    submesh.indexCapacityInput = submesh.indexCountOutput;
    submesh.indexCountOutput = 0;
    submesh.indices = new uint32_t[submesh.indexCapacityInput];
  }
  CHK_XR(xrGetSubmeshDataANDROID(snapshot, submeshes.size(), submeshes.data()));

  // Destroy the scene mesh snapshot since we have finally grabbed the submeshes data. In
  // next iteration app can: create a new one to get latest mesh data
  CHK_XR(xrDestroySceneMeshSnapshotANDROID(snapshot));
  snapshot = XR_NULL_HANDLE;

  // Iterate the submeshes and get the vertex positions, vertex normals and
  // vertex semantics data for the selected submeshes
  for (uint32_t i = 0; i < submeshes.size(); i++) {
    XrSceneSubmeshDataANDROID& data = submeshes[i];
    XrVector3f* vertexPositions = data.vertexPositions;
    XrVector3f* vertexNormals = data.vertexNormals;
    XrSceneMeshSemanticLabelANDROID* vertexSemantics =
      new XrSceneMeshSemanticLabelANDROID[data.vertexCountOutput];
    for (uint32_t j = 0; j < data.vertexCountOutput; j++) {
      vertexSemantics[j] =
        static_cast<XrSceneMeshSemanticLabelANDROID>(data.vertexSemantics[j]);
    }
    XrPosef pose = submeshesPoses[i];

    // *** Use the data as per needs ***

    // Release the allocated memory for the data buffers when done using
    delete [] vertexSemantics;
    delete [] data.vertexPositions;
    delete [] data.vertexNormals;
    delete [] data.vertexSemantics;
    delete [] data.indices;
  }

  // ...
  // Finish frame loop
  // ...
}

CHK_XR(xrDestroySceneMeshingTrackerANDROID(tracker));

New Object Types

New Commands

New Structures

New Enums

New Enum Constants

  • XR_ANDROID_SCENE_MESHING_EXTENSION_NAME
  • XR_ANDROID_scene_meshing_SPEC_VERSION

  • Extending XrObjectType :

    • XR_OBJECT_TYPE_SCENE_MESHING_TRACKER_ANDROID
    • XR_OBJECT_TYPE_SCENE_MESH_SNAPSHOT_ANDROID

  • Extending XrStructureType :

    • XR_TYPE_SCENE_MESHING_TRACKER_CREATE_INFO_ANDROID
    • XR_TYPE_SCENE_MESH_SNAPSHOT_CREATE_INFO_ANDROID
    • XR_TYPE_SCENE_MESH_SNAPSHOT_CREATION_RESULT_ANDROID
    • XR_TYPE_SCENE_SUBMESH_DATA_ANDROID
    • XR_TYPE_SCENE_SUBMESH_STATE_ANDROID
    • XR_TYPE_SYSTEM_SCENE_MESHING_PROPERTIES_ANDROID

Issues

Version History

  • Revision 3, 2025-05-15 (Salar Khan)

    • Initial extension description