Architecting Spatial Content on Android XR with Unity

All right.

Welcome, everyone.

This is Architecting Spatial Content on Android XR with Unity.

My name is Jennifer Stanton.

I work at Unity as a senior software engineer.

I work on the XR Features team, specifically on AR Foundation and the related packages that bring augmented and mixed reality to Unity.

So, I'm Levana Chen from Google's XR Game Engines team.

Our teams focused on delivering plugins and the packages that to bring enhanced XR functionality and the seamless game engine integrations.

One of the biggest challenges we hear from you is that supporting multiple XR platforms with the same codebase is a lot of work.

AR Foundation is Unity's high-level extended reality API layer.

You can target multiple XR platforms without major code rewrites, having to manage multiple codebases, and it gives you one scripting API for AR features on any platform.

This allows you to focus on creating your app, game, or experience, and Unity takes care of the platform details for you.

One way that AR Foundation accomplishes this is through OpenXR extensions.

These are a set of open, industry standard APIs defined by the Khronos Group.

OpenXR has created Spatial Entities as well.

It's a framework for digital content to represent and interact with the physical world.

Spatial Entities provide a unified model for representing real-world elements as digital objects.

I'm excited to share with you some of the latest info on AR Foundation so that you can begin to create your journey begin your journey to create experiences for XR.

Next, I'll go over AR Foundation's capabilities so your digital content can interact with the real world.

Then I'll go over project setup, including packages and other dependencies, so you can implement this in your projects.

And then I will wrap things up by going through some feature unique to Android XR.

So, some of the ways that AR Foundation represents the real world digitally are planes.

And these are flat surfaces in the physical environment, such as floors, walls, ceilings, and even tables and couches.

Meshes are 3D structures representing the geometry of the physical world.

And then we also have ways of interacting with the real world, ray casts.

Those determine where a ray intersects with digital content.

Think of it as a laser pointer, and you get information about where that laser pointer is shining.

We also have anchors, which help stabilize and relocate a point in space.

I will be going deeper into anchors later in this presentation.

So, all four of these are considered AR Foundation Features.

There are also a handful more features as well that are not mentioned in this slide.

Planes and anchors, specifically, are considered trackables since they extend the ARTrackable class, and their managers track updates from the run time.

So, let's go into planes.

Planes, we have the ARPlane component, which provides real-time identification of flat surfaces.

They have semantic labels that you can query so you know if your plane is a floor, a table, a wall, and so on.

Planes also have dynamic boundaries, so, as the user moves through the real world and can see more of the surfaces, the plane boundaries will dynamically expand, merge, and refine in real time.

We can also attach a mesh collider to the plane to handle physics.

Here you can see an example of plane detection.

This is AR Foundation samples ray cast scene running on an Android XR device.

It detects real-time, dynamic, 2D polygons.

You can build an app that has this functionality by following our project setup guide or by using one of our samples or templates.

Next, we have meshes.

Meshes are an AR feature that provide 3D meshes of the environment.

We can attach a mesh collider to these generated meshes, and this allows your digital content to interact with real-world objects.

Meshes allow for convincing object occlusion.

So, a virtual object can appear to be behind a physical object.

If your digital object can move in space, you can use the same geometry to find a path that avoids real-world objects.

Next, we have ray casting.

Ray casting projects virtual rays into your 3D scene to look for intersections without you having to worry about the complex math.

The project A projected ray could potentially intersect with multiple 3D objects in your environment.

You can configure the ray cast to only hit certain types of digital content.

For example, you could configure it to not hit planes and only intersect with meshes.

Now, let's take a look at the relevant packages.

These packages all work together to communicate with each other and create a full XR experience.

As mentioned, AR Foundation handles features like planes and anchors.

The OpenXR plug-in handles native APIs on various XR platforms, so you would want to use this package if your device uses OpenXR.

The OpenXR plug-in can optionally be paired with OpenXR Spatial Entities package to access Spatial Entities extensions specifically, and more on that later.

If you know that you'll be targeting Android XR devices specifically, like the XR headsets and wired XR glasses, the Unity OpenXR Android XR package is needed for that.

And because all of this follows the plug-in pattern, the impact to your code should remain minimal.

Not all features are available in all AR Foundation packages.

For example, if you look at this chart, meshing is not currently available in the Unity OpenXR Spatial Entities package.

XR Plug-in Management allows you to manage which OpenXR features are selected in your project.

So if you're in the Unity editor, you would go to Edit, Project Settings, and then XR Plug-in Management would be over on the left sidebar.

And here you can find OpenXR features and feature groups, so you can select and deselect which features and feature groups that you want to include in your build.

For example, in this screenshot, we're looking at all of the ones that come with the Android XR OpenXR package.

And then you can mix and match these with spatial entities, for example, if you wanted to.

So, here is the Package Manager window.

I've highlighted some of the packages that you might see in a typical XR project setup.

As you can see, we have AR Foundation, which has been mentioned.

We also have the OpenXR plug-in.

We have the Android XR package.

I've also highlighted some others from other platforms because you can certainly make a project that compiles to multiple platforms.

And then we also have on the bottom right some other packages as well.

Composition Layers, those work with OpenXR devices; Core Utils, very useful for anything XR related; and then XR Hands and XR Interaction Toolkit to get your input, which is pretty essential.

So next we have project validation.

So, again, this is part of XR Plug-in Management.

Project validation can detect duplicate features, as well as other issues in your project that you need to fix to get it running.

But in this screenshot, I've highlighted the duplicate features.

So, what this is basically saying is two of the same features, such as I think we have anchors, maybe planes and raycasts here.

The user has selected to include duplicates of these features from multiple packages, and it's just prompting the user to select one or the other.

So, that brings us to local anchors and persistence.

Anchors ensure that digital content stays at desired locations in the environment regardless of changes to the AR session's tracking state.

AR platforms automatically update the poses of anchors over time to ensure that they remain in a consistent position, and anchors the best way to make sure that your content remains in place if you walk away from your anchor and then back towards it.

And what about when the user closes their app and restarts it later and comes back to it in a separate session?

Persistent anchors enable you to save anchors in an AR session and then load them in subsequent sessions.

They survive app life cycles and device restarts.

Pictured here is our persistent anchors sample in the AR Foundation Samples project.

The sample scene has a menu that has all of the supported functionality on that platform, such as saving and loading anchors.

And so that brings us to the life cycle of persistent anchors.

First you would create an anchor locally, so it's not saved anywhere yet, not saved to persistent storage, just a local anchor.

And then you would persist it to storage.

After that, you can restart the app, and then you can load it from storage.

Some platforms including Android XR also support erasing or un-persisting anchors.

Spatial Entities provider supports automatically loading all saved anchors so that you don't have to manually request each anchor by its ID.

In XR Plug-in Management settings, you can select the Automatically Load Saved Anchors option on the Spatial Entities: Anchors feature.

And saving an anchor only takes a few lines of code.

First, we begin in the Unity editor and make sure that your scene is ready by adding the AR Anchor Manager component to your XR Origin game object.

The manager keeps the scene updated with tracked anchors from the run-time.

So, that brings us to the code.

To add an anchor, you can simply call TryAddAnchorAsync and pass in the pose of where you want to place your anchor.

async void CreateAnchorAsync(ARAnchorManager manager, Pose pose) {
    var result = await manager.TryAddAnchorAsync(pose);
    if (result.status.IsError()) {
        // handle error
        return;
    }
    var anchor = result.value;
}

Next, we have removing anchors.

void RemoveAnchor(ARAnchorManager manager, ARAnchor anchor) {
    var result = manager.TryRemoveAnchor(anchor.trackableId);
    if (result.status.IsError()) {
        // handle error
        return;
    }
}

void DestroyAnchorComponent(ARAnchor anchor) {
    Destroy(anchor);
}

And similarly, it's pretty much just one line of code.

You call TryRemoveAnchor, you pass in the trackable ID of the anchor.

When the anchor was created, it was already given a trackable ID.

And then we have TrySaveAnchorAsync.

async void SaveAnchorAsync(ARAnchorManager manager, ARAnchor anchor) {
    var result = await manager.TrySaveAnchorAsync(anchor);
    if (result.status.IsError()) {
        // handle error
        return;
    }

    // Save this value, then return it as an input parameter
    SerializableGuid guid = result.value;
}

You pass in an AR anchor as input.

This function returns a SerializableGuid that identifies the anchor in persistent storage.

You can later use the GUID when loading the anchor.

And that brings us to loading anchors.

async void LoadAnchorAsync(ARAnchorManager manager, SerializableGuid guid) {
    var result = await manager.TryLoadAnchorAsync(guid);
    if (result.status.IsError()) {
        // handle error
        return;
    }

    ARAnchor anchor = result.value;
}

The function to call here is TryLoadAnchorAsync, and you pass in the serialized GUID from before.

It will return an AR anchor as a result.

And that's pretty much it.

These functions allow you to easily manage both local and persistent anchors.

So, anchor tracking is a fairly expensive task on most AR platforms.

When working with anchors, consider the following best practices: avoid creating multiple anchors very close to each other, consider if you can use a nearby anchor instead of creating another one, remove anchors when you're no longer using them, and, finally, if you enable another AR feature to help calculate the desired pose for an anchor, consider whether you can disable that feature once the anchor is successfully created.

And now, I'll pass it over to Levana.

Thanks, Jennifer.

And, so next, let me walk you through what's Google's contribution on Android XR journey in Unity.

So, besides the Unity OpenXR plugins that Jennifer has introduced so far, Google provides Android XR SDK for Unity with additional capabilities that's available on Android XR platform.

This SDK contains a lot of features and packages, and in this talk, we will focused on Android XR Extension for Unity package.

So, similar to OpenXR Android XR and OpenXR Spatial Entities, the Android XR Extension for Unity package is actually a OpenXR provider.

And this package's architecture looks like this, and they can working in parallel or independently.

So, like all other OpenXR provider, Google's package built upon directly on standard OpenXR OpenXR interface.

So, here we shows a bunch of features as a comparison among the all other providers.

So, you can to help you making decision on when and why to choose one over another.

So, you may have a question about the trackable and the spatial entity marker shows in this table.

So, here is the side note: so, for those who already build apps and experience on Android XR before, we have delivered this features through the exten- trackable extensions and which only work on Android XR.

And then later, as Jennifer mentioned, working as OpenXR open group and then released Spatial Entities Extension on the right side and provide a consistent and cross-platform experience.

And currently, we are deprecating the trackable extension in favor for the Spatial Entities framework.

Not only the later much more performant, but also for the cross-platform support.

And then, let's take a closer look to the spatial entities.

So, besides Jennifer has mentioned about the planes, ray casting, anchors, you can also find this feature supported by spatial entity providers, such as markers, which can detect, decode standard 2D barcodes and tracking their position in 3D space, and the image it will recognize known 2D graphics and tracking them in 3D space.

And then the bounding box, which will identify the physical object, like keyboard, laptop, or mouse, and tracking their bounding box in 3D pose.

And in the next example, we will configure the bounding box feature provided by Google's Extensions package.

So, here this view is pretty much the same for all the OpenXR features.

So, you just need to first switch to Android XR platform and then enable the OpenXR provider and select OpenXR feature you want to build with, for example, the spatial entity bounding box feature was shown here, and then fix the project validation issues.

It will help you to find all the dependency required by a given feature and then solve the conflicts caused by multiple feature are contributing the same sub-system.

Sometimes you Mostly you only need to choose one feature if you want to use, for example, anchor, plane, or bounding box here.

And the last you can just build the sample scene or your test scene.

To access the trackable object at run time, you can simply add the AR Bounding Box man-- manager component, it's like the it's the similar component as the AR Anchor Manager, and then use the AR Foundation interface to access it.

And alternately, you can also importing the spatial entity sample provided by our extension package as a template to start with.

So here is the major difference when you use AR Foundation feature on Android XR.

To help with understanding the environment around your user, your app usually needs access to the sensitive data around the environment.

And to protect the user, Android XR restricts the system-- the data access behind the permissions.

You may already be familiar with that, like the normal Android permission call, such as the camera access or the location access, things like that.

And then the best practice here is to disable the manager component at startup, either from the editor, as you can see on the right side, or from script, such as the Awake event.

void Awake() {
    planeManager = GetComponent<ARPlaneManager>();
    planeManager.enabled = false;
    boundingBoxManager = GetComponent<ARBoundingBoxManager>();
    boundingBoxManager.enabled = false;
    permissionUtil = GetComponent<AndroidXRPermissionUtil>();
}

void OnEnable() {
    StartCoroutine(WaitForPermissions());
}

And then once you've granted all the required permission, you can then enable the manager from script, like what the WaitForPermissions helper does here.

IEnumerator WaitForPermissions() {
    planeManager.enabled = false;
    boundingBoxManager.enabled = false;

    yield return new WaitUntil(() =>
        permissionUtil.AllPermissionGranted());

    // Enable after granted permissions and
    // restart subsystems.
    planeManager.enabled = true;
    boundingBoxManager.enabled = true;

    // Save permission status.
    permissionGranted = true;
}

By doing so, you can avoid a subsystem initialization failure caused by insufficient permission.

And this is a special feature here.

As Jennifer walked us through about the anchor using the AR Foundation, to expand anchor capability on Android XR, our package also provide a new feature called shareable anchors.

This feature replicate what AR Foundation anchor feature currently support and also come with some additional functionalities, such as geospatial anchors.

And as you can imagine, anchor is the real bridge connecting the virtual and the physical world.

And while Jennifer has walked us through the creating, saving, loading them, but all of this happened in a local space, such as your user's living room.

But what about the rest of the world?

And how could we share it with each other?

And here is the Geospatial API comes in.

It allows you to create a geospatial anchor with based on a real-world location, like your favorite bookstore or a meeting spot in a large park.

So, the Geospatial API can give you accurate geopositional tracker through VPS, also known as a virtual positioning system, such like a GPS, but use the outdoor Street View data to provide much more accurate tracking results.

So, some of you might already be familiar with this view if you have experience with ARCore's geospatial feature before.

So, to access Geospatial API on Android XR, we need first to connect to the backend hosted on the Google Cloud platform.

Since the geospatial backend is actually powered by ARCore service on Google Cloud, so you can follow exactly the same step to configure your Google Cloud project with ARCore API access.

All this will then enable the usage of Android XR Google Cloud feature that provided by and-- our extension package.

Once you have enabled the Geospatial API and grabbed the API key from Google Cloud project, you can then enable this feature and configure the authentication through the OpenXR settings menu showed it's here, and-- or alternately, you can manually configure it through the scripting API called TrySetAuthenticationAsync.

async Task TrySetAuthentication() {
    googleCloudFeature =
        OpenXRSettings.Instance
            .GetFeature<XRGoogleCloudFeature>();
    var result = await googleCloudFeature
        .TrySetAuthenticationAsync(
            strategy, credential);
    if (result.IsSuccess()) {
        // Ready to use for geospatial tracker.
    } else if (result.IsUnsupported())  {
        // Google Cloud feature is not supported
    } else if (result.IsKeylessError()) {
        // Confirm if keyless has been set up properly.
    } else {
        // Handling other common OpenXR errors.
    }
}

To be noticed here, that Android XR platform right now only supported API key authentication, but more credential support will be coming to the in the future.

So, once you have authenticated, you can then use a geospatial manager to access geospatial capabilities.

It's highly recommended to check the Google Map first about the VPS coverage before the integration.

As such, you must also have the networking access when using the feature.

So, the XR Geospatial Manager, it works similar to the AR Session component provided by AR Foundation.

So it come with a static state property to indicate the system support and the tracking state of the geospatial tracker.

You can register state change event by a callback like this.

void OnGeospatialStateChanged(XRGeospatialStateChangedEventArgs eventArgs) {
    bool isInitFailed = eventArgs.CurrState == XRGeospatialState.InitializationFailed;
    if (isInitFailed && XRGeospatialManager.InitializationResult.IsCloudAuthError()) {
        // Handle cloud authentication issues.
    } else if (isInitFailed) {
        // Check XRGeospatialManager.InitializationResult for other common OpenXR errors.
    } else {
        // Iterate runtime state change, and handle accordingly.
    }
}

private void OnEnable() {
    XRGeospatialManager.OnStateChanged += OnGeospatialStateChanged;
}

And once the geospatial state become running, you can use this manager instance for all the following functionality.

First, such as to check the VPS availability with the given GPS location, here we use Unity's location API to get the GPS value and then check the VPS availability.

async Task CheckVPSAvailability(CancellationToken token) {
    var location = Input.location.lastData;
    try {
        var result = await geospatialManager.TryCheckVPSAvailabilityAsync(
            location.latitude, location.longitude);
        token.ThrowIfCancellationRequested();
        if (result.status.nativeStatusCode == (int)XrResultAndroid.GeospatialCoordinateInvalid) {
            // Invalid coordinates, try with another location.
        } else if (result.status.IsError()) {
            // Handle common OpenXR error: result.status.ToOpenXRStatus()
        }  else if (result.value == XRVPSAvailability.Available)  {
            // You may use this pose to attach geospatial anchors.
            var currentGeospatialPose = new XRGeospatialPose(location);
        }
    }
}

So this is basically a replacement if you don't want to check the VPS support through the map, but you need to go outdoor to check the actual GPS status.

And then, you can also convert the pose between the Unity pose and the geospatial pose by TryLocate.

// Geospatial Pose -> Unity Pose
var geospatialPose =
    new XRGeospatialPose(Input.location.lastData);
if (geospatialManager.TryLocate(
        geospatialPose, out XRPoseResult result)) {
    if (result.IsPositionValid ||
        result.IsPositionTracked) {
        transform.position = result.Pose.position;
    }
    if (result.IsRotationValid ||
        result.IsRotationTracked) {
        transform.rotation = result.Pose.rotation;
    }
}

// Unity Pose -> Geospatial Pose
Pose pose = new Pose(
    transform.position, transform.rotation);
if (geospatialManager.TryLocate(
      pose, out XRGeospatialPoseResult result)) {
    if (result.IsPositionValid ||
        result.IsRotationValid) {
        // You may use for anchor attachment.
    }
}

And taking the result from either VPS check or the pose conversion, you can then attach the spatial anchor to a given geospatial pose or upon a known surface type, such as terrain or rooftop.

async void PlaceAnchor(XRGeospatialPose pose) {
    try {
        Result<ARShareableAnchor> anchorResult;
        XRGeospatialSurfaceType typeToPlace = /* Choose your target type */;
        if (typeToPlace == XRGeospatialSurfaceType.Default) {
            anchorResult = geospatialManager.TryAttachGeospatialAnchor(pose);
        } else {
            pose.Altitude = 0;
            anchorResult = await geospatialManager.TryAttachSurfaceAnchorAsync(typeToPlace, pose);
        }
        if (anchorResult.status.IsSuccess()) {
            // ARShareableAnchorManager will track the anchor and update it.
        } else {
            // Handle OpenXR failures.
        }
    }
}

And then when you put all of these together, you can build some truly magic spatial experience with the real-world in front and center.

So here, we place an arrow to navigate our audience to the targeted place in the real-world.

Okay.

So, Android XR SDK is not the only package that we provided to Unity developers.

Google's Android XR SDK for Unity now also including the Android XR Interaction Framework and Android XR Audio package.

And we are super excited to expanding the Android XR SDK with more feature support and including more Spatial Entity as well.

Alrighty, so that brings us towards the end of the session.

You can follow Unity's roadmap online to see what's coming next.

Don't forget, we have Unity 7, the next generation of Unity, coming early next year.

And in the meantime, you can try out these tools and create a project using the Mixed Reality template or take a look at the AR Foundation samples project on our GitHub.

And also, to learn more about Spatial Entity and our Unity package support, you can just visit the page on Android developer site.

And then, thank you all.

We can't wait to see what you can build with AR Foundation and the Spatial Entity on Android XR.