Let agents generate UI in your Android apps with A2UI

Hi, everyone, and welcome to this talk where we're going to be talking about how to generate UI in your Android apps using our brand new A2UI library.

My name is Nick Butcher, I'm a product manager working on Jetpack Compose.

Hey everyone, I'm Anton, I'm an engineer on the Compose team.

I'm working on this new Jetpack Compose A2UI renderer that we're very excited to tell you about today.

Yes, now with the rise of AI capabilities, many apps are looking at how AI can help you to improve or build new experiences for their users.

One of the exciting experiences that some people are looking at is generative UI, which is how, at runtime, agentic back ends can create brand new UIs for your users, which we think is very interesting.

But unfortunately right now, many agentic back ends' responses are trapped in walls of text, like this.

So, here's an example where I've asked an a- agent to generate me a new feature for booking a trip.

So, here's the prompt.

And as you can see, it's given actually quite a decent answer, but it's trapped in this wall of text which I can't really easily interact with and it looks nothing like the rest of my application.

Generative UI allows you to create an experience like this instead, where you can generate a new UI on the fly in response to dynamic input, opening up possibilities for more customization, or more new experiences.

So, here's an example of a UI that's created using like standard, native components.

So, you can absolutely build this kind of experience today yourself, but you'd have to define your own like format for interchange with the back end, your own kind of event stream, and functions, and so on.

A2UI, which stands for Agent-to-UI, is an open protocol which is a JSON format for defining all of these things.

So, one of the things I really like about A2UI is it's a declarative format.

The agent is just returning a JSON representation of the UI to render.

It's not returning any code that your client then needs to execute, which can open you up to certain security vulnerabilities.

It's just declarative.

So, what other concepts do we need to know about A2UI?

Just two main concepts: catalogs and surfaces.

Catalog has components like text, button, row, and column that the agent can use to build the UI.

A catalog also has functions that the agent can use for formatting text, validating user input, or opening URLs.

A catalog is a contract that defines what components and functions your app supports, and the agent can only build with those.

This means, as Nick mentioned, the agent never executes arbitrary code in your app.

It composes a UI from the building blocks that you design and trust.

One more important concept is a surface.

A surface is one piece of UI that the agent controls, like this trip planner example on the slide.

The agent describes the UI for a surface in JSON using the A2UI protocol messages, like createSurface, updateComponents, and using the components and functions defined in your catalog.

So speaking of catalog, catalog is a very fundamental concept in A2UI, cuz it controls what components that the agent can use to construct an interface.

The A2UI spec defines something called the basic catalog, which is a set of common components you can use for constructing most UIs.

The Jetpack Compose A2UI renderer provides an implementation of the basic catalog using Material 3 components.

Most of the catalog components are ready to use out of the box.

There are three media components: image, video, and audio player, where you need to plug in your library of choice.

For example, you may choose Coil for image loading and Media3 for the video and audio player.

But the real power of A2UI and our new renderer library is that you can create your own, completely custom components and catalogs, and fully control what the agent can use to build the UI.

We'll come back to catalogs later.

Okay, so let's take a look at what this looks like.

So, if we can switch to the demo machine, please.

This is our demo app, which we're going to use, and we're going to build up the A2UI renderer to show you how easy it is.

So it's a set of prompts, as you can see in these little chips in the middle of the screen, that you can send to an agent.

So, if we click on one of the prompts, it's sending that prompt to a real, live agent back end, so fingers crossed for the demo, and in addition to the prompt, we also append some system instructions.

So, we tell the back end about the A2UI protocol, about the catalog of components, in this case, the basic catalog we're using right here, that it can use to build the UI, and we give it some examples of how to build this.

The chat session implementation in our demo app already knows how to extract raw JSON A2UI messages from the agent's responses.

So, in this case, the agent replied with some text followed by A2UI protocol messages: createSurface, updateComponents.

But at the moment, our app just shows this raw JSON and doesn't render it as UI yet.

Okay, before we dive in and implement the code, if we can switch back to the slides just for 1 second, let's take a step back and tell you about the system design, which we're going to use to implement this.

Oh, here we go.

Yeah, so here's the plan.

So, the AI agent streams a response into our app that is received by the ChatSession class.

The ChatSession extracts raw A2UI JSON messages from the agent's responses as it's streaming in and passes them to the A2UI message processor.

The message processor is configured with a parser that converts those raw JSON messages into a semantic representation.

The processor then validates the messages against the catalog definition and maps them to a set of surface models.

The surface models, these are the things that define, how the UI looks and, the pieces of UI that the agent controls.

Those surface models are then rendered as native Compose UI using the A2UI surface composable provided by our library.

The communication between the agent and the UI is bidirectional, so when the user interacts with the UI, the user actions and errors flow back to the agent.

So, in summary, there are three main building blocks: the catalog, the message processor, and the surface.

Let's write the code for it.

Yes, let's build it if we can switch machines.

Okay, so the very first step is adding the dependency.

Here you can see we've already included the Material 3 A2UI dependency, and we've already synced the project so that we don't have to sit and watch Gradle.

This is the top-level dependency that defines the implementation of the basic catalog using those Material 3 components, but there are other finer-grained dependencies underneath.

So, if you don't want to use the basic catalog, which we'll describe why later, you can use those more fine-grained ones.

But this will bring-- pull in the rest transitively.

Okay.

So, we said there are three things.

The first step was to define a catalog.

So here in our application, we're going to create a factory function to help build an instance of that Material 3 implementation of the basic catalog.

So we use this library function here, material A2UI basic catalog, which is versioned as well.

And as we said, that this, provides all of the common components you need, except we don't want to force you to use certain image-loading or media-rendering libraries, so you have to provide those yourself.

So here, for example, Anton has already created this, wrapper around Coil's AsyncImage composable, which we'll use to load images, as well as a little extra logic for our demo to look up images from Wikipedia.

And so, very easily, we can plug this into, the catalog components.

And similarly, we have similar implementations for rendering, video and audio, where we're going to use the Media3 library, to render them with ExoPlayer.

So very simply like this, we can plug in, our own components to it.

The next thing that the catalog needs, is a set of, standard functions.

So, this is functions that the client can provide.

Again, we ship an implementation of this along with the basic catalog, but we need to provide it with a context, which you will use to do certain actions, like opening a URI if a user taps on an item.

And like that, that's our first step completed.

We've created a catalog.

The next step we have to do was set up the message, processor.

So, this is the thing that actually parses, the responses and so on.

So, we'll switch over to our application's view model, where we'll first, we'll actually instantiate that catalog we set up, previously, cuz we're going to need that.

And then we're going to create this message parser, message processor object, which needs access to the catalog in order to, validate the responses against the components that we want to use.

One other component we need to create is a parser, which is actually the object responsible for doing the parsing of the, JSON messages as they come back.

So, here is where we create the ChatSession, object.

So, as I mentioned before, the ChatSession already extracts the raw A2UI messages from the agent's responses.

And when every new A2UI message is ready, it calls this onA2uiMessage callback.

So, all we need to do is pass each of those messages when they arrive to the processor.

We call processInput function, give it a parser and a message.

So, the processInput function parses each message and then queues up the work for the processor to validate it against the catalog and update the surface models.

So, the processor exposes the current set of active surface models as, activeSurfaces StateFlow.

The ChatSession has some logic to add the surface models to the UI states of the agent replies to render.

So, we just pass this activeSurfaces StateFlow as a parameter to the ChatSession.

Also, as I mentioned, the processInput, function queues up the work for the processor.

But to actually start processing, we need to, launch a coroutine on a background thread.

So, we'll use a view model scope for that and set the default dispatcher.

And in that coroutine, we just need to call processor.collectMessages.

So, while this coroutine is active, the processor will be handling the messages, from the message queue.

So, one more thing we need to do is when the user clears the chat history, we also need to clear the current surfaces.

So, we call clearSurfaces.

Okay.

So we've done the first two steps.

We created the catalog, we created the processor to parse the messages.

The next step we need to do is actually render those responses in our UI.

So we're collecting those those surfaces as they get parsed by the parser.

And so now in our UI, we'll pass those on in this agent reply object and we'll simply iterate over them.

And then we'll use this A2UI surface composable and then pass it the model each of the models as we iterate through them in order to render the content where we want it to go.

One note here is that the agent may choose to use something like a list if it chooses to display, you know, a large amount of content.

So we need to set the bounded height a maximum height on this component to make sure it renders properly.

And with that, we have implemented like a basic A2UI renderer.

So now if we go back to our app, relaunch it, and hit that prompt again, then hopefully the agent should use that in order to, yeah, here we go, to render a real UI.

So we just code that live.

This isn't like smoke and mirrors.

But yeah, it's using real images.

It's using tabs, dialogues, date pickers, sliders, you know, all standard Material components that you would you know, build your app with rather than just showing a textual response.

Because this is a real app using real Material components, it you know, does the things real components do.

It has like native performance.

It does things like respond to dark mode here.

It the these are the Material components.

So if you have an outer Material theme composable where you've customized the colors, shapes, typography, and so on, then these components will pick them up so it looks like the rest of your application.

As Nick mentioned, since these are native Compose components, they have native semantics.

This means all of Compose's built-in accessibility features are supported.

So things like TalkBack, font scaling, keyboard navigation work out of the box.

Now, because you're invoking an agent to generate a new UI at runtime, there can be some latency.

You know, you might have to wait for the agent to respond.

The library does what it can in order to minimize this latency.

So if we use this kind of we'll inject some some some waits in order to show you a bit more slowly to show what happens how we minimize that.

Yeah, since it may take some time for the agent to stream the full complete response, we use something called progressive rendering.

So as soon as the root component arrives, we start displaying the UI for the surface.

All container components show loading placeholders for their children if the children haven't arrived yet.

When the child components arrive, they get animated in using Material 3 motion scheme.

Okay.

And if we can switch to the next prompt just to show another feature of the library.

So here's a another prompt where we're booking a table for dinner.

And here the agent is choosing to use this slider component, yeah it's used it.

So here you can see that you can drag around and interact with the UI the agent has created on the fly.

And notice how it's not just changing the number on the component, there's also this format string further below where it's counting it.

Like tell me that's not doing a round trip back to the agent just to update that, right?

This is a two-way binding.

So the slider component is bound to this surface's in-memory data model.

All the components that bind to the same path in the data model get updated automatically when the value changes.

So the slider reads the value to display, and also when I drag the slider, it writes it back to the model to update it.

So all of this happens locally without a round trip to the agent.

But we can also ask the agent to update data.

So let's make it 6 people.

Pay attention to the slider when the agent replies.

It just jumped to 6.

So let let's see what we received from the agent.

Instead of sending a whole new description of the UI, the agent just sent a one small message updateDataModel that updated the path for the slider.

Since the slider is bound to that path, it updated in place.

So each surface's data model is built on Compose snapshot state, and each component property bound to a path is a derived state.

So whenever the user or the agent updates the value in data model, all the components that are bound to the same value get recomposed automatically.

Okay.

So say we want to take action and we want to hit the reserve button, what's going to happen here, Anton?

Nothing happens at the moment.

The button does dispatch an action, but we don't send it to the agent yet.

So let's fix it.

Okay, so just like we had to hook up into the processor to process the inputs, we also will need to hook into any outputs, any actions that the user takes on the UI the agent has created.

So to do that, we use the outboundEvents flow that the processor expose exposes, and collect on this.

So this is your chance where you can plug in any logic that you want based on, you know, when the user taps an action or performs something with the UI.

So right here we're sending any actions up back up to the agent to then decide what to do, but you can also at this point decide whether you kind of call your own back end, or navigate to a different screen in your application, it's up to you.

So here Anton's plugged it in, and so now when we hit the confirm reservation action, it's going to send that action that you took back out to the agent.

And this is where, you know, you could on the back end even decide to then invoke some functionality.

So let's look at what we actually send to the agent.

So this is the A2UI action message.

It has a context, and the context contains a set of properties that the agent asked to be sent when this action is performed.

So when I click the button, this particular properties are resolved at that moment based on the current values in the data model, and that's what we send to the agent.

Okay, and so far the agent has been pretty cooperative with us, but we're talking about AI here, and AI can make mistakes.

So what Let's take a look at what happens if the agent returns a you know a bad you know makes a mistake in its UI that it describes back to us.

So this last prompt we have, we're actually asking the agent to make an error just so we can demo this to you.

Let's see what happens.

Yeah.

As you can see, one of the components is marked with an error.

And now it got fixed.

So this is the agent self-correction loop.

Before each component reaches the UI, the we validate that message against the catalog schema.

If the component doesn't match the schema, it's marked as an error.

The renderer the basic catalog implementation shows that error isolated to that specific component while the rest of the UI is still displayed.

Behind the scenes, the renderer sends an error message to the agent to let the agent know what went wrong.

So in this case, we sent a message that said component type unknown type that does not exist is not found in the catalog.

The agent then had a chance to respond send a follow-up response that updated the existing UI and the UI got fixed in place.

Yeah.

In the current implementation of the Material basic catalog, we show the errors right away, but in your custom component, you may decide to continue showing a loading indicator for a component if there is an error while waiting for the agent to fix it.

And so this way the error may be completely hidden from the user.

And speaking of like catalogs and custom components, so far we've just been using the basic catalog as it ships out-of-the-box, but you can of course customize it to your own needs.

So you can see that when we use the factory function to create the catalog, they're defaulted values for all of the different components.

So you can easily plug in your own components if you want.

So for example, if we want to plug in our own if we have our own design system in our app and its own button, we can easily plug that in.

So we just override this parameter when we're creating an instance of the catalog, and we'll create this component definition.

So if we create a new class to implement that, you can see there's an interface here, the A2UI basic catalog of type button.

So if we implement this, we can plug in our own component.

Yeah.

So since we're overriding a component in the basic catalog, we need to implement this A2UI basic catalog button interface.

So this interface already has some information about the component like the name, description, and the property schema.

So all we need to do to implement our own version is to implement this composable function.

It takes a few parameters.

So these parameters are values received from the agent after getting validated.

So these are things like the child ID, which is which references a component that will be the content of the button.

It also has a variant that describes the type of a button, action definition, and some other parameters.

In production implementation of the button, we would need to use all of these parameters, but for our demo purposes, we only need the child and the action.

So let's implement this composable.

Okay, so in this composable function, we're going to emit our own custom like button type here.

And as you can see, we can use the parameters that were given and validated by A2UI.

So for example, in the onClick function, we can dispatch that action.

We can use this library type DispatchAction and pass it the action that A2UI defined.

So we're using our own button, but we still want to render the child content, the label, or whatever the agent decided to use.

So the way we do that is we use this observeA2uiComponentState function and pass it the childId in order to get a handle onto the child content that should be rendered.

And then what we'll do is we'll use this A2uiComponent composable function and then pass it that child component state to render it.

And here you can see we're only rendering that the child content if it loaded successfully, but you can also have your own logic here for if there if it's loading or if there's an error state, you can decide what it is you want to show.

So if we go back to one of our example prompts in our demo app and now we replace the button using, you know, the standard Material button.

So subtle, hopefully you can see.

You can see we're using a very custom button here as well.

Cool, so that's how to you can, you know, replace some of the library types, but what if you want to extend the library with your own custom types?

We can do that very easily by kind of adding to the catalog itself.

So Anton's going to create a new class here to implement a star rating component.

So, we're going to add something to our application to let us rate restaurants while we're on this hypothetical trip.

So, because we're not replacing an existing component, we use this more generic interface, this A2UI component interface, which, you can see, has a few parameters that we need to override.

So, the first two are the name and description.

So, this describes what our component is and does.

Now these are very important because when we serialize the catalog and send it to the back end to tell it what it can use to build a UI, this is what the agent will use to decide whether to not to use this component when it's responding.

So, you need to be quite descriptive and tell the AI you know exactly what this does.

The next property we need to override is this properties property, which is basically telling which basically defines what the data this component deals in.

So, we're doing a star rating bar so you can rate a restaurant, and there's going to be two properties that we need.

It's going to be the value, like how many stars you the user has selected, which is going to be a mutable property that the user can interact with and change, and the maximum number of stars.

So, here we can see Anton's use this dynamic number-type property for the value.

So, this that's like you know the actual how many stars are selected.

Which again has its own kind of description telling the agent what this value is for.

And then we have a static-type property, so we're just using this A2UI property of type number, to define the maximum number of stars that this component can show.

And then once we've defined those two properties, we just need to return them.

So, overriding this properties property, we'll do a list of, return those.

So, that's how you tell the agent this is what this component deals in.

Yeah.

As Nick mentioned, this property definitions are used to be sent as part of the catalog schema for the agent to know how to use this component.

But this property, the same property definitions are also used to extract typed values at runtime.

So, there is a single source of truth.

So, let's see what we need to do to implement this.

So, first, to access the let's say the max property, since the max property is a static property, it means the agent sends the value for it inline in the properties object.

So, to access it, we just need to get it from the properties object by referencing it using the proper- max property object as a as the key.

Since this property is defined as a number, we also need to convert it to an int and set the default.

The next thing is the value property.

Since value property is a dynamic property, it means it's referencing something in the data model.

So, to access it, we need to use properties.bind API and pass the property object as a key as a parameter.

So, what this does, it returns the current value stored in the data model for this property and also subscribes for future updates.

So, whenever the data model is updated, this component will recompose automatically.

So, similarly, we also converted to an int and set some default.

One more thing we need to do is set up two-way binding so that we can change the rating.

To do that, we use the bindUpdater API.

We pass the same property object that defines the property, and as a result, we get So, the bindUpdater returns a lambda that allows us to mutate a value for this property in the data model.

Now we are ready to emit our star rating component.

Okay.

So, now that we've defined our own custom component, we then need to add it to the catalog that we're using.

So, previously, we were just using the basic catalog as is.

So, now what we need to do is actually create a new catalog, which will take all the basic catalog and add in our custom component.

So, a catalog is defined made up of three things.

It's an ID, a set of components, and a set of functions.

For the ID, we'll use an a URI to define, you know, a uniquely identifies catalog as well as the version that we're working with.

And then for the components, we'll take all of the components from the basic catalog, and then we'll use the add operator to add in that custom component that we just created.

And then for the functions, we'll just use the functions from the basic catalog.

And so like that, we've kind of extended the basic catalog to add in our new custom component type.

See, it's pretty easy to create a custom catalog.

So, the agent learns about how to use the catalog from the system instructions that we send to it.

So, since we now created a custom catalog, let's also update the system instructions.

So, we'll pass the catalog object to our function that builds the system instructions.

So, here it is.

So, up until now, we were sending a static definition of the schema for the basic catalog since we were using this standard catalog.

Now that we have a custom catalog, we can dynamically generate the schema using the toJsonSchemaString function.

So, this will create a full complete JSON representation of this catalog, including the star-rating component that we just added.

The system instructions also include some example responses for the agent.

So, we also change the catalog ID here, so that the examples reference the correct catalog.

Okay.

And with that, we have extended the basic catalog to add our own custom types.

So, if we redeploy that and try one of these prompts, which should hopefully give us a rating UI.

And then the agent replies.

And here, you can see this star component is this brand new component that the agent had never seen until we just implemented it right now.

This isn't part of the basic catalog, we just created this.

When I click on the star, that's when the bindUpdater lambda is called and the value is updated in the data model.

And if we submit the ratings, we can take a look at the action and the context of the action will include the values that we just set.

Okay, if we could switch back to the slides, now.

So, that was a lot that we went through quite quickly, but hopefully, we showed how you can easily go from agents responding as a wall of text into them actually creating real native UIs.

Hopefully, we showed you that the integration is pretty easy.

There are these three major components that you need to do: you need to create a catalog of components, implement some parsing logic, and then render it in your UI.

And hopefully, we showed that the library does a lot for you out of the box, things like the progressive rendering or validating your schema, the self-correction loops or error handling, it does a lot.

And hopefully, we demonstrated quite quickly today that it's very easy for you to customize it.

We showed how the basic catalog already responds to your Material theme if you're using Material, but it's really simple to extend or replace components within the catalog.

But I think some of the most power comes from if you actually create your own catalog.

If you want the most control over what UI the agent can actually use, then you can create completely custom components.

If you want to give it like, you know, complete free rein, you can give it very basic components, things like text labels and buttons, and say, "You create the UI."

If you want more control, you can create more elaborate components and say, "These are the only things that you can use."

So, you're in control of what the agent can do and show to your users.

So, this is the new A2UI renderer library and Jetpack library that we've just launched.

implementation("androidx.compose.material3:material3-a2ui:1.0.0-alpha02")

It's currently in Alpha, Alpha 02 went out just recently.

So, follow these links to check out.

We have documentation on developer.android.com, as well as a couple of samples.

We have a basic sample showing the basic catalog, as well as a more elaborate sample, and this JetPacker sample app we've just created, which you're going to see again in the ADS keynote coming up later today, demonstrating how you can create these more kind of elaborate components to control what the user sees.

Like I said, we're currently in Alpha, so it's a great time to check it out and give us your feedback and your feature requests, and tell us like, you know, how you would like to use this, or I'd love to use this if it could do X or Y.

So please, do check out the library and give us feedback.

And that's what we got.

And I can't wait to see what you and your agents build with it.

Thanks very much.

Thank you.