Powering interactive surfaces with Remote Compose

Hi everyone.

So my name is Nicolas Roard, I'm the tech lead on the Remote Compose project.

I'm John Hoford.

I'm a developer on Remote Compose.

And I'm John Zoeller, and I do developer relations on Wear OS.

All right, let's get started.

So, I'll give you first a quick overview of what exactly is the kind of problem we're trying to solve with Remote Compose, right?

So usually when you talk or you when you think about UI on Android, you think about the UI as part of your application and your application process, right?

There are a lot of different places where sometimes you want to take your UI, package it, and send it somewhere else like to a remote surface.

So what we mean by remote surface, it's pretty much like those examples, like watch styles, widgets, could be even your own app or a different app, all right?

And so, all of those are essentially examples of remote surfaces.

So somewhere, somewhere else, could be your app, could be a server, somewhere else knows what to send to that remote surface.

And…

What about my speaker?

Between those two, there is essentially a distance.

And that distance can be literally an actual distance, could be a network connection, right?

Could also be just a process boundary.

You could be on the same device.

But whatever it is, it has a cost.

And simplest thing you could think of taking that UI and sending it is just to take a screen copy, right, and sending it.

Problem with that, it's not super fast, right?

So what we do with Remote Compose instead is that we send a document representing that UI.

And keep that in mind, being small has a lot of advantages.

It's important.

We saw it on the first pixel that you sent, but it actually is true of any kind of updates you want to do on that surface, right?

You got the round trip.

So you get something on that remote surface like a an event, a click, a touch.

You have to go back to whoever created it to get a new version.

And that's, that's not good.

So in many ways, that is what we are trying to solve with Remote Compose.

So we saw the document being sent.

Beside the document, there's a couple of, you know, two a couple of areas that are very important.

One is how you create that document.

And the second one is how you play it.

I mentioned that we're creating a document.

We're not sending a screen copy, and we're also not sending an application.

So we have three different ways to help you create that document.

One is a Compose API, two is procedural Kotlin API, KMP, and three is a JSON format.

They all emit the same bytes.

So we provide a set of tools to create those documents and with the Compose API, we can actually take advantage of the existing tooling in Android Studio.

So for example, you can use the preview to see live what you're developing, directly in the IDE.

But we're also working on some new powerful tools like a document inspector.

Let's let's take a quick look at what the document actually is.

It's fundamentally a list of operations encoded in a compact binary representation, and those operations describe a standalone user interface that can be sent to that remote surface.

So one very important characteristic of those documents is they are small.

And as we saw it, small is good.

So typically we're talking about a few kilobytes, right?

But even more complex UI with something with a very adaptive layout, we are still talking about a few maybe dozens of kilobytes, right?

And we package all of the assets into that document.

So if you think about it, the only thing that can really take space on the document are if you start packaging bitmaps.

So keep that in mind.

This is, after all, the entire reason why we are sending a document and not a screen copy.

When we're talking about being small and being it important, it's not just the communication.

It's kind of a central corner case of the philosophy behind Remote Compose.

As we saw, it obviously has an impact on the time it takes to get the first pixel on screen, but oops, sorry.

It also means that we can use, in equal capabilities equal capacities of the surface, we can use more documents than otherwise, right, while still being efficient.

And that's something we'll see that we take advantage of when we support widgets.

The players themselves tend to be pretty small too, and very little dependencies.

And the fact that they are small and few dependencies means we can extend the reach to where we can actually play Remote Compose documents.

And that's specifically important for constrained platforms, as you will see.

So I talked a lot about the transport and the distance, but Remote Compose by itself does not really deal with the transport.

That's on you.

What we do is we package all of that into, you know, a byte buffer that you can then send with the appropriate method to the remote surface.

For widgets, we help you, we do provide a connector for you.

But generally, it's a useful thing to keep in mind, you are in charge of the transport.

And once a player received that document, however you sent it can extract the list of operations and convert them in draw commands, in layout trees, expressions, and all of that together can create a really rich user experience.

So, one thing to keep in mind is, by handing you the power to decide how you want to transfer the document, it means that you can reuse Remote Compose as a tool in a lot of scenarios, some that we may not even think of, right?

Once the document is loaded by a player, it runs by itself.

And that's kind of the entire idea.

We want the document to create a UI that is somewhat autonomic, that can doesn't have to go back to whoever created it, right?

And all of that together, we can support, you know, resizing, scrolling, touch events, you know, deciding to show or hide components, animation, some touch event, and some touch handling, can all be done in the document without having to go back to the player.

And sometimes, if you need to go back, sorry, to the creation, you can actually request an action to say, "Please give me a new document."

We provide three official players mostly via the AndroidX libraries in Remote Compose.

One is a Compose player, so you can use it in your Compose application.

We also have a View player if you happen to still use View.

And we also have a RemoteViews implementation that is shipping in the Android platform since API 36, and that allows you to create Android widgets running on launcher using Remote Compose.

We also have some experimental players if you are interested, some C++ running on Skia, TypeScript, and even Compose Multiplatform.

So, the idea is the same technology that you are using can be essentially used to render on a lot of different surfaces and platforms.

So, to sum it up, the distance that I've been talking about is still there, but we are trying to make it matters only when it actually does.

Yeah.

So, let's talk a little bit about the capabilities that you'll get inside of a Remote Compose document.

So, at at its base, there's a Canvas, and it's all the features of a Android Canvas.

But with a few additional APIs in there.

But one of the special things is that all the floats that you have in that Android Canvas, they can be replaced by expressions.

That allows you to do animations, as you see here, dynamic scaling, flexible layouts.

A lot of different things can be done with it.

A little bit more about expressions.

You have a rich set of operators from normal math, trig stuff, array access, spline interpolation.

And one of the cool things is that any value that changes can be animated, so you can animate the change in the state of any value.

In this particular drawing you see here, each one each pendulum picks a different type of animation, and all of those are available in Remote Compose.

On top of that, we have the basic components, Compose stuff, stuff you know.

It produces a very familiar API, box, row, column, that sort of stuff.

But on top of that, we have adaptive components.

One of the nice things about that is that since you're not in control of your document at that point, these can handle objects disappearing and appearing and going away, coming back because of resizing, etc., all without going back to the source of the document.

And then on top of that, we have component animation.

All the changes can be animated, so if something goes away, you can animate it.

And you can write your own custom animations.

In this case here, you see some of them are using shaders, all sorts of things.

One of the other things that we are capable of is interaction while on there, so it's not just pretty pictures.

You can in click on things, have touch events, scrolling, that sort of stuff.

Also important is accessibility, so all your accessibility APIs are there.

One of the things you're seeing here is I'm building a full stopwatch with kind of interesting UIs on it, and all of this is happening on the player side.

It's all within the document.

Everything you see, no going back to the server for this.

Now, if you built it within an application, in this case, the blue parts are an application and the black parts are a Remote Compose document.

You can actually interact between the two of them.

Here, I'm incrementing values, and you're seeing them being received by the application, what we call the host, and I can also move a slider, and it will change the values inside the document.

So, I can go both directions for that.

So, let's dive into how to create one of these documents.

There are really three APIs: Compose, Kotlin, and JSON.

The Compose API, you kind of know what it's like.

It's It has a few twists to it and we'll go go through that in detail.

The Kotlin API is very procedural.

It You sort of one command in the document, so it's sort of one-to-one.

The JSON is a JSON syntax that you can We have converters that will convert it into Remote Compose.

It's also useful if you're working with an LLM because they are able to read the documents and write the documents.

So, let's actually go into the Compose and build something.

In this case, you'll see that you're It's a composable and a remote composable.

We have a canva-- not a canvas, but a remote canvas and remote modifiers, but the modifiers themselves are kind of the same, the things you're you know and love.

Now, all the commands inside of it are actually generating remote command Remote Compose document commands as it goes through.

In this case, drawing a circle and then adding some floats and expressions using time in those expressions.

And when I go and turn it on in animation, notice this is a live preview from the Android Studio and it actually will run live for you.

You see those expressions there.

Let's kind of get into a little bit of what that's all about and how it kind of works.

On the left, you'll see that there's a sort of what it looks like a spreadsheet.

It's a table of the values in floats.

And it also has a dependency tree, so you know where one thing connects to another.

Whenever something changes, its dependents are marked as dirty to be cleaned up, and to be ad-- adjusted.

And as it runs, it'll clean up all of their dependencies and only evaluate the ones that change.

So, it's very efficient and in a case where let's say a object like the size changes, then that dependency changes, and it has to reflect it.

It makes it a very, very efficient way of computing, cuz you're you're not You're going top to bottom, and you're going very efficiently.

So, let's kind of expand on that clock a little bit just to show that we could build real things, slightly real things.

In this case here, what we're doing is we're going to add some tick marks on the edges, and you'll see some for loops, so you can actually sort of for loop in it local and right.

Then, we add some hour hands and minute hands, and then go go on to the sec-- add seconds, then cap it off with a couple of different buttons just to make it to finish it off, and we have a live clock.

You know, it's actually showing the time.

The other interesting is you can write that same thing in JSON, and if it was in JSON, you can easily interact by with an LLM saying, "Change the minute hand to green," and it would kind of figure out what you mean.

So, it's very nice that way.

Okay.

Right.

So, hopefully, that gives you a sense of the capabilities of Remote Compose.

Let me cover a little bit the developer support we have for you, right?

So, I mentioned those are principally AndroidX libraries, so everything that is on DAC, you know where to find it.

You'll get the latest releases, the documentation, and all of that.

As usual, we saw John demonstrate what we can do in Android Studio.

As I think the coolest thing for me is really the live preview that really accelerate the development time.

One thing we are working on as well is an actual public kind of open specification for the format, right?

We are not trying to have something that's completely hidden, it's out there.

You can see how it works, how it's actually built and encoded.

One of the common things that you are using your applications are animated vector drawables, and, you know, it's very convenient clearly.

And remember what I was mentioning about bitmaps.

We are very interested into you sending vectors and not bitmaps, right?

So we are building a couple of tools to help you take a Lottie, for instance, possibly other format, and convert it into a RemoteCompose component that you can use in your RemoteCompose file.

I mentioned earlier that we're also working on additional tools, and specifically on and the live like a inspector for the documents because it's one thing to have the developer experience to create, but it's another to be able to inspect the result of that creation.

And in our case, that's a document.

So, here's a quick sneak peek, a preview on the tool we're working on.

It's a RemoteCompose document inspector, and it lets you inspect the documents as the name says, and apply several useful tools and visualization to it.

One of the nice example is you can open a document and execute the different operations that are in the document step-by-step to try to better understand how everything ties together.

But we have additional tooling.

For example, I really like this one, that immediately visualize your your document in different, you know, densities and sizes, so you can very quickly see if what you're building is going to adapt nicely to the different changes in sizes, for instance.

One of the latest things we're working on as well is a really full-blown layout inspector, so, the type of layout inspection capabilities you are used to in Android Studio, but here, applied to the operations and to the documents, right?

And we can go even further in terms of 3D.

I mentioned earlier, and hopefully by now, I convinced you that small is a feature.

It's important.

We really want small documents.

It helps everyone.

Please do it.

So, what can we do for that?

Let's take a common example of UI, right?

Something where you got a bunch of rows.

The rows themselves are, you know, you got a name, a little graph, and maybe a button.

Pretty typical.

The size of that UI is already pretty small.

We are talking about around 25 kB, but we have various functionalities in RemoteCompose to actually help you make it even smaller.

So, let's go through them.

First thing we do and this one, it's for free, we'll do it for you, if you are sending the same strings, we notice it.

We can shave off some bytes by pointing to the same string.

But we provide more stuff, like loops.

So here, I used this loop operation to draw my little graph, right?

And that saves a few more bytes.

We have functions as well.

So, you could create a function, and here, I'm doing that for that graph again, and I can then simply call that graph multiple times.

And again, that's going to save me some bytes.

And for component, we have templates, right?

Similar ideas as for the functions, but more for component, right?

And so, I can replace all of those different components, all of those different rows, by simply invoking that template with the correct parameters, and that saves a lot on the document.

And the last thing we support out of the box as well is compressing the document.

Generally, you can expect like to save about half of the size of the document this way.

One interesting thing is that the header of the document is a bit special.

We don't compress it.

That's very handy if you want to very quickly sift through a ton of documents without having to decompress them.

So bottom line, documents they are small, you know, out-of-the-box, but you can make them a lot smaller as well.

And you can see, with this little example, how easy it is to literally reduce it by an order of magnitude, as small is a feature.

I talked about widgets and specifically launcher widgets in the Android platform.

So let's cover a little bit what what we are doing and what's already out there.

So the first thing we can do is we can replicate an existing widget, like that clock, purely in RemoteCompose, and it would essentially behave the same, just in RemoteCompose.

But we can do more, right?

So if you want to do more adaptive UI, widgets that resize and correctly animate, we can do that now, right?

Because we got the right primitives to do this in contrast to what you could do with remote views.

We give you a lot more flexibility and power, essentially.

And talking about power, here's a quick example of crazy stuff you could do, right?

This shows you a full game in a widget.

And I would point to the actual size of that widget.

We're talking about 4 KB.

I mentioned that transport is not something we directly deal with in Remote Compose.

Although for widgets specifically, we do provide a RemoteCompose Widget connector class that let you very easily take a document and send it to launcher.

That said, if you already have widgets, we encourage you to continue using Glance, because the latest version of Glance provide automatically a back-end to emit Remote Compose on the platform.

And now, to talk about Wear OS.

Thanks, Nico.

What's up, everybody?

I'm John Zoeller.

I'm here to talk to you about Wear OS.

I wrote like 30 jokes for this presentation, but I told them all to my mom, and she said I would get fired if I told any of them.

So, I'm going to skip that and just do the presentation instead.

So, let's talk about Remote Compose and some of the great advantages of it on Wear OS.

First up, we're going to talk about the new Wear OS Workout Tracker.

Now, we used Remote Compose to build this, and we know that building a full-featured, high-quality workout tracker is really difficult for Wear.

It takes some time.

So, we kind of did it for you, and we included some really nice new modern workout features, such as live metrics and being able to control media.

And thanks to Remote Compose, we were able to decouple the UI from all the execution, so it is able to run very efficiently and helps not to drain users' batteries.

Without Remote Compose, if we did this with just normal Compose, we wouldn't be able to have live metrics refreshing on the screen on a second-to-second basis.

It would take actually a minute, because you would have to wake up the watch in ambient mode just to show a metric.

So, this is a great advantage.

And, actually, this is a GIF, believe it or not.

This is not a picture.

I know it hasn't moved yet, and it'll move in about 45 seconds when we get to the next metric update.

But that's the advantage of Remote Compose.

And guess what?

It's actually available, Wear Workout Tracker is.

The ASICS Runkeeper app built their new application with it, and you can try it out today by following this link and joining their open beta test.

Next up, we have Wear OS Widgets.

This is another great example of Remote Compose in action.

These replace the old ProtoLayout Tiles library, so you no longer need to support two different types of UI frameworks for different devices.

You can do it all with Remote Compose.

And because they're part of Jetpack Glance, widgets on Wear OS are part of the unified Android ecosystem of widgets.

All right, and as part of this ecosystem, RemoteCompose widgets' UI are completely decoupled from any specific app process, so they can be safely serialized and rendered on Wear OS devices using the same technology as the rest of the Android stack.

And Remote Compose also brings modern graphics and responsive layouts to widgets, and it does this all while being very efficient and good on your battery.

And with Wear OS 7, hardware manufacturers are already adopting vertically scrolling trays.

So, this is the Samsung Galaxy Watch, and you can see how widgets are vertically scrolling through the trays here.

If you'd like to learn more about migrating your tile to a widget, you can check out this link.

On older devices that rely on the horizontal carousels for tiles, widgets will automatically be displayed as a full-screen tile.

And, also, they are fully backwards compatible to Wear OS 4.

And, also, this link will teach you more about that backwards compatibility.

And we have a few other partners who have also adopted widgets already.

You can take a look.

And that's it for me.

Yep, back to you Nico.

Thank you.

All right.

So, that does cover our talk.

We reviewed some of the capabilities of Remote Compose.

There's there's a few more that we didn't talk about.

It's a very rich framework.

I encourage you to take a look at the available documentation and the upcoming documentation.

I would say our aim is really to build like a future where you could really use Remote Compose to address all those different remote surfaces.

And you saw some of them, right, like the wear, widgets.

But there is a lot more coming up.

I would also say this is an ongoing project.

It's in active development.

We, you know, we develop all of that in the open.

You can see what we're doing.

But, I would say, we really want to hear from developers, right?

We are building that for you.

So, if you have additional ideas, things you are missing, talk to us.

We are more than happy to hear from that.

I would finish this with this, the entire presentation you just saw, obviously, is done in Remote Compose, animations included.