Mastering on-device intelligence with ML Kit and LiteRT in Google Play services

Good evening, everyone.

We are about to begin the talk.

So, when people say "on-device AI" today, they often mean LLMs.

And earlier today, you saw several talks about Gemini Nano that were covered with ML Kit, Firebase, and many other of Google's products.

This session is about the other half: classical ML and the model you bring yourself.

You've probably used it a few times today without even knowing it.

Think about using any of your apps where you blurred your background or used a barcode scanner or use some kind of image editing.

A lot of that has been on classical ML, and typically even runs on-device.

And if you've ever added any of these features to your own app or APK, you've often hit this trade-off: do I basically use a turnkey API, or do I bring my own model?

What's going to be the impact to my app size, right?

Today, you'll see you don't have to choose between a turnkey or bringing your own model.

You can actually do both with one lean runtime.

My name is Chintan Parikh.

I am product manager for LiteRT, and with me is Fergus, engineering lead for Android ML services.

Hi.

And by the end, you'll see how you can build one app running two models on every camera frame at about 30 frames per second.

So, there's four parts to this talk.

Part one is we're going to talk about Google's on-device AI solutions focusing on classical ML, that is ML Kit and LiteRT.

Second, we'll go over what's new in LiteRT and Play services as well.

This is also going to cover the new CompiledModel API from LiteRT and the devices we support.

Three, we'll show you how to build a new app.

Fergus is going to go over building an app with ML Kit and a custom model all in one APK.

And finally, we are going to show you how to upgrade an existing app using Agent Mode.

And also, you could even scaffold a new app with this agent support.

All right.

So first, let's start with, what is LiteRT?

LiteRT is Google's on-device inference runtime for your own models.

It uses the same .tflite file format that you if you've used before with TensorFlow Lite.

It's reengineered for performance and also supports multi-framework.

What that means is you can bring in models from PyTorch, TensorFlow, and even JAX to be able to lower it down to flat buffer format for LiteRT.

Now, since 2022, TensorFlow Lite in Google Play services has been Android's official on-device runtime for custom models.

Today it gets a major upgrade, and that's what a lot of this talk is going to talk about.

The upgrade is going to be bringing the latest LiteRT CompiledModel API, including new improved GPU inference that is going to be 1.4x faster, and also enables you to build smaller apps in much more leaner APK sizes.

So, why deliver it through Play services?

There's a couple of advantages, and this is definitely a big pain point that I'm seeing with a lot of talks that I'm hearing where app size and APK bloat is becoming a real challenge, right?

So, gets you a small runtime.

Runtime is not in your APK, so you rely on a thin client, and that makes your APK about 8 MB smaller.

The second benefit is efficient.

You have one copy of the runtime, so the runtime being LiteRT runtime is shared across all the apps.

So, you can imagine if you have numerous apps, they all could be saving you 8 MB each.

That adds up really fast when you're trying to save disk space and have a leaner app.

Second, it's maintained.

Google ships the bug fixes, it addresses the GPU driver challenges that you won't have to do, and also any other security updates are all coming in via automatic updates.

And as I mentioned, the GPU is faster, so you get 1.4x faster GPU, and we will go into that in more detail as well.

And from a coverage standpoint, we can ship to older Android devices.

LiteRT in Play services supports all the way back to Android 7 from 2016.

And as Fergus will show you also, the other new devices that we are expanding to in the slides to come.

So, this is the delivery vehicle, and let's talk about the next step, the runtime.

So, LiteRT isn't new, and it's not experimental.

TensorFlow Lite and LiteRT already run in over 250,000 apps today.

It also runs on over 3.8 billion devices today.

So, a lot of the apps you may be using, you may not even know, it all runs this.

And this also equates to over a trillion inferences every day, right?

At that scale, we've certainly hit every odd chipset issue, we have been able to address ROM issues or GPU driver issues, and we have a workaround it just for you.

LiteRT is also what Google runs on its flagship apps as well, across apps, platforms, and even several devices.

YouTube Short effects, Meet background segmentation, Photo Unblur, are some examples where real-time video and heavy image models all run on LiteRT.

This is also extremely important as making use cases snappy and engaging to users becomes extremely critical, right?

So, this is the same runtime that we have hardened for these flagship features, and this is what you get through Play services.

So, let's also touch on why go on-device at all.

Latency, privacy, working offline, or having low costs, if any of these matter to you, run it offline, right?

There may be even more reasons, but these are some of the most obvious reasons that that that we have heard for running on on-device.

Running on-device is also not always free.

It does have some nuances to it and some challenges.

So, we'll also go over some of these challenges and have a path forward for you as well, right?

So, performance: As you saw, right, some of the examples from the previous apps, latency and memory budgets are always tight, right?

Real-time video and audio needs extreme efficiency.

Even simple interactive features need to be snappy.

Otherwise, you lose user engagement.

Hardware diversity: There are hundreds of SoCs out there with different CPU configurations, GPU configurations.

You can't write code for all of these, right?

How do you solve for any of that?

APK size: Bundling a native runtime also makes every download bigger that can impact your conversion rates.

Customization: Model training, quality, model size, these are all going to be extremely critical before you even deploy.

So yes, building a great on-device app can be tough.

But in every one of these situations, we do want the platform to do the work for you and not the other way around, right?

And Google's on-device portfolio exists to address these challenges, and we're going to go over that next.

So, we'll focus on two simple solutions today, right?

There's two easy ways to add on-device AI via Play services.

Path 1 is the turnkey path using ML Kit, right?

These are ready-to-use vision, language APIs, limited ML expertise needed, many models are all provided and all delivered through Play services.

And we will go into some examples of these as well in the next slides.

The custom path, this is through LiteRT.

LiteRT allows you to bring your own model, run it on CPU, GPU.

The runtime comes from Play services, too.

So, it's in your-- so, you don't bundle it in your APK.

These aren't either-or.

They can always be better together as well.

And that's exactly what we'll build today in the next couple of minutes.

So, but first, let's touch on what ML Kit covers.

Here's what ML Kit covers out of the box.

So, if you are already thinking some of these capabilities meet your need, that's a great starting point, and you are already, have a head start.

We have coverage for vision, for barcodes, documents, text, pose, many of these capabilities, language APIs like translation, smart replies, and even entity extraction, right?

The key point is many of these vision, language APIs all run with LiteRT under the hood.

So, if ML Kit does what you need, you can start here.

And when it doesn't, you have the opportunity to bring your own model.

And here's the key point: ML Kit and your custom model are not going to be two runtimes, right, as I mentioned.

They're going to be, think of it as two front doors leading to the same LiteRT that'll be running under the hood.

So, LiteRT will be the Play the runtime that will be orchestrating both of these capabilities.

So, if your app already uses from Play Store, for ML Kit, go for it.

Otherwise, we will go over the LiteRT path as well.

So, let's also touch on what's new with LiteRT.

As I mentioned, LiteRT is an evolution of TensorFlow Lite, right?

So, the biggest improvements are we have a faster GPU.

The faster GPU supports OpenCL, OpenGL, and also has cross-platform support even for Metal backends.

So, you can certainly use this for multiple platforms as needed.

It's also simpler.

We have one unified workflow for CPU, GPU, and NPU acceleration.

CPU and GPU are currently in Play services, whereas NPU is bundled through LiteRT today, and that is work in progress.

So, previously, each accelerator meant a different delegate setup, right?

You only have to do it once now.

So there's one API with automatic fall back to CPU, and you do not need to do dedicated-delegated setups.

And you can just have accelerator calls that can manage this much more simply and with less code.

It's also powerful, the same model runs on multiple platforms, as I mentioned.

This is going to be critical, the flexibility part of it.

So the .tflite format stays the same.

If you had an existing app, you can still use this format.

If you're building a new app, you can carry over this file format once again.

What's also interesting is that we support the Interpreter API, which used to be in the TensorFlow Lite.

So if you are looking to migrate in phases, the Interpreter API is included, part of the new LiteRT as well, so that makes it easy for you to migrate.

So that's the runtime.

Let's talk about what you can build with this.

So here's what Android apps are building today.

Everything from camera effects, photo editing, smart typing, games and avatars, right?

A lot of these apps are going to need real-time capabilities.

The GPU is sending frames directly from the camera, getting the things to be snappy and real-time will be important.

Some of these photo editing models are also getting heavier.

And same with the AR/VR capabilities that you have also seen down in some of the demos with the glasses, a lot of them all run LiteRT under the hood.

It's the same runtime now behind Google's flagship features, and also in Play services, so you get all of this without bundling anything to your APK.

Now why don't we build an app, and Fergus will show us how.

Let's do that.

Thank you, Chintan.

So, I'm going to talk a little bit about what's new in LiteRT in Google Play services.

So just before we build, one more piece of news, everything you just saw now runs beyond just the phone, LiteRT in Play services is also available on Wear OS and Android Automotive OS.

So you can have on-device models running in your watch or other wearable device, or in the car itself.

Previously it was possible to use an app on your phone that would be a companion app to Wear, but now you can have it actually running on the wearable device itself, and the same with Android Automotive.

And the so you have the same API, the same .tflite model, but more surfaces.

There's no need for a separate implementation per form factor, you can just use the same implementation for all these different devices.

And then the beauty of the Play services distribution paradigm is that we can actually backport this to run on older devices.

So for Android Automotive OS, we support as far back as Android 11, released in 2020, and for Wear OS, we're going from Wear OS 4 and later.

Okay.

So, now I'm going to talk about two different paths, building a new app and upgrading an existing app.

That that's going to cover the rest of the talk.

So, I'll start with an example app that combines the use of ML Kit with a custom GPU model running on LiteRT.

And then for upgrading an is- existing app, there'll be three small changes, a Gradle change to update the dependencies, an init change to enable LiteRT in Play services, and then switching to the from the Interpreter API to the CompiledModel API.

But we'll do that the new app first, because the upgrade makes a lot more sense once you've seen what you're upgrading to.

So, building a new app.

Here's the app we're going to build.

It's going to have a live camera view, where you stay sharp, and the background is going to get an artistic style.

So, we're going to take the camera frame.

The images from the camera frame, we're going to stream that through ML Kit selfie segmentation.

That's one of the existing ML Kit turnkey APIs.

And that's going to produce a foreground mask that will basically say which of the pixels in the image correspond to the person in the image.

And then we're going to take our custom model, which is like a style transfer type model, it's going to apply a style to an image.

And we're going to apply that to the whole of the camera frame, but it's really just the background part that we are concerned about, because in the end we're going to take the original camera frame and the mask, and the style background, and with a little bit simple logic combine those, so that we have a composited frame where we kept the subject, and the background has the style.

So, we want to have a We're making use of turnkey APIs where they exist, and custom APIs where they differentiate our app.

So, this is that the first part where we invoke the ML Kit selfie segmentation API.

// 1. Configure and create segmenter client
val segmenter = Segmentation.getClient(
    SelfieSegmenterOptions.Builder()
        .setDetectorMode(
            SelfieSegmenterOptions.STREAM_MODE)
        .build())

// 2. Prepare image & run inference
fun processFrame(mediaImage: Image) {
  val image = InputImage.fromMediaImage(
      mediaImage, /* rotationDegrees= */ 0)
  segmenter.process(image)
      .addOnSuccessListener { mask -> … }
}

So, this is just like a regular API, right?

This is just like the bread and butter for Android developers.

You know, you just you look at the API docs or you just ask Gemini to build it for you.

But, you know, it's fairly straightforward.

You construct a client you pass in some options when you construct the client.

In this case, we're specifying streaming mode so that we can process a stream of video frames.

And then, there's a you call this process method there, and the callback from that is going to get the mask that's is returned, computed by selfie segmentation.

And the input is the image from the camera.

And all these APIs using application-level types, not raw tensors, so you're really programming at a very high level and that's really straightforward.

For that, the second part, the custom model, we're going to take a PyTorch model.

I think Chintan got this one from from one of the things like Kaggle or Hugging Face…

Right.

Yeah, and or you can convert your own, you know.

You can build your own model.

Sorry, going back.

Yeah, so this is what this the sort of stack looks like.

You take a PyTorch model, you can convert and quantize that, and then you get a LiteRT model.

You then you distribute that with your app, however you like.

It can be bundled as part of the app as an asset or can be downloaded separately.

Either way works.

And that that then sits on top of the LiteRT runtime in Play services.

And then, there are CPU kernels for optimizing on CPU, and ML Drift is the name of the GPU accelerator.

Yeah, so as I said, you get the TFLite model you can either train the model yourself, take an find an existing model, either either way it works.

Converting from PyTorch is just a few lines of Python.

import litert_torch

model = my_models.Stylizer().eval()

# Convert with torch.export under the hood
edge_model = litert_torch.convert(
    model, sample_inputs)
edge_model.export("stylizer_f32.tflite")

So, you pip install LiteRT-Torch, you call litert_torch.convert, pass it some sample_inputs that can be random inputs, and you export a TFLite file.

By the way, LiteRT-Torch is the new name for ai-edge-torch, if you're familiar with that.

And then quantization is, again, very straightforward.

# Python
# previously ai_edge_quantizer
from litert_quantizer import quantizer, recipe

qt = quantizer.Quantizer("stylizer_f32.tflite")

# int8 weights, float activations
qt.load_quantization_recipe(
    recipe.dynamic_wi8_afp32())

qt.quantize().export_model("stylizer_int8.tflite")

You import the quantizer and the recipe from LiteRT Quantizer.

You when you construct the quantizer, you pass in the original floating-point model, you pass in a recipe that this is that that's the standard recipe we use for we recommend for GPU, which uses 8-bit integer weights, but keeps the activations as 32-bit floating point.

And then, you just call quantize and then export_model to save that save the output model.

And that that'll be a quantized model that you can then run with and that'll be a much smaller weights, and will execute more efficiently.

Okay, so yeah, the CompiledModel API the GPU is where that really shines.

So, as Chintan mentioned, you know, a significant speed up on legacy TFLite GPU.

The exact numbers will vary a bit from model to model, but 1.4 times is what you can reasonably expect.

Up to 10 times faster than CPU inference, so that's the, you know, the big win there, right?

You if you are mo- most of you who are using models today are probably running them on CPU, and missing out on like a factor of 10 in performance by you can get from GPU.

So, that's a real incentive.

What makes it fast?

Okay, so there's a bunch of things that we support, for example, asynchronous execution so you can take a an AHardwareBuffer and pass that directly or an OpenGL buffer from that camera render pipeline, pass that straight to a tensor buffer.

Zero-copy optimize zero-copy we support, so basically you can when you're trying to process things, you know, with a low-latency budget like when you're having to process every frame in a matter of, you know, a few tens of milliseconds, then even just simply copying a memory buffer can be a non-trivial overhead.

So, for those really high real-time requirement applications you want to use the zero-copy optimization, we support that.

And then, as, as Chintan mentioned, it's cross-platform with support for OpenGL, OpenCL, and Metal, as the GPU back ends.

Last but by no means least, the ML Drift optimizer is open-source.

It's available under Apache license.

It's recently been open-sourced by Google.

Okay, and then, here's, here's the app code.

// Kotlin
// Build once, off the main thread.
private val model: CompiledModel =
    CompiledModel.create(
        context.assets,
        "stylizer_int8.tflite",
        // Simple GPU setup!
        CompiledModel.Options(Accelerator.GPU),
        environment
    )

// Reuse buffers across frames.
private val inputBuffers  = model.createInputBuffers()
private val outputBuffers = model.createOutputBuffers()

fun stylize(input: FloatArray): FloatArray {
    inputBuffers[0].writeFloat(input)
    model.run(inputBuffers, outputBuffers) // sync
    return outputBuffers[0].readFloat()
}

As you can see, it's quite small and straightforward.

So, you create the model once, and then you reuse it in every frame.

So, there, CompiledModel.create passes in the .tflite file.

You just specify that you want to run on GPU, and then you construct the input and outputBuffers.

And then, in the main loop, where you're actually processing every frame, you call this model.run passing the input, output.

After you write to the inputBuffers, call model.run, and then read from the outputBuffers.

Okay.

Oh, yes, and then in the, if you just specify GPU there, it will run on the GPU.

If you also specify CPU and GPU there, it'll try running the GPU.

If the GPU is not available for whatever reason, then it will fall back to CPU.

So, for some cases, the CPU fallback might not be efficient enough for your app, and that's when you want to use just specify the GPU like that.

More commonly, you want to have a CPU fallback.

Okay.

So, we've built it with bundled LiteRT, that's not, not yet using Play services.

So, let me show you how to use LiteRT in Play services.

So, there's three changes: the first one is the dependency change.

// Easier with CompiledModel API: 3 steps

// 1. app/build.gradle: replace litert with play-services-litert
- implementation 'com.google.ai.edge.litert:litert:...'
+ implementation 'com.google.android.gms:play-services-litert:17.0.0'  // also pulls in litert-api

You just replace the, in your Gradle file or wherever else, whatever build system you're using, wherever else you specify the Maven package dependencies, that's what you're going to change.

And you remove the litert dependency or any of the older TensorFlow Lite Maven package dependencies, and replace those with this play-services-litert.

That also pulls in the litert-api package.

Now, if you are writing a middle layer that just wants to depend on LiteRT, but doesn't really care whether the run time is coming from the app or whether it's coming from Play services, you can just depend on LiteRT API, and take an environment as an input parameter.

So, the second step is the environment parameter.

That's where you provide the dependency injection, essentially, to specify that you are using, you want to use Play services LiteRT.

So, you import the gms.litert.LiteRt, and call the initialize function, passing in the context, which is your Android activity.

// 2. Initialize the LiteRT Environment from Play services (context = your Activity)
- val environment = Environment.create(...)
+ import com.google.android.gms.litert.LiteRt
+ val environment = LiteRt.initialize(context).await()

// 3. Pass environment whenever you call CompiledModel.create

// That's it! The rest of your app uses the normal CompiledModel APIs from litert-api.

And that's an asynchronous thing, so you have to call .await or have the rest of it be based on a callback.

And then, lastly, the once you've got that environment, you then just pass that environment whenever you call CompiledModel.create.

And that will be what makes it use Play services LiteRT runtime from Play services.

And so, that's it.

That's the rest of your app uses the exact same LiteRT CompiledModel APIs from LiteRT API.

The rest of your inference code doesn't change for using Play services.

And you can actually do that migration incrementally.

You don't have to do it all at once, you can migrate one component at a time.

If you have a large app, it makes sense to migrate each component that's using LiteRT, and then you can migrate one at a time, test one at a time.

The only caveat is that you only get the size win, the space savings only land when you remove the last dependency on the bundled runtime.

And, if you are, the other caveat is that you shouldn't mix objects between LiteRT in Play services and bundled LiteRT.

So, if you-- if you're getting tensors or compiled models or interpreters from the LiteRT in Play services, you shouldn't pass those to the bundled LiteRT or vice versa.

But otherwise, mixing the these things is works just fine.

Okay.

So, that was building a new app.

Now I'm going to talk about how to upgrade an existing app.

So, you might have an existing app that's using the Interpreter API.

This is the This is what we call the older TFLite API that was that uses interpreter as the main class.

So, that's the code on the left using the Interpreter API, and then on the right, you see the new CompiledModel API code.

// Dependencies (Gradle):
// play-services-tflite-java and/or litert (bundled)

// Init
TfLite.initialize(context).await()

// Load
val interpreterOptions =
    InterpreterApi.Options().apply {
      runtime = TfLiteRuntime.FROM_SYSTEM_ONLY
    }

val interpreter =
    InterpreterApi.create(modelFile, interpreterOptions)

// Prepare inputs/outputs
val inputs = …
val outputs = …

// Execute
interpreter.run(inputs, outputs)

// Dependencies (Gradle):
// play-services-litert

// Init
val environment =
  LiteRt.initialize(context).await()

// Load
val modelOptions = CompiledModel.Options(Accelerator.CPU)

val model =
    CompiledModel.create(modelPath, modelOptions, environment)

// Prepare inputs/outputs
val inputBuffers = model.createInputBuffers()
val outputBuffers = model.createOutputBuffers()

// Execute
model.run(inputBuffers, outputBuffers

So, you're just changing TfLite.initialize to LiteRt.initialize, and that now returns an environment.

And you then pass the environment to CompiledModel.create.

So, that's a small change, no APK growth.

And you'll see, on the next slide, we'll show you how to do it with GPU, and that's just one more option.

So, this is the additional code you needed for this to enable GPU.

// Dependencies (Gradle):
// play-services-tflite-java and/or litert (bundled)

// Init
val initOptions = TfLiteInitializationOptions.builder()
    .setEnableGpuDelegateSupport(true).build()
TfLite.initialize(context, initOptions).await()
val useGpu =
    TfLiteGpu.isGpuDelegateAvailable(context).await()

// Load
val interpreterOptions =
    InterpreterApi.Options().apply {
      runtime = TfLiteRuntime.FROM_SYSTEM_ONLY
    }
if (useGpu) {
    interpreterOptions.addDelegateFactory(
        GpuDelegateFactory());
}
val interpreter =
    InterpreterApi.create(modelFile, interpreterOptions)

// Prepare inputs/outputs
val inputs = …
val outputs = …

// Execute
interpreter.run(inputs, outputs)

// Dependencies (Gradle):
// play-services-litert

// Init
val initOptions = LiteRtInitializationOptions.builder()
    .setEnableGpu(true).build()
val environment = LiteRt.initialize(context, initOptions).await()

// Load
val modelOptions =
    CompiledModel.Options(Accelerator.CPU, Accelerator.GPU)

val model =
    CompiledModel.create(modelPath, modelOptions, environment)

// Prepare inputs/outputs
val inputBuffers = model.createInputBuffers()
val outputBuffers = model.createOutputBuffers()

// Execute
model.run(inputBuffers, outputBuffers)

As you see, the code in the new API for enabling GPU is a little bit simp-- yeah, simpler than the code previously.

So, the first call to a setEnableGpu in LiteRT initialization options that you pass then to LiteRt.initialize, that will cause the code for the GPU accelerator to get loaded into your app.

And then, when you construct the modelOptions, and pass those to CompiledModel.create, that's where you specify which accelerator will be used for it for a particular model, whether you want other model to run like only on GPU, or GPU with CPU fallback, or just on CPU.

All right.

Yeah, as you can see, enabling GPU is a lot simpler with the new API.

So, what do you actually gain?

Chintan will tell you.

All right.

All right, So, thanks Fergus.

So, let's talk about what does all of this get us, right?

So, remember we said we're going to build an app, so let's see what we actually built, right?

So, that's me, anyways, and what we've done here is what's happened now is we've used ML Kit is used to find me, so the segmentation is being done by a turnkey API like ML Kit.

So, that's just finding my head over there.

And then LiteRT uses a style another model similar to style transfer to paint the background.

And the other benefits as we mentioned are you get approximately 8 MB in space savings, you have one runtime now that is used between all your apps, or at least like the that that is shared here, and you get automatic updates, and you also have a faster GPU that gets you close to 30 frames per second on something that needs to be real-time for a rich user experience.

What's also good is you don't have to do all of this by hand.

There is also a new skill that we have just launched in Android is, basically, we've published an Android LiteRT skill that upgrades your on-device ML in minutes, right?

So, you could do a couple of things with this: you could upgrade an existing app, you could scaffold a new app, you can also upgrade your app to use this new Play services integration as well.

So, all this is available.

The link is on the slide, it's also on GitHub, and it'll show up in our documentation and our blog as well.

And yeah, so that's that should be useful as well for anyone who is looking to make the change soon.

So, Fergus will then tell us where this is going next.

So, we're committed to on-device AI with LiteRT.

We continue to invest significantly with improving the GPU acceleration and model coverage, and continuing to provide regular updates for LiteRT 2.x, both for the open source bundled LiteRT and also for the Play services LiteRT Maven package that is LiteRT in Play services.

And we're also going to continue to provide backwards compatibility.

The Play services runtime will keep supporting apps built with older SDKs.

Yeah.

You know that's that's really important for our users, that's really important to us.

For the LiteRT 1.x SDKs, it's those are in maintenance mode now, so critical security fixes only.

And NNAPI, which we deprecated back in 2024, is going away, and that's likely to be removed starting in 2027.

So, if you have any uses of NNAPI, please switch those over to using GPU acceleration or NPU acceleration with bundled LiteRT.

Okay.

Yeah.

So, the sooner you move to 2.x and CompiledModel, the less you'll have to do later.

So, three things to take away.

So, build.

Start with ML Kit, and if that already does what you need, then that's great.

But if it doesn't, then use LiteRT for your custom models.

Upgrade to LiteRT in Google Play services for the APK savings and the shared runtime.

And consider using the CompiledModel API for faster, simpler GPU inference.

Yeah.

Everything's documented at our LiteRT documentation site on AI Edge at the URL just there.

So, yeah.

Thanks for joining us, and we can't to see-- wait to see what you guys build.