Inside Gemini Nano and Gemma 4 powering on-device intelligence

Sara Smoot

Hello, welcome welcome to the talk.

My name is Sara, and I am an engineering lead on the Gemma on the modeling side.

So what I'm here to talk to you today about is our models.

I'm going to talk about capabilities, how we think about designing for mobile.

I'm going to talk about the techniques that we build into the model to make them run efficiently on mobile.

I'll go through how you can access that and how we integrate with AICore.

And then, I'm going to end with some demos just to show off the model and give you an idea of what the model is capable of and what it can do.

So to understand where we're going, it helps to understand where we've been.

We launched Gemma 1 in 2024.

It was we launched a 2 billion and 7B parameter model, text only, very very small sizes compared to what was going on at the time.

Following Gemma 1, we launched Gemma 2, which introduced some additional capabilities.

We introduced coding.

We also introduced a large-size model, a 27B, which really expanded the capabilities of the Gemma series.

Gemma 3 launched a short while later.

Again, we added more capabilities, multimodal, and then a smaller model, a 270 million parameter model, as we realized that smaller models really had a lot of use cases and we're really trying to help enable as many online mobile use cases as possible.

So with each release, our capabilities are going up, and that's intentional.

What we're doing is we're designing for single-device hardware, and we're trying to I mean this includes phones, laptops, GPUs, and our goal isn't making a bigger model, although we did go up to 12- to 27B, our goal is to fit more capabilities with each iteration onto the same footprint, because we want to stay on-device, we want to stay local.

So the way we've done that is by building in better efficiency.

And I'm going to talk about what we've done for Gemma 4 specifically and how we did this.

We launched the E2B, I'm going to explain that "E" in a minute, we launched E4B, we've launched a 12B unified model, explain why we did that one, and then we also launched an MoE model.

All to help prove efficiency and give developers more flexibility for the apps and things that they want to build.

So looking at the capabilities a little bit more closely, here you can see the capabilities that we've added over time.

Couple things that I want to point out, the context length.

We've added we have a much longer context length than in previous Gemma versions.

Gemma 4 has up to 128 K context.

The larger models go up to 256.

I will note that ML Kit does put a cap on the context length, so make sure you check your what your budget is before you expect that you can launch with 128.

But this should be enough to let you read long documents, let you do some expanded reasoning, function calling, and also give you enough budget to do multimodality, use use the vision and audio.

Video is a new capability that we have added in, starting Gemma 3n, actually, and we've leaned into it a little bit more for Gemma 4.

So now we have multilingual support.

We're still training on multiple multiple languages, so you have both built-in language support, and you have pre-trained support.

And all of this is coming in now a brand-new Apache 2 license, which we're really excited about, because this is going to give a lot more flexibility with what you can build with it, and also just makes the it makes the friction a lot less.

It's much more seamless to build and download the model.

Sara Smoot

Okay, so vision, what do we do with vision?

Vision for this for Gemma 4 specifically, it has variable aspect ratio.

So what that does is it takes an image, and rather than doing some squishing and squishing it into an awkward square, we preserve the actual aspect ratio of the model.

So this means your text isn't getting distorted, objects aren't getting distorted.

It actually dramatically improves the capability of the model.

Shapes stay true, OCR is improved a lot with this variable aspect ratio.

The vision also is able to read charts, it can read spreadsheets, it can read graphs.

It can also translate from visual cues, so you can take pictures of words, and it'll be able to translate those for you.

It also reasons about the physical world.

It can look at a picture, it can understand where things are, it can understand what's in the image.

It can do quite a lot.

And the other little-known feature about our Gemma models is you can set set the visual token budget.

So, right now, it I mean check and see what it is in ML Toolkit, but you have like a large range.

If you want to spend more of your power and energy on higher resolution image understanding, you can set that.

If you If that's not a use case that you want to really leverage, you can reduce the image token budget by quite a lot and reclaim a lot of that power and the You know, in the model compute cycles for your your text applications.

Audio is a relatively new mode for us.

We first introduced it in Gemma 3n.

We are now fully leaning into it for Gemma 4.

Audio is only in our smallest models: our E2B and our E4B and our 12B model.

It will transcribe fully based off of the audio file.

It translates speech.

It supports, out of the box, 35+ languages.

We also have pre-trained it on over 140 languages, so if you want to be able to, for a language that's not one of our 35, you should be able to do a fine-tune and get really nice support for a language that isn't supported out of the box.

We've added agentic reasoning.

This is also new to Gemma 4.

It plans.

It does multi-step.

It uses tools.

It follows instructions precisely.

It returns structured JSON.

It can do all of the things that, you know, you expect out of an agentic agentic model.

I will say watch the Make sure you're using If you're doing If you're not using ML Toolkit, you're doing things by yourself, make sure you're paying attention to the chat templates.

Chat templates are tricky.

They need to be handled with care.

If you're using ML Toolkit, all of this is done for you.

And lastly, Gemma continues to deliver great text understanding.

And with the longer context, we are now able to analyze code, long documents, translate.

In fact, with the coding functionality, Android Studio now supports Gemma 4 in offline mode.

So, if you want to use Android Studio offline, it'll be powered by Gemma 4 to do your coding tasks.

And just key point: E2B and E4B are small enough that it will actually it'll fit and run well on your phone.

So, it's a lot of capability packed into a really small footprint.

This is actually a very difficult engineering challenge, so we're very proud of this on the Gemma team.

Here are a few examples demonstrating capabilities.

On the left, we show that the model is able to find the laptop and return coordinates of where the laptop is.

And then, the app is able to An app is able to draw a pointer, show show where the laptop is.

In the center, the model is able to identify objects and then draw bounding boxes around those objects.

I There's some interesting applications for this: object detection, object avoidance.

This is a really nice capability.

On the right, is showing how the model can read text.

So, this translates as "Caution, there's a cat."

It also understands what's going on in the image, and it sees this cat in this really funny position, and it can translate and points out that it's a punchline that the text underneath says, "Something may happen."

Which I think it's kind of cute.

So, pointing, object detection, it does OCR, and it translates, all with no special vision pipeline.

Okay, let's get into the eval numbers.

You know, I Demonstrating capability is always important, but a lot of people look at these, and I think here that you can see the strength of the models.

If you look at those last two columns, E2B beats Gemma 3 27B, which is a much bigger model, on reasoning, coding, and agentic.

So, look at the live the LiveCodeBench.

We've got 44 for Gemma 4 versus 29 for Gemma 3 27B.

AIME is also much higher.

Tau2, which is an agentic eval, also much higher, showing a lot of strength there.

If we go up just a few more parameters, our E4B model solidly beats our 27B model on every benchmark.

So, going from Gemma 3 to Gemma 4, we've packed a lot more capability into the same number of parameters.

Sara Smoot

So, how do we do this?

Making the model more capable without making better making it bigger is a challenging problem.

If you just look at the hardware, phones have much less memory.

I mean, we're looking at 12 to 16 GB versus the 80-ish that you might get on a server.

Bandwidth is also much, much less: 50 to 80 GB versus terabytes that you would get in a server.

And then there's severe power constraints.

So all of these we have to keep in mind you know while running the model.

And each technique that I'm going to show you next targets each one of these constraints and helps make the model more optimal for living on the mobile hardware.

All right, the first thing that we do, we don't have one architecture for each size.

Each each model size has its own separate architecture.

So our E2B, E4B have some special elements, which I'm going to primarily focus on for the rest of my talk, but I think it's helpful to know that we have our our 12B model doesn't have a vision encoder.

And so with this model, we took out the vision encoder to simplify both deployment and minimize the number of parameters that the model needed.

That one's typically targeted towards our our laptop hardware.

Our large models are are there to demonstrate the most capability.

These are come in both full dense and a mixture of experts model.

The mixture of experts model could also run on laptops, and we built it because of the lower latency that you get from from the expert architecture.

So for the rest of the talk, I'm going to be focusing on the mobile models, which are the E2B, E4B and talk about what we've done there to make them run really well on your phones.

All right, this is where the "E" comes from, Per-Layer Embeddings.

This is a an interesting technique that we have, an innovative technique that we've added to the models.

And if you think about the way models work, models need more parameters to be smarter.

And we're looking We're getting better at parameter efficiency, but, generally speaking, you want the model to be better, you need more parameters.

And so PLE, what this does is it says, "Okay, we need more parameters."

But instead of adding more parameters directly to the model, we have an extra embeddings table.

And this embeddings table is read in for each layer.

So what you can do with this is once you have your the tokens that you know that the model is going to need, you can pull in the per-layer embeddings that you need for a given given input sequence, and then you can load just those embeddings into the model.

So what it does It gives you an effective 2 billion parameter models, although the number of parameters itself is quite a bit bigger.

That's why we call it effectively 2 billion parameters, cuz only that 2 billion gets loaded in to your memory.

The embeddings table is generally saved in flash, not in RAM.

So you really are limiting both bandwidth, because you're not having to transfer as many parameters, but you're still getting the full model capability of a much bigger model.

It also takes up less RAM on the device.

The next thing we added is a drafter model.

The drafter model that we use for Gemma is a Multi-Token Prediction model.

And it's a form of speculative decoding.

What this does is it gives you It predicts more tokens per pass than than a general autoregressive model will do.

So rather than give Typical autoregressive models will just do one token at a time, you've got to go through the whole model every token.

What these drafter what MTP does is it will predict three or four tokens at a time, and then we use the main model to validate those tokens.

And if those tokens are accepted, they get used, and then we can typically get, you know, two, three, four tokens at a per pass rather than just a single token at a time.

This is a common technique.

Drafter models are out there.

What we have done that that is different and special for Gemma is that we share the KV cache from the base model.

So for E2B we do the full prefill, we build the cache up, and then our drafter model uses that KV cache.

So you're actually saving about 130 milliseconds on a hundred-token a thousand-token prompt, because you're not having to re- regenerate a prefill for your drafter model.

And so our drafter model is about 30 to 40 times faster than than other drafter models out there.

If you look at the numbers, you can see the speed ups when you use the when we use the drafter model.

We get a 40 to 60% higher acceptance rate than most drafter models.

We have a 40% 40 40% lower decode latency by by using the drafter model, and we also use 30 to 40% less energy by using this drafter model.

So you get faster responses, a longer battery life, and no no no code changes.

It's a fairly straightforward and well-supported tool.

Sara Smoot

Our next innovation goes, into quantization.

Typically, quantization is also not a new technique, but we take extra special care for our mobile models to make sure we have a quantization that works really well with the mobile hardware.

And there's a couple of things we think about with our QAT models.

The first is, we do QAT, which, for those of you who don't know, is, quantization-aware training.

And this means that when we do our our training, we consider and recognize the fact that we're going to quantize.

And so it helps the model understand that the parameters are going to be bucketed, and it helps preserve quality.

And it also helps preserve quality, and, usually, we can get the model down to a fraction of the size.

So if you go from a 16-bit model to a 4-bit model, and preserve all of the quality, we've really, really saved a lot of memory.

So the E2B comes in two different quantizations.

We have a Q4_0 format.

This is a more general-purpose quantization.

This is customizable.

This works really well with other quantization libraries.

It allows you to play and have a quantization that might work well for your use case.

We also have a very special quantization, we call it our mobile quant, that is very aggressive in its quantization, and we've done a lot of research into making this one optimal.

It's not as supported, in most runtimes, and so, it is with ML Kit and on Android Studio, so you can use it.

You can use it there.

Some of the innovations that we've built in: we use static, activation quants.

So when you're doing quantization, typically the model, if you're doing dynamic range activation, you need to calculate the ranges on the fly and then use that to do the quantization.

Static range quants, we've done that for you, and so that is just savings you don't have to precompute the ranges.

The other thing we've done is channel-wise quantization.

We structured and compress the model in a channel-wise way that works really well with most mobile hardware.

We have targeted 2-bit quantization.

So the two-bit quantization, generally is quite lossy.

It's very hard to do PTQ.

And we've done research, and we've understand our models well enough that we know where the parameters can be compressed down to 2-bit, and done done well with quantization-aware training.

This saves a lot of storage, and also while preserving almost all of the quality.

So our, our special quant model is around less than a gig, which is, you know, quite a bit smaller than even our the 4-bit quants that we have out there.

The other thing that we have built into our custom quantization is we've done embedding and KV cache quantization.

There's lots of techniques on how to do both of these.

We've we've done the research, and this is what will works really well with our model.

And we've bundled that up into this, into these checkpoints.

So all of this dramatically reduces your active memory.

It, saves you a lot on bandwidth in having to move parameters back and forth.

And it also lets you manage and keep, you know, your long chats and things without running out of space.

Sara Smoot

All right, so we integrate through AICore.

So what you get with AICore is the Gemini Nano 4 model.

Gemini Nano 4 is Gemma built and run through AICore.

AICore does some nice things for you.

It it's the most hands-off path, and actually, I'm going to talk about the other path real quick.

The other path is going through LiteRT, which is a more custom, hands-on path.

If you're going to go this route, the model is going to be bundled with your app.

It won't be using the shared, model that's that's hosted on-device.

So you'll have to be aware of, your memory.

You're going to have to be aware of your power consumption.

You're going to have to be a lot more hands-on with the resources on the phone and how you manage it.

But it does get you full control.

You can use fine-tunes with LiteRT, and then it gives you more options for customization.

If you go through AICore, you can use the ML toolkit APIs, and this is the most hands-off path.

And when you use this, you get a lot all this functionality that I've talked about, the per-layer embeddings is handled appropriately, and we the runtime supports having parameters not in RAM, having the embedding parameters not in RAM.

It already has MTP built in, so that will work automatically.

It's also using our mobile quant.

And it's running on a runtime that is designed to host and work well with that quantization.

All right, so it actually can run and do things in 2-bit precision, where we need it, which saves which makes it really quite quite efficient.

If you want to go deeper, there's two other talks they either have come or are coming.

David's ML Kit session covers the Gemini Nano 4 path, how to and how to use the model through th- through the ML Kit, and then Chintan and Fergus's LiteRT session is going to cover how to use LiteRT and how to develop with LiteRT.

So, this is what the request flow looks like for Gemma.

There's an API surface that goes through some input processing, and this is all built-in for you.

Part of the input processing is safety signals.

You get some built-in safety with AICore.

It checks for We build a lot of safety into the model.

This is an extra layer of safety for mobile apps.

It'll go through feature config and then through our on-device runtime.

On the output, it also checks safety again.

It checks the output signals to make sure it's producing safe output.

This is, again, an additional layer of security beyond what we built into the model.

It also does updates and it's shared across apps.

So, if you're building multiple apps in particular, or you don't want to have to manage a bundled model, this is really the great way to go, and it's you're going to get a lot of functionality built-in straight out of the box.

Sara Smoot

Okay, I'm going to jump into the demo.

This is a voice assistant.

I'm going to show it on a laptop.

Some things that I want you to notice, it's fully local, it's running on the device.

It's cascaded using VAD.

It goes through STT, then it goes into Gemma, and then it goes through TTS, which is text text-to-speech model.

It's streaming, and it's also being run on the E2B model.

So, this is the exact it's running on a laptop, but this is the exact same model that's running on a phone.

All of this could be done on the phone.

This particular demo is built with LiteRT.

Should be able to be built with ML Kit as well.

Watch for latency, there's no server round trip, and watch how it handles interruptions.

Hey Gemma, tell me a little bit about yourself.

I'm Gemma 4, large language model developed by Google DeepMind.

I process text and images to provide information and complete tasks.

Cool.

Can you tell me a little bit about Shakespeare?

William Shakespeare was an English playwright and poet widely regarded as the greatest writer in the English language.

Actually, hold on.

Do you know what?

I really want to know about the solar system.

Can you tell me about that?

Solar system consists of the sun and everything gravitationally bound to it, including eight planets, dwarf planets, moons, asteroids, and comets.

That's cool.

Can you explain how a black hole works?

Black hole forms when a massive star collapses under its own gravity, creating a region of spacetime with gravity so strong that nothing, not even light, can escape.

That's so cool.

Thank you.

You are welcome.

Do you have any other questions I can answer for you?

All right.

There's the audio assistant.

Okay, next up.

This is going to be a local assistant using the video capabilities.

It's fully offline and to like really drive that point home, th- the demo is done while camping, supposedly out in the wilderness.

Things that you want to pay attention to this is built off the E4B model.

It talks naturally.

We're going to show it how to do things.

It's using tool calls in the background, and it's able to access different things on the phone.

And it's pretty playful.

Hey Gemma, do we have internet now?

No, we don't have an internet connection right now.

What is this warning about?

That's in Spanish.

That sign is a warning about the puma habitat.

It advises that pumas are part of the local environment and that you should be cautious if you encounter one.

What is the derivative of this function?

The derivative of x squared plus 7x plus 10 is 2x plus 7.

It's funny.

Oh, hello.

Hey Cindy.

Gemma, when is our group hike tonight?

Your group hike is scheduled for 6:00 p.m.

Today.

Is this a puma?

No, that is a dog.

Good boy.

Sara Smoot

All right, start building.

These are the links up here, these are actually links to the checkpoints.

If you want to go and look directly at the checkpoints and look at our model code, it's all available.

You can use ML-- there's Gemma, Gemini Nano 4, ML Kit link.

The checkpoints are on Hugging Face, our code is in the Transformers codebase.

You can go see what we've done.

You can also get the checkpoints off of Kaggle.

Just again I want to remind you, please check out David's ML Kit presentation, if you want to know more about ML Kit.

And then we have Chintan's and Fergus's LiteRT presentation also later.

If you want to see how to access and use the model in your apps through LiteRT.

Thank you.

I'm really excited to see what you all are going to build.