Agentic coding workflows with small models

All right.

Hi, everyone.

My name is Jose, and I'm here with Ian and Zoe.

We all know about LLMs.

We use them for everything: finding recipes, summarizing YouTube videos, and coding.

There are the really big LLMs, such as the Gemini models, that are capable of truly incredible things, but in this talk, we are going to talk about small language models.

They are also LLMs because something that has 4 billion of something cannot be small, but we'll make this distinction from now on, Frontier Models and Small Language Models, the rocket and the bike.

We'll discuss small models not necessarily as an alternative to frontier models, but mostly to complement them.

We'll explore all the benefits of small language models, especially when they are open and open weights and downloadable, but let's look at cost for a second.

For this talk, I've been running lots of benchmarks, and one of them was a dataset of 200 medium and hard tasks using almost 131 million tokens for less than $9.

I used Gemma 4 for this, so I could have just downloaded it and run it on my own hardware for free, but I ran it on a commercial gateway just for a fair comparison.

So, how much would a frontier model cost?

Around 20 to 200 times more depending on token caching and other factors.

So, the bicycle might not be the fastest means of transportation, but it's definitely the most efficient.

It's true.

So now, the obvious question is what's the difference in quality and accuracy?

In this particular data set, Gemma 4 can solve 53% of the tasks.

This means two things.

If you're using a frontier model to run these tasks on the left, it might be overkill.

And if you're using a small model to run these tasks on the right, you'll have to guide the model towards a solution or use a larger one.

In this talk, we want to help you get more out of every token.

Zoe will walk you through how to choose a small model, and Ian will talk about the Gemma models so that you understand how to frame your thinking about what they can enable.

Then I'll quickly show you how to try Gemma 4 in Android Studio, and finally, we'll show you how to fit small models into your daily workflows and also real Android development benchmarks to help you build an intuition for what they can do.

With that, over to Zoe.

Thank you so much, Jose.

So, using one model for everything is a bit like cracking a nut with a sledgehammer, right?

It is efficient, but it's a little bit disproportionate.

So, when you think about small models, it's almost like you're expanding your tool set.

To know what you have to bring into your workflows, you need to open the tool set, see what's available, what are their specs, what are they good for.

So, to choose small models, you need to know small models.

And the type of model that most people think of whenever they're thinking of a language model is an autoregressive language model.

This is autoregressive because based on an input chunked in tokens, it chooses the most likely next token out of a list of candidates.

And then, that most likely next token that has been chosen goes back into the input, and on we go, which is capable of producing coherent and intelligent-sounding responses, but it does have some caveats, including, but not limited to, working linearly.

So, we can work nonlinearly.

A diffusion model has the same approach, but it works a block of tokens at a time.

It's almost like a sketch that's coming into focus.

And in this case, DiffusionGemma works on chunks of 256 tokens at a time.

So, basically, you get a pass that works on the whole block, being able to generate results with less latency.

So, okay, our tool set has models that can write, right?

Wrong.

So, basically, embedding models essentially turns objects like an image, or text, or, you know, any other concept into a vector.

So, that provides us a mapping of let's just say coordinates into a high dimensional space that allows us to compare two things conceptually.

So, for example, if a hot dog is a sandwich, they will likely lead close or end up in a close space.

Because of this, being able to check whether two concepts are related becomes a let's just say a little bit of math instead of a model pass, which arguably is also a bit of math, but So, this paradigm goes even further, right?

And there are smaller and specialized models that are capable of doing one job, and they do it really well.

Some examples here is, for example, we have function models that turn natural language into API calls, vision models that I'm sure you know what they are capable of doing by the name, or, for example, classifiers that can give scores to an input based on certain parameters.

Okay, so, there is a lot to choose from, right?

And there's a lot of different types of models, and they have different dimensions, different sizes, different context windows, different capabilities, different memory footprints, and once you open the toolbox, and you account for every possibility, well, the options grow exponentially to the point that any DIY fan would be absolutely ecstatic or frozen by decision paralysis.

So, how do you actually choose a small model?

Let's start by opening a toolbox of open small models that can run locally, the Gemma models.

Over to you, Ian.

Thank you, Zoe.

So, as Jose mentioned at the start, a small language model can do maybe about 50% of tasks out of the box.

But why specifically may you choose a local model?

So, it comes down to about four things.

So, the first one is privacy.

So, for instance, you might have, for instance, API keys, proprietary code, or things that just literally cannot leave your laptop.

The second one is latency.

So, when the model is actually loaded into memory, either graphics memory or unified memory, then you get a response as soon as it's finished computing.

So, no round-trip times to servers, no additional latency cost there.

Next is zero marginal cost.

So, models are cost-effective when you get to control where they run.

So, in the case of running something on a local machine, for instance, it might be doing additional retries in an error loop.

It doesn't cost you anything, and there's no kind of surprise API costs for that part of that.

And then finally, the fourth one is offline access.

So, imagine you can use this on a plane, or a train, or a bus, or in the tube, basically anywhere where you have the model running on your hardware.

So, that sounds great, but there are kind of two major factors that we need to consider when making these decisions.

So, one is memory, and then the second one is speed.

So, if you look at this diagram here, if you look at the two bottom models, the 26B and the 31B, they both take over 50 GB of video RAM to run in BF16, which is a 16-bit quantization.

So, the challenge here is that wouldn't fit in a lot of hardware.

So, what we do so we do a process called quantization that shrinks that precision down to say a 4-bit into a float.

Now it's a bit more reasonable.

We're talking about 14 GB and 17 GB, right?

That would now fit on something like unified memory Mac.

Now the challenge with this is in normal post-training quantization, what will happen is as you compress those weights, you lose a bit of the accuracy in the model from the base model, and we have a technique to combat that.

It's called quantization-aware training.

And what happens when we do that is as the model is trained, it is trained to know that it's going to be on these smaller sizes, and therefore there's actually less loss compared to that.

So, we get a 75% reduction in RAM requirements, and now it can fit on those devices.

And we go one step further.

So, the mobile quantization approaches that we have that work on a lot of mobile phones can shrink those models down even further by understanding how all the different layers operate and what sizes they can operate at.

So, we have kind of a mixed quantization bag.

So, we go from an 11.4 Gemma E2B down to a 1.1 GB.

So, now it's actually much more reasonable on kind of mid to high-end phones.

The next is speed.

So, when you use an autoregressive model, as Zoe mentioned, you're predicting the next set of tokens.

So, in like a coding task, you might find a lot of these tokens are actually quite predictable, like it's the end of a like a method signature, or it's an opening or closing brace.

So, what you can do is you can use a technique called speculative decoding.

And what that will do is it will use a smaller drafter model to predict a number of tokens which then get validated by a bigger model.

And the reason this works in the Gemma 4 architecture is that they share the same KV cache.

So, think of this as like the model's kind of short-term memory.

So, what will happen is a small drafter model will go predict three or four tokens ahead of time, and the bigger model will then look at that same KV cache and will basically validate the output.

And what this can result in is about 3x speed up in terms of performance with no actual loss of quality of the output of the model, because it's just predicting things and then deciding whether it's pass or fail, whether this is correct or not.

So, again, it depends a little bit on the task you're doing.

So, this doesn't work so well on things like creative writing, because there's more variability, but it does work very well on like predictable things like JSON structures and coding.

So, how would you actually evaluate one of these models?

So, the most obvious thing you might do is go look at a benchmark like, you know, LiveCodeBench, or MMLU, and you look at a score.

But that will give you only kind of like a high-level indicator about the model's capability.

What you really need to do is evaluate like the whole stack in which it's running in.

So, this is the model that you're using, the weights and the quantization approach, it's the inference of the runtime, so you might use something like llama.cpp or Ollama or LiteRT, the agent harness that you're using, whether that's like Pi or OpenCode or Antigravity, and then the context window, which is kind of the space for all the data that you want your model to be able to process.

And the combination of all these things results in whether it's kind of viable.

So, to give you a quick example, we were doing some testing with Gemma 4 on Android Bench, and we were running an Ollama version with a default 32k context.

And what we were finding is in one of the tests, where it was looking for a bug, the compaction of the agent harness would move that file out of memory, and then the model wouldn't be able to complete the task, and it would fail.

So, just by increasing the context from 32k to 256k, now it's able to keep that in memory, and the model is able to pass that particular test.

And we saw a doubling of the success rate on that particular job.

So, just goes to show that even one of these small factors can have a big impact as to whether this model is suitable or not.

So, how do you actually do your own evaluations?

So, here's kind of three quick suggestions.

So, one is you could take a Git history, so a bunch of PRs you've already got that's successful, and you could use them as prompt inputs to the model and see how well it kind of performs on those.

The next is you need something like a deterministic grader that will help you evaluate whether the output of the model is correct or not.

We've got two really good examples of that.

We've got a compiler and we've got unit tests.

Whatever gets generated, you can run it through that, and it will tell you kind of pass/fail as to whether you've got a success out of that particular task.

And then the third one is log trajectories.

So, when you're running an agent, you want to be able to trace the different behaviors that the agent has, see where it succeeds and where it fails, and you can use that to evaluate how well the model is able to interact and use tools.

And I want you to try and change your framing of the question from, you know, "Is this model as smart as this big frontier model?"

To, "Is this task possible on this hardware with this particular configuration?"

And to kind of figure that out, we've invented a framework called Are we there yet?

Or AWTY.

And the whole principle of this framework is to kind of score on that axis all these different tasks in terms of readiness.

So, the model itself, the inference engine that you're using, the harness, and then the hardware configuration.

And that will tell you whether you're ready yet.

So, for instance, you might be looking at like how reliable that particular task is.

If you run it, you know, 10 times in a row, does it do the same thing each time?

You might be looking at the latency.

Maybe you need it to be a real-time response.

So, you know, what are your thresholds for like the speed of doing the task?

And then you might be actually looking at like the RAM requirements.

So, it's not just, you know, the RAM that the device has, it's the RAM that the device-- the carve-out that you can take, and the other applications and other tooling that you'll also need to run on that hardware.

So, can you take up 50% or can you only take up 10% of the RAM?

And you get this final kind of readiness score.

So, if it's green, that means that configuration is ready to go.

If it's orange or yellow and or red, that means maybe you need to change something about the configuration.

Maybe you need a faster accelerator for lower latency, maybe you need a bigger model that's more capable.

So, yeah, I suggest having a look at this.

We've got a preview at the moment and we'd love to get your feedback on how useful this tool is.

So, Zoe is going to cover a bit more about how we set up some of the evaluations later, but I'm going to now hand over to Jose, who's going to show you a very quick way to get started with Gemma 4.

Thank you, Ian.

So, yeah, the fastest way to try Gemma 4 today is probably through Android Studio.

So let me show you how.

Let's switch to the laptop, please.

There you go.

So, if I go to the model providers page in the settings and I select Gemma, this gives you a pretty clear idea of how big each model is.

I just have to download the models onto my hard drive.

So, this opens or starts a llama.cpp server, and it's correctly configured for you, from the get-go.

But you can also feel free to use your own llama.cpp server or Ollama or LM Studio, whatever you want.

For that, you just come here, and add a local provider.

And also, it makes a lot of sense to use, small models through a remote provider.

So, you can add one here, as long as it's OpenAI or Anthropic compatible, you can connect to it.

All right, so, let's see.

So, now, oh, by the way, so, the way you select a model in Android Studio is through this drop-down, obviously.

So, I have Gemma 4 26B selected, and I can ask, "What is a ViewModel?"

Okay.

So, one very important thing you need to remember is that, whenever you change model, the whole conversation is sent again to the new model from the beginning.

So, if you generate a really long conversation with a model, and you want to switch to a new one, you should probably start a new conversation, or compact it, or whatever.

So, now, back to the slides, and to Zoe, to learn about the different workflows you can run with small models.

Thank you so much, Jose, and thank you, Ian, for the overview on Gemma models, and whether we're there yet or not.

So, we've seen what types of models are there, what their specs are, and now it's a matter of understanding what are some types of jobs that you can put them to do.

The first type of jobs that, likely, you're already doing are, you know, coding workflows like debugging, or like troubleshooting, or something that, I'm sure, everybody, or, at least, I will very deeply sarcastically miss doing by hand, which is increasing test coverage.

Right?

So, these are, you know, some of the things that I'm pretty sure everybody's already doing, even, likely, without realizing.

But I would like to encourage you, or I would I'd want to encourage you to think beyond that, right?

So, considering that small models can be open-weight and run locally, what is something that could be giving you back time?

Right?

We're talking about not using the rocket, so why are we using a brain for that?

So, if we could go beyond that, what are some things that small models could do?

What are some workflows that you could bring them into that could make your life easier and give you back more capacity to do things that, you know, maybe you want to do more of?

So, for example, something like making sure that your documentation is always up to date with the changes that you made, or something like ensuring that you have automatic regression tests that are a part of a pre-commit hook, or even making sure that the context that may your agent is receiving as part of your workflows is optimized for it, and it works even faster.

So, these are some examples, right?

And, at the end of the day, the sky's the limit to some extent, or architecture is the limit, and there are a lot of things that you could think of how to implement it.

But so, having the specs, having the tools, having the workflows, are you still show-- not sure where to begin?

Well, that's what benchmarks are for.

And, as Ian said, a generic leaderboard, especially one for Python, is going to tell you very little about whether a small model is going to be helpful for your Android workflows.

Just very much like a drywall drill like might not tell you much about a concrete drill.

But you should invest in your own evals.

You should invest in testing your own architecture, but before you do that, you might not want to put all the resources.

It takes time, it takes money, it takes effort, so looking into publicly available benchmarks can give you an idea and a short list before you decide to invest on your own evals.

And there's three things that you should look at whenever you evaluate the benchmark, right?

It's a bit meta, but, basically, the first thing is, what is the type of task that this model is doing, right?

How specified it is.

Does it look anything like the type of task that you will give it?

The second one would be the scope of the task.

So, is it writing documentation, is it resolving a PR, is it something that looked like looks like the type of job that you have?

And, finally, is the complexity of the system akin to the complexity that you're going to give?

And here is where we can see why a Python benchmark is not going to be very representative of Android.

That's why we build Android Bench, which is a benchmark that captures the Android ecosystem and Android work.

The first version has 100 PR-sized Android tasks, and you can run it yourselves today through Harbor to check any agent, model, and skill combination.

We launch 2.0 with more complex long-running tasks to understand what is the capabilities of frontier models to work in jobs like the ones that you give it today.

So, let me hand it over to Jose now to show you the results that we got through this and some ways that you can optimize working with small models.

Okay, thank you, Zoe.

All right, so let's look at the tasks that a small model can solve and where they stra-- start to struggle.

And let's, arbitrarily, define four categories of tasks: small, medium, big, and long-horizon.

Long-horizon tasks as are are tasks that might take a human days or weeks to complete.

For example, a migration, a big refactoring, And spoiler alert, you cannot do those with a small model.

At least in one go.

We'll talk about that later.

Okay, small tasks.

They are jobs that require reading and writing maybe one or two files, and they don't require much knowledge.

For example small models are pretty good at Compose, at the at least the basics of Compose.

So, things like there must be 8 dp between elements, or use a theme instead of hardcoded colors, or extract strings.

Those are very easy for a small model.

This one, however, covers a more advanced topic, but the task is still easy.

@Composable
 fun AnimatedDrawer() {
   val offsetX = remember { Animatable(0f) }
-  Box(modifier = Modifier.offset(x = offsetX.value.dp)) {}
+  Box(modifier = Modifier.graphicsLayer { translationX = offsetX.value }) {}
 }

The difficulty, in this case, is about scope, not about knowledge.

So, the prompt actually tells us which composable to optimize and what's wrong with it: it's recomposing on every frame.

So, this makes the job pretty easy even for a small model, such as Gemma 4.

It took, in this case, 3 minutes, and it is a one-line change.

All right.

So, I think we have a pretty good idea of what a small model is.

Let's look at mi- sorry, a small task is.

Let's look at medium tasks.

Small models should be able to do most of them.

In this task, from from the app Pocket Casts, the user-agent is missing from a in a request, and also there's an error in the retry logic.

So, this one is a little bit harder because first we're requesting two things in this at the same time, and there are no hints about where these things live in the code base.

But it passed.

It took 10 minutes.

It modified three files.

And, by the way, it cost 6 cents through OpenRouter.

Now, let's look at a pa- a task that failed from the same dataset.

This is an example of Now in Android with a pretty comprehensive report or prompt, but it doesn't mention a class name or a file, so the agent needs to spend more effort into understanding the architecture of the app, the data layer, the sync mechanism and in this case, etc. So, in this particular run, it failed.

Everything was implemented almost correctly, but it just assumed that the value of the initial sync would be -1, but it's actually 0.

val isInitialSync = currentVersion == -1

val isInitialSync = currentVersion == 0

That's it.

This bug would have been caught if the agent had just run the unit test, the existing unit tests.

So, this shows how smaller models, in general, tend to stop a little bit earlier than frontier one ones.

We just asked the agent to fix the issue, not to verify it.

So, maybe we should modify the task so that it contains a verification step.

We'll talk about that in a minute as well.

So, all right, let's talk about big tasks.

About half of the big tasks, in this definition, pass every time.

Some of the ones that fail are actually too long and complex for a small model, not much to do there.

But there are some that when you look at what actually happened, you realize that the model was 90 or 90 per- percent there.

So, let's pick one of them.

I picked this one because it's a flaky one.

It passes around half of the time that it's run.

So, I consider this to be at the frontier of what a small model can do.

Or, at least, a model from April.

It's about parsing malformed URLs in the Coil library.

As you can see, the PR description is pretty comprehensive, which is great, but it missed a requirement.

So, let's quickly look at it.

All runs checked for typical URL encodings and errors, but only some, half of them, added tests for badly encoded characters, so that test case there.

The ones that didn't failed the evaluation.

So, all runs are almost perfect, but this goes to show again that small models don't always go the extra step.

They follow instructions, but they don't try to impress you like frontier models do.

You know how you can set the effort in an agent nowadays?

Just consider that it's set too low when you use a small model.

So, let's define big tasks as those that might be completed by small models around 50% of the time, which is what Gemma 4 31B can reach against the Android Bench v1 dataset, which is what Zoe talked about before.

So, before talking about long-horizon tasks, can we do every anything to help small models with those tasks?

Absolutely, and these things don't require any effort at all.

First, if you use a model such as Gemma 4 locally as Ian mentioned, make sure that it's configured correctly.

For example, if you use Ollama to serve Gemma 4, it defaults to a context window of 32,000 tokens, when it actually supports eight times more.

Of course, if you use Gemma 4 through Android Studio, you don't have to configure it.

It comes correctly configured by default.

In our set of evals, that made the enabling the full context made the results go from 26 to 53%.

Next, use Android CLI, the Android CLI tool, especially for searching documentation.

This doesn't always apply, as models are pretty good at the basics, but whenever you use a new or a niche API, having the documentation is key.

Much better than using web search tools, in our experience.

Next, add verification hints.

We talked about this.

Tell the model how to run tests, what type types of tests are available, how to manage emulators, etc. You can do this in an AGENTS.md file, which would make the instructions available to all agents, but, I mean, that's a good thing, anyway.

You should have it.

But also watch out for super long AGENTS.md files, because if you try different agents, you'll see that some of them have massive system prompts and tool descriptions, and they seem to degrade the performance of small models.

We actually tried running these benchmarks with OpenCode, and it performed a lot lower than pi, probably because it uses bigger tool descriptions and system prompts.

Next, add skills, but also remove skills.

Small models might need a nudge when using new and niche APIs, but it's better not to fill their context with general Compose skills, for example.

So, what about long-horizon tasks?

They are tasks that span dozens of files, require like serious architectural decisions, and are normally handled by a thinking model that creates a plan and starts a swarm of sub-agents that execute the different tasks.

Examples of long-horizon tasks would be migrating a whole app from XML to Compose, creating a complex app from scratch, or converting cross-platform apps to native Android.

If we look at the results from Android Bench V2, which focuses exclusively on long-horizon, we see that frontier models are scoring 3 to 40%.

So, for small models, I think it's safe to say it's out of their league.

However, routing the initial analysis through a frontier model and then cascading simpler tasks to smaller models would be far more token and credit efficient.

Let's see how we can do this.

Back to Zoe.

Thank you so much, Jose.

We'll get to the cascade, but before before I show you some ways, or before we deep dive into this, I want to take a step back and show you other ways in which orchestration of different models can help you.

So, orchestration can, for example, help you never be blocked, right?

So, whether you're seeing a provider go down, right, or whether you hit a API rate limit, or whether you are on a plane or a train, like Ian was saying, you can set up a fallback model and have make it so that whether that is online, or locally, you can keep going if anything happens.

The other thing that you could do is through orchestration is get better answers.

For example, you could have a small model work on its own and call an advisor whenever it needs a little bit of support, right, to get that help.

You could also set up a council in which different types of models, or different models, are all executing the same task at once, and then you get to evaluate the different responses and how the weights and biases of each model have affected the outcome.

You could also use a separate model as the Let's just say as the tests.

So, it's like a peer review.

And you could see it through this.

But And, of course, finally, orchestration can help you be cost-efficient through something like that cascade.

Basically, you can set it up manually, so you could set one model for planning, you could set a different model for execution, another one for tests.

There's multiple ways to do it, either on the cloud or you could set up a LiteLLM proxy to do it locally.

Or you could just use a classifier to route it automatically to the right type of model based on the request, which is the proxy idea that Ian mentioned, but automatic.

So, let's wrap up by going back to the question that Jose asked at the beginning of the talk, which is, what's the difference in quality and accuracy between small models and frontier models?

And the answer with As with many things in computer science is, it depends.

However, this is something that I You know, I would like to bring back with you is that this is something you can engineer.

This is something that you can make a difference in depending on how you set it up and depending how you play with the pieces.

So, the first piece here is, pick the right tool, right?

There's multiple types of tools.

Get to know them, explore them, right?

And then find out which jobs are they good for or even which jobs they could be doing.

The second one is make sure to adjust the scope that the model is getting in order to do the job.

So, for example, things like the instruction for the task, things like providing the right context for the task and only the right context for the task, and ways to check its own work.

Then, of course, you need to set-- test the whole setup and not just the model in isolation, because this is going to make a difference and maybe the results that you're getting could be tweaked if you set it up with, let's just say a different configuration of elements.

And at the end of the day, you need to work on setting up or creating your own evals.

And finally, you can orchestrate the bicycle and the rocket so that you never have to choose between one or the other and spend effectively with this.

But most importantly, you can continue to explore, continue to build, continue to have fun, right?

And you can begin that today by setting up a shortlist of models through Android Bench and the findings you have there, or starting with Gemma 4 on Android Studio today.

So, thank you very much.