Building a Formula E co-driver on Android with ML Kit and Gemini Nano

Hi there.

My name's Alastair, and today, I'm really happy to be speaking to you about a little bit of a project we undertook last year with Formula E.

And I have Oli here to join us today to give a bit of a talk about what we did together.

So, in-- I work in Google Cloud in the CTO's office, and we're often looking to try and push the boundaries of what's possible on mobile today, and Cloud, and how they can work together.

I'm going to show you a quick video that'll give you a bit of an intro into what we got up to.

The Gen 4 car has 600kW of racing beast.

All electric power, permanent all-wheel drive.

I'd describe it as bloody fast.

Could we use an agent on a Pixel phone that's connected to the telemetry data that's coming off the car?

As the driver goes up the hill at Goodwood, they will be able to, actually, talk to the car.

That's pretty cool!

Hey Gemma, how are my tires?

Tire pressures are looking good, around 1.4 bar.

There's a phone which is plugged directly into the ECU, which is reading many, many data points of what I'm doing as a driver.

This is the first time this has happened in a race car.

I just think there's not really a better place to showcase it.

Hey Gemma, are we ready?

Ready when you are.

So, what we saw there was a project where we wanted to try bring a co-driver agent into the cockpit with the driver.

And fundamentally, what we're trying to do here is push the limits of what is possible with agents at the edge because this is a great example of a really difficult challenge.

The kind of challenges you see in a race car is obviously high heat.

Race cars are notorious for getting hot, and that's, you know, part of the biggest challenge.

Patchy connectivity.

So, we're actually doing this at Goodwood Festival of Speed, where like, kind of, you know, hundreds of thousands of motor sport fans gather to come and watch cars race up the hill.

And as you can imagine, when there's hundreds of thousands of people in a field, connectivity can be pretty challenging.

Bluetooth as well.

Like, this car's basically a Faraday cage, so you can only imagine the kind of challenges to even be able to just connect into the helmet of the race driver.

And then, finally, the low latency requirements.

This driver's going over 200 km an hour, and so it's really, really important that the answers that they get back from their co-driver agent are really quick and snappy.

And fundamentally, we're really interested in this because we believe agents are going to become ubiquitous.

We're already starting to see it.

They're not just in our computers anymore.

They're in our phones, our watches, our glasses, our smart homes, and even now our cars.

And so, fundamentally, we really wanted to see what we could achieve in this hard scenario like this.

And when we were, kind of, brainstorming how we might build this, do we build a little microcomputer with a little accelerator in it to have a model there?

We kind of had this sudden realization that all the ingredients that we needed kind of existed already, and it was Android.

Android's the kind of perfect host for this because you've got ADK, which is our Agent Developer Kit.

This is a really nice library that lets you build agents directly into your application.

There's billions of users, so, and, you know, different form factors that we can run this on really easily, so it's really easy to test this and distribute the code itself.

Developer Tools that let you debug and try different things and find what's going wrong.

On-device AI, Google Pixel phones now ship with a TPU right in the phone itself, optimized for the on-device model that lives right in the phone, making it really easy using APIs like ML Kit to get access to a really powerful model.

Then, we also have things like text-to-speech and speech recognition, fully offline APIs that allow you to get really powerful conversational experiences.

And then, finally, obviously, the connectivity.

Phones are built for connectivity, whether that's through cell networks, Wi-Fi, or other means.

And so, one of the challenges, though, is agents at the edge can't do everything, and so how would they work with agents, potentially, in the Cloud?

And what if the solution was hybrid?

What if an agent living in a phone can actually talk straight to an agent in the Cloud to be able to perform more complicated actions?

You can imagine, for example, let's say the driver wants to know what race position they are.

Well, the car itself isn't going to know what position it is in the race, but maybe an agent in the Cloud is.

And we have a protocol called agent-to-agent, which means that you can create agents on the edge and in the Cloud and have them converse right with each other to perform more complex actions.

So, how did we build it?

So, I'm going to run you through kind of the ingredients of this system.

Starting off at the harness.

Now, the harness here is basically an Android application running Agent Developer Kit for Android.

The second part is a model.

You need a model to power it, and for us, we were using Gemini Nano via ML Kit, a really powerful on-device model right at your grasps.

Then, the data.

Oli's going to talk a little bit about this in a minute, but you have the data coming in from the car, but also, potentially, the speech recognition from the driver.

And then, finally, you also have the actions that the agent can do.

This could be text-to-speech to be able to talk back to the driver and ask follow-up questions or give the information required, or as mentioned earlier, the agent to agent, communication with a with an agent in the cloud to perform some of the more complex actions.

I'd now like to invite Oli to the stage to give a little bit of talk of how how we actually did this with Formula E.

Thanks, Alastair.

So, we've got our ingredients, now for the hard part: How do we actually integrate an AI agent into a Formula E car?

So, it starts here at the VCU, or the vehicle control unit.

Now, every sensor and every component on the car report back to it over something called the CAN bus.

And the car is generating thousands of points of telemetry per second, stuff like speed, acceleration, throttle, braking, and more.

So, in order to plug-- So, in order to read this telemetry, what we decided to do was plug a Raspberry Pi directly into the VCU of the car via this CAN connection.

Now, the Raspberry Pi does a few different things.

First, it decodes the raw signals, it transforms them, and downsamples them to about 10 Hz, which is plenty of information for a co-driver.

The Raspberry Pi then publishes this refined data over MQTT to a Pixel 10 Pro sitting in the cockpit of the car.

And this is one input that the app hosting our agent application can accept.

Second is Android speech recognition.

So, when the driver speaks, the app converts the input into text, and Gemini Nano is able to reason over both of these inputs.

Now, the agent can make two decisions: First, it can use Android's text-to-speech engine to talk back to the driver.

And second, as Alastair mentioned previously, it can talk back to the cloud agent via the A2A protocol.

Now, around this, agent flow is quite simple: Say the hotword, in this case "Hey, Gemma.", ask a question, and get an answer.

The agent stays listening for about 5 seconds afterwards so you can ask a follow-up question without having to start over, just like talking to a real race engineer.

The app is also a multi-turn agent, which means that you that it can access the conversation history to allow itself to be more conversational.

So, you might be expecting the code for this to look a little bit complicated, right?

Well, it turns out it's actually quite simple.

The first few lines call the onDeviceModel using ML Kit and wrap it as Gemini Nano.

val generativeModel: GenerativeModel = Generation.getClient()
val onDeviceModel = GenaiPrompt.create(
    generativeModel = generativeModel,
    name = "gemini-nano",
)
val agent = LlmAgent(
    name = "race_engineer",
    model = onDeviceModel,
    staticInstruction = /* System prompt */,
    instruction = Instruction {
        /* Dynamic context */
    },
    tools = listOf(
        /* Custom tools */
    )
)

Then, we define our agent with ADK.

Give it a name and two types of instructions: The first is the static instruction, and that's its personality and who it is.

And this can include stuff like how it talks back to the driver and some information about the racetrack.

The second is the dynamic instruction.

Now, this gets generated on every query with the latest telemetry summary, so the agent always knows what the driver is doing at any moment in time.

It also includes the current conversation history, as previously mentioned.

On top of this, we also define some tools, for example, giving it the ability to talk back to previous session data, giving it the ability to talk back to the cloud agent, and also giving it the ability to talk back to the driver using text-to-speech.

Now let's see a quick demo of the agent in action on our sim rigs in the Formula E office.

Hey Gemma, how aggressive is my steering?

Smooth and measured.

Peak lock was 48 degrees, and you were moving the wheel 11 degrees a second on average.

Hey Gemma, what is my peak G-force?

Peak G-force was 2.2 G, through the corners.

Hey Gemma, how was my run?

Strong run, Dan.

Top speed 306 km/h, averaging 186 over 35 seconds.

You were flat out 58% of the time, and peaked at 2.2 G.

So that's it.

That's Gemini Nano on a phone talking to a driver, answering their questions in real time.

A sim rig is one thing, a Formula E car is something else altogether.

So how did we put it all together?

First we had to make a trip to Spark Racing Technology in Paris, where our cars are made.

Now this is me on the left wiring the Raspberry Pi into the actual VCU of the Formula E car.

Everything we've built so far ran on emulated data and was replayed from a test session, so we didn't really know whether it would work with the real thing.

So we plugged it in, powered it on, waited, waited a little bit more, and finally the telemetry started coming through on the device, proving that the data was moving through the system as we expected.

On the right, you can see the finished system, and you can see that the phone is actually implemented into the cockpit, right next to where the driver will be sitting.

So, we were all ready for Goodwood.

So, what actually happened on the day?

Well, remember those four challenges that Alastair talked about earlier?

Goodwood really tested two of them.

Now, the first was the heat.

Now, in Britain, we usually only plan for any weather except for the sun.

So, naturally, what we got was one of the hottest weeks on record, right bang in the middle of a heat wave.

The perfect time to put a computer inside a Formula E car.

But here's the twist.

The phone was actually fine.

Android didn't really suffer at all.

It turns out it was the Raspberry Pi that struggled the most.

And as we couldn't plug it directly into the car, we had to use something called a UPS, which, if you don't know, is basically basically looks like a load of batteries stuck to the underside of the Pi.

But luckily, we managed to fix this by sporadically topping it up throughout the day, and without this, the car probably wouldn't have gone up that hill with no AI agent at all.

Next was Bluetooth.

Now, at that point, we'd never actually tested with the driver's helmet, which connects to the app via Bluetooth.

And it was proving to be a little bit of a challenge.

So, in order to test this, we sat down on the grass, and it looks like we're having a really good time, right?

Got our laptops out, having a nice chat.

Well, in reality, that was probably one of the most stressful periods of my career so far.

We were desperately trying to get that code to play nice and get the Bluetooth helmet connected to the app before the drive went up the hill in the afternoon.

But thanks to some last-minute hot fixing and Android's great debugging tools, we managed to fix it just in time, and the car went up the hill with the AI agent working great.

Now, I'll hand back to Alastair, who's going to talk to us a little bit more about the lessons learned from the project.

Thanks.

Thank you, Oli.

And so, what did we learn on this project?

Well, the first one is that Android is a really perfect host for your agents.

I highly encourage you to use the versatility of Android, the APIs available to you, the on-device models, ADK.

Try them out because you'll be surprised how powerful the experiences you can get from such a simple thing.

Integration hell still exists.

Most people in this room have probably had this at some point in their career.

You build all the sub-components separately, and then when you come to finally plug it up together, something doesn't quite work.

And the driver's racing helmet was exactly this.

But thankfully, we were able to get it fixed on the day, and it really proves that testing is still really critical.

And then finally, Gemini Nano.

Gemini Nano was a real hero here.

It's a really effective model, and it runs really quickly on your device and can do perform really complicated tasks, but it doesn't just do that straight away.

There's some real thinking behind that.

You have to do prompt engineering.

You've got to test it.

You got to think about the way you want to feed the dynamic instruction versus the static instruction.

You got to think about the tools that you're going to give it and how you can best instruct it to do that.

But with the right kind of testing and evaluations, you can get a lot from a model like Gemini Nano right on the device.

Thank you very much, everyone.

We'll be in the booth later, so you can come and join in and ask us some questions.