Agentic coding: A roundtable with industry leaders

Hey, everyone.

You've heard from a lot of Android Googlers this week, or this last two days.

So, we thought it would be really nice to close out the conference by hearing from some people from outside, from the community, from the Android Dev community.

We're going to be talking about agentic coding and sort of real-world impact of that.

So we'll start by introducing ourselves.

My name's Ben, Android DevRel.

You've already heard from me once this conference, but let's go down the panel.

Hey, folks.

My name is Vinay, and I support AI tooling efforts at Databricks.

Excited to be here.

Hello, I'm Luis Artola.

I work for Google Maps.

I lead teams that are using AI for building agentic experiences and improving the software development cycle.

Hey, I'm Annyce Davis.

I work at Ford.

I lead some engineering teams who develop our subscriptions and loyalty products.

Hey, folks.

I'm Ty Smith.

I'm at Uber supporting our developers.

That's been mobile for a really long time.

In the last couple years, it's been a big focus on this new exciting time we're in with AI tooling.

Hey, folks.

I'm Kaushik Gopal.

I work as a principal engineer at Instacart, and I happened to start this podcast on Android development called Fragmented.

These days, like Ty said, we've changed strategies a little and, you know, we talk about AI development.

All right.

So, we've got 30 minutes.

I think it's going to race by, so we're going to jump straight into the hard-hitting questions.

We've always, you know, on Android DevRel, said we want to make it faster and easier to build apps.

Now AI's come along, and, you know, you've heard us up here saying, "It is indeed.

We think it's faster and easier to build apps."

But I want to put it out there, are we actually shipping apps faster, or are we just generating a whole bunch of code that sort of goes nowhere?

Let's go, Vinay.

Okay.

Yeah, we're all looking at each other.

Fantastic.

So, if you look at just pure numbers, we definitely see that the output in terms of PRs, in terms of lines of code, we've seen a massive acceleration, and as a result, we also are hitting scaling challenges when it comes to even things like where we are storing our code, right?

And so that acceleration is very real.

What is not obvious is how it impacts outcomes, and I think the whole industry is trying to figure out what the right metric is around outcomes, how to know how it impacts the bottom line.

But I definitely see it in action.

I think we've shipped more code in the last six months than the previous, and again, I'll keep this ambiguous, but like a very long period of time.

And as a result, I think our strategy has changed around how to measure productivity, even, right?

Because there's there's various way to think about it.

We are trying to figure out how to raise the median, right?

Because when you look at the outliers and the median, there's a 5x to 10x delta between those two categories of people, and our focus is raise the floor, raise the median, and make those people as productive as the peers around them that we see are as productive.

We see a combination of roles.

Some folks are moving much much faster.

There's people who have a hundred PRs a day that are just doing incredible work.

But that's a type of space that they're operating in.

That's maybe more greenfield, or they're more kind of fluent with the tools.

We also see folks that are much, you know, more in the brownfield, or the platform work, and they're supporting a different type of work that maybe aren't as accelerated because they're having to spend more time with a human judgment and making sure things aren't aren't going to break.

And the combination of these two, it you know, it's significantly faster than it was before, but I think the real interesting topic for us is even though we're shipping more, is the business actually shipping more to the end users?

That's an area that we've been investing in over the last year to really understand are we shipping features faster, not just shipping code faster?

Early signals say we are, but it has been a very different look than just the pure PRs.

I think like the like what's happened is, very clearly, people are shipping far like, you know, the number of PRs, the counts have like, you know, increased.

But I think even right now a lot of the thinking is like, "How do we take AI and sort of this tool that we have and turbocharge existing processes?"

Right?

Like, a lot of how we used to ship code before feels like we always thought of like a human as like someone who would shepherd this all the way to production.

But I think what we're seeing now is like, yeah, that works.

We're shipping more.

But we're running into every single bottleneck that you otherwise in a previous world wouldn't have seen, right?

Cuz we never ran into those issues in the past, but now, like, we're turbocharging one part of like, you know, a multiple-step cycle, and I think that's where it's getting interesting, which is like, "Okay.

We are delivering more, but now it's like a fire hose of effects."

So, we're running into other challenges.

That's a very interesting point, right, because when you think about it, there are many different steps, but from a high-level perspective, you can bucket them in three stages, right?

There's coding, which is clearly being accelerated.

There's production readiness that includes a number of things like legal, and privacy, and things that allow you to get something out of the door, and then there's launching, right?

And then, shipping really means different things depending on where you are, right?

If you are like a completely server side product, probably shipping means a more streamlined process, whereas when you are shipping something that goes to the consumer, right, like getting that in experimentation is just like the middle of a journey, right?

You still have to do a lot of experimentation, and measurement, and that's where it ties to what you were saying is that it feels like you are in a metropolitan area, in a freeway, and you have a choke point, and then you start accelerating and you are like, "Yes, great.

I'm going really fast," but now you hit the next choke point.

And what it's making like AI is making very evident is that you are seeing the friction points more clearly, and some of them have nothing to do with coding or technology.

It has to do with processes, and humans, and how do you optimize that with AI.

That's a really interesting challenge on how to streamline end to end.

One interesting way that we're thinking about that the space a little bit is, how much agency does the person or team have in shipping it fully?

So, if we look internally, you know, there's a lot of teams that maybe are building internal tools, or things that aren't shipping to the customer.

They're shipping much faster, right?

They have full agency to getting that out into their customers immediately.

If you have an external product that then still has to go through experimentation with a user, or, you know, app store rollout times, or, you know, working with design and product, all these other bottlenecks are adding up, and the people writing the code have a lot less agency to move fast on that.

And so, they're they're just hitting the rest of the bottlenecks in the system.

And so, I think splitting those into the two different areas really highlights where your business processes could be improved.

Yeah, I agree, and I feel honestly, engineering was never the bottleneck.

I've been saying this for years.

Engineering pretty much has a very buttoned up way of doing things, and it was always from the business, "What is the problem?"

You get 10 different answers to that.

And then, "Okay.

Now, let's translate that sort of to some solutions with product.

Now, let's marinate on that for a while.

Oh, let's maybe talk to design a bit.

Kick it over there."

By the time it gets to engineering, we more or less knew what we need to do.

And then, we just try to go to Stack Overflow, and copy-paste some stuff, and ship that thing out the door.

So, with AI, where I'm seeing the biggest speedups are just around that compliance work.

Those things where we've always wanted to modernize XYZ, or we've always wanted to like quickly upgrade the Spring Boot version.

We're doing that so fast.

But the actual feature delivery, I can't say that I've really seen significant improvement.

So, you've kind of all touched on it that maybe the bottleneck has moved, and you're running into these processes that used to not really be the bottleneck, but are now the bottleneck.

Except those processes probably came from hard learned lessons in the past, right?

So, are you feeling the pressure to kind of change those, and I think the most obvious one is code review.

What do we think the future of code review is?

What are we doing currently?

So yeah, I'd love to hear.

You know, it goes back to this like point I made.

I feel a lot of these bottlenecks are because we thought of like Like, if you had a magic wand, so to speak, and you get to reinvent software engineering, and you had AI today, right, would we use a lot of the same Would we do the same things?

Like, for example, code review was because we needed a human to like pass things, like, you know, do that final cross-check.

I think that's changing, too, like, you know, the way we're thinking about code reviews now is, maybe we should be open to the idea that, you know, if you're like refactoring the name, or you're trying to refactor code, and you know that you have great tests, which everyone clearly has.

We've got the AI to write them, it's fine.

No, like maybe, you know, are you going to do a much better job than that agent?

I don't know, like so, maybe just don't spend the time reviewing that.

Then again, if you're going to ship a bug and, you know, you get a call from your CEO basically saying, "Hey, this thing is costing us X number of dollars," maybe identify areas and systems where like, "Hey, this is a critical area.

This is not a critical area," and then try to re-imagine.

Like, I don't think that's how we operate today, right?

Today, it's like, "Hey, there's a PR.

I'm the engineer.

I'm focused on this thing.

It's going to go up.

I ask I bug Vinay for like a code review like middle of the night, cuz I just want to get this thing merged."

Versus, maybe we should rethink a lot of those.

I fully agree, and I think of code reviewing as a verification problem, And what I mean by that is that in a world where I have full confidence in my ability to verify something, would I still code review 100% of my PRs?

Probably not, right?

So, there are two purposes that code reviews serve.

One is obviously verifying that we're doing the right thing, and the second is education, right?

Like people around you knowing what you're up to and whether they agree with that decision or not.

And so there's definitely a category of work that we should feel comfortable in accelerating and making sure that it makes its way into the code and even releases.

Now, this might not apply to 100% of the cases, but I think it's a very high, double-digit number that it should apply to.

And I think we are getting ready to make those changes and we all as a community need to make sure that our systems are ready for that, because it's coming.

There's no way around it.

The top of funnel is so wide with various surfaces allowing you to create more PRs.

And you're as fast as your slowest step of the process.

And for a lot of teams that that's code reviews, and the teams that are moving faster have honestly figured out and accepted this reality.

Yeah, and it's like if we're honest with ourselves, I want to say 70% of PRs that I've seen looks good to me.

Like if we just searched all of the code bases in the world, the PR comments would just be, "looks good to me", LGTM, all over the board.

So, part of it is like as engineering professionals, sometimes we give ourselves more credit than we deserve.

We really think we're like in there.

"Did they use command pattern?"

No, we're just like, "Okay.

Yeah, it looks good to me."

So, then why can't I have an agent do that?

I really think like that's the future that we're going to see where like what is the looks good to me, like kind of what is the characteristics, what are the behaviors that made you feel that way?

And then like let's make sure that the agent looks for the same thing.

Or like on the flip side, also try to think about different ways of reviewing, right?

Like I know of people like who are on Twitter these days, they say this thing, right?

Where it's like maybe when you verify a PR now, get your agents to do like visualize it in a different way, right?

Hey, show me a before.

Show me an after.

Show me the tests you wrote as against reviewing code like line-by-line.

But it does challenge You were going to say something, Luis?

No, I think it's a thought-provoking thing, right?

Like that we're going to get to a place where you feel really comfortable letting CLs, like or PRs, going through without human intervention.

So, I'm going to play devil's advocate a little bit, and I want to say that when we talk about friction, I think there is a tendency to think that friction is bad, but there is good and bad friction.

And I think that I feel that reviews are still in the category of good friction.

It's a matter of how we make it less onerous to developers, right?

And sometimes it's not about how do we make that review, at that point in time, better or more efficient?

It's how can you make that upstream the development more precise with AI, right?

Because if you just Depending on how you use AI, like good inputs produces better outputs, but bad inputs produces really slop, right?

So, I think that is not I think that to get to that appealing, you know, end state, it has to be a holistic improvement of the process with AI all the way from the beginning, using AI to have a better understanding of the architecture, to have like really solid specifications with, you know, less effort from developers, so that the code that it produces is higher quality.

And therefore, the reviews are less have less friction for developers.

I think that the end state of this that wouldn't be that controversial is most folks see that all this code will be landing without human review eventually.

I think in the meantime, we're in this evolving situation where the models are getting better, and we're less worried about them making a technical mistake, and more worried about them making a like a business mistake or the judgment or the ideation that came into it was the problem.

And so, that also goes into the signal where you would want to separate that out during the review stage.

Like should we be doing line-by-line review in the PR versus reviewing the intent that they're, you know, trying to push after?

We're thinking about this maybe in a few different ways.

One is how do we give more relevant information to the user so they can start to prioritize that?

The other is what are those safer areas where we can experiment and we can move faster in these sort of programs?

Like Kaushik was saying, not all of the areas need the same amount of rigor.

We can identify this and we can start letting some go faster, lower the risk while keeping, you know, areas that are more compliance-oriented or, you know, very foundational with the higher friction until we really get to that point of trusting not just the models, but that the inputs going into them

Were the right decisions.

Like refactoring new Android module in the Uber app, maybe okay.

Giving free rides to all Android developers, probably want to check that code.

I'll have to check that one with our CEO.

Sorry, Kaushik.

But I also think that this is a solved problem.

We already have like feature flags.

We already have blue-green deployment.

We already have a lot of the pieces.

The challenge is like we've been too busy to really do it.

We already have behavioral tracking.

We have all this sort of data, but they're in all these different systems.

We already track incidents.

We have just all so much data, but putting it all together into one like comprehensive trace, that's the piece that's missing.

Once we finally figure out what that kind of engineering context service looks like across all these different companies, then we don't have to worry anymore because even if the PR just gets merged, it's safe.

Because we know, okay, we've got that behind a feature flag.

We've got blue-green deployment.

We have all of the observability in place.

And then you can feel super confident with just saying like, "Oh, put it in the pipeline."

Something bad happens, we have all these guards and gates along the way.

I think the speed of being able to respond is also really important, right?

If you know, let's let's pick on Android for a minute, if you're putting out a change in Android and that needs to ship through the Play Store and then get rolled out to your consumers, your ability to immediately hotfix that is really inhibited versus if you're doing something that has an internal deployment with internal observab- observability and an agent that's on it immediately, you have a lot more opportunities to move faster there.

So, the one of the questions I had for the you folks is there's also this problem that there's work that needs to be done on these app, but then you have these greenfield apps which are easier, then you have like these brownfield apps, and then there's this moment where you need the time to put in a lot of these checks you know, so that you can proliferate the use of AI.

I know like a bunch of you work on like massive apps, right?

So, I'm curious how have you been thinking about it?

Is it like, hey, CEO, give me like 3 months, let me just get this app AI-ready, or I mean, you know, like add it every single day?

Yeah.

They never do that anyway.

I don't think so.

I mean, most companies will never have that luxury.

I think that we just have to dedicate it the same way we used to do like tech debt, where it's like, use 20% of your time towards this, it's like use 5% of that to improve a prompt.

And then even understand what does the prompt do, and make make the context better.

So, like we kind of been talking about this, a lot of it, I think, is that engineers still don't understand how it works.

It still feels very like magical and mystical, and so they're not like configuring it and tuning it the way that they could best get get better value, efficiency out of it.

So, I think that there does have to be a point where leaders say, "What do we need to do to level up our team so they can understand how this works, and then we can get more?"

It's also a continuity of it.

Because it's changing so fast, if, you know, I had asked a team to go spend a week right now learning the latest and greatest, and put those into practice and then they didn't have any time to continue learning or experimenting, and we checked in next year, they'd be running behind significantly from what the new capabilities are.

So, it needs to be this all we, you know, we're kind of building the plane while we fly it, where we're putting in the guardrails, and we're learning, and we're adjusting as as things are happening, and teams need to stay on top of this.

And at large organizations, there's teams like some of ours that try to distill some of that information and make it, you know, a little more related relatable and dig- digestible.

But it's a challenge for everyone in the industry right now.

I'd love to mix it up a little bit.

You A lot of you work in like developer infrastructure.

Thousands of PRs, are your like testing infrastructure, your servers, all melting?

And I'd love to hear some battle stories.

Ty?

Yes.

They're on fire.

My CI cries to s- cries my cries me to sleep every night.

Pretty much all of our developer infrastructure teams right now are are battling scaling problems in compute and efficiency problems.

There's, you know, code reds going on.

They see this huge amount of volume increase.

We're doing architectural changes with agents.

We're you know trying to do things like as quickly as possible expand our remote build execution and caching with Bazel, looking at things like you know, how does all of main need to be green at all times, or can we start to like forward fix with agents in certain areas, looking at the volume coming in from engineers, trying to do things like self-healing CI, or like lower feedback loops, and all the teams are just actively investing in all of these to try to keep up with the growth while keeping some level of efficiency.

But it's extremely challenging.

Yeah, I agree with everything that Ty just said, but I think it also forces us to reflect and honestly question if we were to 10x from this point onwards, are we ready?

And for most people, the answer is probably not.

And as a result, as we think about the future, we are intentionally trying to match our ambition with the scale that the infra can support.

And wha-- one thing that AI has done is that it's made all of us more ambitious because we've already seen that meaningful jump, and so we are honestly rethinking our systems.

Right, we're thinking about what other system is going to break, or we anticipate is going to break, and what can we do to just get ahead of it so that it's not a surprise?

And in reality, we're trying to avoid surprises.

Right, as long as we feel we are in control, everyone's happy.

The minute we encounter a surprise, I know it's bad news for everyone.

Right?

And so this is a good point in time to reflect and look back, and audit, and figure out what is going to break next, because it's going to happen.

It's inevitable.

We just want to get ahead of it.

There was an interesting quote I heard from the CEO of Inflow.

She was describing that this the timeline it took for Google to get to the, whatever, billion lines of code that they are now and it, you know, took 20 20-ish years, and with the acceleration of this that like a lot of our companies are going to be looking at a similar scale in just a short number of years, based on the acceleration of AI.

And I don't think all the companies are ready to have Google-level infrastructure for development.

Well, on that topic then, do your companies implement some sort of spend limit?

Like, how do you manage just some developers just burning tokens for everything now, you know, use the best model possible all the time.

Like, is there any strategies you found or is it better to just let them free because experimenting leads to sweet results?

I'd say last year was the year of experimentation.

Do all the things.

And maybe even the first half of this year was just use it reasonably.

And now, since September, we have token limits.

So we actually have limits for developers on the token usage.

And then we have like instrumentations, so they can understand like, are you having good, healthy sessions?

What does it mean to have a healthy session?

Are you having this many turns per session?

What is a turn?

So, we're really trying to like use the tooling to also educate developers.

I think that's just a first step.

You know, you got to know you have a problem before you can fix it.

And so, like them at least seeing this is not free.

Before, it really felt like AI was free.

And people were just using it.

I'd like, "Change my branch," you know, like, "Oh, no problem Annyce, I'll change that branch for you, $5," right?

So like now, we're getting to the place where like, "Okay, I could just go to the command line and like do that myself, right?"

So I think that we're going to start to like really have engineers be educated on the actual cost.

More so like I wouldn't just spin up ECS instances and just like run away from them.

Maybe when AWS first came out, we were like, "Yeah, I can spend my own instance."

And we were just like costing the company so much money.

I think the same thing is happening now with the tokens.

I think that practical limits also present really interesting opportunities, right?

Because when you think about how, in the beginning, we started using LLMs, you would throw everything at them, right?

They would introspect the code, figure things out, and do magic for you, right?

But that came with a cost.

And we're actually real-- you know, the industry is realizing that.

And that actually is forcing developers to think about and organizations to think about, how can you optimize that, right?

What Why is, say for instance, I'll give you a concrete example: If you have a generator that is producing user interface components, and they need to comply with a specific structure, and unit tests, and things like that, why would you have the models generate that on the fly, if you can actually have it generate a deterministic tool that they can just create that for you, right?

So, right there, there is a lot of opportunities to use the limits of tokens and usage to your advantage, and actually become more effective.

Yeah, and along those themes, we absolutely have introduced limits, at least like you know, at Instacart.

But I think the goal is not necessarily to limit how much you use AI, it's to bring some kind of hygiene, or like what you're saying, introspection, right?

Like when should you probably use certain models versus other models.

You know, you probably don't need like fable or Astra to you know tell you the time or change a Git branch.

Right?

You can probably, you know, spend some time to write a script and it's free forever.

No tokens wasted there.

Right?

So, I think if you can add some kind that the constraint is meant not necessary to limit the use, but more to sort of help you think beyond just like throwing everything at it.

Right?

I like what you said, last year felt like the year of experimenting.

This year feels like, okay, how do we effectively use this thing?

Yeah.

The stories here we resonate a lot with as well.

Especially going from more experimentation to, you know, a balanced set of efficiency and a accountability.

That comes from a variety of investments.

There's things like education for the developers and visibility.

You know, like Annyce was talking about with like the active session cost and information.

Investments in like safe defaults that, you know, help avoid wasteful interactions, making sure that the skills that they're using are efficient looking for regressions and things in those.

And giving accountability on the cost that they're spending.

So, you know, we do have token limits.

And when a developer reaches those, they can you know, one click a button and put in a justification and their manager can approve.

So if there's great work going on, like we want them to go through that.

But, you know, if they're maybe doing something more wasteful or experimental, they couldn't justify to their manager why, you know, why they should be spending money on that, then it helps them kind of keep that in check as well.

And that results not just in not doing work, but looking and saying, is this the right model choice?

Is this the right weight for it?

You know, obviously, the cost of these tokens have started coming down with some of the competitive pressure.

But in general having our engineers kind of just thinking about it is being good stewards of the money that we're giving them is, I think, the right healthy approach right now.

And then long-term, being able to remove that decision from them entirely.

Right?

You want automatic model routing and things like that, where they never have to think about it.

But in the meantime, this is a healthy state.

Yeah.

I think a lot of our focus has been on doing things such as the model router so that we take that decision away from the developer and try to map it to the right mod map the task to the model.

But it's also about having the right tripwires.

And so, we think about token limits in a few ways.

We never want to block the developer.

But we do want to make sure that there are moments where we can pause for a second and ensure that it's being spent on the right areas and in the right manner.

And so there are daily limits as an example, purely as a tripwire that a developer can unblock on their own.

All they have to do is go to the portal, click a button and say, "Hey, I Nothing surprising happened.

I'm in control."

Right?

And that's just one simple lever because everyone is looping and these loops can run overnight and we don't want this to run up a cost.

So, this is very reasonable.

On the other hand, we also want to make sure that our developers have the best models available.

But for those category of models, we've created a frontier budget.

And so, there is a monthly budget in place.

It's high enough.

You're not blocked.

If you need to be unblocked, there is an exception process.

Goes to your manager.

It's a moment where you describe what you're doing.

And we're not the decision-makers as a platform team, because those decisions should be close to where the work is happening.

In terms of using the frontier model, again, we give you full flexibility with a ceiling.

And the reason that works is that ultimately this is bringing predictability in the process.

Right?

We want to make sure that when we are budgeting our token spend, and this is a challenge for everybody, we've accounted for what we were planning and these levers allow us to do that.

You know what I find fascinating in these conversations?

I think it's because all of us work at companies.

And so, enterprises.

And so, we have to pay for every single token.

Right?

But there's an entire side of the internet where people are on subscription pricing.

And so, for them it's such a different calculus.

Right?

But I think you know, the it's a matter of time before you know everyone is in the same pool where you have to start thinking about every single token.

And that's when, you know, the other shoe is probably going to drop in terms of being a little more responsible, like we have to do right now in companies.

Or you work at a frontier lab where you get unlimited budget.

And then you just keep operating and you know creating a lot of these really exciting like AI native ways of working that are all over Twitter.

All of our companies want to mirror that, but they don't think they want to spend the tokens that…

We have people who work at Google here.

I'm very curious.

How does that work?

What's the budget from Google?

Do you want to take it, Luis?

I don't think I'm the right person to talk about the economics of it.

Google is a very large company with many different orgs.

But that actually wraps us up perfectly, cuz we are right on time.

That was convenient.

Saved by the bell.

That is A+ moderating, folks.

The PR people are…

So, anyway, thank you to our panel and yeah, thanks for coming to ADS.