Trace me if you can: Automated instrumentation and field profiling

Rahul Ravikumar

Good afternoon, everyone.

My name is Rahul.

I'm an engineer on the Android Developer Experience team.

I focus on app and library performance, and this talk is about tracing.

Before we dive deep into tracing, I want to establish some shared vocabulary so you can follow along.

So anytime I refer to tracing in this talk, I'm talking about explicit tracing, manually instrumenting code so you know what the app is doing.

I'm not talking about automatic profilers that I'll go into.

So manual tracing is the art of leaving breadcrumbs in your code base, so you can reason about it later when you have a performance problem or any other problem for that matter.

It's usually way better than logs.

So, I don't know if you've seen a Logcat output of a large app.

It's usually a giant wall of text that you can't reason about very easily.

Tracing, on the other hand, is a lot nicer.

It's easier to visualize.

You have durations.

You have some structure.

You have some tooling, so you can even slice and dice and see exactly what happened when.

And so it's a lot nicer.

And you can, you know, point your agents at a trace and say, ask questions.

You can use tools like Trace Processor, benchmarks, and even convert your traces into metrics.

So, these are all things that you can do with traces that it's hard to do with Logcat.

Profiling, on the other hand, and I talk about automatic profiling, is usually in response to a problem that you know of.

So, for example, you might have noticed that your app starts slow, or something is janky, and you want to narrow it down, and you want to figure out what is going on.

And at this point, if you don't have manual traces already to help you, this is where automatic profiling comes in.

You might use something like ART method tracing where you're instrumenting every single method entry and exit.

Or, you might be looking at stack sampling with tools like Simpleperf.

So, rather than using every single method entry and exit and having a high overhead, you look at call stack samples, and you see what is on the hot path of your app.

This is the year 2026, RAM is a problem everywhere, so you might be looking at memory, memory leaks and heap profiles.

So, this is a yet another profiler that you can take advantage of.

You know, you can look at heap retention paths.

You can look at what code is allocating memory, and those are all the kinds of analysis that you can do with this.

Right.

Now, we have that vocabulary.

So now let's talk about why we built a new library.

Tracing APIs, the 1.0 APIs were originally introduced in Android Jelly Bean.

That was in 2012.

That's over 14 years ago.

Since then, the APIs have not evolved as much as you think.

We added a few extensions.

Async events and counters got added in Android 10.

That was in 2019.

And that was it.

The API hasn't evolved since.

However, the ecosystem has evolved quite a lot.

So, platform tracing has improved by leaps and bounds, but none of that has trickled down in terms of API additions to you.

That is something that we want to change.

The other big problem is that tracing has a fixed tracing back end in 1.0, which means anytime you add a trace section in your code, it always ends up in the system trace buffer.

Atrace is the Android tracing subsystem that owns that data.

So, if you wanted to take your traces and convert them into field telemetry, that is really really hard, because now, a, you have to realize how atrace stores that data.

It's a well documented, but potentially hard to consume format.

So, that's something we want to fix.

So, we want to give you the tools to take your tracing data wherever you want.

And we also had a restriction in tracing 1.0 where labels were restricted to 128 characters.

Now, that was probably a reasonable restriction in 2012, but in the year 2026, that seems rather odd.

What makes it worse is that you can also not add any extra metadata to your trace, which is extra problematic.

And then, there's the overhead.

So, anytime you're tracing your app, you have to be extremely cognizant about the overhead, because the act of observing a system should not change what the system is doing under the hood.

So, if there's a race condition that you want to reproduce, tracing should not make the race condition go away.

So, you want to make sure you preserve all the semantics that you're doing otherwise.

Tracing 1.0 had a relatively high overhead, and we wanted to improve that.

Okay.

I talked a lot about the problems.

Now, let's look at some of the problems visually and see how they compound in practice.

So, here we have a Perfetto trace.

It's for an app, com.example.app.

We have a main thread, and we have a single slice called initializeComponent.

Now, ideally, we want some more details.

So, let's add a detail.

So now we know that initializeComponent calls performInitialize.

We still don't know what component is being initialized.

Now, the only tool in our tool belt with Tracing 1 is the name, so we add it to the name of the slice.

Now, remember, tracing slice names are restricted to 128 characters, so it gets truncated in your trace.

Now, to add insult to injury, the overhead of truncating is also now overhead that you incur when you're tracing.

So now, you run into this problem pretty quickly.

So, this is something we really wanted to fix.

You saw the Perfetto trace, and you saw what it looks like visually.

Now, let's look at from the perspective of writing the code to emit those trace sections.

Again, we have the initializeComponent API.

It takes a context and a name, calls performInitialize.

So far, so good.

Now, we want to add a trace with the Tracing 1.0 API, so you do that by importing androidx.tracing.trace and use a trace block.

You give it a name.

Here, it's performInitialize.

But like we wanted to do before, we wanted to know the name of the component, so we'll use Kotlin's string interpolation, and we'll add the name.

And here's the first problem.

So, here we have the 128-character limit.

Rahul Ravikumar

We can fix this with Tracing 2.

Instead of using androidx.tracing.trace, you use Tracer.

The API looks remarkably similar.

You have instead of trace, you have Tracer.global.trace.

You have a category.

We'll talk about categories in more detail later, but fundamentally, you have a name.

And then, you still have the block of code that you're trying to trace.

And now, you no longer have the limitation of 128-character limit.

I talked about tracing overhead.

I want to go in a bit more detail and see how we've improved this.

On a Pixel 6a running Android 16, the act of inserting a single begin and an end section took around 1,200 nanoseconds.

Now, that might not seem like a lot, but if you have really hot code that's adding a lot of trace sections, then it is time away that your app may be otherwise getting to spend on rendering frames.

So, we don't want to take that time away from you.

We actually improved things quite a bit with the AndroidX Perfetto SDK.

This was an experimental SDK that we introduced for Composition Tracing, and this actually improved the state of affairs a lot.

Now, we really never wanted you to ship this because this is the Perfetto C SDK with JNI bindings, and the C SDK is about 4 MB inside your APK.

This is not something that we ever wanted you to ship.

So, now with Tracing 2, we have a purpose-built library.

We've improved overhead a lot more, and now you'd be happy to know that Tracing 2 is 7x faster, which means you have more time to do the things that you wanted to do, and you know, infra takes less time.

What does something like this make possible that was not possible before?

So, one of the examples is Metro's runtime tracing feature.

So, Metro, for those who don't know, is a dependency injection library.

It supports Kotlin Multiplatform.

It uses a Kotlin compiler plugin.

It generates all the code for dependency injection, and it integrates with Tracing 2.0 to make it easier to visualize the cost of every single @Inject in your code.

So, here's an example.

Let's say we have an app graph.

In Metro, everything is scoped to a graph, and so we have a single field, AnalyticsTracker.

We also have a provides binding.

We have a network client with the OkHttpClient as impl, and then the implementation of AnalyticsTracker comes from the AnalyticsTrackerImpl, and we need an instance of the network client, which the graph knows how to provide.

So, under the hood, the compiler plugin generates code that looks like this.

So, if you have the AnalyticsTracker, the getter creates an instance of the network client and passes it to the constructor of AnalyticsTrackerImpl, which is good.

But now with tracing enabled, now you can see the cost of creation of not just the tracker impl, but every single dependency that it needs.

So now, we make it possible for you to see not just the cost of instantiating the type, but also all its dependencies in a trace, and that is really powerful.

So, using this information, Wise, which is a financial app, improved their app startup by 45%.

They were able to defer things that were not necessary to initialize during their app startup, and they also identified some wrongly scoped items, where things that should have been a singleton were not actually singletons by accident, and these are things that you can see and easily fix in your code.

If you look at their startup metrics, they're really impressive.

Not only did they improve the time to initial frame, but they also improved time to full display.

Another partner, BandLab, uses the same information on CI.

So, not only do you know the cost of every single @Inject binding, you also know which thread you know, incurred that cost.

So, once you know that, then you can find all the expensive @Inject bindings on the main thread and in your CI, you can run a query to say, hey, give me a list of all the expensive bindings and then attribute it back to the change that introduced them and then you can now blame the actual change and try and fix it.

Rahul Ravikumar

Okay.

So far, I've talked about all the things that we've improved vis-a-vis Tracing 1, but now I'm going to talk about new features that are in Tracing 2 that were never previously possible.

So, one of the core features is called Context Propagation and because modern apps are getting more and more complex, workloads run across multiple threads, multiple coroutines, and multiple processes.

You want to be able to visualize how a request propagates between every single thread versus process.

And that's what Context Propagation makes possible.

So, if I just gave you a list of slices and without telling you anything about what caused what, they're useful, but they're not super easy to reason about.

But instead if I told you that this was the causal chain of events that happened, then it makes much more easy for you to figure out what was going on and then associate the cost of each and every single slice.

This is what Context Propagation makes available.

You'd be happy to know that Context Propagation is fully supported and so, here's a real example.

Looking back at the com.example.app Perfetto trace, let's say performInitialize was taking too long and you wanted to defer the cost or move it to a different thread.

Something you can totally do.

So, let's say I moved performInitialize to a different thread.

Now, you've lost that hierarchical nature of the slice.

So, now you can no longer tell that initializeComponent was the thing that caused performInitialize to happen unless you looked at the code.

In an ideal world, you'd want an arrow to tell you that A caused B and then you can sort of reason about the causal chain of events.

You might have even seen a system trace where we actually do this for binder transactions on the platform.

So, here you can see that the Compose benchmark makes a binder transaction to system_server to prepare a frame.

So, this is stuff that's always been possible from the perspective of platform tracing, but we've never made the tools available to app developers.

So now, Kotlin coroutine Context Propagation is fully supported in Tracing 2.0.

You don't have to do anything.

You just get it out of the box.

So, here we have a single suspend method called fetchProfile.

It calls a client.get, which is another suspending method and then, it switches to Dispatchers.Default to deserialize the response.

Very cookie-cutter coroutines code, something you've probably dealt with every day.

Now, let's say we want to trace this.

We add a trace section.

So here, instead of trace, we'll use the traceCoroutine API given it's a suspending block of code and then, for the inner call before we context switch to Dispatchers.Default, we'll add another inner trace section using the same traceCoroutine API.

Now, in a trace, you will see something like this.

You will see that our work started in worker-1 and then, it actually context-switched and moved to worker-3 and you'll see helpful arrows in the trace that point that.

Not only that, if you zoom in to the square in the middle, you can see that deserialize actually started in worker-1 and the call to withContext is what caused the context switch to happen and then, it continued on to worker-3 and then, fetchProfile resumed on worker-3 because the coroutines machinery wants to minimize the number of context switches that happen.

You can see exactly when a coroutine suspended, when it resumed, and you can see the full causal chain of events.

Another thing that Tracing 2 makes possible is the ability to add custom metadata to a trace.

So, in the past, the only tool that you had was the name of the trace section, but now along with that, you can actually add a metadata block that allows you to add as many metadata entries as you want.

And you can add that to the trace.

When you look at the actual trace, here you can see that all the metadata that you added is actually in the trace for all the tools to consume and use.

So now, you can embed all you know, all the additional information that you've always wanted to.

So, I gave you a bunch of like abstract features that tracing makes possible, but where all this shines is a new feature in Jetpack Compose called Recomposition Tracing.

This is shipped as part of Compose 1.13, so this is ready for you to use today.

So, here's an example.

This is Composition Tracing that you're all probably familiar with.

This has existed since the early days of Compose.

It tells you the relative cost of a composable and all its, you know, subslices, and the cost of each on the thread that they were occurring in.

So, this is pretty straightforward.

What it doesn't tell you is the context.

It doesn't tell you why this happened.

It doesn't tell you the reason why things recomposed.

So, if I were to zoom out, that's the region of interest that we were just looking at in the previous slide.

And on the left, you can see that there is a dispatch input event that happened.

Maybe that's what changed some state, and maybe that's what caused recomposition.

But it's unclear.

You don't know for sure.

However, with recomposition tracing, now we add custom metadata into the trace that tell you exactly why recompositions happened.

Not only that, so it's very easy for you to tell now that these events are related.

And then if you click on them, it'll tell you exactly what state changed.

It'll tell you that it was a state, it was a mutable state value that was switched to false, which was previously true.

Not just that, we'll even tell you the exact call stack that caused it.

So, here you can see that, AbstractClickableNode.performClick happened, which caused mutable state to change, which caused the recomposition to happen.

So, this is really, really cool, something we haven't been able to do before.

These tools are also available, sorry, all of this functionality is also available in the new Android Performance Analyzer tool, if you want to take some of the agentic tools, you know, for a spin.

And right.

So, I told you that context propagation is fully supported for Kotlin coroutines.

But what about code that uses a mixture of coroutines versus non-coroutines code, or maybe you're running threads on your own, or maybe you have some brownfield code which is too old that uses callbacks for some reason?

Okay.

So, the good news is you can still get all the, you know, goodness of context propagation there.

So, let's say we have, you know, a brownfield API that uses a callback and an executor.

The implementation uses executor.execute, calls runCatching, which is the implementation detail, gets the response, and then tells the callback about the response.

Okay.

Now, if I wanted to trace this, then I do Tracer.global.trace, give it a label, and then surround the block.

But given I want to also preserve the causal chain of events, I'll accept a new, optional type, which is called propagation token.

And what I'll do is I'll just pass the token along to the inner trace API.

So, whoever's calling this, method can optionally give me the propagation token, so I can preserve the causal chain of events that happen.

Okay.

Now, the caller of that API can be a suspending API.

So, here I have a suspend fun profileRequest, which is now going to call the API that uses a callback and an executor.

Here, because I'm bridging two worlds, where I'm bridging the suspending world with the callback world, I'm going to use the suspendCoroutine builder.

I'll get a continuation, and then I'll just invoke the API, and then resume the continuation in the callback, once I get the response.

And then here's the executor that I'm defining is called profileRequestContext.

It's a single-thread executor.

And because traceCoroutine embeds a propagation token inside the coroutine context, I can obtain the propagation token from inside the coroutine context, and then pass that along as input to the brownfield API, so I can still preserve the causal chain of events.

And in the trace, it makes it very obvious.

So, here you have something that started in worker-3 that switched to the single-threaded executor, once we did the context switch, and then resumed back on worker.

So it's pretty neat.

Rahul Ravikumar

Okay.

One of the other things that you can do is, customize Tracer.global.

So far, the Tracing 2.0 API gives you really good defaults for what Tracer.global should be, but if you ever wanted to customize it, you can.

And also, Tracing 2.0 supports the JVM and Android because we want tracing to be fully supported for both Java and Android libraries.

So, the three steps for initialization are, you create a trace sink.

A trace sink is effectively how trace packets are stored.

It defines the wire format and the location of the output traces where they are stored.

We give you an out-of-the-box implementation for a trace sink that supports the Perfetto format, but you can obviously write your own trace sink.

This goes back to trying to solve the problem that atrace was the only back end that was available in Tracing 1.

We wanted to fix that.

So, now, if you want to write an OpenTelemetry sink or you want to bring your own sink, that is totally possible.

You can choose to override that.

Once you have a sink, you can create a trace driver, give it the sink.

And then you can also optionally choose which categories you want to, you know, turn on.

So, this is the part where categories really shine.

So, you can decide that in release, some categories are more important than others, when compared to debug, and you can say those are the categories that you want to turn on.

Or you can also just say, "I want to turn on all categories irrespective of where they came from."

And that's totally okay.

You can also add custom attributes that are global to your trace.

So you can embed things like what the build version was, what flavor, or any other metadata that you want to attach to a trace can be attached using the addAttribute API.

Finally, you can register a tracer by calling setGlobalTracer.

This is what binds Tracer.global to this instance of tracer, and subsequently, this is what is available for the process.

On Android, because processes have a much better defined lifecycle, all you have to do is to implement AbstractTraceDriver.Factory, override its create method, and you do exactly what you did before.

So you create the sink, and then you create the driver.

You don't have to register here.

We'll just do the registration for you, because we know that this is what happened first.

Okay.

All right, I want to reiterate that tracing is multi-platform ready.

This is a very intentional choice because we know Compose is running on multiple platforms, and we want to make it possible for you to trace in as many places as possible.

Today, we support the Android and JVM, but our implementation leaves it completely open for us to support future platforms if we wanted to.

Rahul Ravikumar

You'd be happy to know, R8, which is the tool that is used to shrink and optimize source code on Android, uses Tracing 2.0.

And by taking advantage of the traces, we've optimized R8 to actually shrink and optimize your code faster by 25%.

This means shorter build times and you know, shorter times to wait for optimization passes.

I won't go over every single optimization that we made in R8, but I just want to go over one, just to give you a taste of like what tracing makes possible.

So, R8, at its core, is a source-to-source transformer.

So, you get a bunch of JARs, and you shrink and optimize them, and you produce another set of JARs that you can dex.

So here, AGP starts by reading JARs, which is not just your app, but all its dependencies.

So, it uses Gradle workers to start reading all the input JARs.

You have many, many JARs that you want to read.

And then, once you finish the reading of the JARs, R8 starts to parse the code, and to build a model of what your code represents, and all its entry points, so it can start optimizing and shrinking your code.

Now, that's the slice, await, that represents when R8 is truly ready to actually start the process of optimization, right?

You might be seeing some dead space in the middle, and you might be wondering why we have that, you know, unused CPU time.

And the reason is because the way that the old API worked was we waited for all the workers to have read all their files first before we parse anything.

Obviously, we can fix this, which is exactly what we did.

So in AGP 9, you can see that the moment we finished reading the input JARs, we started processing them right away.

And we didn't have to wait for every single worker to finish reading their inputs.

This actually made the amount of time that we wait to read all the inputs shrink a lot.

So this is an obvious optimization that you can see is super obvious when you look at a trace, but not as obvious when you're just looking at source code to see what is worth optimizing for.

Another tool that we use every day as an AndroidX engineer is Metalava.

Metalava is a tool that we use to make sure that any APIs that we ship have the API compatibility guarantees that you expect, and are fully usable by external people without any footguns that we want to avoid.

This is a tool that runs on every single public API and restricted API, and so anytime that we save here is actually time that we get back as an AndroidX engineer.

Metalava also used tracing to improve post-submit times by a whopping 15%.

And going into the trace to explain what happened, so Metalava runs two types of checks: the first one is where it checks restricted APIs, and the second one is where it checks public APIs.

But you might be noticing a pattern.

We are doing a whole bunch of setup for running those checks twice.

So, what happens is all the program inputs are read, we build the PSI, which is the program structure interface, or, you know, exactly what the shape of your code is that can be reasoned about.

And then, we build the AST before we start applying all the lint checks that we want to.

And we are doing this expensive work twice.

How did we fix this?

We just did it once.

So, instead of doing all that setup work twice, we just did it once, and we reused those inputs for both restricted checks as well as public checks, therefore getting all those savings.

Yeah.

Okay.

Rahul Ravikumar

So, now that I have sort of given you a taste of what Tracing 2.0 makes possible, it's important to know that while tracing is important, knowing what to trace is also equally important.

You don't want to sprinkle traces everywhere.

You want to be very careful about what you pick, and I'm going to try and give you some guidelines on what is worthy of tracing.

So, the first thing that you should do is to find unexplained gaps in your trace.

So, anytime you're doing when your threads are in running or runnable state, and you see obvious gaps in your system trace, and you can't reason about them, and you don't know what your app is doing, that might be worth investigating.

At that point, I might take a method trace overlaid with the system trace, and see what is worth tracing, and I might add those relevant trace sections.

You also want to prioritize via user journeys.

So, you want to look at app startup.

You want to look at your core app user journeys, and you want to sort of figure out whether you can explain them by looking at a trace.

So, anytime you can't reason about something, that's worthy of tracing.

You should also look at deep call stacks or unexplained CPU usage from a trace and try and figure out exactly what is going on, and then, you should convert those parts of your codebase into traces.

So, you can trace them.

And then, anytime your threads are sleeping, you want to make sure that you can understand why they're sleeping.

They should be waiting for I/O, or they should be waiting for a binder transaction, or they should be waiting on some external input events that you expect to happen.

But if you have, you know, unexplained sleep, then those are extremely worthy of investigation because obviously there might be something lurking there.

And then, once you have you know, sprinkled all the trace sections in that code, you should convert them into benchmarks so you can track changes in them.

So, as your code base evolves, you can see if the metrics are shifting, if your dashboards actually reflect big changes, because then you know that something important changed.

Or you can skip all the previous steps and just have your agent do it.

Take these, you know, guidelines, make it a skill, and then point your favorite agent at it, and then have it take a go.

Yeah.

Rahul Ravikumar

So if I have convinced you that tracing is a worthy endeavor and you should do it, here's the project setup.

It's pretty straightforward.

Tracing is just a simple you know, just a traditional Java library, so you define a dependency on androidx.tracing:tracing-wire, which is the Perfetto-compatible trace driver plus tracing combo, and the current stable version is 2.0.5.

If you're a Java library, it's the exact same thing; again, this is a fully multiplatform API.

We want to extend it to other platforms when it makes sense.

And yeah, I just wanted to do a quick recap.

So, I talked about all these new cool features that we added.

So Context Propagation is really, really powerful.

We have the ability to embed custom information, metadata, call stack information, and all sorts of other things.

I have not talked about many, many things.

You can obviously change the tracing format, you can change your trace sink, you can, you know, take your tracing data to OpenTelemetry.

Those are all things that you can do.

Tracing is multiplatform.

This is again a very, very conscious decision because we want as many people to be able to take advantage of this.

And all the tooling that we ship whether it is Perfetto, whether it is AndroidX Benchmarks, whether it is the new performance analyzer, or the old Android Profiler, they all support Tracing 2.0.

With that, thank you so much for listening.

I hope you have a great dev summit, and thank you.