Memory troubleshooting with Firebase Crashlytics

Polina Govorkova

Hello, everyone.

Welcome to Memory troubleshooting with Firebase Crashlytics.

Wait, what a corporate title.

Let's mix it up a little bit.

Keeping apps lean and mean with Firebase Crashlytics.

Oh, that sounds better.

My name is Polina, and I am a senior software developer on the Firebase Crashlytics team, and your presenter today.

At some point in your development cycle, you will feel like you've solved all the bugs.

Those JSON parsing typos, the forgotten null pointer checks, all gone!

And you might ask yourself, "Am I done?"

Well, I have got bad news for you, because you're not.

Because what you're going to be left with is, actually, the hard problems.

And I claim today that the two categories of the hardest problems you're going to be facing are going to be responsiveness and memory.

The good news, we have got your back.

Today, we will walk through existing and brand new tooling in Android and Crashlytics to help you root-cause responsiveness issues manifesting through Application Not Responding errors, also out of memory exceptions, memory leaks, and native memory corruption.

Let's dive in.

Why are responsiveness and memory issues so notorious?

Take a classic bug: you declare a string, set it to null, and, at some point, try to read its length.

What do you get?

A good old NullPointerException.

But life is, actually, easy here, because the exception stack trace points to the exact line of code that broke your app.

Look at that stack trace on the left.

java.lang.NullPointerException
  at MainActivity.onCreate(MainActivity.kt:42)
  at android.app.Activity.performCreate
  at android.os.Handler.dispatchMessage
  at android.os.Looper.loopOnce
  at android.app.ActivityThread.main

We can tell right away that the issue is right there on line 42 in the MainActivity.kt.

And this approach is true for most trivial bugs.

Now, when a responsiveness or a memory issue hits, like this Application Not Responding on the right, the stack trace almost never points to the true culprit.

"main" prio=5 tid=1 Blocked
  | group="main" sCount=1 dsCount=0 flags=1
  | state=S schedstat=( 1450239 48192 145 )
  at DataManager.getData(DataManager.kt:25)
  - waiting to lock <0x0a1b> held by thread 12
  at MainActivity.onResume(MainActivity.kt:50)
  at android.app.Activity.performResume
  at android.os.Handler.dispatchMessage
  at android.os.Looper.loopOnce
  at android.app.ActivityThread.main

Best case, it only shows you the effect, which is some innocent thread or allocation that just happened to be running at the time when app froze or ran out of memory.

All while the real root cause happened way earlier, or, maybe, somewhere else entirely.

Well, luckily, when it comes to this issues, you don't have to guess in the dark, because Firebase Crashlytics has the answer for us.

But before we roll our sleeves with the troubleshooting, raise your hand if you're familiar with Crashlytics.

Oh, awesome.

Love seeing all those hands.

So, for those who are new here, Firebase Crashlytics is a real-time error reporting solution that provides field data, along with contextual and AI-driven insights, troubleshooting tips, and clear signal about your new releases, all to help you fix issues faster.

And Crashlytics is well-loved by our developers, rocking the status of the most adopted crash reporting solution on mobile, powering over 370,000 native 370,000 apps, including 57% of top Android apps.

To see how Crashlytics helps us track down bugs in action, let's look at a concrete example.

Today, I want to tell you the story of a ChromaClip.

It's a video color palette extractor that I've been building, and I want to tell you how I tackled the responsiveness and memory bugs while I was on the way of developing it.

So, the idea behind ChromaClip is simple: users pick a video, maybe a gorgeous sunset clip or a scene with great color grading, and the app analyzes the frames to generate a custom color palette, which then users can use for their designs, digital art, mood boards, anything.

So, our story starts right before my 1.0 release.

I wanted to make sure it's rock-solid before the launch, so I was running through final tests locally across couple of devices, picking the video, tapping the button to extract the color palette, and, just my luck, every Android developer's favorite dialogue, isn't it?

ChromaClip isn't responding.

So, this is an Application Not Responding error, otherwise known as an ANR.

Polina Govorkova

So, what do we do in this situation, aside from panicking?

Following our first instinct, since the test phone is plugged right into the laptop, we open Android Studio and check Logcat.

Well, we see the stack trace for the main thread.

14:32:15.105  1205-1250/system_process E/ActivityManager: ANR in com.chromaclip.app (com.chromaclip.app/.MainActivity)
    PID: 14205
    Reason: Input dispatching timed out
    Dumped ANR trace to /data/anr/traces.txt
"main" prio=5 tid=1 Native
  | group="main" sCount=1 dsCount=0 flags=1 obj=0x72a8c3d0 self=0x759fb92000
  | sysTid=14205 nice=-10 cgrp=default sched=0/0 handle=0x7623910548
  | state=S schedstat=( 1450239120 4819203 1453 ) utm=132 stm=13 core=4 HZ=100
  | stack=0x7fed2a0000-0x7fed2a2000 stackSize=8MB
  at sun.misc.Unsafe.park(Native method)
  - waiting on an unknown object
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:190)
  at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279)
  at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
  at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
  at com.chromaclip.app.extractor.PaletteExtractor.extractColors(PaletteExtractor.kt:42)

What does it, actually, tell us?

First, right on top, we get confirmation that it is an ANR, running in the MainActivity.

What else?

We see the reason, that is input dispatching timed out.

That means the user tapped on the screen.

Yes, we did.

But the main thread was so tied up that it couldn't process that event within the Android's five-second watchdog timeout.

Then, we also see that the system dumped the file into traces.txt and right below that, logcat prints the stack trace for our main thread.

Notice that state equals S over there.

Our main thread isn't even actively burning CPU.

S stands for sleeping.

Why is our main thread sleeping while we're waiting?

Let's look closer at the stack frames.

Right here in the middle of the trace, we see our main thread is parked inside Unsafe.park waiting on an unknown object.

Why don't we know what it's waiting for?

Well, it is actually true that for some ANRs, like when a thread is blocked by a classic Java synchronized lock, the main thread stack trace is kind enough to tell us what it's actually waiting for, what's holding the lock.

But when you use a higher level concurrency primitives under the hood, all logcat tells us is that we're waiting on an unknown object.

So, we have no idea which what object we're waiting for, which background worker thread we're waiting on specifically, or why it's taking over 5 seconds.

And since we are testing locally, we could pull that traces.txt of the device and manually scroll through dozens of sleeping threads, and possibly hunt down the culprit or perhaps burn the tokens asking AI.

In some cases, it would actually work, and in truth, it would work for this case.

But not only do I really not want to read raw thread dumps by hand-- honestly, that's so 2019-- I know that this approach isn't even viable at all once we are in production.

We just don't have direct access to traces.txt file in the user's phones, right?

So, good thing we have integrated with Firebase Crashlytics.

Because it removes the need for search and gets to the bottom of the ANR even in production.

How?

That's where Android's ApplicationExitInfo API comes in.

Starting in Android 11, whenever the system flags an ANR, or terminates your app process, the OS records rich exit metadata, including the exact reason for the crash, the memory state, and a complete system trace for every thread running in your process at the moment.

And the best part, you don't need to install any extra plugins or write a single line of code.

As long as you're on Crashlytics SDK 18.3.5 or higher, it just works out-of-the-box.

So, as soon as your app restarts, the Crashlytics SDK would automatically query the ApplicationExitInfo, capture the full multithreaded dump alongside the device telemetry, and send it to the Crashlytics back end.

Let's open up the Crashlytics console and see what it tells us.

This is the Firebase Crashlytics console issue details view for the ANR, and we see right away under the stack trace tab the DefaultDispatcher worker one.

DefaultDispatcher-worker-1 (Root blocking)

dalvik.system.VMRuntime.allocateNode (Native method)
android.graphics.Bitmap.createBitmap (Bitmap.java:1124)
android.graphics.Bitmap.createBitmap (Bitmap.java:1091)
com.chromaclip.app.extractor.FrameDecoder.getUncompressedFrame (FrameDecoder.kt:112)
com.chromaclip.app.extractor.PaletteExtractor$processVideo$2.invokeSuspend (PaletteExtractor.kt:65)
kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:33)
kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:106)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
java.lang.Thread.run (Thread.java:923)

This is in fact the thread that surfaced at the very top by Crashlytics that is the bottleneck for our main thread, and Crashlytics marks it with a badge called Root blocking.

As the tool tip on the badge explains, this worker thread is indeed the one that blocks the thread that triggered the ANR.

And by the way, Crashlytics has smart badges for other culprits too.

So say, if thread was stuck reading a giant file or performing a really slow database query, then if it triggered an ANR, the thread would be tagged as I/O root blocking.

And if two threads were stuck waiting on each other, then Crashlytics would tag them as deadlocked.

Now, let's look at the stack frames itself within the root blocking thread.

So, we've got our PaletteExtractor processVideo calling FrameDecoder getUncompressedFrame highlighted right there in blue, and that's completely stuck inside createBitmap just trying to allocate memory for a single frame.

Since we know that it isn't waiting on another lock, and we know that it's not in a network call, our prime suspect has got to be memory, and we can actually confirm it right away by checking the device's free memory right here in the Crashlytics console as well.

Look at the two places where Crashlytics surfaces this.

Down in the data tab for the specific event right next to the stack trace tab we were just exploring, and also in the top right showing the average across all events.

And in both cases, oof, looks like there's barely anything, not even 100 KB.

So, let's think.

What happened here is because the process was starved for memory, the Android runtime spent all its time frantically running garbage collection, trying to find enough space for that next bitmap, and never getting it.

Freezing our worker thread and starving the UI for over five seconds.

So, here is a quick tip for you.

Before you get, Whenever you get an ANR paired with this kind of, low free RAM, it's your smoking gun that ANR is actually either a memory leak or a bloat problem.

Now, the question is our app the victim or is it also the one that's hoarding the memory?

So, let's step back into our project's issue list and see if there is anything else that feels relevant.

Polina Govorkova

And look right there.

At the bottom, we see the list of the issues, and right above our ANR with the eight events, we've got OutOfMemory exception.

So, we might be the culprit after all.

And that's actually a huge clue.

Because on Android, a Java OutOfMemoryError, or OOM for short, doesn't just mean that the phone was low in RAM.

It means that our own app process blew right past its Android runtime Java heap limit.

So, at this point, my working theory is our ANR and OOM are actually the two sides of the same coin.

Depending on the timing during a test run, ChromaClip would either get stuck thrashing inside the garbage collection long enough to trigger an ANR, or completely exhaust heap, resulting in an OOM.

Let's investigate the OutOfMemory exception to know for sure.

Starting with the stack trace.

java.lang.OutOfMemoryError: Failed to allocate a 256 byte allocation
 at android.graphics.Bitmap.createBitmap
 at android.view.LayoutInflater.inflate(LayoutInflater.java:532)
 at com.chromaclip.app.SettingsActivity.onCreate(SettingsActivity.kt:21)

Wait a second.

Bitmap.createBitmap, that's familiar.

Didn't we see that one before?

Well, except look at where it's called from this time.

It failed to allocate 256 bytes when opening settings activity.

We're falling for the same trap again.

If your app crashes because it can't inflate a tiny settings icon, settings activity isn't going to be the villain.

It's just the unlucky caller that asked for a drop of memory when there was none.

So, to find the real culprit, we need to zoom out and see everything that was sitting in memory when the crash happened.

In other words, we need a Java heap dump.

At your desk, if you know how to trigger an OOM, you can manually grab a Java heap dump with Android Studio memory profiler.

But when an OOM happens unpredictably, across your test runs or out in the wild, again, this method just won't work.

Polina Govorkova

So, this is another shining moment for Firebase Crashlytics.

In addition to the application exit info, Crashlytics now integrates with Android's ProfilingManager API.

ProfilingManager API in Android 17 is supercharged with system-triggered memory profiling, and that's exactly what we need right here.

That means that right when your app hits an OutOfMemory error or breaches system memory limits, the OS would capture a heap snapshot in the background, and then in Crashlytics, the triggered collection comes for free out of the box as long as you are on Crashlytics SDK 20.1.0 or higher.

And then, Crashlytics would automatically group your OOM and Memory Limiter kills, we'll talk more about them later, in the dashboard.

And when you opt in to heap dump collection via Firebase console or the CLI, Crashlytics would start automatically collecting those captured heap snapshots, de-obfuscating them for you, and basically, you'll get a full x-ray of everything sitting in your app's Java heap at the moment of the crash.

So, here is what our ChromaClip OOM looks like in the Crashlytics console once the heap is captured.

At the top, you see an explanation of why Java exhausted the heap and why it happened.

Then, you will also see useful troubleshooting tips and pointers to the external resources.

Down in the stack trace tab, right above that misleading settings activity there, Crashlytics actually flags that OOM stack traces are not deterministic, so that people don't have to waste time looking at the stack trace like we just did.

Then, Crashlytics attaches the de-obfuscated Perfetto file, the heap dump, for the session.

And from here, we can download the file.

We can browse all dumps in Google Cloud Storage, or we can open it in the Perfetto UI.

So, clicking Analyze in the Perfetto UI opens Perfetto heap dump explorer.

Now, raise your hand if you have seen this one already.

All right, got a couple of hands.

To me, this view is very intimidating, but, in this case, the issue is actually quite simple.

Look at the set of massive bars spanning all the way almost through the entire Java heap.

So, for the most of it, it's an array of pair objects, each wrapping a color and some string instance.

Oh!

I actually know what's going on, I think.

Our code is allocating a color plus some supporting data for every single pixel in every frame, and holding into millions of them in a list.

So, no wonder Settings activity couldn't allocate those 256 bytes.

Well, at least one mystery is solved.

Let's fix it.

And at this point, we've got a couple of options.

So, since Perfetto already pointed us to the PaletteExtractor, we could open our IDE and rewrite the loop by hand, or use the AI coding agent.

Or we can bypass the manual heap inspection altogether, and leverage Android CLI.

# 1. Install Android CLI & Perfetto skills
$ android skills add android-cli perfetto

# 2. List recent heap dumps from Crashlytics
$ android signals heaps list --kind oom --timeframe=90d

# 3. Download a heap dump as a Perfetto trace
$ android signals heaps download <heap-dump-id> \
    --output ./heaps

# 4. Query top retained objects in the heap dump
$ trace_processor query ./heaps/oom.perfetto-trace ...

# 5. Fetch matching Crashlytics events
$ android signals heaps describe <heap-dump-id>

So, installing Android CLI and Perfetto skills will let you list and download deobfuscated Crashlytics heap dumps, query the top retained objects from these heap dumps through the trace_processor, and fetch matching Crashlytics issue data like custom keys, logs, breadcrumbs, associated rollouts, and more, all from your terminal.

Even better, look at the bottom of the slides.

Don't run this commands yourself.

Again, ask your favorite AI coding agent in natural language to inspect the latest OOM heap dump using Android command and ship the fix.

# Or ask your favorite LLM in natural language
> Using `android` command, inspect the latest heap dump
  for my OOM issue and propose a fix
● Ran android signals heaps list · download · describe
● Analyzed ./heaps/oom.perfetto-trace (perfetto)
● Updated PaletteExtractor.kt (+6 -4 lines)

And it will have sufficient context to do the whole flow for you.

So, listing OOMs, downloading the trace, running the analysis, and generating the code patch directly in your project.

And look at this patch in PaletteExtractor file.

// PaletteExtractor.kt
fun extractPalette(framePixels: IntArray): SparseIntArray {
-   val pixelColors = mutableListOf<Pair<Color, String>>() // 8.3M objects per 4K frame!
-   for (px in framePixels) {
-       pixelColors.add(Color.valueOf(px) to "#%06X".format(px))
-   }

+   val colorHistogram = SparseIntArray() // Downsampled & packed ARGB ints!
+   for (i in framePixels.indices step 16) {
+       val argb = framePixels[i]
+       colorHistogram.put(argb, colorHistogram.get(argb) + 1)
+   }
+   return colorHistogram
}

Instead of million of object allocations, we now just downsample the frame and count raw integers directly.

So, let's apply the fix, close the issues, bump our version, and keep testing.

Fingers crossed, not only should our out-of-memory error disappear, but since we won't be thrashing in garbage collection anymore, hopefully, that nasty ANR will go away, too.

So, we deployed the patch build across our test devices, pick a new clip, and start extracting palettes.

And the UI freezes again.

Yet another ANR.

Interestingly, testing more, we discovered that some devices In some of them, we still see that ANR, but for other devices, the app now just randomly crashes, and pretty frequently, too, more frequently than with those OOMs.

But we fixed the OOM, so what is going on?

Polina Govorkova

Well, back to Crashlytics we go.

Good news first.

Look at our issue list.

Our out-of-memory exception is completely gone, like we were hoping for.

The bad news?

Now that the ChromaClip is surviving just a little longer, it has the time to uncover what else was hiding underneath.

And so, our ANR didn't go away.

And now we've got two native crashes at the bottom thankfully with just a couple of events.

And right at the very top, with 11 events, we have the Memory Limiter kill, thank goodness, already badged as having the heap dump.

Wait.

What is a Memory Limiter kill?

Can I get a raise of hands who guys, who of you have heard of it?

All right.

A few rare hands.

Let's learn together.

If you haven't seen Memory Limiter kill before, that's because it's brand new on Android 17.

And a Java OOM only triggers when you blow past the managed Java heap limit, but when your app total memory footprint also includes native RAM, like graphic buffers and C++ allocations, before, there was no per-app ceiling.

So, we would have the low memory killer, which was global.

It would step in when some app would gobble up the system RAM all the way until the whole phone choked.

Now, starting Android 17, the Memory Limiter caps each app's individual memory footprint.

So, when the process consumes the disproportionate amount of RAM, the OS terminates that specific app.

And Crashlytics captures all those details and groups terminations together in the dashboard.

So, clicking into Memory Limiter kill in Crashlytics, just like with the OOM, you get an explanation of why the OS stepped in, so you don't need to memorize what I just said.

You also get tips on how to tackle it next, and links straight to Android memory enforcement docs.

And just like with OOM, the stack trace blames an innocent frame when the OS pulled the plug, and Crashlytics dashboard highlights it.

Now, Memory Limiter kills happen primarily in native code because native environments are missing that layer that handles memory safety and allocations.

Otherwise, it would have been an OOM.

So, our first question is, are our Kotlin objects holding onto huge native-backed allocations?

Or is it a raw C++ code that's leaking on its own?

Well, let's click Analyze in Perfetto UI to find out.

Look at the Overview tab in the Perfetto heap dump explorer.

We see that under general information, anon RSS plus swap shows us that total process memory hit 4 GB.

Okay.

Yet, below that, we see that under Bytes Retained by Heap, our entire managed heap, including native memory tied to Kotlin objects like bitmaps, is only 42 MB.

Okay.

So, in 10 seconds, we ruled out all the Kotlin and Java code.

So, the culprit has to be leaking from a native C++ layer.

And since ChromaClip has a thin C++ layer for frame processing, we don't have to look far.

Pointing our coding agent or Android Studio's native profiler at NativePaletteEngine.cpp processFrame, the bug is right there.

// NativePaletteEngine.cpp
void NativePaletteEngine::processFrame(const uint8_t* rgba, size_t size) {
    uint8_t* scratch = new uint8_t[size]; // Temporary frame buffer
    transformColorSpace(rgba, scratch, size);

    // BUG: Allocated every frame, but never freed!
+   delete[] scratch; // Free native buffer when done
}

So, on every single video frame, our C++ code allocates a temporary scratch buffer, not so temporary if you don't release it, what a rookie mistake.

So, time to connect the dots.

What happened?

Once we fixed the Java leak, the C++ leak will have the time to accumulate and run wild.

On Android 17, the Memory Limiter caught it and killed the app.

But on the older devices where Memory Limiter was not there yet, it actually manifested as an ANR.

So, finally, it all makes sense.

Let's add the delete[] scratch over here, bump our version, and ship the update.

With Memory Limiter kills and ANRs finally gone, we should be good to ship to 1.0.

Exciting.

And thanks to how fast and stable ChromaClip is now, our app takes off.

Thousands of people are uploading and remixing the videos every single day.

But with all the traffic, remember those mysterious native crashes that we saw earlier at the bottom of the dashboard?

Well, they're running wild too.

Suddenly, there are way more of them.

And our job is not done just yet.

Polina Govorkova

So, opening that native crash in Crashlytics, look at the stack trace.

There isn't a single line of code from our own app.

RenderThread crashing deep inside libc.so while freeing the memory.

And another tip for you here, when a system thread crashes inside memory allocator with none of your app code inside, that's a textbook sign of native memory corruption.

So, our C++ code corrupted or freed the buffer earlier, and render thread just tripped over it later.

So, how do we catch the real culprit at the exact moment corruption happened?

Android gives us two powerhouse tools, and Crashlytics supports both of them out of the box.

First, the GWP-ASan, which is a lightweight safety net built specifically for production.

It samples a tiny fraction of your native heap allocations and surrounds them with guard pages so that CPU and memory overhead is nearly zero.

Across thousands of user sessions in the wild, when a bug touches freed or out of bounds memory on a sampled allocation, GWP-ASan catches it right away.

However, because GWP-ASan is, by nature, probabilistic, when you test things locally, you don't want to run your test phone for 16 or more hours to hit the random sample.

So, what do you get then?

For deterministic local debugging, there you have the ARM MTE, or Memory Tagging Extension.

MTE uses a hardware pointer and memory tags to check every allocation, and it's available on ARM v9 devices like Pixel 8 and newer.

So, pair it with GWP-ASan in production, and you get the best of both worlds.

And as you might have expected, when either tool catches the bug, Crashlytics automatically badges the issue and lets you filter by GWP-ASan or MTE and pins info card with the exact corruption cause in the dashboard.

So, look at what happens in the Crashlytics console when GWP-ASan catches the crash.

Next to the regular stack trace tab, we've got the memory stack trace.

And at the top, we see the exact corruption cause, Use After Free.

And instead of showing render thread crashing in libc.so, GWP-ASan now gives us the full biography of that buffer inside our own effect_pipeline.cc.

So, under Allocated By, we see where the buffer was born.

And right under Deallocated By, we see the exact culprit that freed it from under the render thread.

And that is hiding on line 208 on our effect pipeline.

So, we open our native code in the EffectPipeline, and the bug is grinning at us.

// effect_pipeline.cc
void EffectPipeline::onFilterSwitched(FilterPresetId preset) {
    // BUG: Freed on worker thread while RenderThread is still reading it!
+   std::lock_guard<std::mutex> lock(lut_mutex_);

    releaseLutBuffer();       // Frees old color-grading LUT
    loadFilterPreset(preset); // Allocates new color-grading LUT
}

Whenever a user switches the color filter in the background thread, we free the old buffer while the RenderThread is still reading it.

Okay, so we add the mutex lock around the buffer swap, ship the update, and finally watch both GWP-ASan reports and all those mysterious crashes vanish almost overnight.

And our crash-free users metric skyrocketed to 100%.

Woohoo!

Remember, back at the start of the talk we asked ourselves: "Am I done?"

Right before getting hit by basically every single responsiveness and memory boss in the book.

Well, the ANRs, the Out of Memory errors, the memory limiter kills, and the native memory corruption crashes are now all solved.

And even if the stack trace lies and the culprit is hiding somewhere across the threads in C++, Android and Firebase Crashlytics give you the tool to track down and ship with confidence.

Polina Govorkova

So, let's recap what you can use right when you get back to your laptop: ANR thread insights and other relevant metadata via ApplicationExitInfo; automatic collection of OOM and memory limiter kill triggers via ProfilingManager; and full GWP-ASan and Arm MTE support for native memory corruption.

And launching in private preview, field heap dumps on Android 17 or higher.

If you want to be among the first ones to try the automatic heap dump collection on Firebase Crashlytics and analyze them with Perfetto and Android CLI, please grab a quick photo of the slide, and sign up for the link.

So, now that we have gone through most of the talk, I want to thank you for your joining me today and for your time.

May your main thread stay unblocked, your heap stay clean, and your next release be the stable one yet.

So, happy debugging.

Thank you so much, again.