Elevating camera experiences across devices with CameraX and on-device intelligence

Hello, everyone.

Welcome to our session.

I am Aman, and with me is Scott.

Today, we are really excited to talk about how you can elevate your camera experiences across devices with CameraX and on-device intelligence.

So, let's get started.

Think about how you use your phones.

We often use the phone camera to capture moments and share with our loved ones.

The camera has become central to the smartphone experience.

In fact, a good camera is often a key factor while purchasing smartphones.

Today, we'll cover two broad themes for building premium camera experiences.

First, we'll explore some CameraX features, and then, we learn about some on-device machine learning techniques.

Let's begin with CameraX.

Jetpack CameraX powers the camera experience on Android.

Apart from CameraX, we also have Camera1 and Camera2, which are part of the Android platform framework.

So, which library should you use?

Camera1 was the original camera API on Android.

It's now deprecated, so new apps should either use Camera2 or CameraX.

Camera2 succeeded Camera1.

So, if you require low-level camera control for complex use cases, then Camera2 is a good option.

However, it requires handling device-specific configurations yourself.

And finally, Jetpack CameraX builds on Camera2.

It provides a consistent API and handles device-specific configurations automatically.

It's the recommended choice for most developers.

We have a very important call out to make.

We are planning to disable Camera1 and this will start with Play policy enforcements next year in Android 18.

If you're still using the legacy Camera1 API, you will need to migrate to either CameraX or Camera2.

So, why should you use CameraX?

First, it supports Android 7 and above, thus making it compatible with almost 99% of active Android devices.

Second, it's easy to use and provides simple APIs for common use cases like preview, capture, and recording with less code.

Third, it provides a consistent experiences across devices by reducing device-specific issues such as preview stretching, which is quite common on foldables and tablets.

And finally, it simplifies implementation of advanced OEM capabilities such as night mode and portrait.

Now, I would like to invite Scott to talk about some of the CameraX features.

Over to you, Scott.

Thanks thanks, Aman.

Hello, everyone.

My name is Scott.

And today, we will cover several CameraX feature to enable you to build a professional-grade camera experience.

And first up is Adaptive Viewfinder.

It is designed to completely take the pain out of handle preview across different form factor and screen sizes.

Users now have a variety of devices, including foldable, phone, and laptop, and XR devices.

So, it's more important than ever to build a camera experience adapt to the new form factors.

And camera preview is notoriously difficult to get right, especially on the widescreen foldable.

On these devices, when unfolded, its inner camera is mounted horizontally, while its back camera is mounted vertically.

So, if you assume all camera sensor are vertically mounted, then you will lead to a spatial mismatch and causing the distortion or the sideways.

And CameraXViewfinder, it designed to completely handle the complexity for you, so ensuring your camera preview work seamlessly across all the form factors.

And let's see how to use the CameraXViewfinder composable.

var surfaceRequest by remember {
    mutableStateOf<SurfaceRequest?>(null)
}

LaunchedEffect(...) {
    preview.setSurfaceProvider { surfaceRequest = it }
    cameraProvider.bindToLifecycle(..., preview)
}

surfaceRequest?.let { request ->
    CameraXViewfinder(
        surfaceRequest = request,
        contentScale = ContentScale.Fit
    )
}

First, you declare the surfaceRequest content variable to hold the surface request.

Sorry.

And second, second, in LaunchedEffect, you set the surface provider to catch that a surface request object.

And then, you bind to lifecycle, and then start the camera.

And finally, whenever the state update with a valid surface request and trigger recomposition, you set the request directly to the CameraXViewfinder.

And here, you can set the contentScale to fit or crop, depending on your preference.

And that's it, you are all set.

In CameraX 1.7, we have added several new features to the CameraXViewfinder.

This including new gesture like tap to focus or pinch to zoom.

It also supports Screen Flash, which lets you use a screen to brighten your selfie.

Implementing this feature is easy.

CameraXViewfinder(
    surfaceRequest = viewModel.surfaceRequest,
    contentScale = ContentScale.Fit,
    isTapToFocusEnabled = true,
    isPinchToZoomEnabled = true,
    onScreenFlashReady =
        {
           imageCapture.screenFlash = it
           imageCapture.flashMode = FLASH_MODE_SCREEN
        }
)

You just enable the tap to Focus and pinch to Zoom to true.

And for the Screen Flash, you get the Screen Flash instance from the onScreenFlashReady callback and then pass to the ImageCapture and set a flash mode to FLASH_MODE_SCREEN.

And that's it.

We have seen how CameraX enables camera Preview that works seamlessly across all form factors.

Now let's take step further with a camera capture on Android XR.

So there are three categories of XR devices: first, XR Headsets.

These are fully enclosed and immersive headsets that using camera to capture the physical world, display inside the headsets.

And second, Wired XR Glasses.

We have in- announced the XREAL Aura in the keynote in this category today.

And so as you can see in the keynotes, these are the see-through glasses connected to the compute puck and letting you view the virtual content overlay on the real world.

And the third, Audio and Display Glasses.

These are the lightweight glasses.

They can combine audio, camera, and display for hands-free and glanceable experiences.

And the first two category, XR headset and Wired XR Glasses, they also support Camera2 and CameraX, just like a regular phone.

So today we will focus just audio and Display Glasses.

On these glasses, apps run on the Android phone, host Android phone, instead of glasses, but here access to the both of the phone camera and the glasses camera.

The key question is how do you know which camera you are talking to?

So it's simple.

With standard Context, your CameraX accesses the phone camera.

And with the ProjectedDeviceContext, CameraX accesses the glasses camera.

So different Context will lead to different target camera.

And remember camera and glasses permission are separate.

Phone and glasses permission are separate, so you need to request request glasses permission separately.

The other option is can also use the Projected Activity, which also run on the host Android devices, but your UI is projected onto the glasses.

And here the rule flips.

If you're using standard Context, you will access the glasses camera.

If you want to access the phone camera, then you need to use the HostContext.

And let's look at code, see how we can capture using the glasses camera.

// In a coroutine
val imageCapture = ImageCapture.Builder().build()

ProjectedContext.isProjectedDeviceConnected(activity, coroutineContext)
    .collectLatest { isConnected ->
        if (isConnected) {
            val projectedContext = ProjectedContext.createProjectedDeviceContext(activity)
            val cameraConfig = CameraXConfig.Builder.fromConfig(Camera2Config.defaultConfig())
                .setRepeatingStreamForced(false)
                .build()
            val cameraProvider = LifecycleCameraProvider.createInstance(projectedContext, cameraConfig)
            cameraProvider.bindToLifecycle(activity, DEFAULT_BACK_CAMERA, imageCapture)
        }
    }
// Shutter button clicked
imageCapture.takePicture(...)

First, you invoke isProjectedDeviceConnected to see if the glasses are connected or not.

And then you call createProjectedDeviceContext to get a projectedContext.

And second, here you use the LifecycleCameraProvider to initialize CameraX, instead of ProcessCameraProvider.

So ProcessCameraProvider is a singleton that you cannot change the Context af- after initialization.

So using LifecycleCameraProvider allow you to change the Context anytime you want.

And next, you need to disable additional repeating stream in CameraX.

This is needed because today's glasses support only single stream.

And the API is currently internal, but we will make it public soon.

And that's it.

From here is standard CameraX, you just bind the ImageCapture, and you're now capturing through the glasses.

Okay.

Next, let's see how Next, see how CameraX helps you find the feature combination that work together and get the best from the device.

Modern camera apps are defined by powerful features and user expect that camera to support feature like HDR, 60 FPS, and Preview Stabilization, all at the same time.

But as developer, we know combine these features together isn't always straightforward.

Checking individual support feature support doesn't guarantee they will work together.

And this uncertainty has forced developer to be conservative and limiting what user can experience on capable devices.

To solve this, we introduced the Feature Group API, which is powered by the Camera2 feature combination query framework.

And there are two way to use it: first, use requiredFeatureGroup to check and enable feature combination when it is supported.

And second, you can provide your feature priority by preferredFeatureGroup, and then CameraX choose the best supported so combination for you.

And here is the example.

On the right, on the left, sorry, with enable nothing, just a regular phone, regular recording.

On the left, we enable HDR, 60 FPS, Preview Stabilization all together.

And you can see the quality is improved significantly with the brighter, smoother, and the shake-free video.

And let's look at the code.

cameraProvider.bindToLifecycle(
    lifecycleOwner,
    cameraSelector,
    SessionConfig(
        useCases = listOf(preview, videoCapture),
        // HDR_HLG10 > FPS_60 > Preview Stabilization.
        preferredFeatureGroup =
           listOf(HDR_HLG10, FPS_60, PREVIEW_STABILIZATION),
    )
)

So here we use the preferredFeatureGroup to provide our feature priority list, which is HDR, and then 60 FPS, and then Preview Stabilization.

And CameraX will check the combination and select the best ones supported by the device.

With CameraX 1.6 above, we're adding 4K Recording and Video Stabilization right into the Feature Group API.

// CameraX 1.6+ supports:
// 4K, video stabilization & Effect
val sessionConfig = SessionConfig(
   useCases = listOf(preview, videoCapture),
   requiredFeatureGroup =
     setOf(UHD_RECORDING, VIDEO_STABILIZATION),
   effects = listOf(overlayEffect)
)

Combining 4K and Video Stabilization is supported by many premium devices.

Now with Feature Group API, you can enable them together with confidence.

On top of that, you now have flexibility to apply the real-time effect with Feature Group API, which was not possible previously.

Now let's talk about how you can get high quality image capture in CameraX.

CameraX now support DNG RAW capture.

It give you access to the unprocessed, uncompressed RAW image directly from camera sensor for professional editing.

And next, we have improved our Night Mode extension.

This extension enabled to capture bright and clear photo in low light without using flash.

And we have introduced a new Night Mode Indicator API.

It will monitor when the current environment is dark and it's suitable for Night Mode, and you can use it to show the Night Mode indicator.

And Ultra HDR is now supported in CameraX extensions.

This brings a wider dynamic range and vibrant color to the Night Mode, and Night Mode with Ultra HDR is currently supported on many recent premium devices.

You can see the difference here from normal, night, and Night Mode with Ultra HDR.

So, let's see how to implement this.

val cameraInfo = cameraProvider.getCameraInfo(cameraSelector,
        ExtensionSessionConfig(ExtensionMode.NIGHT, extensionsManager))

val imageCapabilities = ImageCapture.getImageCaptureCapabilities(cameraInfo)

val imageCapture = ImageCapture.Builder().apply {
       if (imageCapabilities.supportedOutputFormats
                .contains(OUTPUT_FORMAT_JPEG_ULTRA_HDR)) {
           setOutputFormat(OUTPUT_FORMAT_JPEG_ULTRA_HDR)
       }
     }.build()

val extensionSessionConfig = ExtensionSessionConfig(
    ExtensionMode.NIGHT, extensionsManager, preview, imageCapture
)

cameraProvider.bindToLifecycle(...,  extensionSessionConfig)

First, you create an ExtensionSessionConfig for the Night Mode, and use it to get the CameraInfo instance.

And next, you use the CameraInfo to get the image capture capability, and then query if the Ultra HDR is supported.

And then you set the output format to Ultra HDR.

And finally, you create the ExtensionSessionConfig again with the use cases, and bind to lifecycle.

Now you're ready to go to capture with the Night Mode with Ultra HDR photos.

We also bring the Motion Photos support to CameraX.

Motion Photos capture the sound and movement around the photo, preserving the moment, not just a still image.

We standardize this across different phone, Android devices, and make Motion Photos easier to support.

Motion Photos is coming to CameraX in 2027 Q1.

Once it's available, simply set the Motion Photos mode in ImageCapture.Builder, and now you can start capturing the Motion Photos.

val imageCapture = ImageCapture.Builder()
        .setMotionPhotoMode(
             ImageCapture.MOTION_PHOTO_MODE_ON,
             customConfig
         )
        .build()

cameraProvider.bindToLifecycle(.. imageCapture)
imageCapture.takePicture(...)

Next, let's explore how CameraX enable high speed video recording.

Recording fast-moving action with clarity is an essential feature by modern camera apps.

It's especially useful for fast-paced scenarios like sports.

This is achieved through high-speed capture at 120 or 240 fps.

There are two ways to use it.

First, encoding at the same frame rate, then you will creating a true high-frame-rate video.

The other option is to encode it at 30 fps to create a ready-to-play slow-motion video.

For example, one second recorded at 120 fps takes four second to play at 30 fps.

So, this stretch of time creates a slow-motion effect.

And to implement high-speed recording, you create a HighSpeedVideoSessionConfig with your videoCapture.

val recorder = Recorder.Builder().build()
val videoCapture = VideoCapture.withOutput(recorder)

val frameRateRange = cameraInfo.getSupportedFrameRateRanges(
       HighSpeedVideoSessionConfig(videoCapture, preview)).first()

val sessionConfig = HighSpeedVideoSessionConfig(
    videoCapture,
    preview,
    frameRateRange = frameRateRange,
    isSlowMotionEnabled = true
)

cameraProvider.bindToLifecycle(
     lifecycleOwner, cameraSelector, sessionConfig)

// Start recording slow motion videos.
val recording = recorder.prepareRecording(context, outputOption)
      .start(executor, {})

And then you set isSlowMotionEnabled to true to encode it at 30 fps to create a slow-motion video, or set it to false to create a high speed frame rate video.

And when slow motion is enabled, Google Photos automatically identify it, and add a slow-motion badge on the sum-- to the thumbnail.

And let's see how it work in Google Photos.

Here the slow motion gives a dramatic effect to that cat.

My cat never want me to touch it.

So, sorry.

Finally, let's see some improvement in our concurrent API, Concurrent Camera API.

Concurrent API let your app stream from the front and back camera simultaneously.

This year, we introduced some improvement to the Concurrent Camera API.

You can dynamically update the camera composition after binding to the lifecycle and opening a camera.

So, this including changing the position, and size, and rendering order.

You can also add styles like a rounded corner, or border width, and border color.

And here's how to do it: just use the CompositionSettings to configure styling like rounding corner, border width, and color, and use a setZOrder to adjust the rendering order.

val concurrentCamera =
        cameraProvider.bindToLifecycle(list(config1, config2))
// CameraX 1.7 features: rounded corner, border width & color
val pipSettings = CompositionSettings.Builder()
    .setOffset(0.2f, 0.2f)
    .setScale(0.35f, 0.35f)
    .setRoundedCornerRatio(0.15f)
    .setBorderWidthRatio(0.02f)
    .setBorderColor(Color.WHITE)
    .setZOrder(1)
    .build()
// CameraX 1.7 features: dynamical composition update
concurrentCamera.setCompositionSettings(
listOf(mainSettings, pipSettings))

And the next, you just invoke the setCompositionSettings on the active concurrent camera to update the composition dynamically.

And that's it.

You are okay to go to update the composition dynamically.

Okay.

Now we have gone through all the features.

We have an important call-out here, so we highly recommend you to upgrade to CameraX 1.6 and all above.

These can avoid crash on upcoming devices that support the new format, Eclipsa Video, and also ensure your CameraX extension continue to work on the feature future Pixel update.

With that, we wrap up our first session.

We have covered CameraX features that help you build a professional-grade camera experience, and now I would like to invite Aman to cover our next session on on-device AI.

Thank you, Scott, for walking us through the CameraX features.

Now, let's take a look at how you can leverage On-Device Machine Learning for building intelligent camera experiences.

On-device machine learning enables a broad range of camera use cases across two main areas.

First is pre-capture.

Here, you can analyze your camera preview in real time for doing things like object detection or you can enhance the preview itself.

Second is post-capture.

Once you have captured your image, you could enhance its quality or do things like generating a description for the image.

Now, let's dive into some of the on-device machine learning solutions we have for your camera apps.

The first on-device ML library that we're going to talk about is ML Kit.

ML Kit was primarily built to solve common problems in vision and natural language domains such as text recognition, face detection, and language translation.

Last year, we introduced the ML Kit sorry.

Here are some examples of ML Kit Vision APIs, for example face mesh detection, subject segmentation, and real-time object detection.

Yes.

Last year we introduced the GenAI APIs.

These APIs harness the power of Gemini Nano to enable capabilities like text summarization, image description, and much more.

Here is an example of image description using the Prompt API.

We pass the image bitmap with the text prompt to the model, and the result description reads as a small green Android robot with a cactus-like design sits on a black surface.

And all of this happens on the device.

Pretty cool, right?

Adobe Photoshop has been using the Prompt API to improve the user experience in their app.

While editing assets, users often have to manage multiple layers in Photoshop app.

Using the Prompt API, now the app is able to automatically rename the unorganized layers.

The app also uses the Prompt API to generate captions and hashtags for images, which is pretty useful for sharing on social media.

Let's look at the code for image description using Prompt API.

// 1. Initialize the on-device GenerativeModel client
val generativeModel = Generation.getClient()

// 2. Check whether Gemini Nano is available on the device
val status = generativeModel.checkStatus()
when (status) {
    FeatureStatus.UNAVAILABLE -> {}
    FeatureStatus.DOWNLOADABLE -> {}
    FeatureStatus.DOWNLOADING -> {}
    FeatureStatus.AVAILABLE -> {}
}

First, you need a generativeModel client object, then check whether Gemini Nano is available or downloadable on your device.

Next, generate a content request by combining your image and the text prompt, and finally supplies this request to the generateContent call to run the inference on Gemini Nano and get your image description.

// 3. Build a multimodal request with the image and text prompt
val request = generateContentRequest(
    ImagePart(bitmap),
    TextPart("Describe this image in detail.")
) {}

// 4. Run inference on-device and get the description
val response = generativeModel.generateContent(request)
val description = response.candidates.first().text

Yes, the next on-device ML library we have is MediaPipe.

MediaPipe offers a wide range of machine learning solutions for your applications.

You can quickly visit the MediaPipe website to try out different samples such as face detection, object detection, image segmentation, and much more.

All of these use cases sound very similar to ML Kit, so which library should you use?

While MediaPipe and ML Kit solve similar problems, there are few key differences.

First is platform support.

MediaPipe works across Android, iOS, Web, and Python, whereas ML Kit focuses only on Android and iOS.

Next, MediaPipe offers model customization, whereas ML Kit provides ready-to-use APIs.

And finally, while MediaPipe is open source, ML Kit is a Google SDK.

Here is an example of gesture recognition using the MediaPipe Vision APIs.

This API detects the hand gestures in real time from camera input.

This could be used, for example, to start or stop the camera recording using simple hand gestures while standing at a distance from the phone.

Let's look at the code.

So first, you need to build the GestureRecognizerOptions object, providing the model path, live streaming mode, and a result listener.

// 1. Build GestureRecognizerOptions
val options = GestureRecognizerOptions.builder().setBaseOptions(
    BaseOptions.builder().setModelAssetPath(MP_RECOGNIZER_TASK).build())
    .setRunningMode(RunningMode.LIVE_STREAM)
    .setResultListener { result, image -> /* Handle detected gesture */ }
    .build()

// 2. Create the GestureRecognizer
val gestureRecognizer = GestureRecognizer.createFromOptions(context, options)

// 3. Convert CameraX ImageProxy to MediaPipe MPImage
val bitmap = rotateBitmap(imageProxy.toBitmap(),
imageProxy.imageInfo.rotationDegrees)
val mpImage = BitmapImageBuilder(bitmap).build()

// 4. Run gesture recognition on the image
gestureRecognizer.recognizeAsync(mpImage, frameTime)

Then create a gesture recognizer object from the options.

Next, get the CameraX ImageProxy and convert it into a bitmap, and use the bitmap to generate a MediaPipe mpImage object.

Finally, you need to supply the MediaPipe mpImage object to the recognizeAsync call to detect gestures in real-time.

Next up, we have On-Device Image and Video Enhance available through the Google Play services API.

This model provides capabilities like tonemapping, deblurring, and Super Resolution on most premium devices.

Here is an example of an image tone mapped and upscaled using this model.

You can clearly see the richer colors in the sky and the sea and improved sharpness in the enhanced image.

Let's look at the code for Image Enhancement.

// Step 1: Verify device support & install module
val enhancementClient = Enhancement.getClient(context)

suspend fun prepare(): Boolean {
    if (!enhancementClient.isDeviceSupported().await()) return false
    if (!enhancementClient.isModuleInstalled().await()) {
        check(enhancementClient.installModule().await()) {
            "Enhancement module install failed"
        }
    }
    return true
}

First, you need to query whether the underlying device supports it or not, and if necessary, you need to install the enhancement module on your device.

Next, create the EnhancementOptions object by specifying the bitmap and the enhancements that you want to opt for.

// Step 2: Select Enhancement options & create session
val options = EnhancementOptions(
    width = bitmap.width,
    height = bitmap.height,
    enhancementMode = EnhancementMode.BITMAP,
    isTonemappingEnabled = true,
    isDeblurAndDenoisePhotoEnabled = true,
    isDeblurAndDenoiseVideoEnabled = false,
    isUpscalePhotoEnabled = false,
    isUpscaleVideoEnabled = false
)

val enhancementSession = enhancementClient
    .createSessionAsync(options, enhancementExecutor)

Next, create a client session with Google Play services API, then define a callback to handle your resultant bitmap, and finally call the process function with the bitmap and the options to enhance your image.

// Step 3: Process Enhancement
val enhancedBitmap = suspendCancellableCoroutine { continuation ->
    val enhancementCallback = object : EnhancementCallback {
        override fun onBitmapProcessed(result: Bitmap) {
            if (continuation.isActive) continuation.resume(result)
        }
        override fun onError(statusCode: Int) {
            if (continuation.isActive) {
                continuation.resumeWithException(
                    Exception("Bitmap processing failed: $statusCode")
                )
            }
        }
        override fun onSurfaceProcessed(timestamp: Long) {}
    }

    enhancementSession.process(bitmap, options, enhancementCallback)
}

We have recently integrated the Video Enhancement APIs into Jetpack Media3.

You can check out our media talk to learn more about the Image and Video Enhancement APIs.

The last On-Device ML feature that we're going to cover is Low Light Boost.

Capturing in low light is a common problem for smartphone users.

Think of users capturing image or taking video calls in dimly lit surroundings.

Low Light Boost solves this problem by brightening up your camera preview in real-time.

There are two variants of Low Light Boost: hardware and software.

The hardware version uses the OEM hardware to enhance the dimly lit frames without using machine learning.

Whereas, in contrast, the software version uses a On-Device ML model to enhance the frames.

Let's quickly take a look at how we can integrate CameraX with Software Low Light Boost, which is also known as Google Low Light Boost.

Google Low Light Boost requires two key surfaces: first, you need an in-- input surface for the raw camera frames and an output target surface for the enhanced output frames.

To integrate with CameraX, simply wrap the Google Low Light Boost pipeline inside a CameraEffect.

CameraX automatically handles the surface lifecycle and routing seamlessly for you.

Finally, we're launching the Low Light Boost Effect in CameraX in Q4 2026.

// Integrates SW Low Light Boost
val lowLightBoostEffect =
    LowLightBoostEffect(
        CameraEffect.PREVIEW or
        CameraEffect.VIDEO_CAPTURE or
        CameraEffect.IMAGE_CAPTURE
    )

val sessionConfig = SessionConfig(
    useCases = listOf(preview, imageCapture),
    effects = lowLightBoostEffect
)

cameraProvider.bindToLifecycle(.. sessionConfig)

With this, you'd be able to create a Low Light Boost object and up- add it to your session config, then bind the session config to your camera lifecycle, and it just works on supported devices.

That brings us to the end of both the sections.

Let's quickly summarize what all we covered today.

Today we talked about how you can elevate your camera experiences across devices with CameraX and on-device machine learning.

First, with CameraXViewfinder, you can enable your camera previews to adapt to different form factors.

Next, we saw that CameraX enables you to capture on XR devices as well.

Then the Feature Group API helps you to find the optimal combination of hardware features.

We also explored upgrades to the Image Capture and learnt about High-Speed Video Recording for slow motion videos.

Finally, we learned how to enhance the Concurrent Camera preview dy-- dynamically.

Next, we looked at some of the on-device machine learning solutions such as ML Kit and MediaPipe for building intelligent vision experiences.

Finally, we saw on-device Image and Video Enhancement along with Low Light Boost for brightening up your camera preview in real-time.

As mentioned earlier, we are planning to disable Camera1, hence we highly encourage you to migrate to CameraX.

To support this migration, we recently launched the CameraX skill.

We also have a migration guide and the camera samples repo to guide you.

You can visit this link to access more resources about the features we covered today.

It includes link to documentation, blog, migration resources, all in one place.

I'll take a second, pause.

Okay.

Thank you, everyone.

This brings us to the end of our talk.