Build smarter apps with machine learning
Machine learning (ML) is a programming technique that provides your apps the ability to automatically learn and improve from experience without being explicitly programmed to do so. This is especially well-suited for apps that utilize unstructured data such as images and text, or problems with large number of parameters such as predicting the winning sports team.
Android supports a wide variety of machine learning tools and methods: The ML design guides, Google's turnkey machine learning SDK ML Kit, TF Hub for finding pre-built cutting edge models, TF Lite Model Maker to train an existing model with your own data, and ML Kit custom models and Android Studio for integrating these models into your app. The choice is yours.
Key ML Development Areas
When developing ML apps for Android, your main tasks are designing products, implementing inference algorithms, and deploying existing ML models. Depending on your circumstances you may also retrain existing models or build and train new ones.
Design for Machine Learning
Similar to other technologies, applying machine learning as a solution requires product managers, designers and developers to work together to define product goals, design, build and iterate. Google has produced two guides in this area:
- The People + AI Guidebook provides best practices to help your team make human-centered AI product decisions.
- The The Material Design for Machine Learning spec contains a collection of design guidelines and patterns for machine learning-powered features such as object detection and barcode scanning.
Build and Train a Model
Machine learning requires a model that's trained to perform a particular task, like making a prediction, or classifying or recognizing some input. You can select (and possibly customize) an existing model, or build a model from scratch. Model creation and training can be done on a development machine, or using cloud infrastructure.
Explore pre-trained models
TensorFlow Hub contains a large repository of TensorFlow Lite models from Google and the wider research community that are optimized to run on Android. There are two model collections worth highlighting which works well with Android inference tooling:
- ML Kit Image Classification Collection which is designed to work with ML Kit Image Labeling and Object Detection and Tracking APIs.
- Android Studio ML Model Binding Collection contains models that are compatible with Android Studio 4.1 beta's ML model binding feature.
In addition, other pretrained models are available through ML Kit and Google Cloud. Read more about them in the next section.
Learn how to create your own models with TensorFlow
Firebase AutoML Vision Edge offers a graphical interface for producing custom classifiers.
For a deeper hands-on development experience, you can use these TensorFlow resources:
- TensorFlow Lite Model Maker is designed to make common custom models easy. It currently covers:
- Tensorflow Tutorials
Inference
Inference is the process of using a machine learning model that has already been trained to perform a specific task.
A key decision you’ll face as an Android developer is whether inferencing runs on the device, or uses a cloud service that's accessed remotely. Here are some of the issues you should take into account when making this decision:
| Issue | On-device inference | Cloud-based inference |
|---|---|---|
| Latency | Lower latency enhances the realtime experience | Asynchronous communication and available bandwidth can affect latency |
| Resources | The particular device's resources, like processing power and storage, can limit performance | Cloud-based resources are more powerful and storage is more plentiful |
| Offline/Online | The ability to operate offline is a plus for running with poor or non-existing network infrastructure | A network connection is required |
| Cost | Battery usage, model download time for end users | Bandwidth for data transfer for end users, computing charges for developers |
| Privacy | User data never leaves the device | Data may leave the device, additional precautions may be necessary |
The following table shows the available development options for each kind of inference:
| On-device inference | Cloud-based inference |
|---|---|
ML Kit
Custom TensorFlow Lite ModelsIf you have downloaded a model from TensorFlow Hub or have trained an image classifer with TensorFlow Lite Model Maker you can implement it in one of two ways depending on the composition of the input image. If it contains:
For non-image classifier use-cases, refer to the TensorFlow Lite example apps on how to implement custom TensorFlow Lite models in various use cases ranging from Style Transfer to Question and Answer. Firebase MLUse AutoML Vision Edge to train your own custom vision model on Google Cloud and run the resulting model on Android |
Firebase MLGoogle Cloud APIs
|
Deployment
Deployment is the process of packaging and updating your ML model for use on Android when doing on-device inference. There are three options available:
- Include the model with your Android app
- Your model is deployed with your app like any other asset. Updates to the model require updating the app. There are two ways you can add a model to your app:
- Include the model in the app's APK file. This is the most basic option.
- Include the model in an Android App Bundle and use feature modules.
- Provide the model at runtime
- This enables you to update your model independently of your app. This also makes A/B testing easier. You can serve your custom model using the Firebase ML Custom Models function or host the model download with your own infrastructure.
- A combination of both
- It is not unusual for developers to package an initial version of their model with their Android app so that user does not need to wait for their model to download while updating the model to a new version.
For selected pre-trained ML Kit models, namely text recognition and barcode scanning, developers can use the shared model provided by Google Play Services resulting in smaller APK sizes.
Developer Stories
Making the impossible possible
Adding ML to your Android app opens up a new way to build applications that were too difficult to get right in a wide variety of conditions (such as reliable barcode scanning) or that were not even previously possible (for example, image detection and text sentiment).
Lose It!
Lose It! is a weight loss calorie tracker app. It helps you lose weight by logging all the food you eat so you know how many calories you have consumed. Lose It! uses the ML Kit text recognition API to scan nutrition labels to pull in the data when users are entering a new food that isn’t in their library.
PlantVillage
PlantVillage helps farmers detect diseases in Cassava Plants. Penn State University and the International Institute of Tropical Agriculture uses their custom TensorFlow models running offline on mobile devices to help farmers detect early sign of plant diseases.
Fishbrain
The Fishbrain app provides local fishing maps, forecasts, and connects millions of anglers. Fishbrain uses ML Kit Custom Model to deliver updated custom TensorFlow Lite models.