Use Stream Protect for latency-sensitive streaming apps

This document explains the Stream Protect SDK (Software Development Kit).

Background

Mobile streaming continues to evolve. Traditional streaming applications like YouTube and Twitch cache data at user devices. This allows users to watch videos on Wi-Fi networks with high latency. Today's new streaming applications include Google Stadia, Nvidia GeForce Now, Microsoft Xbox Live, and Amazon Luna. These applications have higher requirements in reliability and latency of the user's Wi-Fi networks.

Unfortunately, the high number of multiple Wi-Fi networks often causes interference for each other. Neighboring Wi-Fi sources can create unstable networks for those latency-sensitive streaming applications.

Stream Protect

To fix the issue, we collaborated with vendors and created a new feature called Stream Protect. Stream Protect predicts the transmission timing of streaming video frames between the client device and the Wi-Fi router. This helps reserve the Wi-Fi channel when the client device needs to avoid potential interference.

Overview of Stream Protect SDK

As shown in Figure 1, the Stream Protect SDK provides a set of APIs for use in the client application. Due to security and fair usage concerns, we only provide simple APIs like isFeatureSupported, enable, and disable. In enable, the developers must provide StreamProtectOptions to set the expected FPS and clock used for FrameInfo. We also allow developers to use OnStreamProtectEventListener to asynchronously monitor the internal state transition of Stream Protect.

The workflow is as follows:

1. The client app requests StreamProtectClient with StreamProtect.getClient(). When the client app starts a streaming session, it can enable Stream Protect via enable(...) API. For each new video frame arrived at the client, we pass the frame information with submitFrameInfo(...).
2. When the session ends, the app stops Stream Protect via disable() API.

3. Optionally, the client app can use isFeatureSupported() API to check

   if the client device supports the Stream Protect feature. If the feature isn't supported, all other APIs report FeatureNotSupportedException to the client app via the asynchronous listener Task.OnFailureListener, and the app handles these exceptions accordingly.

4. We also expect the client app to always enable(...) before APIs submitFrameInfo(...) and disable(). Otherwise, FeatureNotEnabledException is thrown to the listener.

5. Finally, in case any internal events or errors happen in Stream Protect such as a disconnected network, the client app is recommended to set up an onStreamProtectEventListener callback to asynchronously monitor the state of Stream Protect.

For a detailed explanation of each API, see more details in javadoc.

SDK usage example

The following example illustrates Client App implementation and behavior.

Client app implementation

import com.google.android.gms.streamprotect.StreamProtect
...
class ClientAppActivity : Activity() ... {
  private val listener: OnStreamProtectEventListener
  private val streamProtectClient: StreamProtectClient
  private val options: StreamProtectOptions

  init {
    options = StreamProtectOptions.Builder()
        .setExpectedFps(60)  // Expecting 60 FPS
        .setClockType(ClockType(ClockType.MONOTONIC))
        .build()
    listener = OnStreamProtectEventListener { event -> // Handles events.
    }
  }

  override fun onCreate(savedInstanceState: Bundle?) {
    ...
    streamProtectClient = StreamProtect.getClient(context)
    // (Recommended) Checks if the feature is supported by the device.
    streamProtectClient.isFeatureSupported
      .addOnSuccessListener {
        isSupported -> {
          // (Optional) Takes immediate action if Stream Protect is not supported.
        }
      }
      .addOnFailureListener {
          // Handles errors, for example Stream Protect APIs are not yet available
          // on the device.
      }
  }
  ...
  // Streaming session starts.
  fun start(...) {
    // Even though the client app wants to enable the feature after checking if
    // the device supports it, enable(...) can still fail, for example when the
    // device is not using Wi-Fi, when the app is in the background, etc.
    streamProtectClient.enable(options, listener)
      .addOnSuccessListener(
        enabled ->
        {
          // (Optional) Takes immediate action if Stream Protect is not enabled.
        }
       )
      .addOnFailureListener(
        e -> {
          // Handles errors, for example Stream Protect APIs are not yet available
          // on the device.
        }
      );
  }
  // Streaming session stops.
  fun stop(...) {
    // If not enabled, we get FeatureNotEnabledException;
    // Or if not supported, we get FeatureNotSupportedException;
    // Or Stream Protect APIs are not available.
    streamProtectManager.disable()
      .addOnFailureListener(e -> Log.e(TAG, e));
  }
  // Streaming session gets a new video frame.
  fun onFrameReceived(FrameInfo frameInfo) {
    if (hasStreamProtectStarted) {
    // If not enabled, we get FeatureNotEnabledException;
    // Or if not supported, we get FeatureNotSupportedException;
    // Or Stream Protect APIs are not available.
    streamProtectClient.submitFrameInfo(frameInfo)
      .addOnFailureListener(e -> Log.e(TAG, e));
  }
}

import com.google.android.gms.streamprotect.StreamProtect;
…
public ClientAppActivity extends Activity {
  private StreamProtectClient streamProtectClient;
  private OnStreamProtectEventListener listener =
      new OnStreamProtectEventListener() {
        @Override
        public void onEvent(StreamProtectEvent event) {
          // Handles events.
        }
      }
  private StreamProtectOptions options =
      StreamProtectOptions.Builder()
        .setExpectedFps(60)  // Expecting 60 FPS
        .setClockType(new ClockType(ClockType.MONOTONIC))
        .build();
  …
  public void onCreate(...) {
    StreamingSession streamingSession = new StreamingSession();
    streamProtectClient = StreamProtect.getClient(context);
    // (Recommended) Checks if the feature is supported by the device.
    streamProtectClient
      .isFeatureSupported()
      .addOnSuccessListener(
        isSupported -> {
          // (Optional) Takes immediate action if Stream Protect is not supported.
        }
      )
      .addOnFailureListener(
        e -> {
          // Handles errors, for example Stream Protect APIs are not yet available
          // on the device.
        }
      );
  }
}

public StreamingSession extends SurfaceView implements ... {
  …
  // Streaming session starts.
  public void start(...) {
    // Even though the client app wants to enable the feature after checking if
    // the device supports it, enable(...) can still fail, for example when the
    // device is not using Wi-Fi, when the app is in the background, etc.
    streamProtectClient.enable(options, listener)
      .addOnSuccessListener(
        enabled ->
        {
          // (Optional) Takes immediate action if Stream Protect is not enabled.
        }
       )
      .addOnFailureListener(
        e -> {
          // Handles errors, for example Stream Protect APIs are not yet available
          // on the device.
        }
      );
  }
  // Streaming session stops.
  public void stop(...) {
    // If not enabled, we get FeatureNotEnabledException;
    // Or if not supported, we get FeatureNotSupportedException;
    // Or Stream Protect APIs are not available.
    streamProtectManager.disable()
      .addOnFailureListener(e -> Log.e(TAG, e));
  }
  // Streaming session gets a new video frame.
  private void onFrameReceived(FrameInfo frameInfo) {
    if (hasStreamProtectStarted) {
    // If not enabled, we get FeatureNotEnabledException;
    // Or if not supported, we get FeatureNotSupportedException;
    // Or Stream Protect APIs are not available.
    streamProtectClient.submitFrameInfo(frameInfo)
      .addOnFailureListener(e -> Log.e(TAG, e));
  }
}

Client app behavior

Stream Protect is backed by a code module (around 1MB) that must be downloaded by Google Play service prior to first use. The download may take several minutes before Stream Protect is ready to use. Attempts to use the APIs before the module is finished downloading will result in an ApiException. Keep this in mind when you design and implement your app.

Also, to have a smooth developing experience, make sure you have updated to the latest version of Google Play Service. Once the Stream Protect code module sits in the user device, it's automatically updated in the background and users don't need to wait for updates to install it.

Google Play's data disclosure requirements

In May 2021, Google Play announced the new developer-provided disclosure for an app's data collection, sharing, and security practices. The following information can help you complete the requirements for this data disclosure in regards to your app usage of the Stream Protect SDK.

We aim to be as transparent as possible in supporting you; however, as the app developer, you are solely responsible for deciding how to respond to Google Play's Data safety section form regarding your app's end-user data collection, sharing, and security practices.

Data collected automatically

The Stream Protect SDK collects the following data automatically.

Data collected depending on your usage

The Stream Protect SDK doesn't have optional features that the developer can configure or invoke to collect other end-user data.

Additional information

Stream Protect SDK uses the collected data listed in this section for Stream Protect performance analysis and enhancement. In your data disclosure, make sure to also account for how you specifically use the collected data.

Supported device list

The SDK requires a minimum of Android version 29 to run, as well as device-specific firmware support.

Currently, the SDK only supports Chromecast with Google TV.