MediaSync

Kotlin |Java

class MediaSync

kotlin.Any
↳	android.media.MediaSync

MediaSync class can be used to synchronously play audio and video streams. It can be used to play audio-only or video-only stream, too.

MediaSync is generally used like this:

MediaSync sync = new MediaSync();
  sync.setSurface(surface);
  Surface inputSurface = sync.createInputSurface();
  ...
  // MediaCodec videoDecoder = ...;
  videoDecoder.configure(format, inputSurface, ...);
  ...
  sync.setAudioTrack(audioTrack);
  sync.setCallback(new MediaSync.Callback() {
      @Override
      public void onAudioBufferConsumed(MediaSync sync, ByteBuffer audioBuffer, int bufferId) {
          ...
      }
  }, null);
  // This needs to be done since sync is paused on creation.
  sync.setPlaybackParams(new PlaybackParams().setSpeed(1.f));
 
  for (;;) {
    ...
    // send video frames to surface for rendering, e.g., call
    // videoDecoder.releaseOutputBuffer(videoOutputBufferIx, videoPresentationTimeNs);
    // More details are available as below.
    ...
    sync.queueAudio(audioByteBuffer, bufferId, audioPresentationTimeUs); // non-blocking.
    // The audioByteBuffer and bufferId will be returned via callback.
    // More details are available as below.
    ...
      ...
  }
  sync.setPlaybackParams(new PlaybackParams().setSpeed(0.f));
  sync.release();
  sync = null;
 
  // The following code snippet illustrates how video/audio raw frames are created by
  // MediaCodec's, how they are fed to MediaSync and how they are returned by MediaSync.
  // This is the callback from MediaCodec.
  onOutputBufferAvailable(MediaCodec codec, int bufferId, BufferInfo info) {
      // ...
      if (codec == videoDecoder) {
          // surface timestamp must contain media presentation time in nanoseconds.
          codec.releaseOutputBuffer(bufferId, 1000 * info.presentationTime);
      } else {
          ByteBuffer audioByteBuffer = codec.getOutputBuffer(bufferId);
          sync.queueAudio(audioByteBuffer, bufferId, info.presentationTime);
      }
      // ...
  }
 
  // This is the callback from MediaSync.
  onAudioBufferConsumed(MediaSync sync, ByteBuffer buffer, int bufferId) {
      // ...
      audioDecoder.releaseBuffer(bufferId, false);
      // ...
  }

The client needs to configure corresponding sink by setting the Surface and/or AudioTrack based on the stream type it will play.

For video, the client needs to call createInputSurface to obtain a surface on which it will render video frames.

For audio, the client needs to set up audio track correctly, e.g., using android.media.AudioTrack#MODE_STREAM. The audio buffers are sent to MediaSync directly via queueAudio, and are returned to the client via Callback.onAudioBufferConsumed asynchronously. The client should not modify an audio buffer till it's returned.

The client can optionally pre-fill audio/video buffers by setting playback rate to 0.0, and then feed audio/video buffers to corresponding components. This can reduce possible initial underrun.

Summary

Nested classes
abstract	`Callback` MediaSync callback interface.
abstract	`OnErrorListener` Interface definition of a callback to be invoked when there has been an error during an asynchronous operation (other errors will throw exceptions at method call time).

Constants
static Int	`MEDIASYNC_ERROR_AUDIOTRACK_FAIL` Audio track failed.
static Int	`MEDIASYNC_ERROR_SURFACE_FAIL` The surface failed to handle video buffers.

Public constructors
`MediaSync()` Class constructor.

Public methods
Surface	`createInputSurface()` Requests a Surface to use as the input.
Unit	`flush()` Flushes all buffers from the sync object.
PlaybackParams	`getPlaybackParams()` Gets the playback rate using `PlaybackParams`.
SyncParams	`getSyncParams()` Gets the A/V sync mode.
MediaTimestamp?	`getTimestamp()` Get current playback position.
Unit	`queueAudio(audioData: ByteBuffer, bufferId: Int, presentationTimeUs: Long)` Queues the audio data asynchronously for playback (AudioTrack must be in streaming mode).
Unit	`release()` Make sure you call this when you're done to free up any opened component instance instead of relying on the garbage collector to do this for you at some point in the future.
Unit	`setAudioTrack(audioTrack: AudioTrack?)` Sets the audio track for MediaSync.
Unit	`setCallback(cb: MediaSync.Callback?, handler: Handler?)` Sets an asynchronous callback for actionable MediaSync events.
Unit	`setOnErrorListener(listener: MediaSync.OnErrorListener?, handler: Handler?)` Sets an asynchronous callback for error events.
Unit	`setPlaybackParams(params: PlaybackParams)` Sets playback rate using `PlaybackParams`.
Unit	`setSurface(surface: Surface?)` Sets the output surface for MediaSync.
Unit	`setSyncParams(params: SyncParams)` Sets A/V sync mode.

Protected methods
Unit	`finalize()`

Parameters
`audioData`	ByteBuffer: the buffer that holds the data to play. This buffer will be returned to the client via registered callback. This value cannot be `null`.
`bufferId`	Int: an integer used to identify audioData. It will be returned to the client along with audioData. This helps applications to keep track of audioData, e.g., it can be used to store the output buffer index used by the audio codec.
`presentationTimeUs`	Long: the presentation timestamp in microseconds for the first frame in the buffer.

Exceptions
`java.lang.IllegalArgumentException`	if the audioTrack has been released, or is invalid.
`java.lang.IllegalStateException`	if setting the audio track is not supported, e.g. not in the Initialized state, or another audio track has already been set.

Parameters
`cb`	MediaSync.Callback?: The callback that will run. Use `null` to stop receiving callbacks.
`handler`	Handler?: The Handler that will run the callback. Use `null` to use MediaSync's internal handler if it exists.

Parameters
`listener`	MediaSync.OnErrorListener?: The callback that will run. Use `null` to stop receiving callbacks.
`handler`	Handler?: The Handler that will run the callback. Use `null` to use MediaSync's internal handler if it exists.

Exceptions
`java.lang.IllegalStateException`	if the internal sync engine or the audio track has not been initialized.
`java.lang.IllegalArgumentException`	if the params are not supported.

Exceptions
`java.lang.IllegalArgumentException`	if the surface has been released, is invalid, or can not be connected.
`java.lang.IllegalStateException`	if setting the surface is not supported, e.g. not in the Initialized state, or another surface has already been set.

Exceptions
`java.lang.IllegalStateException`	if the internal player engine has not been initialized.
`java.lang.IllegalArgumentException`	if params are not supported.

MediaSync

Summary

Constants

MEDIASYNC_ERROR_AUDIOTRACK_FAIL

MEDIASYNC_ERROR_SURFACE_FAIL

Public constructors

MediaSync

Public methods

createInputSurface

flush

getPlaybackParams

getSyncParams

getTimestamp

queueAudio

release

setAudioTrack

setCallback

setOnErrorListener

setPlaybackParams

setSurface

setSyncParams

Protected methods

finalize