Page from network and database

Provide an improved user experience by making sure that your app can be used when network connections are unreliable or when the user is offline. One way to do this is to page from the network and from a local database at the same time. This way, your app drives the UI from a local database cache and only makes requests to the network when there is no more data in the database.

This guide assumes that you are familiar with the Room persistence library and with basic usage of the Paging library.

Coordinate data loads

The Paging library provides the RemoteMediator component for this use case. RemoteMediator acts as a signal from the Paging library when the app has run out of cached data. You can use this signal to load additional data from the network and store it in the local database, where a PagingSource can load it and provide it to the UI to display.

When additional data is needed, the Paging library calls the load() method from the RemoteMediator implementation. This is a suspending function so it is safe to perform long-running work. This function typically fetches the new data from a network source and saves it to local storage.

This process works with new data, but over time the data stored in the database requires invalidation, such as when the user manually triggers a refresh. This is represented by the LoadType property passed to the load() method. The LoadType informs the RemoteMediator whether it needs to refresh the existing data or fetch additional data that needs to be appended or prepended to the existing list.

In this way, the RemoteMediator ensures that your app loads the data that users want to see in the appropriate order.

Paging lifecycle

Figure 1. Diagram of the lifecycle of Paging with PagingSource and PagingData.

When paging directly from the network, the PagingSource loads the data and returns a LoadResult object. The PagingSource implementation is passed to the Pager through the pagingSourceFactory parameter.

As new data is required by the UI, the Pager calls the load() method from the PagingSource and returns a stream of PagingData objects that encapsulate the new data. Each PagingData object is typically cached in the ViewModel before being sent to the UI to display.

Figure 2. Diagram of the lifecycle of Paging with PagingSource and RemoteMediator.

RemoteMediator changes this data flow. A PagingSource still loads the data; but when the paged data is exhausted, the Paging library triggers the RemoteMediator to load new data from the network source. The RemoteMediator stores the new data in the local database, so an in-memory cache in the ViewModel is unnecessary. Finally, the PagingSource invalidates itself, and the Pager creates a new instance to load the fresh data from the database.

Basic usage

Suppose you want your app to load pages of User items from an item-keyed network data source into a local cache stored in a Room database.

The RemoteMediator loads data from the network into the database and the PagingSource loads data from the database. A Pager uses both the RemoteMediator and the PagingSource to load paged data.
Figure 3. Diagram of a Paging implementation that uses a layered data source.

A RemoteMediator implementation helps load paged data from the network into the database, but doesn't load data directly into the UI. Instead, the app uses the database as the source of truth. In other words, the app only displays data that has been cached in the database. A PagingSource implementation (for example, one generated by Room) handles loading cached data from the database into the UI.

Create Room entities

The first step is to use the Room persistence library to define a database that holds a local cache of paged data from the network data source. Start with an implementation of RoomDatabase as described in Save data in a local database using Room.

Next, define a Room entity to represent a table of list items as described in Defining data using Room entities. Give it an id field as a primary key, as well as fields for any other information that your list items contain.

Kotlin

@Entity(tableName = "users")
data class User(val id: String, val label: String)

Java

@Entity(tableName = "users")
public class User {
  public String id;
  public String label;
}

Java

@Entity(tableName = "users")
public class User {
  public String id;
  public String label;
}

You must also define a data access object (DAO) for this Room entity as described in Accessing data using Room DAOs. The DAO for the list item entity must include the following methods:

  • An insertAll() method that inserts a list of items into the table.
  • A method that takes the query string as a parameter and returns a PagingSource object for the list of results. This way, a Pager object can use this table as a source of paged data.
  • A clearAll() method that deletes all of the table's data.

Kotlin

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  suspend fun insertAll(users: List<User>)

  @Query("SELECT * FROM users WHERE label LIKE :query")
  fun pagingSource(query: String): PagingSource<Int, User>

  @Query("DELETE FROM users")
  suspend fun clearAll()
}

Java

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertAll(List<User> users);

  @Query("SELECT * FROM users WHERE mLabel LIKE :query")
  PagingSource<Integer, User> pagingSource(String query);

  @Query("DELETE FROM users")
  int clearAll();
}

Java

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertAll(List<User> users);

  @Query("SELECT * FROM users WHERE mLabel LIKE :query")
  PagingSource<Integer, User> pagingSource(String query);

  @Query("DELETE FROM users")
  int clearAll();
}

Implement a RemoteMediator

The main role of RemoteMediator is to load more data from the network when either the Pager runs out of data or the existing data is invalidated. It includes a load() method that you must override to define the loading behavior.

A typical RemoteMediator implementation includes the following parameters:

  • query: A query string defining which data to retrieve from the backend service.
  • database: The Room database that serves as a local cache.
  • networkService: An API instance for the backend service.

Create a RemoteMediator<Key, Value> implementation. The Key type and the Value type should be the same as they would be if you were defining a PagingSource against the same network data source. For more information on selecting type parameters, see Select key and value types.

Kotlin

@OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
  private val query: String,
  private val database: RoomDb,
  private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
  val userDao = database.userDao()

  override suspend fun load(
    loadType: LoadType,
    state: PagingState<Int, User>
  ): MediatorResult {
    // ...
  }
}

Java

@UseExperimental(markerClass = ExperimentalPagingApi.class)
class ExampleRemoteMediator extends RxRemoteMediator<Integer, User> {
  private String query;
  private ExampleBackendService networkService;
  private RoomDb database;
  private UserDao userDao;

  ExampleRemoteMediator(
    String query,
    ExampleBackendService networkService, RoomDb database
  ) {
    query = query;
    networkService = networkService;
    database = database;
    userDao = database.userDao();
  }

  @NotNull
  @Override
  public Single<MediatorResult> loadSingle(
    @NotNull LoadType loadType,
    @NotNull PagingState<Integer, User> state
  ) {
    ...
  }
}

Java

class ExampleRemoteMediator extends ListenableFutureRemoteMediator<Integer, User> {
  private String query;
  private ExampleBackendService networkService;
  private RoomDb database;
  private UserDao userDao;
  private Executor bgExecutor;

  ExampleRemoteMediator(
    String query,
    ExampleBackendService networkService,
    RoomDb database,
    Executor bgExecutor
  ) {
    this.query = query;
    this.networkService = networkService;
    this.database = database;
    this.userDao = database.userDao();
    this.bgExecutor = bgExecutor;
  }

  @NotNull
  @Override
  public ListenableFuture<MediatorResult> loadFuture(
    @NotNull LoadType loadType,
    @NotNull PagingState<Integer, User> state
  ) {
    ...
  }
}

The load() method is responsible for updating the backing dataset and invalidating the PagingSource. Some libraries that support paging (like Room) will automatically handle invalidating PagingSource objects that they implement.

The load() method takes in two parameters:

  • PagingState, which contains information about the pages loaded so far, the most recently accessed index, and the PagingConfig object that you used to initialize the paging stream.
  • LoadType, which indicates the type of the load: REFRESH, APPEND, or PREPEND.

The return value of the load() method is a MediatorResult object. MediatorResult can either be MediatorResult.Error (which includes the error description) or MediatorResult.Success (which includes a signal stating whether or not there is more data to load).

The load() method must perform the following steps:

  1. Determine which page to load from the network depending on the load type and the data that has been loaded so far.
  2. Trigger the network request.
  3. Perform actions depending on the outcome of the load operation:
    • If the load is successful and the received list of items is not empty, then store the list items in the database and return MediatorResult.Success(endOfPaginationReached = false). After the data is stored, invalidate the data source to notify the Paging library of the new data.
    • If the load is successful and either the received list of items is empty or it is the last page index, then return MediatorResult.Success(endOfPaginationReached = true). After the data is stored, invalidate the data source to notify the Paging library of the new data.
    • If the request causes an error, then return MediatorResult.Error.

Kotlin

override suspend fun load(
  loadType: LoadType,
  state: PagingState<Int, User>
): MediatorResult {
  return try {
    // The network load method takes an optional after=<user.id>
    // parameter. For every page after the first, pass the last user
    // ID to let it continue from where it left off. For REFRESH,
    // pass null to load the first page.
    val loadKey = when (loadType) {
      LoadType.REFRESH -> null
      // In this example, you never need to prepend, since REFRESH
      // will always load the first page in the list. Immediately
      // return, reporting end of pagination.
      LoadType.PREPEND ->
        return MediatorResult.Success(endOfPaginationReached = true)
      LoadType.APPEND -> {
        val lastItem = state.lastItemOrNull()

        // You must explicitly check if the last item is null when
        // appending, since passing null to networkService is only
        // valid for initial load. If lastItem is null it means no
        // items were loaded after the initial REFRESH and there are
        // no more items to load.
        if (lastItem == null) {
          return MediatorResult.Success(
            endOfPaginationReached = true
          )
        }

        lastItem.id
      }
    }

    // Suspending network load via Retrofit. This doesn't need to be
    // wrapped in a withContext(Dispatcher.IO) { ... } block since
    // Retrofit's Coroutine CallAdapter dispatches on a worker
    // thread.
    val response = networkService.searchUsers(
      query = query, after = loadKey
    )

    database.withTransaction {
      if (loadType == LoadType.REFRESH) {
        userDao.deleteByQuery(query)
      }

      // Insert new users into database, which invalidates the
      // current PagingData, allowing Paging to present the updates
      // in the DB.
      userDao.insertAll(response.users)
    }

    MediatorResult.Success(
      endOfPaginationReached = response.nextKey == null
    )
  } catch (e: IOException) {
    MediatorResult.Error(e)
  } catch (e: HttpException) {
    MediatorResult.Error(e)
  }
}

Java

@NotNull
@Override
public Single<MediatorResult> loadSingle(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  String loadKey = null;
  switch (loadType) {
    case REFRESH:
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Single.just(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Single.just(new MediatorResult.Success(true));
      }

      loadKey = lastItem.getId();
      break;
  }

  return networkService.searchUsers(query, loadKey)
    .subscribeOn(Schedulers.io())
    .map((Function<SearchUserResponse, MediatorResult>) response -> {
      database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
        }

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    })
    .onErrorResumeNext(e -> {
      if (e instanceof IOException || e instanceof HttpException) {
        return Single.just(new MediatorResult.Error(e));
      }

      return Single.error(e);
    });
}

Java

@NotNull
@Override
public ListenableFuture<MediatorResult> loadFuture(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  String loadKey = null;
  switch (loadType) {
    case REFRESH:
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Futures.immediateFuture(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Futures.immediateFuture(new MediatorResult.Success(true));
      }

      loadKey = lastItem.getId();
      break;
  }

  ListenableFuture<MediatorResult> networkResult = Futures.transform(
    networkService.searchUsers(query, loadKey),
    response -> {
      database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
        }

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    }, bgExecutor);

  ListenableFuture<MediatorResult> ioCatchingNetworkResult =
    Futures.catching(
      networkResult,
      IOException.class,
      MediatorResult.Error::new,
      bgExecutor
    );

  return Futures.catching(
    ioCatchingNetworkResult,
    HttpException.class,
    MediatorResult.Error::new,
    bgExecutor
  );
}

Define the initialize method

RemoteMediator implementations can also override the initialize() method to check whether cached data is out of date and decide whether to trigger a remote refresh. This method runs before any loading is performed, so you can manipulate the database (for example, to clear old data) before triggering any local or remote loads.

Because initialize() is an asynchronous function, you can load data to determine the relevance of the existing data in the database. The most common case is that the cached data is only valid for a certain period of time. The RemoteMediator can check whether this expiration time has passed, in which case the Paging library needs to fully refresh the data. Implementations of initialize() should return an InitializeAction as follows:

  • In cases where the local data needs to be fully refreshed, initialize() should return InitializeAction.LAUNCH_INITIAL_REFRESH. This causes the RemoteMediator to perform a remote refresh to fully reload the data. Any remote APPEND or PREPEND loads wait for the REFRESH load to succeed before proceeding.
  • In cases where the local data doesn't need to be refreshed, initialize() should return InitializeAction.SKIP_INITIAL_REFRESH. This causes the RemoteMediator to skip the remote refresh and load the cached data.

Kotlin

override suspend fun initialize(): InitializeAction {
  val cacheTimeout = TimeUnit.MILLISECONDS.convert(1, TimeUnit.HOURS)
  return if (System.currentTimeMillis() - db.lastUpdated() <= cacheTimeout)
  {
    // Cached data is up-to-date, so there is no need to re-fetch
    // from the network.
    InitializeAction.SKIP_INITIAL_REFRESH
  } else {
    // Need to refresh cached data from network; returning
    // LAUNCH_INITIAL_REFRESH here will also block RemoteMediator's
    // APPEND and PREPEND from running until REFRESH succeeds.
    InitializeAction.LAUNCH_INITIAL_REFRESH
  }
}

Java

@NotNull
@Override
public Single<InitializeAction> initializeSingle() {
  long cacheTimeout = TimeUnit.MILLISECONDS.convert(1, TimeUnit.HOURS);
  return mUserDao.lastUpdatedSingle()
    .map(lastUpdatedMillis -> {
      if (System.currentTimeMillis() - lastUpdatedMillis <= cacheTimeout) {
        // Cached data is up-to-date, so there is no need to re-fetch
        // from the network.
        return InitializeAction.SKIP_INITIAL_REFRESH;
      } else {
        // Need to refresh cached data from network; returning
        // LAUNCH_INITIAL_REFRESH here will also block RemoteMediator's
        // APPEND and PREPEND from running until REFRESH succeeds.
        return InitializeAction.LAUNCH_INITIAL_REFRESH;
      }
    });
}

Java

@NotNull
@Override
public ListenableFuture<InitializeAction> initializeFuture() {
  long cacheTimeout = TimeUnit.MILLISECONDS.convert(1, TimeUnit.HOURS);
  return Futures.transform(
    mUserDao.lastUpdated(),
    lastUpdatedMillis -> {
      if (System.currentTimeMillis() - lastUpdatedMillis <= cacheTimeout) {
        // Cached data is up-to-date, so there is no need to re-fetch
        // from the network.
        return InitializeAction.SKIP_INITIAL_REFRESH;
      } else {
        // Need to refresh cached data from network; returning
        // LAUNCH_INITIAL_REFRESH here will also block RemoteMediator's
        // APPEND and PREPEND from running until REFRESH succeeds.
        return InitializeAction.LAUNCH_INITIAL_REFRESH;
      }
    },
    mBgExecutor);
}

Create a Pager

Finally, you must create a Pager instance to set up the stream of paged data. This is similar to creating a Pager from a simple network data source, but there are two things you must do differently:

  • Instead of passing a PagingSource constructor directly, you must provide the query method that returns a PagingSource object from the DAO.
  • You must provide an instance of your RemoteMediator implementation as the remoteMediator parameter.

Kotlin

val userDao = database.userDao()
val pager = Pager(
  config = PagingConfig(pageSize = 50)
  remoteMediator = ExampleRemoteMediator(query, database, networkService)
) {
  userDao.pagingSource(query)
}

Java

UserDao userDao = database.userDao();
Pager<Integer, User> pager = Pager(
  new PagingConfig(/* pageSize = */ 20),
  null, // initialKey,
  new ExampleRemoteMediator(query, database, networkService)
  () -> userDao.pagingSource(query));

Java

UserDao userDao = database.userDao();
Pager<Integer, User> pager = Pager(
  new PagingConfig(/* pageSize = */ 20),
  null, // initialKey
  new ExampleRemoteMediator(query, database, networkService, bgExecutor),
  () -> userDao.pagingSource(query));

Handle race conditions

One situation that your app needs to handle when loading data from multiple sources is the case in which local cached data becomes out of sync with the remote data source.

When the initialize() method from your RemoteMediator implementation returns LAUNCH_INITIAL_REFRESH, the data is outdated and must be replaced with fresh data. Any PREPEND or APPEND load requests are forced to wait for the remote REFRESH load to succeed. Because the PREPEND or APPEND requests were queued before the REFRESH request, it is possible that the PagingState passed to those load calls will be out of date by the time they run.

Depending on how the data is stored locally, your app can ignore redundant requests if changes to the cached data cause invalidation and new data fetches. For example, Room invalidates queries on any data insertion. That means that new PagingSource objects with the refreshed data are provided to pending load requests when new data is inserted into the database.

Solving this data synchronization problem is essential to ensuring that users see the most relevant, up-to-date data. The best solution depends mostly on the way that the network data source pages the data. In any case, remote keys allow you to save information about the most recent page requested from the server. You app can use this information to identify and request the correct page of data to load next.

Manage remote keys

Remote keys are keys that a RemoteMediator implementation uses to tell the backend service which data to load next. In the simplest case, each item of paged data includes a remote key that you can easily reference. However, if the remote keys do not correspond to individual items, then you must store them separately and manage them in your load() method.

This section describes how to collect, store, and update remote keys that are not stored in individual items.

Item keys

This section describes how to work with remote keys that correspond to individual items. Typically, when an API keys off of individual items, the item ID is passed as a query parameter. The parameter name indicates whether the server should respond with items before or after the provided ID. In the example of the User model class, the id field from the server is used as a remote key when requesting additional data.

When your load() method needs to manage item-specific remote keys, these keys are typically the IDs of the data fetched from the server. Refresh operations don't need a load key, because they just retrieve the most recent data. Similarly, prepend operations do not need to fetch any additional data because refresh always pulls the newest data from the server.

However, append operations do require an ID. This requires you to load the last item from the database and use its ID to load the next page of data. If there are no items in the database, then endOfPaginationReached is set to true, indicating that a data refresh is needed.

Kotlin

@OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
  private val query: String,
  private val database: RoomDb,
  private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
  val userDao = database.userDao()

  override suspend fun load(
    loadType: LoadType,
    state: PagingState<Int, User>
  ): MediatorResult {
    return try {
      // The network load method takes an optional String
      // parameter. For every page after the first, pass the String
      // token returned from the previous page to let it continue
      // from where it left off. For REFRESH, pass null to load the
      // first page.
      val loadKey = when (loadType) {
        LoadType.REFRESH -> null
        // In this example, you never need to prepend, since REFRESH
        // will always load the first page in the list. Immediately
        // return, reporting end of pagination.
        LoadType.PREPEND -> return MediatorResult.Success(
          endOfPaginationReached = true
        )
        // Get the last User object id for the next RemoteKey.
        LoadType.APPEND -> {
          val lastItem = state.lastItemOrNull()

          // You must explicitly check if the last item is null when
          // appending, since passing null to networkService is only
          // valid for initial load. If lastItem is null it means no
          // items were loaded after the initial REFRESH and there are
          // no more items to load.
          if (lastItem == null) {
            return MediatorResult.Success(
              endOfPaginationReached = true
            )
          }

          lastItem.id
        }
      }

      // Suspending network load via Retrofit. This doesn't need to
      // be wrapped in a withContext(Dispatcher.IO) { ... } block
      // since Retrofit's Coroutine CallAdapter dispatches on a
      // worker thread.
      val response = networkService.searchUsers(query, loadKey)

      // Store loaded data, and next key in transaction, so that
      // they're always consistent.
      database.withTransaction {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query)
        }

        // Insert new users into database, which invalidates the
        // current PagingData, allowing Paging to present the updates
        // in the DB.
        userDao.insertAll(response.users)
      }

      // End of pagination has been reached if no users are returned from the
      // service
      MediatorResult.Success(
        endOfPaginationReached = response.users.isEmpty()
      )
    } catch (e: IOException) {
      MediatorResult.Error(e)
    } catch (e: HttpException) {
      MediatorResult.Error(e)
    }
  }
}

Java

@NotNull
@Override
public Single>MediatorResult< loadSingle(
  @NotNull LoadType loadType,
  @NotNull PagingState>Integer, User< state
) {
  // The network load method takes an optional String parameter. For every page
  // after the first, pass the String token returned from the previous page to
  // let it continue from where it left off. For REFRESH, pass null to load the
  // first page.
  Single>String< remoteKeySingle = null;
  switch (loadType) {
    case REFRESH:
      // Initial load should use null as the page key, so you can return null
      // directly.
      remoteKeySingle = Single.just(null);
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Single.just(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when
      // appending, since passing null to networkService is only
      // valid for initial load. If lastItem is null it means no
      // items were loaded after the initial REFRESH and there are
      // no more items to load.
      if (lastItem == null) {
        return Single.just(new MediatorResult.Success(true));
      }
      remoteKeySingle = Single.just(lastItem.getId());
      break;
  }

  return remoteKeySingle
    .subscribeOn(Schedulers.io())
    .flatMap((Function<String, Single<MediatorResult>>) remoteKey -> {
      return networkService.searchUsers(query, remoteKey)
        .map(response -> {
          database.runInTransaction(() -> {
            if (loadType == LoadType.REFRESH) {
              userDao.deleteByQuery(query);
            }
            // Insert new users into database, which invalidates the current
            // PagingData, allowing Paging to present the updates in the DB.
            userDao.insertAll(response.getUsers());
          });

          return new MediatorResult.Success(response.getUsers().isEmpty());
        });
    })
    .onErrorResumeNext(e -> {
      if (e instanceof IOException || e instanceof HttpException) {
        return Single.just(new MediatorResult.Error(e));
      }

      return Single.error(e);
    });
}

Java

@NotNull
@Override
public ListenableFuture<MediatorResult> loadFuture(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter.
  // For every page after the first, pass the last user ID to let it continue
  // from where it left off. For REFRESH, pass null to load the first page.
  ResolvableFuture<String> remoteKeyFuture = ResolvableFuture.create();
  switch (loadType) {
    case REFRESH:
      remoteKeyFuture.set(null);
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Futures.immediateFuture(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Futures.immediateFuture(new MediatorResult.Success(true));
      }

      remoteKeyFuture.set(lastItem.getId());
      break;
  }

  return Futures.transformAsync(remoteKeyFuture, remoteKey -> {

    ListenableFuture<MediatorResult> networkResult = Futures.transform(
      networkService.searchUsers(query, remoteKey),
      response -> {
        database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
        }

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getUsers().isEmpty());
    }, bgExecutor);

    ListenableFuture<MediatorResult> ioCatchingNetworkResult =
      Futures.catching(
        networkResult,
        IOException.class,
        MediatorResult.Error::new,
        bgExecutor
      );

    return Futures.catching(
      ioCatchingNetworkResult,
      HttpException.class,
      MediatorResult.Error::new,
      bgExecutor
    );
  }, bgExecutor);
}

Page keys

This section describes how to work with remote keys that don't correspond to individual items.

Add remote key table

When remote keys are not directly associated with list items, it is best to store them in a separate table in the local database. Define a Room entity that represents a table of remote keys:

Kotlin

@Entity(tableName = "remote_keys")
data class RemoteKey(val label: String, val nextKey: String?)

Java

@Entity(tableName = "remote_keys")
public class RemoteKey {
  public String label;
  public String nextKey;
}

Java

@Entity(tableName = "remote_keys")
public class RemoteKey {
  public String label;
  public String nextKey;
}

You must also define a DAO for the RemoteKey entity:

Kotlin

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  suspend fun insertOrReplace(remoteKey: RemoteKey)

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  suspend fun remoteKeyByQuery(query: String): RemoteKey

  @Query("DELETE FROM remote_keys WHERE label = :query")
  suspend fun deleteByQuery(query: String)
}

Java

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertOrReplace(RemoteKey remoteKey);

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  Single<RemoteKey> remoteKeyByQuerySingle(String query);

  @Query("DELETE FROM remote_keys WHERE label = :query")
  void deleteByQuery(String query);
}

Java

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertOrReplace(RemoteKey remoteKey);

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  ListenableFuture<RemoteKey> remoteKeyByQueryFuture(String query);

  @Query("DELETE FROM remote_keys WHERE label = :query")
  void deleteByQuery(String query);
}

Load with remote keys

When your load() method needs to manage remote page keys, you must define it differently in the following ways as compared to basic usage of RemoteMediator:

  • Include an additional property that holds a reference to the DAO for your remote key table.
  • Determine which key to load next by querying the remote key table instead of using PagingState.
  • Insert or store the returned remote key from the network data source in addition to the paged data itself.

Kotlin

@OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
  private val query: String,
  private val database: RoomDb,
  private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
  val userDao = database.userDao()
  val remoteKeyDao = database.remoteKeyDao()

  override suspend fun load(
    loadType: LoadType,
    state: PagingState<Int, User>
  ): MediatorResult {
    return try {
      // The network load method takes an optional String
      // parameter. For every page after the first, pass the String
      // token returned from the previous page to let it continue
      // from where it left off. For REFRESH, pass null to load the
      // first page.
      val loadKey = when (loadType) {
        LoadType.REFRESH -> null
        // In this example, you never need to prepend, since REFRESH
        // will always load the first page in the list. Immediately
        // return, reporting end of pagination.
        LoadType.PREPEND -> return MediatorResult.Success(
          endOfPaginationReached = true
        )
        // Query remoteKeyDao for the next RemoteKey.
        LoadType.APPEND -> {
          val remoteKey = database.withTransaction {
            remoteKeyDao.remoteKeyByQuery(query)
          }

          // You must explicitly check if the page key is null when
          // appending, since null is only valid for initial load.
          // If you receive null for APPEND, that means you have
          // reached the end of pagination and there are no more
          // items to load.
          if (remoteKey.nextKey == null) {
            return MediatorResult.Success(
              endOfPaginationReached = true
            )
          }

          remoteKey.nextKey
        }
      }

      // Suspending network load via Retrofit. This doesn't need to
      // be wrapped in a withContext(Dispatcher.IO) { ... } block
      // since Retrofit's Coroutine CallAdapter dispatches on a
      // worker thread.
      val response = networkService.searchUsers(query, loadKey)

      // Store loaded data, and next key in transaction, so that
      // they're always consistent.
      database.withTransaction {
        if (loadType == LoadType.REFRESH) {
          remoteKeyDao.deleteByQuery(query)
          userDao.deleteByQuery(query)
        }

        // Update RemoteKey for this query.
        remoteKeyDao.insertOrReplace(
          RemoteKey(query, response.nextKey)
        )

        // Insert new users into database, which invalidates the
        // current PagingData, allowing Paging to present the updates
        // in the DB.
        userDao.insertAll(response.users)
      }

      MediatorResult.Success(
        endOfPaginationReached = response.nextKey == null
      )
    } catch (e: IOException) {
      MediatorResult.Error(e)
    } catch (e: HttpException) {
      MediatorResult.Error(e)
    }
  }
}

Java

@NotNull
@Override
public Single<MediatorResult> loadSingle(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional String parameter. For every page
  // after the first, pass the String token returned from the previous page to
  // let it continue from where it left off. For REFRESH, pass null to load the
  // first page.
  Single<RemoteKey> remoteKeySingle = null;
  switch (loadType) {
    case REFRESH:
      // Initial load should use null as the page key, so you can return null
      // directly.
      remoteKeySingle = Single.just(new RemoteKey(mQuery, null));
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Single.just(new MediatorResult.Success(true));
    case APPEND:
      // Query remoteKeyDao for the next RemoteKey.
      remoteKeySingle = mRemoteKeyDao.remoteKeyByQuerySingle(mQuery);
      break;
  }

  return remoteKeySingle
    .subscribeOn(Schedulers.io())
    .flatMap((Function<RemoteKey, Single<MediatorResult>>) remoteKey -> {
      // You must explicitly check if the page key is null when appending,
      // since null is only valid for initial load. If you receive null
      // for APPEND, that means you have reached the end of pagination and
      // there are no more items to load.
      if (loadType != REFRESH && remoteKey.getNextKey() == null) {
        return Single.just(new MediatorResult.Success(true));
      }

      return networkService.searchUsers(query, remoteKey.getNextKey())
        .map(response -> {
          database.runInTransaction(() -> {
            if (loadType == LoadType.REFRESH) {
              userDao.deleteByQuery(query);
              remoteKeyDao.deleteByQuery(query);
            }

            // Update RemoteKey for this query.
            remoteKeyDao.insertOrReplace(new RemoteKey(query, response.getNextKey()));

            // Insert new users into database, which invalidates the current
            // PagingData, allowing Paging to present the updates in the DB.
            userDao.insertAll(response.getUsers());
          });

          return new MediatorResult.Success(response.getNextKey() == null);
        });
    })
    .onErrorResumeNext(e -> {
      if (e instanceof IOException || e instanceof HttpException) {
        return Single.just(new MediatorResult.Error(e));
      }

      return Single.error(e);
    });
}

Java

@NotNull
@Override
public ListenableFuture<MediatorResult> loadFuture(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  ResolvableFuture<RemoteKey> remoteKeyFuture = ResolvableFuture.create();
  switch (loadType) {
    case REFRESH:
      remoteKeyFuture.set(new RemoteKey(query, null));
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Futures.immediateFuture(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Futures.immediateFuture(new MediatorResult.Success(true));
      }

      // Query remoteKeyDao for the next RemoteKey.
      remoteKeyFuture.setFuture(
        remoteKeyDao.remoteKeyByQueryFuture(query));
      break;
  }

  return Futures.transformAsync(remoteKeyFuture, remoteKey -> {
    // You must explicitly check if the page key is null when appending,
    // since null is only valid for initial load. If you receive null
    // for APPEND, that means you have reached the end of pagination and
    // there are no more items to load.
    if (loadType != LoadType.REFRESH && remoteKey.getNextKey() == null) {
      return Futures.immediateFuture(new MediatorResult.Success(true));
    }

    ListenableFuture<MediatorResult> networkResult = Futures.transform(
      networkService.searchUsers(query, remoteKey.getNextKey()),
      response -> {
        database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
          remoteKeyDao.deleteByQuery(query);
        }

        // Update RemoteKey for this query.
        remoteKeyDao.insertOrReplace(new RemoteKey(query, response.getNextKey()));

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    }, bgExecutor);

    ListenableFuture<MediatorResult> ioCatchingNetworkResult =
      Futures.catching(
        networkResult,
        IOException.class,
        MediatorResult.Error::new,
        bgExecutor
      );

    return Futures.catching(
      ioCatchingNetworkResult,
      HttpException.class,
      MediatorResult.Error::new,
      bgExecutor
    );
  }, bgExecutor);
}

Refresh in place

If your app only needs to support network refreshes from the top of the list as in the previous examples, then your RemoteMediator does not need to define prepend load behavior.

However, if your app needs to support loading incrementally from the network into the local database, then you must provide support for resuming pagination starting at the anchor, the user's scroll position. Room's PagingSource implementation handles this for you, but if you're not using Room you can do this by overriding PagingSource.getRefreshKey(). For an example implementation of getRefreshKey(), see Define the PagingSource.

Figure 4 illustrates the process of loading data first from the local database, and then from the network once the database is out of data.

The PagingSource loads from the database into the UI until the database is out of data. Then the RemoteMediator loads from the network into the database, and afterward the PagingSource continues loading.
Figure 4. Diagram showing how PagingSource and RemoteMediator work together to load data.

Additional resources

To learn more about the Paging library, see the following additional resources:

Codelabs

Samples