Dynamically Retrieving Frames in Android

Photo by pine watt on Unsplash

In today’s mobile apps, watching videos is everywhere, from social media to gaming. But sometimes, developers want to do more than just play videos — they need to grab specific frames from them. This could be for making thumbnails, analyzing frames one by one, or adding cool editing features. In this article, we’ll show you how to do just that in Android apps, breaking down the steps in simple terms so you can grab frames from playing videos smoothly and easily.

For this demonstration, we’ll be using ExoPlayer, a widely used media framework for Android. However, it’s essential to note that the method we’ll discuss for retrieving frames from playing videos is not tied to any specific framework. You’re free to choose the media framework that best suits your project’s needs. Before we dive in, ensure you’ve added the following dependencies to your app’s build.gradle file:

def media3_version = "1.1.1"
implementation "androidx.media3:media3-exoplayer:$media3_version"
implementation "androidx.media3:media3-exoplayer-dash:$media3_version"
implementation "androidx.media3:media3-ui:$media3_version"
implementation "androidx.media3:media3-session:$media3_version"

Next, in the class where you intend to work with video playback (likely an Activity or Fragment), you can initialize an ExoPlayer instance as follows:

private val exoPlayer by lazy {
    ExoPlayer.Builder(this)
        .build()
}

Don’t forget to add a PlayerView to your layout and associate it with the ExoPlayer instance:

private fun setPlayer() {
    binding.playerView.player = exoPlayer
}

With these steps completed, you’ll be all set to start extracting frames from playing videos in your Android app. Let’s delve into the process.

Using MediaMetadataRetriever

The MediaMetadataRetriever class offers a straightforward solution for retrieving frames and metadata from media files in Android. This class provides a unified interface for such operations, making it a convenient choice for developers. To create an instance of MediaMetadataRetriever, no parameters are needed in its constructor, simplifying its initialization process. You can initialize it as follows:

private val retriever by lazy(::MediaMetadataRetriever)

Once initialized, setting a data source to the retriever is necessary to start fetching frames. For this project, let’s assume we’re using a local video file stored in the resources of the sample application. We can achieve this using the `setDataSource(FileDescriptor fd, long offset, long length)` method. Here’s an example of how to set the data source using a local video:

val assetFileDescriptor = resources.openRawResourceFd(R.raw.sample_video)
retriever.setDataSource(
    assetFileDescriptor.fileDescriptor,
    assetFileDescriptor.startOffset,
    assetFileDescriptor.length
)

However, if you need to work with a video file located at a URI, the `setDataSource(Context context, Uri uri)` method is available as an alternative.

Finally, let’s integrate this with our ExoPlayer setup. In the loadAndPrepareVideo() function, we set the data source for both the MediaMetadataRetriever and the ExoPlayer, ensuring synchronization between frame retrieval and video playback:

private fun loadAndPrepareVideo() {
    val assetFileDescriptor = resources.openRawResourceFd(R.raw.sample_video)
    retriever.setDataSource(
        assetFileDescriptor.fileDescriptor,
        assetFileDescriptor.startOffset,
        assetFileDescriptor.length
    )

    val video = getRawResourceUriString(R.raw.sample_video)
    val mediaItem = MediaItem.fromUri(video)
    exoPlayer.setMediaItem(mediaItem)
    exoPlayer.prepare()
}

private fun getRawResourceUriString(@RawRes rawResourceId: Int): String {
    val packageName = packageName
    return "android.resource://$packageName/raw/" + resources.getResourceEntryName(rawResourceId)
}

With these steps completed, you’re ready to retrieve frames from a playing video using MediaMetadataRetriever in conjunction with ExoPlayer.

Listening to Frames in ExoPlayer

To effectively retrieve frames from a playing video, we need to listen for changes in the playback state of ExoPlayer. This allows us to synchronize frame extraction with the video’s playback. Begin by registering a listener to ExoPlayer for playback changes:

exoPlayer.addListener(this)

Next, implement the onIsPlayingChanged method in your activity or fragment. This method will be triggered whenever the playback state of ExoPlayer changes. Here’s an example implementation:

override fun onIsPlayingChanged(isPlaying: Boolean) {
    super.onIsPlayingChanged(isPlaying)
    startProcessJobIfNeeded(isPlaying)
    cancelProcessJobIfNeeded(isPlaying)
}

We’ll now proceed to implement the functions cancelProcessJobIfNeeded and startProcessJobIfNeeded. Starting with cancelProcessJobIfNeeded, this function is responsible for cancelling the coroutine job if the video playback is paused:

private var job: Job? = null

private fun cancelProcessJobIfNeeded(isPlaying: Boolean) {
    if (!isPlaying && job != null) {
        job?.cancel()
        job = null
    }
}

Moving on to startProcessJobIfNeeded, this function initiates a coroutine job to continuously retrieve frames while the video is playing. Adjust the delay parameter to control the iteration frequency of frame retrieval:

private fun startProcessJobIfNeeded(isPlaying: Boolean) {
    if (job == null && isPlaying) {
        job = lifecycleScope.launch {
            while (isActive) {
                getCurrentFrame()
                delay(50) // Adjust delay as needed
            }
        }
    }
}

It’s important to note that this approach is based on the best available practice found online, as ExoPlayer doesn’t provide built-in support for listening to frame progress iteratively.

Finally, the main function responsible for retrieving the current frame is getCurrentFrame. This function calculates the current position of the video, retrieves the frame at that position using MediaMetadataRetriever, and scales the bitmap to match the dimensions of the PlayerView:

private fun getCurrentFrame(): Bitmap? {
    val currentPosMillis = exoPlayer.currentPosition.toDuration(DurationUnit.MILLISECONDS)
    val currentPosMicroSec = currentPosMillis.inWholeMicroseconds
    val currentFrame = retriever.getFrameAtTime(currentPosMicroSec, MediaMetadataRetriever.OPTION_CLOSEST)
    val bitmap = currentFrame ?: return null
    return Bitmap.createScaledBitmap(bitmap, binding.playerView.width, binding.playerView.height, false)
}

With these functions in place, you’ll be able to dynamically retrieve frames from a playing video in your Android app using ExoPlayer.

Conclusion

In this article, we’ve shown how to extract frames from playing videos in Android apps. By utilizing ExoPlayer and MediaMetadataRetriever, developers can seamlessly integrate frame retrieval functionality into their projects. Whether for video editing, frame analysis, or enhancing user experience, these techniques offer efficient solutions. Remember, while we demonstrated with ExoPlayer, the principles discussed are applicable across various media frameworks. With these tools, you’re equipped to leverage the power of video frames in your Android development endeavors. Happy coding!

Source Code

GitHub — oguzhanaslann/MLPosie: Sample project that demonstrates Pose detection on ML-kit in…

LinkedIn

Love you all.

Stay tune for upcoming blogs.

Take care.