Real-Time Pose Detection in Image with ML Kit in Android

Photo by charlesdeluvio on Unsplash

Machine learning is redefining mobile application development in the fast-paced field of Android development. At the heart of this breakthrough is posture detection, an advanced technique that allows for the detection of human positions inside photographs. In this post, we look at how Google’s machine learning platform, ML Kit, can be used by Android developers to easily incorporate posture detection features into their apps. This tutorial offers a clear road map for using ML Kit for posture detection.

dependencies {
  implementation 'com.google.mlkit:pose-detection:latest-version'
  implementation 'com.google.mlkit:pose-detection-accurate:latest-version'
}

After that, go ahead and add a sample image to your project. So, you can test the pose detection feature on it.

Next, we need to create a PoseDetection client object to process the image and detect the poses in it. This

requires PoseDetectorOptionsBase. The following code snippet shows how to create a PoseDetection client object:

    private val poseOptions =   AccuratePoseDetectorOptions.Builder()
            .setDetectorMode(AccuratePoseDetectorOptions.SINGLE_IMAGE_MODE)
            .build()

    private val poseDetector = PoseDetection.getClient(poseOptions)

Here I use `AccuratePoseDetectorOptions.SINGLE_IMAGE_MODE` hence the pose detection will be done on a single image. There are also `STREAM_MODE`, `CPU` and `GPU` modes available.

After creating the client object, all we need to do is pass an input image object. There are many factory methods to create an InputImage instance, but to keep things simple, I will use only the `fromBitmap` method. And to load the image into the ImageView that is used to display the image, I will use the Glide library. In addition, I will use the listener method, which I discussed in my previous post.

    val imageId = R.drawable.sample_image
    binding.imageView.load(imageId) { drawable -> 
            val bitmap = drawable.toBitmap()
            val inputImage = getInputImageFrom(bitmap)
            poseDetector.process(inputImage)
                .addOnSuccessListener { onPoseDetectionSucceeded(it, bitmap) }
                .addOnFailureListener(::onPoseDetectionFailed)
    }

Here, the detection will be fired right after the image is loaded into the ImageView. Now, we will focus on the `onPoseDetectionSucceeded`. We can basically neglect the `onPoseDetectionFailed`, since handling the failure is not the main focus of this post.

 private fun onPoseDetectionSucceeded(pose: Pose?, bitmap: Bitmap) {
        val pose = pose ?: return
        val allPose = pose.allPoseLandmarks
        if (allPose.isEmpty()) {
            Log.e("Pose detection", "onCreate: no pose detected")
            return
        }
        // rest of the code that we will cover
    }

The pose parameter here contains the detected pose in the image. And the `allPoseLandmarks` contains all the landmarks of the detected pose. Moreover, it’s up to the handler code to decide how to use and display the detected pose. For sake of simplicity, I will use a custom view that is provided by the sample code of ML-Kit. You can check out the details here. At last, we can apply the detected pose landmarks to `GraphicOverlay`.

Of course, do not forget to add the `GraphicOverlay` to the layout file. And make sure that `GraphicOverlay` covers the whole `ImageView` that is used to display the image.

  private fun onPoseDetectionSucceeded(pose: Pose?, bitmap: Bitmap) {
        val pose = pose ?: return
        val allPose = pose.allPoseLandmarks
        if (allPose.isEmpty()) {
            Log.e("Pose detection", "onCreate: no pose detected")
            return
        }
        binding.graphicOverlay.setImageSourceInfo(bitmap.width, bitmap.height, false)
        binding.graphicOverlay.add(PoseGraphic(binding.graphicOverlay, pose))
    }

Now we are done with pose detection. The detected pose will be displayed on the `ImageView` as soon as the image is loaded into it. But there is one more thing that we need to do. That is simply to close the pose detector when the activity is destroyed.

   override fun onDestroy() {
        super.onDestroy()
        poseDetector.close()
    }

Finally, we can just run the app and see the detected pose on the image.

Conclusion

In conclusion, ML Kit simplifies pose detection for Android developers, enabling seamless integration into applications. Its intuitive APIs and comprehensive documentation make detecting poses straightforward, facilitating innovation. In upcoming blogs, I will explore extending this functionality to video and live camera feeds, expanding the scope of creative possibilities for developers.

Source Code

GitHub - oguzhanaslann/MLPosie: Sample project that demonstrates Pose detection on ML-kit in…

LinkedIn

Love you all.

Stay tune for upcoming blogs.

Take care.