Real-Time Pose Detection in Camera with ML Kit in Android

Photo by Reinhart Julian on Unsplash

We’ve already learned about pose detection in Android, where we could analyze body poses from pictures and videos.

• Pose detection on images
• Pose detection on videos

Now, we’re going to take it up a notch: we’ll learn how to do it with live camera feeds. Using CameraX, we’ll track body movements in real time. Let’s get started!

Just a heads-up: we won’t be going into the details of setting up CameraXin this blog to keep the focus on pose detection. If you’re new to CameraX, I recommend checking out the official documentation.

Before we dive into the details, it’s essential to note a crucial step after setting up CameraX. Once you’ve configured CameraX and obtained necessary permissions, the next step is to bind an image analysis to the camera provider object. This allows us to process frames from the camera feed in real-time. The setup involves attaching an ImageAnalysisobject to the camera provider.Here’s a simplified function to achieve this:

private fun poseDetectionImageAnalyzer(
    cameraProvider: ProcessCameraProvider,
    cameraSelector: CameraSelector,
    cameraExecutor: Executor
): ImageAnalysis {
    val imageAnalyzer = ImageAnalysis.Builder()
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
        .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
        .build()

    imageAnalyzer.setAnalyzer(cameraExecutor) { imageProxy ->
        val bitmap = imageProxy.toBitmap()
        val inputData = InputImage.fromBitmap(bitmap, imageProxy.imageInfo.rotationDegrees)
        
        val poseOptions = AccuratePoseDetectorOptions.Builder()
            .setDetectorMode(AccuratePoseDetectorOptions.CPU_GPU)
            .build()
        val poseDetector = PoseDetection.getClient(poseOptions)

        poseDetector.process(inputData)
            .addOnSuccessListener { onPoseDetectionSucceeded(it, bitmap) }
            .addOnFailureListener { onPoseDetectionFailed(it) }
            .addOnCompleteListener { imageProxy.close() }
    }

    cameraProvider.bindToLifecycle(this, cameraSelector, imageAnalyzer)
    return imageAnalyzer
}

private fun ImageProxy.toBitmap(): Bitmap {
    val buffer = planes[0].buffer
    val bytes = ByteArray(buffer.capacity()).apply { buffer.get(this) }
    return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}

Note: After converting ImageProxyobject to bitmap, the remaining steps are similar to other blogs’ solutions. What is new here is after processing the image, we close the ImageProxy object to avoid memory leaks.

Once we’ve successfully detected the pose, drawing it onto the camera feed is straightforward. Here’s the code snippet for handling pose detection success:

private fun onPoseDetectionSucceeded(pose: Pose?, bitmap: Bitmap) {
    val pose = pose ?: return
    val allPose = pose.allPoseLandmarks
    if (allPose.isEmpty()) {
        binding.graphicOverlayCamera.clear()
        return
    }
    binding.graphicOverlayCamera.setImageSourceInfo(bitmap.width, bitmap.height, false)
    binding.graphicOverlayCamera.clear()
    binding.graphicOverlayCamera.add(PoseGraphic(binding.graphicOverlayCamera, pose))
}

This function checks if a pose was detected. If not, it clears the GraphicOverlayin to avoid displaying incorrect information. Otherwise, it sets the image source information and adds a PoseGraphicto the overlay. Make sure to include a GraphicOverlayin your layout for drawing poses efficiently. However, feel free to use alternative methods for drawing if needed. With this setup, you’re ready to visualize detected poses in real-time on the camera feed.

In conclusion, We explored real-time Android pose recognition in this blog, with a particular focus on binding image analysis. In incoming blogs, we’ll add a pose classification feature that can identify particular movements.

Here is a sample repository that implements what we have covered in this article.

GitHub - oguzhanaslann/MLPosie: Sample project that demonstrates Pose detection on ML-kit in…

LinkedIn

Love you all.

Stay tune for upcoming blogs.

Take care.