Friday, 23 April 2021

Camera 2 preview freezes when I uses YUV_420_888 Image Reader as a target surface

I'm working on an application that has requirements to detect objects and faces in a real-time camera feed. for this, I'm using MLkit. I have successfully implemented it object detection part with the back-facing camera with no issue. ML kit recommends using YUV_420_888 with the smallest size possible to have better results. I am using Camera 2 API and here's my code for setting up ImageReader for processing.

class SelfieCaptureFragment : Fragment(R.layout.fragment_selfie_capture) {

    protected lateinit var characteristics: CameraCharacteristics
    protected lateinit var camera: CameraDevice
    protected lateinit var session: CameraCaptureSession
    protected lateinit var requestPermissionLauncher: ActivityResultLauncher<String>
    protected lateinit var captureImageReader: ImageReader
    protected lateinit var analyzeImageReader: ImageReader
    
    /** [HandlerThread] and [Handler] where all camera operations run */
    private val cameraThread = HandlerThread("CameraThread").apply { start() }
    private val cameraHandler = Handler(cameraThread.looper)

    /** [HandlerThread] and [Handler] where all camera still image capturing operations run */
    private val captureImageReaderThread = HandlerThread("captureImageReaderThread").apply { start() }
    private val captureImageReaderHandler = Handler(captureImageReaderThread.looper)

    private val analyzeImageReaderThread = HandlerThread("imageReaderThread").apply { start() }
    private val analyzeImageReaderHandler = Handler(analyzeImageReaderThread.looper)

    companion object {
        /** Maximum number of images that will be held in the reader's buffer */
        const val IMAGE_BUFFER_SIZE: Int = 3
    }

    private fun configureCamera(selectedCameraId: String) {

        lifecycleScope.launch(Dispatchers.Main) {

            camera = openCamera(cameraManager, selectedCameraId, cameraHandler)

            val previewFraction = DisplayUtils
                .asFraction(previewSize!!.width.toLong(), previewSize!!.height.toLong())

            // Initialize an image reader which will be used to capture still photos
            captureSize = characteristics.get(
                CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)!!
                .getOutputSizes(ImageFormat.JPEG)
                .filter { DisplayUtils.asFraction(it.width.toLong(),it.height.toLong()) == previewFraction }
                .sortedBy { it.height * it.width}
                .reversed()
                .first()

            analyzeImageSize = characteristics
                .get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)!!
                .getOutputSizes(ImageFormat.YUV_420_888)
                .filter { DisplayUtils.asFraction(it.width.toLong(), it.height.toLong()) == previewFraction }
                .sortedBy { it.height * it.width}
                .first()

            if (captureSize != null) {

                captureImageReader = ImageReader.newInstance(
                    captureSize!!.width, captureSize!!.height, ImageFormat.JPEG, IMAGE_BUFFER_SIZE
                )

                analyzeImageReader = ImageReader.newInstance(
                    analyzeImageSize!!.width,
                    analyzeImageSize!!.height,
                    ImageFormat.YUV_420_888,
                    IMAGE_BUFFER_SIZE)

                Log.d(TAG, "Selected capture size: $captureSize")
                Log.d(TAG, "Selected image analyze size: $analyzeImageSize")

                val targets = listOf(
                    binding.cameraSurfaceView.holder.surface,
                    captureImageReader.surface,
                    analyzeImageReader.surface
                )

                session = createCaptureSession(camera, targets, cameraHandler)

                val captureBuilder = camera.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
                captureBuilder.set(
                    CaptureRequest.CONTROL_AF_MODE,
                    CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE
                )
                captureBuilder.addTarget(binding.cameraSurfaceView.holder.surface)
                captureBuilder.addTarget(analyzeImageReader.surface)

                session.setRepeatingRequest(captureBuilder.build(), null, cameraHandler)

              
            }
        }
    }
}

This code is identical to what I'm using in the back-facing camera to set up the image reader for the object detection part. I have not yet Implemented any logic related to Image processing to detect faces in the camera steam. Error is in the line where I set up the target. captureBuilder.addTarget(analyzeImageReader.surface) If I comment that line camera preview works fine. I get the following error and warning in logcat over and over when the camera preview freezes.

W/libc: Unable to set property "debug.sf.dequeuebuffer" to "1": connection failed; errno=13 (Permission denied)
E/BufferQueueProducer: [ImageReader-256x144f23m3-23007-3](id:59df00000003,api:4,p:4806,c:23007) waitForFreeSlotThenRelock: timeout

I can't understand why it's not working in the front-facing camera. analyze image size return from the analyzeImageReader is 256x144 which is not a big resolution either plus it's the smallest out of all the supported sizes for the format of YUV_420_888. Any Help will be highly appreciated. I thought I have figured it out all when I implemented objected detection part. this was a very unexpected issue.

Edit: I'm still stuck with this issue so I have set up a small project to showcase the whole picture. Please check the repo and help me find a solution. Thnaks.



from Camera 2 preview freezes when I uses YUV_420_888 Image Reader as a target surface

No comments:

Post a Comment