Thursday, 31 October 2019

Optimization in bitmap creation

I'm writing an application that renders a sequence of pictures received in real-time from a remote TCP connection into an ImageView element. The stream is composed of single frames encoded in PGM format and sent at 9Hz I tought that a very low frame rate like this should be handled easily using a background Service that sends fully decoded bitmap to my MainActivity.

Here's my VideoService (I'm posting just run() method since I think it's the only one of some interest):

    public void run() {
        InetAddress serverAddr = null;

        try {
            serverAddr = InetAddress.getByName(VIDEO_SERVER_ADDR);
        } catch (UnknownHostException e) {
            Log.e(getClass().getName(), e.getMessage());
            e.printStackTrace();
            return;
        }

        Socket socket = null;
        BufferedReader reader = null;

        do {
            try {
                socket = new Socket(serverAddr, VIDEO_SERVER_PORT);

                reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));

                boolean frameStart = false;

                LinkedList<String> frameList = new LinkedList<>();

                while (keepRunning) {
                    final String message = reader.readLine();

                    if (!frameStart && message.startsWith("F"))
                        frameStart = true;
                    else if (frameStart && message.startsWith("EF")) {
                        frameStart = false;

                        final Bitmap bitmap = Bitmap.createBitmap(IR_FRAME_WIDTH, IR_FRAME_HEIGHT, Bitmap.Config.ARGB_8888);
                        final Canvas canvas = new Canvas(bitmap);

                        final String[] data = frameList.toArray(new String[frameList.size()]);

                        canvas.drawBitmap(bitmap, 0, 0, null);

                        //Log.d(this.getClass().getName(), "IR FRAME COLLECTED");

                        if ((data.length - 6) == IR_FRAME_HEIGHT) {
                            float grayScaleRatio = Float.parseFloat(data[2].trim()) / 255.0f;

                            for (int y = 0; y < IR_FRAME_HEIGHT; y++) {
                                final String line = data[y + 3];
                                final String[] points = line.split("\\s+");

                                if (points.length == IR_FRAME_WIDTH) {
                                    for (int x = 0; x < IR_FRAME_WIDTH; x++) {
                                        final float grayLevel = Float.parseFloat(points[x]) / grayScaleRatio;

                                        Paint paint = new Paint();

                                        paint.setStyle(Paint.Style.FILL);

                                        final int level = (int)grayLevel;

                                        paint.setColor(Color.rgb(level, level, level));

                                        canvas.drawPoint(x, y, paint);
                                    }
                                } else
                                    Log.d(this.getClass().getName(), "Malformed line");
                            }

                            final Intent messageIntent = new Intent();

                            messageIntent.setAction(VIDEO_BROADCAST_KEY);

                            ByteArrayOutputStream stream = new ByteArrayOutputStream();

                            bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
                            bitmap.recycle();
                            messageIntent.putExtra(VIDEO_MESSAGE_KEY, stream.toByteArray());
                            stream.close();
                            sendBroadcast(messageIntent);
                        } else
                            Log.d(this.getClass().getName(), "Malformed data");

                        frameList.clear();
                    } else if (frameStart)
                        frameList.add(message);
                }

                Thread.sleep(VIDEO_SERVER_RESPAWN);

            } catch (Throwable e) {
                Log.e(getClass().getName(), e.getMessage());
                e.printStackTrace();
            }
        } while (keepRunning);

        if (socket != null) {
            try {
                socket.close();
            } catch (Throwable e) {
                Log.e(getClass().getName(), e.getMessage());
                e.printStackTrace();
            }
        }
    }

The message is a line coming from the following text:

F
P2
160 120
1226
193 141 158 152 193 186 171 177 186 160 195 182 ... (160 times)
                         .
                         . (120 lines)
                         .
278 248 253 261 257 284 310 304 304 272 227 208 ... (160 times)


EF

In MainActivity I handle this trough this code:

class VideoReceiver extends BroadcastReceiver {
    final public Queue<Bitmap> imagesQueue = new LinkedList<>();

    @Override
    public void onReceive(Context context, Intent intent) {

        try {
            //Log.d(getClass().getName(), "onReceive() called");

            final byte[] data = intent.getByteArrayExtra(VideoService.VIDEO_MESSAGE_KEY);

            final Bitmap bitmap = BitmapFactory.decodeByteArray(data,0,data.length);

            imagesQueue.add(bitmap);

            runOnUiThread(updateVideoTask);
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

updateVideoTask task is defined like this:

    updateVideoTask = new Runnable() {
        public void run() {
            if (videoReceiver == null) return;

            if (!videoReceiver.imagesQueue.isEmpty())
            {
                final Bitmap image = videoReceiver.imagesQueue.poll();

                if (image == null) return;

                videoView.setImageBitmap(image);

                Log.d(this.getClass().getName(), "Images to spool: " + videoReceiver.imagesQueue.size());
            }
        }
    };

Unluckly when I run the application I notice a very low frame rate and a very big delay. I cannot argue what's going on. The only hints I got from logcat are these lines:

2019-05-20 16:37:08.817 29566-29580/it.tux.gcs I/art: Background sticky concurrent mark sweep GC freed 88152(3MB) AllocSpace objects, 3(52KB) LOS objects, 22% free, 7MB/10MB, paused 3.937ms total 111.782ms
2019-05-20 16:37:08.832 29566-29587/it.tux.gcs D/skia: Encode PNG Singlethread :      13003 us, width=160, height=120

even with the sum of all this delay (140 ms) the app should sustain a frame rate of more than 5Hz while am getting 0.25Hz or even worse.

After some investigation I found that moving:

Paint paint = new Paint();
paint.setStyle(Paint.Style.FILL);

out of the nested loops prevent GC from being invoked so frequently and I found another major source of delay in this line:

final String[] points = line.split("\\s+");

it burns out 2ms per time so I decided to go for something less smart but faster:

final String[] points = line.split(" ");

Anyway it's still not enough.. the code between:

canvas.drawBitmap(bitmap, 0, 0, null);

and

sendBroadcast(messageIntent);

still consume more than 200ms ... how can I do better than this?



from Optimization in bitmap creation

No comments:

Post a Comment