Sunday 26 June 2022

Update frames of a video on a HTML page, from incoming raw image data

I have raw image data (1000 x 1000 pixels x 3 bytes per pixel) in Python, that I need to send to a HTML page in realtime, at 20 frames per second (this is 57 MB of data per second!).

I already tried the multipart/x-mixed-replace method (as seen in Sending RGB image data from Python numpy to a browser HTML page), with various encoding: BMP, PNG, JPG. It is quite intensive for the CPU, so I'm trying alternatives.

I am now getting the raw image data directly in JavaScript with binary XHR HTTP requests.

Question: How to (CPU-efficiently) decode binary RGB data from dozens of binary XHR HTTP requests into a <video> or <img> or <canvas> on a HTML page, with Javascript?

oReq = new XMLHttpRequest();
oReq.open("GET", "/imagedata", true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
  var arrayBuffer = oReq.response;
  if (arrayBuffer) {
    var byteArray = new Uint8Array(arrayBuffer);
    // update displayed image
  }
};
oReq.send(null);

Edit: The method given in @VC.One's comment: var byteArray = new Uint8ClampedArray(arrayBuffer); var imgData = new ImageData(byteArray, 1000, 1000); var ctx = myCanvas.getContext('2d'); ctx.putImageData(imgData, 0, 0); works and is lighter for the CPU: 5%-6% for the server sending the data vs. 8%-20% with BMP/PNG/JPG encoding. But Chromium now has two processes in parallel for this task, each of them ~ 15% CPU. So the total performance is not much better. Do you see other potential alternatives to efficiently send raw image data from a Python or C++ HTTP server to Chromium?

Also new ImageData(...) requires a 1000x1000x4 bytes array for R, G, B, A. This requires that I send alpha channel that I don't need; maybe there's a way with ImageData to only pass a RGB (nxnx3 bytes) array?


Edit 2: the real bottleneck is the XHR HTTP requests between my process #1 and process #2 (Chrome) on the same computer for up to 100 MB/sec. Is there a more direct inter process communication possible between process #1 and Chrome? (some sort of direct memory access?)

See Chrome + another process: interprocess communication faster than HTTP / XHR requests?



from Update frames of a video on a HTML page, from incoming raw image data

No comments:

Post a Comment