Saturday 2 January 2021

How to synchronize native OpenGL rendering when using flutter's Texture widget on Android

I'm using a Texture widget, rendering its content from native code using OpenGL ES. In native code I call ANativeWindow_fromSurface and from that create an EGL surface. AIUI what happens is:

  • The ANativeWindow represents the producer side of a buffer queue.
  • Calling eglSwapBuffers causes a texture to be sent to this queue.
  • Flutter receives the texture and renders it using Skia when the TextureLayer is painted.
  • The texture is scaled to match the size of the TextureLayer (the scaling happens in AndroidExternalTextureGL::Paint()).

I'm trying to figure out how to synchronise the OpenGL rendering. I think I can use the choreographer to synchronise with the display vsync, but I'm unclear on how much latency this bufferqueue-then-render-with-skia mechanism introduces. I don't see any means to explicitly synchronise my native code's generation of textures with the TextureLayer's painting of them.

The scaling appears to be a particularly tricky aspect. I would like to avoid it entirely, by ensuring that the textures the native code generates are always of the right size. However there doesn't appear to be any direct link between the size of the TextureLayer and the size of the Surface/ANativeWindow. I could use a SizeChangedLayoutNotifier (or one of various alternative hacks) to detect changes in the size and communicate them to the native code, but I think this would lag by at least a frame so scaling would still take place when resizing.

I did find this issue, which talks about similar resizing challenges, but in the context of using an OEM web view. I don't understand Hixie's detailed proposal in that issue, but it appears to be specific to embedding of OEM views so I don't think it would help with my case.

Perhaps using a Texture widget here is the wrong approach. It seems to be designed mainly for displaying things like videos and camera previews. Is there another way to host natively rendered, interactive OpenGL graphics in Flutter?



from How to synchronize native OpenGL rendering when using flutter's Texture widget on Android

No comments:

Post a Comment