Merge Videos and Photos with Transitions using OpenGL ES and MediaCodec

I tried to find a lot of posts or tutorials about how to work with OpenGL ES 2 in Android, however, I could find the one I want. My problem is to make a Video Slideshow out of videos and photos combination.

I have spent so much time reading and practicing CTS of Google but still couldn't take those codes to make what I wanted. Also, I tried to look closely to a sample project called Grafika which has great examples that I can learn from. As a result, I pretty much understand so many things such MediaCodec, Codec, MediaFormat, MediaMuxer YUV/RGB, FrameBuffer, Shaders(Vertex, Fragment), Surface, SurfaceTexture, Buffer, EGL Stuff such as EGLContext and more.

Actually, I did have some knowledge about OpenGL and Shader, but I struggled to combine all those things together.

Ideas in My Head (That doesn't work)

• Create a decoder which takes a Surface to decode a video file. The decoder has the ability to seek to a specific point in time.
• Create a series of frames of a photo using OpenGL textures.
• Create a shader (main shader) which takes all the shaders of video and photos and then merges it together by using two textures (External Texture). The shader would take care of applying filters like Saturation, Brightness, Contrast, ... to the whole movie.
• Create a class which extends from TextureView and create a thread which has OpenGL context. And let the main shader render to it. (same thing as GLSurfaceView)
• Create another encoder which takes the main shader and encode it into a file with MediaMuxer.

If you have any ideas, it would be great, I think this is one of the most difficult problems to achieve, but I think it would be easy for those who have tons of experience with it.

Most Inspiring Projects

• GPUVideo by Masayuki Suda
• ImageToVideoConverter by Masayuki Suda
• Mp4Composer by Masayuki Suda
• AndroidGPUImage by CyberAgent Advanced Technology Studio
• ImageEditor by CreateChance (Full-features photos to video slide show with transitions but doesn't support videos)
• AndroidTranscoder by ypresto

from Merge Videos and Photos with Transitions using OpenGL ES and MediaCodec