SurfaceTexture’s onFrameAvailable() method always called too late

by

The way SurfaceTexture works makes this a bit tricky to get right.

The docs say the frame-available callback “is called on an arbitrary thread”. The SurfaceTexture class has a bit of code that does the following when initializing (line 318):

if (this thread has a looper) {
    handle events on this thread
} else if (there's a "main" looper) {
    handle events on the main UI thread
} else {
    no events for you
}

The frame-available events are delivered to your app through the usual Looper / Handler mechanism. That mechanism is just a message queue, which means the thread needs to be sitting in the Looper event loop waiting for them to arrive. The trouble is, if you’re sleeping in awaitNewImage(), you’re not watching the Looper queue. So the event arrives, but nobody sees it. Eventually awaitNewImage() times out, and the thread returns to watching the event queue, where it immediately discovers the pending “new frame” message.

So the trick is to make sure that frame-available events arrive on a different thread from the one sitting in awaitNewImage(). In the ExtractMpegFramesTest example, this is done by running the test in a newly-created thread (see the ExtractMpegFramesWrapper class), which does not have a Looper. (For some reason the thread that executes CTS tests has a looper.) The frame-available events arrive on the main UI thread.

Update (for “edit 3”): I’m a bit sad that ignoring the “size” field helped, but pre-4.3 it’s hard to predict how devices will behave.

If you just want to display the frame, pass the Surface you get from the SurfaceView or TextureView into the MediaCodec decoder configure() call. Then you don’t have to mess with SurfaceTexture at all — the frames will be displayed as you decode them. See the two “Play video” activities in Grafika for examples.

If you really want to go through a SurfaceTexture, you need to change CodecOutputSurface to render to a window surface rather than a pbuffer. (The off-screen rendering is done so we can use glReadPixels() in a headless test.)

More Related Contents:

  1. How to reduce latency in MediaCodec video/avc decoding
  2. Draw text in OpenGL ES
  3. GLES10.glGetIntegerv returns 0 in Lollipop only
  4. Appropriate multiplication of matrices for rotation/translation
  5. Modifying camera output using SurfaceTexture and OpenGL
  6. Android: GLES20: Called unimplemented OpenGL ES API
  7. Android OpenGL Texture Compression
  8. Applying Effects on Video being Played
  9. Is Google’s Android OpenGL tutorial teaching incorrect linear algebra?
  10. libgdx SpriteBatch render to texture
  11. Adding watermark bitmap over video in android: 4.3’s MediaMuxer or ffmpeg
  12. Limitation on texture size? Android Open GL ES 2.0
  13. How to deal with different aspect ratios in libGDX?
  14. Android OpenGL .OBJ file loader [closed]
  15. How to use MediaCodec without MediaExtractor for H264
  16. YUV to RGB conversion by fragment shader
  17. Is discard bad for program performance in OpenGL?
  18. Crop video before encoding with MediaCodec for Grafika’s “Continuous Capture” Activity
  19. Why my opengl output differs for various devices?
  20. Quaternion rotation does not work as excepted
  21. Apply custom filters to camera output
  22. Android OpenGL ES Transparent Background
  23. Does the Android emulator support OpenGL ES 3.0?
  24. Converting from GLSurfaceView to TextureView (via GLTextureView)
  25. PCM -> AAC (Encoder) -> PCM(Decoder) in real-time with correct optimization
  26. Android: Deciding between SurfaceView and OpenGL (GLSurfaceView)
  27. Threading textures load process for android opengl game
  28. Emulator: Process finished with exit code -1073741819 (0xC0000005)
  29. How to generate the AAC ADTS elementary stream with Android MediaCodec
  30. Does the Android Emulator support OpenGL ES 2.0?

Leave a Comment