Defined in linux/msm_mdp.h in the downstream kernel.
On Qualcomm systems, there is a special block separate from the GPU called MDSS (mobile display subsystem). It contains a display controller called MDP, which is given access to image data from the GPU and other sources and returns an image buffer ready to be sent directly to the screen. It also supports some additional compositing and rendering features.
MDP comes in three versions:
The framebuffer devices created by Qualcomm's driver are standard Linux fbdev devices. The usual fbdev ioctls and mmap() are implemented and apparently even work on some devices (see https://wiki.postmarketos.org/wiki/Nokia_2720_Flip_(nokia-beatles)#Display). But this doesn't seem to be the right way for three reasons:
fbcat would work on stock. In reality, it returns a blank image.An overlay is a kind of buffer that gets rendered on top of the framebuffer. The actual framebuffer stays black, but the content of the overlay is shown on the display. Advanced MDP versions support multiple overlays with scaling, Z-Indexing and opacity control (this is the compositing feature mentioned above).
Overlays are very useful if you have a buffer from some external source (camera, video decoder hardware, GPU, etc.) and you need to display it without copying the image in memory to achieve high performance.
That's why most image-related custom APIs use ION buffers: they're Linux dma_bufs exported to userspace and may be passed around between drivers. The mainline Linux DRM framework provides a similar API, but Android needed its own for some reason.
MDP3 might not support all of these advanced features, but it does work with one overlay and that's what is used on Bananian or in the recovery menu for example. It's highly likely that B2G uses overlays too.
FBIOGET_VSCREENINFO ioctl)MSMFB_OVERLAY_SET on the framebuffer device with all the struct parameters set. It only creates the overlay and sets its dimensions, but doesn't tell the driver anything about the image.MSMFB_OVERLAY_PLAY on the framebuffer device. You just need to set id to the returned overlay ID and data.memory_id to the ION memory descriptor.MSMFB_DISPLAY_COMMIT (.flags = MDP_DISPLAY_COMMIT_OVERLAY) on the framebuffer device. This displays the frame.MSMFB_OVERLAY_PLAY again. Something might be wrong here)More research is needed here.
When overlays are used, fbcat does not work. So how can the screen be captured?
Just guessing the ID and retrieving the ION memory handle does not work because only the overlay metadata is accessible using the ioctl API.
Qualcomm seems to have designed a special API for this purpose (msm_mdp.h:49), but it is not implemented on MDP3.
The driver code also mentions "writeback panels". That could either be part of the Writeback API or some other screen capturing interface based on virtual displays.
On Android-based systems like KaiOS, there is a screencap command that can dump the screen to a PNG file. It works but the returned image data looks skewed on some KaiOS devices, most notably the Nokia 8110 4G and 800 Tough, probably because the image format is specified incorrectly somewhere.
The command does some binder IPC calls according to strace, so the screen buffer it uses is probably created somewhere in userspace and not inside the display driver.