Get a frame and modify its data

Every video and audio sample in MFormats SDK and MPlatform SDK travels inside a frame object. This article shows how to get a frame and how to read or modify its data, whether the frame is stored in system memory (a CPU frame) or as a texture in video memory (a GPU frame), and how to copy a frame into an OpenGL texture.

Frames in MFormats SDK and MPlatform SDK

MFormats SDK is frame-based: you get a frame, process it, and pass it on. Its frames are MFFrame objects with the IMFFrame interface.

Frames in MPlatform SDK are MFrame objects. They are built on the same core as MFormats frames, so every MFrame also supports the IMFFrame interface. Cast an MFrame to IMFFrame whenever you need a method that is described in this article:

IMFFrame mfFrame = (IMFFrame)myMFrame;

For GPU frames, both SDKs also provide the IMFFrameGPU interface on the same frame object.

The examples in this article use MFormats SDK types. In MPlatform SDK, declare frames returned by IMFFrame methods (for example, MFClone) as IMFFrame instead of MFFrame.

A frame holds its data only while the frame object exists, so any pointer you get from a frame is valid only until you release that frame. In .NET, release every frame with Marshal.ReleaseComObject() when you no longer need it.

Get a frame

MFormats SDK

Source objects such as MFReader and MFLive return frames through the SourceFrameGet and SourceFrameConvertedGet methods. The second one also converts the frame to the format you pass in:

MFReaderClass reader = new MFReaderClass();
reader.ReaderOpen(@"c:\myVideo.mp4", "");

M_AV_PROPS avProps = new M_AV_PROPS();
avProps.vidProps.eVideoFormat = eMVideoFormat.eMVF_HD1080_50i;

MFFrame frame;
reader.SourceFrameConvertedGet(ref avProps, -1, out frame, "");

MPlatform SDK: ObjectFrameGet method

ObjectFrameGet returns the frame that an object is currently processing. It is part of the IMObject interface, so you can call it on any object, whether it is a source or a receiver:

MFrame myFrame;
(myPlaylist as IMObject).ObjectFrameGet(out myFrame, "");

MPlatform SDK: FileFrameGet and FileFrameGetByTC methods

These methods return a frame from a given position of a file: FileFrameGet takes the position in seconds, and FileFrameGetByTC takes a timecode. They are available on objects that support the IMFile interface, such as MFile, MPlaylist and MMixer:

MFrame frameByPos;
(myPlaylist as IMFile).FileFrameGet(10.0, 0, out frameByPos);

MFrame frameByTC;
M_TIMECODE myTC = new M_TIMECODE();
myTC.nHours = 10;
myTC.nMinutes = 23;
myTC.nSeconds = 45;
myTC.nFrames = 15;
(myPlaylist as IMFile).FileFrameGetByTC(ref myTC, out frameByTC);

MPlatform SDK: OnFrame and OnFrameSafe events

The OnFrame and OnFrameSafe events give you every frame that an object processes. See Events for details, and Modify frame data below for how to change frames inside the event.

Check whether a frame is a CPU or a GPU frame

How you read a frame's video depends on where it is stored. Frames are GPU frames when the GPU pipeline is on (gpu_pipeline set to true on MFFactory), or when a frame was created from a texture. Check the eGPUFlags field of the video properties returned by MFAVPropsGet:

M_AV_PROPS props;
int audioSamples;
frame.MFAVPropsGet(out props, out audioSamples);

bool isGpuFrame = props.vidProps.eGPUFlags != eGPUTextureFormat.eGTF_CPU_Frame;

For a GPU frame, eGPUFlags holds the DXGI format of the texture (see eGPUTextureFormat), plus the eGTF_GPU_SeparateFields flag if the two fields are stored in separate textures.

Audio and the frame's other data (timecode, ANC data, strings) are always in system memory, even for GPU frames.

Read data of a CPU frame

Video data

MFVideoGetBytes returns a pointer to the video data and its size in bytes. In MPlatform SDK, the FrameVideoGetBytes method of MFrame does the same.

int cbVideo;
long pbVideo;
frame.MFVideoGetBytes(out cbVideo, out pbVideo);

if (pbVideo != 0 && cbVideo > 0)
{
    // Copy the frame to a managed array (or work with the pointer directly)
    byte[] videoData = new byte[cbVideo];
    Marshal.Copy(new IntPtr(pbVideo), videoData, 0, cbVideo);
}

The layout of the data is described by the frame's video properties (props.vidProps from MFAVPropsGet):

• fccType - the pixel format,
• nWidth, nHeight - the frame size,
• nRowBytes - the length of one row in bytes (the stride),
• for RGB formats, the sign of nHeight: negative means the first row in memory is the top row, positive means the first row in memory is the bottom row.

For a GPU frame, MFVideoGetBytes returns zero size and a zero pointer, because the video isn't in system memory. See Get data from a GPU frame.

Planar and cropped frames

Some frames don't keep their video in one simple block, for example planar YUV frames with separate planes, or frames cut from a bigger frame. For such frames, MFVideoGetBytes returns the first plane only. Use MFVideoGetBytesEx instead: it returns an MF_VID_PTR structure with a pointer, a size in pixels and a stride for each plane.

MF_VID_PTR planes;
frame.MFVideoGetBytesEx(out planes);

IntPtr firstRow = new IntPtr(planes.lpVideoPlanes[0]);
int stride = planes.cbVideoRowBytes[0];
int width = planes.szVideoPlanes[0].cx;
int height = planes.szVideoPlanes[0].cy;
// Row y starts at firstRow + y * stride

MFVideoGetBytesEx always returns rows from top to bottom. For bottom-up RGB frames, lpVideoPlanes[0] points to the top row and cbVideoRowBytes[0] is negative, so the formula above works for every frame.

Audio data

MFAudioGetBytes returns a pointer to the interleaved PCM audio and its size in bytes. In MPlatform SDK, FrameAudioGetBytes does the same. The format is described by props.audProps (nChannels, nSamplesPerSec, nBitsPerSample), and MFAVPropsGet also returns the number of samples per channel.

int cbAudio;
long pbAudio;
frame.MFAudioGetBytes(out cbAudio, out pbAudio);

To work with one channel at a time, use MFAudioChannelGetBytes. It returns the data of a single channel. For 20-, 24- and 32-bit integer audio, the channel data is returned as 32-bit float. Pass 1 as the second parameter to add the channel if the frame doesn't have it yet. Changes you make to the channel data are written back to the frame only when you call MFAudioChannelsUpdate:

int cbChannel;
long pbChannel;
frame.MFAudioChannelGetBytes(0, 0, out cbChannel, out pbChannel);
// ... modify the samples of the first channel ...
frame.MFAudioChannelsUpdate();

VU meters: if you change the interleaved audio returned by MFAudioGetBytes, call MFAudioGain with "recalc_vu_meters" afterwards, so the frame's peak levels match the new data. MFAudioChannelsUpdate updates the levels automatically.

frame.MFAudioGain("recalc_vu_meters", 0.0, 0.0);

All frame information at once

MFAllGet fills an MF_FRAME_INFO structure with the frame's time, media properties, audio pointer and size, and the number of attached data items, objects and strings. For planar or cropped frames, use MFVideoGetBytesEx to access the video, as described above.

Bitmaps and image files

• MFVideoGetHbitmap (MPlatform: FrameVideoGetHbitmap) returns a Windows bitmap handle with a copy of the video. Frames in YUV formats are converted to ARGB32 first. This method works for CPU frames only. Delete the bitmap with the WinAPI DeleteObject function when you are done.
• MFVideoSaveToFile (MPlatform: FrameVideoSaveToFile) saves the video to an image file. The format is taken from the extension: PNG, JPG, BMP, TGA or TIFF, and DPX for 10-bit r210 frames. This method also works for GPU frames.
• MFAudioSaveToFile (MPlatform: FrameAudioSaveToFile) saves the audio to a WAV file, or appends it to an existing one.

[DllImport("gdi32.dll")]
static extern bool DeleteObject(IntPtr hObject);

long hBitmap;
frame.MFVideoGetHbitmap(out hBitmap);
using (Bitmap bitmap = Image.FromHbitmap(new IntPtr(hBitmap)))
{
    bitmap.Save(@"c:\frame.bmp");
}
DeleteObject(new IntPtr(hBitmap));

frame.MFVideoSaveToFile(@"c:\frame.png");

Modify frame data

The pointers returned by MFVideoGetBytes, MFVideoGetBytesEx and MFAudioGetBytes point to the frame's own memory, so you can also write to them, for example to draw on frames. Change the data before the frame goes to the next object:

• In MFormats SDK, make your changes before you pass the frame to ReceiverFramePut (for example, of MFPreview or MFRenderer).
• In MPlatform SDK, use the synchronous OnFrame or OnFrameSafe event with frame data access turned on. The frame goes to the output (for example, a preview or a renderer) only after your event handler returns:

myFile.PropsSet("object::on_frame.sync", "true");
myFile.PropsSet("object::on_frame.data", "true");
myFile.OnFrameSafe += myFile_OnFrameSafe;

private void myFile_OnFrameSafe(string bsChannelID, object pMFrame)
{
    IMFFrame frame = (IMFFrame)pMFrame;
    // ... read or modify the frame data ...
    Marshal.ReleaseComObject(pMFrame);
}

To change a GPU frame, lock it for writing with MFVideoLock, as described in the next section.

Get data from a GPU frame

A GPU frame keeps its video as a Direct3D 11 texture on the SDK's device. You can get its data in three ways.

Copy the frame to a CPU frame

The simplest way is to clone the frame to system memory with MFClone and one of the ForceCPU clone types (see eMFrameClone). Then work with the copy as with any CPU frame:

MFFrame cpuFrame;
frame.MFClone(out cpuFrame, eMFrameClone.eMFC_Full_ForceCPU, eMFCC.eMFCC_ARGB32);
cpuFrame.MFWaitAsync();

int cbVideo;
long pbVideo;
cpuFrame.MFVideoGetBytes(out cbVideo, out pbVideo);
// ... use the data ...
Marshal.ReleaseComObject(cpuFrame);

• The last parameter sets the pixel format of the copy. RGB32 and ARGB32 have 8 bits per channel. To keep 10-bit color, use eMFCC_r210. With eMFCC_Default, the copy keeps the frame's own format.
• For frames taller than 1080 lines, the copy finishes in the background. Call MFWaitAsync before you read the data, as in the example above.
• Changes to the copy don't affect the original GPU frame.

Map the texture to system memory

MFVideoLock copies the texture to memory you can read or write, in the pixel format you ask for, and MFVideoUnlock releases it. Both methods are on the IMFFrameGPU interface.

IMFFrameGPU gpuFrame = (IMFFrameGPU)frame;

M_VID_PROPS lockProps;
MF_VID_PTR lockPtr;
gpuFrame.MFVideoLock(eMFLockType.eMFLT_Read, eMFCC.eMFCC_ARGB32, out lockProps, out lockPtr, "");
try
{
    IntPtr firstRow = new IntPtr(lockPtr.lpVideoPlanes[0]);
    int stride = lockPtr.cbVideoRowBytes[0];
    // lockProps.nWidth x Math.Abs(lockProps.nHeight) pixels, top row first
}
finally
{
    gpuFrame.MFVideoUnlock();
}

• eLockType (see eMFLockType): eMFLT_Read, eMFLT_Write or eMFLT_Read_Write. With write access, your changes are uploaded to the texture when you call MFVideoUnlock. For frames with two field textures, the _FirstField and _SecondField variants lock one field.
• ePixelFormat: eMFCC_Default (the frame's format), RGB24, RGB32, ARGB32, r210, v210, UYVY, HDYC, YUY2, NV12, YV12, YV16 or RGB8. The conversion runs on the GPU.
• pVidProps, pVideoPtr: the format of the locked data and a pointer and stride for each plane. The data is always top row first. Use the returned stride, because it can be bigger than the width multiplied by the pixel size.
• bsHints: src.x, src.y, src.w, src.h lock only this rectangle of the frame. rgb_to_yuv_matrix sets the matrix for YUV output: RGB_YUV_601, RGB_YUV_709 or RGB_YUV_2020.

Only one lock can be active on a frame at a time, so always call MFVideoUnlock, even if your processing fails. A second lock fails while the first one is active. MFVideoLock works for GPU frames only.

Use the texture directly

For processing that stays on the GPU, get the frame's Direct3D 11 texture (ID3D11Texture2D) with MFVideoGetTexture:

IMFFrameGPU gpuFrame = (IMFFrameGPU)frame;

object texture;
gpuFrame.MFVideoGetTexture(0, out texture, "");
gpuFrame.MFVideoWaitExecution("");

IntPtr pTexture = Marshal.GetIUnknownForObject(texture);
// Wrap pTexture with your Direct3D library, for example SharpDX or Vortice
// ...
Marshal.Release(pTexture);

• nField: 0 for the frame, or 1 for the second field of a frame with two field textures. If the frame has no such texture, no texture is returned.
• The texture belongs to the frame and to the SDK's Direct3D 11 device. Keep a reference to the frame while you use the texture, and don't change the texture: to modify a GPU frame, use MFVideoLock with write access.
• Call MFVideoWaitExecution before you use the texture, so the GPU finishes all pending work on the frame.
• To use the texture on your own Direct3D 11 device, open it on that device through its shared handle (IDXGIResource.GetSharedHandle and ID3D11Device.OpenSharedResource).
• MFVideoGetTexture works for GPU frames only.

Copy a frame to an OpenGL texture

MFFrameGetOpenGLTexture copies a frame's video into an OpenGL texture that you created. The copy stays on the GPU and never goes through system memory. The method is on the IMFFactoryOpenGL interface of the MFFactory object, which is available in both MFormats SDK and MPlatform SDK. It is the reverse of MFFrameCreateFromOpenGLTexture.

This method contains the following parameters:

• _pFrame - the source frame. It can be a CPU frame or a GPU frame: CPU frames and YUV frames are converted to an RGB GPU frame first,
• _nField - 0 for the frame, or 1 for the second field of a frame created with MFFrameCreateFromFields. If the frame has no such field, nothing is copied,
• _gluintTexture - your destination texture (see the requirements below),
• _bsPropsList - additional properties (see below).

Requirements:

• The OpenGL context that owns the texture must be current on the calling thread.
• The OpenGL context and the SDK's Direct3D 11 device must be on the same GPU adapter.
• The driver must support WGL_NV_DX_interop2 and glCopyImageSubData (OpenGL 4.3).
• The texture must be GL_TEXTURE_2D with the internal format GL_RGBA8, GL_RGB10_A2 or GL_RGBA16.
• The texture must have exactly the same width and height as the frame. The SDK doesn't resize the video, and the call fails if the sizes differ.

Supported properties:

• pixel_format - the RGB format used when the frame has to be converted first. By default, it is RGB32 for 8-bit frames and r210 for frames with a higher bit depth,
• wait_execution=false - don't wait for pending GPU work on the frame before copying. By default, the method waits,
• rgb_operation=swap_rb - swap the red and blue channels during the copy,
• force_sync=finish|flush - call glFinish() or glFlush() around the copy,
• close_context=true - release the SDK's interop state for the current OpenGL context (see below).

For example, with an OpenGL library such as OpenTK:

MFFactoryClass factory = new MFFactoryClass();
IMFFactoryOpenGL glFactory = (IMFFactoryOpenGL)factory;

// The OpenGL context that owns the texture must be current on this thread.
// Create the destination texture once, with the size of the frames.
M_AV_PROPS props;
int audioSamples;
frame.MFAVPropsGet(out props, out audioSamples);
int width = props.vidProps.nWidth;
int height = Math.Abs(props.vidProps.nHeight);

int glTexture = GL.GenTexture();
GL.BindTexture(TextureTarget.Texture2D, glTexture);
GL.TexImage2D(TextureTarget.Texture2D, 0, PixelInternalFormat.Rgba8, width, height, 0,
    OpenTK.Graphics.OpenGL.PixelFormat.Rgba, PixelType.UnsignedByte, IntPtr.Zero);

// Copy a frame (repeat for every new frame of the same size)
glFactory.MFFrameGetOpenGLTexture(frame, 0, glTexture, "");

Several threads and releasing an OpenGL context

The SDK keeps separate interop state for each OpenGL context, so several threads can copy frames at the same time, each with its own context. State is kept for up to 12 contexts; beyond that, the least recently used one is dropped and set up again automatically on its next call.

Before you delete an OpenGL context with wglDeleteContext, make it current and release its state. Pass no frame and no texture, only the close_context property:

glFactory.MFFrameGetOpenGLTexture(null, 0, 0, "close_context=true");

This applies to contexts used with MFFrameCreateFromOpenGLTexture too. You can also add close_context=true to your last copy call instead of making a separate call.