How create a new frames

You can get frames from source objects (for example, MFReader or MFLive in MFormats SDK, and MFile or MLive in MPlatform SDK), but you can also create a new frame independently from any source. To do this, use the MFFactory object, which supports the IMFFactory interface and, for OpenGL interop, the IMFFactoryOpenGL interface.

MFFactory in MFormats SDK and MPlatform SDK

MFFactory is the same object in both SDKs: MFormats SDK and MPlatform SDK register it under the same class ID, so every method in this article is available in both. The only difference is the namespace and the frame type you work with:

The examples in this article use MFormats SDK types. In MPlatform SDK, declare the frame variables as IMFFrame instead of MFFrame:

using MPLATFORMLib;

MFFactoryClass factory = new MFFactoryClass();
IMFFrame logoFrame;
factory.MFFrameCreateFromFile(@"c:\myLogo.png", out logoFrame, "");

MFrame logoMFrame = (MFrame)logoFrame; // for MPlatform methods that take MFrame

MPlatform SDK also has the older MFramesClass object (IMFrames interface) with the FramesCreateFromFile, FramesCreateFromMem, FramesCreateFromHBITMAP and FramesCreate methods. They return MFrame directly and use the same code as the MFFrameCreateFromFile, MFFrameCreateFromMem, MFFrameCreateFromHBITMAP and MFFrameCreatePlain methods described below, so the same properties apply:

MFramesClass frames = new MFramesClass();
MFrame frameFromFile;
frames.FramesCreateFromFile(@"c:\myLogo.png", out frameFromFile, "");

Keep in mind the following for both SDKs:

• MFFactory is one object shared by the whole process: every new MFFactoryClass() returns the same instance. A property you set on it applies to the whole application, so set it once at startup, before you create other SDK objects. For example, MPlatform SDK's "Color Convert With Matrix Transform" sample turns on the GPU pipeline with PropsSet:

MFFactoryClass m_objMFFactory = new MFFactoryClass();
m_objMFFactory.PropsSet("gpu_pipeline", "true");

• Release every frame you create with Marshal.ReleaseComObject() when you no longer need it. Otherwise, the frame's memory stays allocated until the garbage collector runs.

Create a frame from file

Use the MFFrameCreateFromFile method to create a frame from a local image file. A typical use is a channel logo that you overlay on video.

This method contains the following parameters:

• _bsFileName - the path to the image file (BMP, JPEG, PNG, TIFF or GIF),
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties (see below).

PNG and TIFF images keep their transparency and produce ARGB32 frames. JPEG images produce RGB32 frames. For a GIF, only the first frame is loaded.

Supported properties:

• pixel_format - convert the image to this pixel format, for example pixel_format='UYVY'. Use pixel_format='r210' to load 10-bit images, including DPX files,
• bottom-top=true - store an RGB image bottom-up. By default, it is stored top-down,
• audio_samples - attach this number of samples of silent audio (2 channels, 48 kHz, 16-bit).

For example:

MFFactoryClass frameFactory = new MFFactoryClass();
MFFrame frameFromFile;
string pathToFile = @"c:\myLogo.png";
frameFactory.MFFrameCreateFromFile(pathToFile, out frameFromFile, "");

Create a frame from memory

If you have video data in memory and know its properties, create a frame from it with MFFrameCreateFromMem. The data is copied into the new frame.

This method contains the following parameters:

• _pAVProps - the media properties of the frame. If the video properties aren't set, the frame has no video and can be used only for audio,
• _lpVideo - a pointer to the video data. The data layout has to match _pAVProps. If you pass 0, the frame is filled with a solid color (see below). If you pass -1, the memory is allocated but not filled,
• _lAudioSamples - the number of audio samples per channel,
• _lpAudio - a pointer to the audio data. If you pass 0 with a non-zero sample count, silence is added. Audio is added only if the audio properties are set in _pAVProps,
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties (see below).

If fccType isn't set, the frame uses UYVY for SD formats and HDYC for HD formats (RGB32 when gpu_pipeline is on). For RGB formats, a negative nHeight means a top-down buffer and a positive nHeight means a bottom-up buffer.

For example, to copy a top-down BGRA image from a byte array:

M_AV_PROPS frameProps = new M_AV_PROPS();
frameProps.vidProps.fccType = eMFCC.eMFCC_ARGB32;
frameProps.vidProps.nWidth = 1920;
frameProps.vidProps.nHeight = -1080;        // negative = top-down
frameProps.vidProps.nRowBytes = 1920 * 4;
frameProps.vidProps.dblRate = 25.0;

MFFactoryClass factory = new MFFactoryClass();
MFFrame frameFromMem;
GCHandle pin = GCHandle.Alloc(bgraBytes, GCHandleType.Pinned);
try
{
    factory.MFFrameCreateFromMem(ref frameProps, pin.AddrOfPinnedObject().ToInt64(), 0, 0, out frameFromMem, "");
}
finally
{
    pin.Free(); // the frame has its own copy of the data
}

This method is also the fastest way to create a simple frame, such as a transparent frame or a solid-color frame:

M_AV_PROPS frameProps = new M_AV_PROPS();
frameProps.vidProps.eVideoFormat = eMVideoFormat.eMVF_HD1080_5994i;
frameProps.vidProps.fccType = eMFCC.eMFCC_ARGB32; // ARGB32 is required for transparency
MFFactoryClass factory = new MFFactoryClass();
MFFrame createdFrame;
string colorParameters = "color='Red(128)'";
factory.MFFrameCreateFromMem(ref frameProps, 0, 0, 0, out createdFrame, colorParameters);

Supported properties:

• color - the fill color used when _lpVideo is 0. The older name solid_color also works. If you don't set a color, the frame is black (transparent black for ARGB32),
• no_fill=true - don't fill the frame when _lpVideo is 0,
• bt709=true|false - the matrix used for YUV/RGB conversion. By default, BT.709 is used for frames taller than 625 lines and BT.601 for the rest,
• gpu=true - create the frame on the GPU (the default when gpu_pipeline is on),
• convert_alpha_premul=true - premultiply ARGB32 data by its alpha channel while copying.

The color value has this structure:
color='[color]([opacity])'

where:

• color is a name (Red, Navy, Transparent, etc.), an RGB code (#FFAACC), or an ARGB code (FFAACCEE, as returned by Color.ToArgb().ToString("X8")),
• opacity is optional: a value from 0 (fully transparent) to 255 (solid), or a decimal from 0.0 to 1.0. Red(1) means an opacity of 1 out of 255, so for a solid color use Red(255) or Red(1.0), or leave out the parentheses.

If the planes of a planar image are stored in separate buffers, use the extended variant, MFFrameCreateFromMemEx. See A note on the "Ex" variants.

Create a frame from bitmap

Bitmap objects used in application controls have an HBitmap property. You can create a new frame from this handle with the MFFrameCreateFromHBITMAP method. The bitmap data is copied into the frame.

This method contains the following parameters:

• _hBitmap - the handle of the source bitmap,
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties. The same properties as for MFFrameCreateFromFile are supported: pixel_format, bottom-top and audio_samples.

For example, you can draw on a Bitmap, create a frame from it, and then overlay that frame on a source frame:

[DllImport("gdi32.dll")]
static extern bool DeleteObject(IntPtr hObject);

using (Bitmap bitmap = new Bitmap(1920, 1080))
using (Graphics g = Graphics.FromImage(bitmap))
{
    g.Clear(Color.Navy);
    g.DrawString("LIVE", new Font("Arial", 72), Brushes.White, 40, 40);

    IntPtr hBitmap = bitmap.GetHbitmap();
    try
    {
        MFFrame bitmapFrame;
        factory.MFFrameCreateFromHBITMAP(hBitmap.ToInt64(), out bitmapFrame, "");
        // ... use bitmapFrame, then Marshal.ReleaseComObject(bitmapFrame)
    }
    finally
    {
        DeleteObject(hBitmap); // the frame has its own copy of the pixels
    }
}

Create a frame wrapping an external object

All the methods above copy the source data into the frame's own memory. If you want the frame to use memory that you already own without copying it, use MFFrameCreateWrapper. That memory can belong to a WIC bitmap lock, a locked D3D9 surface, a media sample, an NSSBuffer, or your own object. The frame holds a reference to that object and reads its memory directly, so the object stays alive while the frame exists. The memory itself must stay valid and unchanged for that whole time.

This method contains the following parameters:

• _pVidProps - the video properties of the wrapped memory. Set fccType, nWidth, nHeight and nRowBytes (the stride),
• _cbVideo - the size of the video data, in bytes,
• _lpVideo - a pointer to the video data,
• _pObject - the object that owns the memory. The frame releases it when the frame is destroyed,
• _ppFrame - the resulting frame.

For example, to wrap unmanaged memory owned by a .NET object, pass that object as _pObject. The frame's reference keeps the object from being collected, so the memory is freed only after the frame is released:

[ComVisible(true)]
public class VideoBuffer
{
    public readonly IntPtr Ptr;
    public readonly int Size;

    public VideoBuffer(int size) { Size = size; Ptr = Marshal.AllocHGlobal(size); }
    ~VideoBuffer() { Marshal.FreeHGlobal(Ptr); }
}

VideoBuffer buffer = new VideoBuffer(1920 * 1080 * 4);
// ... fill buffer.Ptr with BGRA pixels ...

M_VID_PROPS vidProps = new M_VID_PROPS();
vidProps.fccType = eMFCC.eMFCC_ARGB32;
vidProps.nWidth = 1920;
vidProps.nHeight = -1080;          // top-down
vidProps.nRowBytes = 1920 * 4;
vidProps.dblRate = 25.0;

MFFrame wrappedFrame;
factory.MFFrameCreateWrapper(ref vidProps, (uint)buffer.Size, buffer.Ptr.ToInt64(), buffer, out wrappedFrame);

The extended variant, MFFrameCreateWrapperEx, can wrap video whose planes are stored separately. See A note on the "Ex" variants.

Create an empty frame

MFFrameCreatePlain allocates a frame with a video buffer of the given size and doesn't set any other properties. It's a low-level method: you have to set the frame properties and fill the buffer yourself.

MFFactoryClass factory = new MFFactoryClass();
MFFrame emptyFrame;
factory.MFFrameCreatePlain(cbVideoSize, out emptyFrame);

In most cases, MFFrameCreateFromMem with _lpVideo = -1 is more convenient: it allocates an unfilled frame that already has the video properties you specify.

Create a frame from a Direct3D texture

If your content already lives in a Direct3D texture or surface, use MFFrameCreateFromTexture to create a GPU frame from it. Direct3D 11 textures (ID3D11Texture2D) and Direct3D 9 surfaces or textures (IDirect3DSurface9, IDirect3DTexture9) are supported.

This method contains the following parameters:

• _pAVProps - the media properties of the frame. Leave the video properties empty to take the size and format from the texture. If you set a different size or pixel format, the texture is scaled or converted on the GPU,
• _pD3DTexture2D - the source texture or surface,
• _lAudioSamples, _lpAudio - optional audio data,
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties (see below).

Whether the content is copied depends on the source:

• A Direct3D 11 texture created on the SDK's own device that needs no conversion is used directly, without a copy. Don't modify it while the frame is in use.
• A Direct3D 11 texture from another device is copied to the SDK's device. This is fastest if the texture is shared (D3D11_RESOURCE_MISC_SHARED) and on the same GPU adapter.
• A Direct3D 9 surface has to be 32-bit ARGB. Its content is copied to a new Direct3D 11 texture.

Supported properties:

• flip (or mirror) - mirror the image: horz, vert or both,
• src.x, src.y, src.w, src.h - use only this rectangle of the texture. If _pAVProps doesn't set a size, the frame gets the size of the rectangle,
• wrap=true - use the texture as is, without scaling, cropping, flipping or conversion,
• gpu=false - return a CPU frame instead of a GPU frame.

flip, src.* and wrap apply to Direct3D 11 textures only.

In .NET, pass the texture as a COM object. For example, with a Direct3D wrapper library such as SharpDX or Vortice:

object texture = Marshal.GetObjectForIUnknown(d3d11Texture.NativePointer);

M_AV_PROPS frameProps = new M_AV_PROPS(); // empty: size and format come from the texture
MFFrame frameFromTexture;
factory.MFFrameCreateFromTexture(ref frameProps, texture, 0, 0, out frameFromTexture,
    "src.x=320 src.y=180 src.w=1280 src.h=720");

This is the Direct3D counterpart of MFFrameCreateFromOpenGLTexture, described below.

Create a frame from two fields

MFFrameCreateFromFields combines two textures, one per field, into a single interlaced GPU frame. It accepts the same texture types and properties as MFFrameCreateFromTexture. Calling MFFrameCreateFromTexture is the same as calling this method with only one field.

This method contains the following parameters:

• _pAVProps - the media properties of the resulting frame,
• _pFirstField, _pSecondField - the textures of the first and second fields. They must have the same format,
• _lAudioSamples, _lpAudio - optional audio data,
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties.

The field order comes from the interlaced video format you set in eVideoFormat, or from eInterlace for a custom format. If neither sets it, frames taller than 486 lines are treated as upper field first (eMI_Field1First), and shorter frames as lower field first (eMI_Field2First).

M_AV_PROPS frameProps = new M_AV_PROPS();
frameProps.vidProps.eVideoFormat = eMVideoFormat.eMVF_HD1080_5994i;

MFFrame interlacedFrame;
factory.MFFrameCreateFromFields(ref frameProps, firstFieldTexture, secondFieldTexture, 0, 0,
    out interlacedFrame, "");

Create a frame from OpenGL texture

Starting with version 2.11.0, you can create a frame directly from an OpenGL texture. The method, MFFrameCreateFromOpenGLTexture, is on the IMFFactoryOpenGL interface of the same factory object in both SDKs. The texture is copied to Direct3D 11 on the GPU through the WGL_NV_DX_interop2 extension, so the pixels never go through system memory.

The workflow is the following:

1. Create or load an OpenGL texture (for example, from an image file).
2. Pass the texture handle to MFFrameCreateFromOpenGLTexture.
3. Receive a ready-to-use frame that can be previewed, processed, or streamed with any MFormats or MPlatform component.

This method contains the following parameters:

• _pAVProps - the media properties of the frame. If you don't set the size, it is taken from the texture,
• _gluintTexture - the OpenGL texture name,
• _lAudioSamples, _lpAudio - optional audio data,
• _ppFrame - the resulting frame,
• _bsPropsList - additional properties: flip, src.* and gpu work as in MFFrameCreateFromTexture, and force_sync=finish|flush calls glFinish() or glFlush() around the copy. Use force_sync if the frame sometimes shows content that isn't fully rendered yet.

Requirements:

• The OpenGL context that owns the texture must be current on the calling thread.
• The OpenGL context and the SDK's Direct3D 11 device must be on the same GPU adapter.
• The driver must support WGL_NV_DX_interop2 and glCopyImageSubData (OpenGL 4.3).
• The texture must be GL_TEXTURE_2D with the internal format GL_RGBA8, GL_RGB10_A2 or GL_RGBA16, and no larger than 16384 pixels on either side.

For example, with an OpenGL library such as OpenTK:

IMFFactoryOpenGL glFactory = (IMFFactoryOpenGL)factory;

// The OpenGL context that owns glTexture must be current on this thread
M_AV_PROPS frameProps = new M_AV_PROPS();
frameProps.vidProps.fccType = eMFCC.eMFCC_ARGB32;
frameProps.vidProps.dblRate = 25.0;

MFFrame frameFromGL;
glFactory.MFFrameCreateFromOpenGLTexture(ref frameProps, glTexture, 0, 0, out frameFromGL, "");

The SDK keeps interop state for each OpenGL context that calls the OpenGL methods. Before you delete an OpenGL context, release that state as described in Get a frame and modify its data. The same article describes the reverse direction: copying a frame into an OpenGL texture.

A note on the "Ex" variants

MFFrameCreateFromMemEx and MFFrameCreateWrapperEx are extended versions of MFFrameCreateFromMem and MFFrameCreateWrapper. Instead of a single video pointer, they take an MF_VID_PTR structure with a pointer (lpVideoPlanes), a size in pixels (szVideoPlanes) and a stride (cbVideoRowBytes) for each of up to four planes. Use them instead of the base methods when your planar data (for example, YUV with separate Y, U and V buffers) isn't in one contiguous block.

• MFFrameCreateFromMemEx copies the planes. Pass the planes in the order of the format: Y, U, V for I420 and I422, and Y, V, U for YV12 and YV16. By default, these formats are converted to packed 4:2:2 (UYVY for BT.601, HDYC for BT.709). Add convert_planar=false to keep the planar layout for a CPU frame. Add full_scale=true if the luma is full range (0-255).
• MFFrameCreateWrapperEx wraps the planes without copying them. It also takes full M_AV_PROPS, optional audio, the object to keep alive, and a properties string whose values become the frame's string properties (read them with MFStrGet). Only the video is wrapped: the audio is always copied. The stride comes from cbVideoRowBytes[0], not from vidProps.nRowBytes.

M_AV_PROPS frameProps = new M_AV_PROPS();
frameProps.vidProps.fccType = eMFCC.eMFCC_I420;
frameProps.vidProps.nWidth = 1920;
frameProps.vidProps.nHeight = 1080;
frameProps.vidProps.dblRate = 25.0;

MF_VID_PTR planes = new MF_VID_PTR();
planes.lpVideoPlanes = new long[] { pY.ToInt64(), pU.ToInt64(), pV.ToInt64(), 0 };
planes.cbVideoRowBytes = new int[] { strideY, strideU, strideV, 0 };
planes.szVideoPlanes = new tagSIZE[4];
planes.szVideoPlanes[0].cx = 1920; planes.szVideoPlanes[0].cy = 1080;
planes.szVideoPlanes[1].cx = 960;  planes.szVideoPlanes[1].cy = 540;
planes.szVideoPlanes[2].cx = 960;  planes.szVideoPlanes[2].cy = 540;

MFFrame planarFrame;
factory.MFFrameCreateFromMemEx(ref frameProps, ref planes, 0, 0, out planarFrame, "");

If a frame's planes are stored separately (for example, a frame created with MFFrameCreateWrapperEx), MFVideoGetBytes returns only the first plane's size. To access all planes, use MFVideoGetBytesEx.