Frames in MFormats SDK
MFormats SDK is a frame-based framework which basically means "grab a frame, process the frame, output the frame". So you always work with frame objects (MFFrame).
How to get frames with MPlatform SDK?
Frames in MPlatform SDK are MFrame objects. These objects support the IMFFrame interface of MFormats SDK (because they have the same core components). It means that you can convert an MFrame object to IMFFrame interface to call some specific method, for example, the MFPrint method.
There are several possibilities to get an MFrame object in MPlatform SDK.
ObjectFrameGet method
With this method, you can get the current frame processed by an object. Once this method is included in the IMObject interface, it is possible to get a frame from any object whether it is a source or a receiver.
MFrame myFrame; (myPlaylist as IMObject).ObjectFrameGet(out myFrame);
FileFrameGet and FileFrameGetByTC methods
With these methods, you can get a frame from the desired position. The FileFrameGet returns you a frame by its position in seconds, and the FileFrameGetByTC returns you a frame by its timecode.
You can use these methods for objects that support the IMFile interface, i.e. MFile, MPlaylist, MMixer objects:
MFrame frameByPos; (myPlaylist as IMFile).FileFrameGet(10.0, 0, out frameByPos);
MFrame frameByTC; M_TIMECODE myTC = new M_TIMECODE(); myTC.nHours := 10; myTC.nMinutes := 23; myTC.nSeconds := 45; myTC.nFrames := 15; (myPlaylist as IMFile).FileFrameGetByTC(ref myTC, out frameByTC);
OnFrame and OnFrameSafe events
With the events, you get access to every frame processed by an object. Here is an article about events.
How to modify a frame data?
Once a frame is received, you can get access to its data by addressing the system memory. So you need to get a pointer to this data and the size of the data both for audio and for video.
You can do it using the FrameVideoGetBytes and the FrameAudioGetBytes methods in MPlatform SDK, and the MFVideoGetBytes and the MFAudioGetBytes methods in MFormats SDK.
FrameVideoGetBytes and MFVideoGetBytes methods return you a pointer to a video data and its size, so you can modify it using some external processing, for example, to draw on frames.
With FrameAudioGetBytes and MFAudioGetBytes methods, you get a pointer to an audio data and its size.
Please note that if you replace the audio data in a frame using the MFAudioGetBytes() method, in order to obtain accurate peak volume values (for VU meters), you must additionally call pFrame.MFAudioGain("recalc_vu_meters", 0.0, 0.0). However, if you retrieve audio data using the MFAudioChannelGetBytes(0...N) method, to obtain correct peak volume values (for VU meters), you must additionally call the MFAudioChannelsUpdate() method.
Please, note, that you should modify a frame data before the frame is sent to a receiver object. So with MFormats SDK, you should do all the modifications before you call the ReceiverFramePut method. And for MPlatform SDK, the best approach for this is using synchronous OnFrame (or OnFrameSafe) events. You can enable it with this code:
myFile.PropsSet("object::on_frame.sync", "true");
myFile.PropsSet("object::on_frame.data", "true");
myFile.OnFrameSafe += myFile_OnFrameSafe;
Then the frame, once processed within the OnFrameSafe event, is sent to an output object (for example, a preview or a renderer device).
GPU_Pipeline
When you work when the GPU pipeline is enabled (gpu_pipeline = "true"), the frames are represented as GPU textures in video memory in your GPU, instead of an array of bytes in RAM. So that, when you try to get the pointer and the size of the frame data, MFVideoGetBytes() method can't return a pointer to RAM memory just because the frame isn't in RAM, but in the GPU memory. The method will return zeros. There are two ways how to handle this situation:
The first one - is to clone your GPU frame as CPU frame via MFClone() method, and work with the received data as with usual frames:
pFrame.MFClone(out MFFrame cpuFrame, eMFrameClone.eMFC_Full_ForceCPU, eMFCC.eMFCC_ARGB32);
Please note that CPU frames do not support 10-bit color, and the cloned frame's color space will be compressed to 8 bits per color channel.
The second approach is more advanced and much complicated - you can get the GPU texture which represents the frame. Having this texture, you can work with is as you desire (DevExpress.Data nuget package is required for this particular example):
(pFrame as IMFFrameGPU).MFVideoGetTexture(0, out Object GPU_Texture, "");
ID3D11Texture2D DX_Texture = (ID3D11Texture2D)GPU_Texture;
DevExpress.DirectX.Common.Direct3D.D3D11_TEXTURE2D_DESC Desc = DX_Texture.GetDesc();