Configuring ChromaKey2 plugin from code

Common Information

This article describes in detail how to configure the Chromakey2 plugin parameters through the code and what actions you need to do for this. The detailed description of all the parameters is written in this article → https://support.medialooks.com/hc/en-us/articles/5317394428562-Advanced-Chroma-Key-2-settings

Both MFormats SDK and MPlatform SDK contain in-built samples which implements the Chromakey2 plugin functionality. You can always explore them yourself. The samples are located here:

• MFormats SDK → "C:\Program Files (x86)\Medialooks\MFormats SDK\Samples\C#\Sample Mixer Chroma Key\Sample Mixer Chroma Key.sln"
• MPlatform SDK → "C:\Program Files (x86)\Medialooks\MPlatform SDK\Samples Basic\C#\Mixer Sample\Mixer Sample.sln"

So, to work with the Chromakey2 plugin, the following steps have to be performed. In general, they are the same for both MFormats SDK and MPlatform SDK:

1. Enable "gpu_pipeline".
2. Create 2 objects that are video sources (they can be MFReader / MFLive for MFormats SDK and they can be MFile / MLive / MPlaylist / MMixer for MPlatform SDK). The first object is a video with a background that will be cut by the Chromakey2 plugin, the second source is a video on which the video with a cut background will be overlaid.
3. Create an instance of the Chromakey2 plugin and an instance of its settings.
4. Create a frame instance, then take one frame from the video with the background that is needed to be cut. This frame will be used by the plugin as a reference point of all the settings.
5. Set the reference frame for the ChromaKey2 plugin, define all the necessary parameters of the Chromakey2 plugin and apply them.
6. Process the video with the background to be cut and overlay this processed video on the second video via .ProcessFrame() method for MFormats SDK and via MMixer for MPlatform SDK.

In the next examples, we will consider how to apply the Chromakey2 plugin to a video with a dinosaur and how to overlay it on Big Buck Bunny video:

1. Video with the background to be cut via ChromaKey2 plugin.
2. Background video on which the video without a background will be overlaid.
3. Processed by ChromaKey2 plugin video (the video with the cut background)
4. The result video (Background video + Processed by ChromaKey2 plugin video):

We will apply the following Chromakey2 parameters to the video with the dinosaur:

When it is clear what and in what order you need to do, we will consider the code in detail.

MFormats

An example of the code for this MFormats SDK:

// 1. Enable "gpu_pipeline":
MFFactoryClass m_objFactory = new MFFactoryClass();
m_objFactory.PropsSet("gpu_pipeline", "true");
Marshal.ReleaseComObject(m_objFactory);

// 2. Create 2 objects that are video sources + create a preview for displaying the result:
MFPreviewClass m_objPreview = new MFPreviewClass();
m_objPreview.PreviewEnable("", 0, 1);

MFReaderClass m_objReader_CK = new MFReaderClass();
MFReaderClass m_objReader_BG = new MFReaderClass();
m_objReader_CK.ReaderOpen(@"C:\GreenScreenVideo.mp4", "");
m_objReader_BG.ReaderOpen(@"C:\BackGroundVideo.mp4", "");
m_objReader_CK.PropsSet("loop", "true");
m_objReader_BG.PropsSet("loop", "true");

// 3. Create an instance of the Chromakey2 plugin and an instance of its settings:
MCHROMAKEYLib.MChromaKey2 Plugin_CK = new MCHROMAKEYLib.MChromaKey2();
MCHROMAKEYLib.MKey2 Plugin_CK_Prms = new MCHROMAKEYLib.MKey2();

// 4. Create a frame instance and take one frame from the video with the background that is needed to be cut:
MFFrame KeyFrame = null;
m_objReader_CK.SourceFrameGet(0, out KeyFrame, "");

// 5 - 1. Set the reference frame for the ChromaKey2 plugin:
Plugin_CK_Prms.KeyFrameSet(KeyFrame, null);
Plugin_CK_Prms.KeyAddPoint(0, 1, 1, "type='RGB' max_keys='1' exclude_mode='0'", out _);

// 5 - 2. Define all the necessary parameters of the Chromakey2 plugin:
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftH, 10.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftS, 68.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightH, 72.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftH, 90.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightH, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Cancel_Brightness, 0.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Cancel_Saturation, 0.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Colors_Saturation, 1.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Colors_Alpha, 1.0d, out _);

// 5 - 3.  And apply them:
Plugin_CK.KeySet(Plugin_CK_Prms);

// Now just play both the files:
Boolean ThreadIsAboutToEnd = false;

M_AV_PROPS m_AV_PROPS = new M_AV_PROPS();
m_AV_PROPS.vidProps.eVideoFormat = eMVideoFormat.eMVF_HD1080_25p;
m_AV_PROPS.vidProps.fccType = eMFCC.eMFCC_ARGB32


System.Threading.Tasks.Task FrameGrabbingTask = System.Threading.Tasks.Task.Run(() =>
{
   while (!ThreadIsAboutToEnd)
   {
      m_objReader_BG.SourceFrameConvertedGet(ref m_AV_PROPS, 0, out MFFrame mFrameBG, "");
      m_objReader_CK.SourceFrameConvertedGet(ref m_AV_PROPS, 0, out MFFrame mFrameCK, "");

      // 6 - 1. Process the video with the background to be cut:
      ((MFORMATSLib.IMFProcess)Plugin_CK).ProcessFrame(mFrameCK, out MFFrame mFrameCK_Out, out _, "");
      
      // 6 - 2. And overlay this processed video on the second video via .ProcessFrame() method:
      mFrameBG.MFOverlay(mFrameCK_Out, null, 0, 0, 1.0d, "", "");

      m_objPreview.ReceiverFramePut(mFrameBG, -1, "");

      Marshal.ReleaseComObject(mFrameBG);
      Marshal.ReleaseComObject(mFrameCK);
      Marshal.ReleaseComObject(mFrameCK_Out);
   }
});

MPlatform

An example of the code for this MPlatform SDK:

// 1. Enable "gpu_pipeline":
MFFactoryClass m_objFactory = new MFFactoryClass();
m_objFactory.PropsSet("gpu_pipeline", "true");
Marshal.ReleaseComObject(m_objFactory);

// 2. Create 2 objects that are video sources + create a preview for displaying the result:
MPreviewClass m_objPreview = new MPreviewClass();
m_objPreview.PreviewEnable("", 0, 1);
m_objPreview.ObjectStart(m_objMixer);

MFileClass m_objFile_CK = new MFileClass();
MFileClass m_objFile_BG = new MFileClass();
MMixerClass m_objMixer = new MMixerClass(); // Mixer is needed for overlaying the videos

m_objFile_CK.FileNameSet(@"C:\GreenScreenVideo.mp4", "");
m_objFile_BG.FileNameSet(@"C:\BackGroundVideo.mp4", "");

m_objFile_CK.PropsSet("loop", "true");
m_objFile_BG.PropsSet("loop", "true");

// 3. Create an instance of the Chromakey2 plugin and an instance of its settings:
MCHROMAKEYLib.MChromaKey2 Plugin_CK = new MCHROMAKEYLib.MChromaKey2();
MCHROMAKEYLib.MKey2 Plugin_CK_Prms = new MCHROMAKEYLib.MKey2();

// 4. Create a frame instance and take one frame from the video with the background that is needed to be cut:
MFrame KeyFrame = null;
m_objFile_CK.FilePlayPause(0.0d);
m_objFile_CK.ObjectFrameGet(out KeyFrame, "");


// 5 - 1. Set the reference frame for the ChromaKey2 plugin:
Plugin_CK_Prms.KeyFrameSet(KeyFrame, null);
Plugin_CK_Prms.KeyAddPoint(0, 1, 1, "type='RGB' max_keys='1' exclude_mode='0'", out _);

// 5 - 2. Define all the necessary parameters of the Chromakey2 plugin:
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftH, 10.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftS, 68.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgLeftV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightH, 72.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_BgRightV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftH, 90.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgLeftV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightH, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightS, 50.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Dist_FgRightV, 50.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Cancel_Brightness, 0.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Cancel_Saturation, 0.0d, out _);

Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Colors_Saturation, 1.0d, out _);
Plugin_CK_Prms.KeyAdjustSet(0, eCK2_Adjust.eCK2A_Colors_Alpha, 1.0d, out _);


// 5 - 3. And apply them:
Plugin_CK.KeySet(Plugin_CK_Prms);

// 6 - 1. Process the video with the background to be cut:
m_objFile_CK.PluginsAdd(Plugin_CK, 0);


// Now just play both the files:
M_AV_PROPS m_AV_PROPS = new M_AV_PROPS();
m_AV_PROPS.vidProps.eVideoFormat = eMVideoFormat.eMVF_HD1080_25p;
m_AV_PROPS.vidProps.fccType = eMFCC.eMFCC_ARGB32

m_objMixer.FormatVideoSet(eMFormatType.eMFT_Convert, ref m_VID_PROPS_Mixer);
m_objMixer.ObjectStart(null);


// 6 - 2. And overlay this processed video on the second video via MMixer:
String StreamID_1 = "Stream_1";
String StreamID_2 = "Stream_2";

m_objMixer.StreamsAdd(StreamID_1, m_objFile_BG, "", out _, 0.0d);
m_objMixer.StreamsAdd(StreamID_2, m_objFile_CK, "", "", out _, 0.0d);

m_objMixer.ElementsGetByIndex(0, out MElement Root_1);
m_objMixer.ElementsGetByIndex(0, out MElement Root_2);

(Root_1 as IMElements).ElementsAdd("", "video", "stream_id=" + StreamID_1 + " x=0 y=0 h=1.0 w=1.0 show=1", out _, 0);
(Root_2 as IMElements).ElementsAdd("", "video", "stream_id=" + StreamID_2 + " x=0 y=0 h=1.0 w=1.0 show=1", out _, 0);

m_objFile_CK.FilePlayStart();
m_objFile_BG.FilePlayStart();