OpenCV is a very popular image processing library that you can add to your video processing based on MFormats SDK.
For this, you need to get a frame data of the current frame and pass the data to OpenCV engine for further processing. Once the processing is done, you receive a modified frame that you can process further with MFormats SDK.
The following example is based on a .Net-wrapper around OpenCV - Emgu CV.
With Visual Studio, you should install EMGU.CV NuGet package to your solution:
Then, you should load an OpenCV cascade classifier:
// Don't forget to copy a corresponding XML file for a cascade classifier
CascadeClassifier cascadeClassifier = new CascadeClassifier(@"haarcascade_frontalface_alt2.xml");
After that, you should get a clone of a source frame before create an OpenCV object:
m_objLive.SourceFrameGet(-1, out pFrame, "");
MFFrame pFrameDraw = null;
pFrame.MFClone(out pFrameDraw, eMFrameClone.eMFC_Video_ForceCPU, eMFCC.eMFCC_ARGB32);
You should use ForceCPU to make a frame in a system (not GPU) memory so you can get access to the frame data.
You should create an Image object of OpenCV from the cloned MFFrame object:
Image<Rgba, Byte> cvFrame = GetCVImageFromMFFrame(pFrameDraw);
Where GetCVImageFromMFFrame method is:
private Image<Rgba, Byte> GetCVImageFromMFFrame( MFFrame _pFrame )
{
M_AV_PROPS avProps;
int lAudioSamples;
_pFrame.MFAVPropsGet(out avProps, out lAudioSamples);
int nFrameWidth = avProps.vidProps.nWidth;
int nFrameHeight = Math.Abs(avProps.vidProps.nHeight);
int nFrameRowBytes = avProps.vidProps.nRowBytes;
int pcbSize;
long pbVideo;
_pFrame.MFVideoGetBytes(out pcbSize, out pbVideo);
IntPtr pnVideoData = new IntPtr(pbVideo);
// Pixel format for OpenCV image and MFormats frame must be the same
Image<Rgba, Byte> cvFrame = new Image<Rgba, Byte>(nFrameWidth, nFrameHeight, nFrameRowBytes, pnVideoData);
return cvFrame;
}
Once the Image object has been created, you can process it with OpenCV methods.
For instance, let's cover face detection. For this, it is necessary to specify minimal and maximal sizes of rectangles for detection and detect faces on the Image using the cascade classifier you've loaded:
// Recognize faces and detect their bounding boxes
int nSizeFactor = Math.Min(cvFrame.Height, cvFrame.Width);
Size szMinSize = new Size((int)(nSizeFactor * 0.2), (int)(nSizeFactor * 0.2));
Size szMaxSize = new Size((int)(nSizeFactor * 0.35), (int)(nSizeFactor * 0.35));
Rectangle[] rcFacesDetected = cascadeClassifier.DetectMultiScale(cvFrame, 1.1, 5, szMinSize, szMaxSize);
If any face is detected, the rcFacesDetected is not empty, and you can draw a rectangle around a face with a code like this:
// Draw corresponding bounding boxes
foreach( Rectangle rcFace in rcFacesDetected)
cvFrame.Draw(rcFace, new Rgba(255, 0, 0, 255), 3);
Because all the functionality of OpenCV used a data of the cloned MFFrame object, you can further use the pFrameDraw object with MFormats methods. For example, to send it to a preview:
m_objPreview.ReceiverFramePut(pFrameDraw, -1, "");
Note that you have 2 MFFrame objects in this case (an original one and a clone), so you should release both objects to avoid memory issues:
Marshal.ReleaseComObject(pFrame);
Marshal.ReleaseComObject(pFrameDraw);
The result of face detection depends on the settings of OpenCV objects. Here is an example of a result using the above code:
In the attachment to this article, there is a sample where the logic described above is implemented. Extract an archive, open Visual Studio and load a solution of the sample. EMGU.CV is quite heavy, so you need to restore NuGet packages for the project when you open the sample the 1st time.