Draw on frames

You can get frame video data and modify it according to your requirements. Then you can pack the modified data back to the frame to obtain a new modified frame. You can draw on frames by using this scenario. The following information relates to MFormats SDK. As for MPlatform SDK, related information is in separated paragraph.

Get frame data

You grab a frame from your source and you have the MFFrame object. To get an access to frame data you should clone input frame with the MFClone method:

MFFrame sourceFrame; 
myReader.SourceFrameGetByTime(-1, -1, out sourceFrame, ""); 
MFFrame clonedFrame; 
sourceFrame.MFClone(out clonedFrame, eMFrameClone.eMFC_Reference, eMFCC.eMFCC_ARGB32);

wherein MFClone method you should set a type of what data should be cloned with eMFrameClone enumeration and a result colorspace with eMFCC enumeration for the cloned frame. In this example, clonedFrame contains reference on sourceFrame data and is presented in ARGB32 colorspace.

You can get a pointer in memory to video data and a size of the data with the MFVideoGetBytes method:

int frameDataSize;
long framePointer; 
clonedFrame.MFVideoGetBytes(out frameDataSize, out framePointer);

By this step, you have an access to video data of the frame. You can modify it according to your requirements. If the pointer is constant, all the actions with memory data are reflected on the frame.

Get image from the frame data

You can create a Bitmap object from the pointer and data size. To make this action correct you should get information about frame resolution and how many information is required for the single line of an image.

To get video properties you should call:

int audioSamplesNumber; 
M_AV_PROPS mediaProperties; 
clonedFrame.MFAVPropsGet(out mediaProperties, out audioSamplesNumber);

M_AV_PROPS structure contains a vidProps field that is M_VID_PROPS structure. This field contains all the required information - width (nWidth), height (nHeight) and single row information capacity (nRowBytes):

int frameWidth = avProps.vidProps.nWidth;
int frameHeight = Math.Abs(avProps.vidProps.nHeight);
int frameRowBytes = avProps.vidProps.nRowBytes; 
System.Drawing.Imaging.PixelFormat framePixelFormat = System.Drawing.Imaging.PixelFormat.Format32bppRgb; 
IntPtr framePointer = new IntPtr(pbVideo); 
Bitmap frameBitmap = new Bitmap(frameWidth, frameHeight, frameRowBytes, framePixelFormat, new IntPtr(pbVideo));

Absolute value for height is taken because in some video formats (like ARGB32) the frames are reverted so the height is negative that is not acceptable for Bitmap creating.

Please also note, that the height of the rgb picture shows the line order of the picture: if the height is positive, it means that the picture starts from the bottom if it is negative - it means from the top.

So when you create a bitmap you need to flip it if your frame has a positive height. You can do it this way:

frameBitmap.RotateFlip(RotateFlipType.RotateNoneFlipY);

Modifying the image

To modify the image you should convert it to the Graphic object (in .Net):

Graphics frameGraphicData = Graphics.FromImage(frameBitmap );

And modify it according to your requirements - draw lines, rectangles, ellipses. You can make all the possible actions with the data. For example, in Draw Frame Sample this code is used to draw lines:

frameGraphicData.DrawLines(new Pen(curveLine.PenColor, curveLine.PenSize),  curveLine.PenPath.ToArray());

To avoid memory leakage you should release the objects that were created in this process:

frameBitmap.Dispose(); 
frameGraphicData.Dispose();

And as result of this action, you have modified video data of your input frame. You can use the modified frame like any frame for any receiver - preview, output device or encoding.

What about MPlatform SDK?

To draw on frames with MPlatform SDK, you should use OnFrame (OnFrameSafe) events. To get access to frame data you should specify "object::on_frame.data" property to "true". And to make sure that frames are not skipped while drawing is processed, you should make OnFrame events synchronous:

myFile.PropsSet("object::on_frame.sync", "true");
myFile.PropsSet("object::on_frame.data", "true");
myFile.OnFrameSafe += myFile_OnFrameSafe;

Then all the actions are similar to MFormats approach, but all of them are processed withing the OnFrameSafe event:

private void myFile_OnFrameSafe(string bsChannelID, object pMFrame)
{
    painter.DrawFrame(m_movie, pMFrame as MFrame, panelPreview.Width, panelPreview.Height, m_mousePosX, m_mousePosY);
    Marshal.ReleaseComObject(pMFrame);
    GC.Collect();
}

And the code of DrawFrame method looks like:

public Bitmap MFrame2Bitmap(ref MFrame _mFrame, out M_VID_PROPS _vidProps)
{
    int cbSize;
    long pbVideo;
    _mFrame.FrameVideoGetBytes(out cbSize, out pbVideo);

    M_AV_PROPS avProps;
    _mFrame.FrameAVPropsGet(out avProps);
    Bitmap bmpPicture = null;
    // Create a bitmap from frame
    try
    {
        bmpPicture = new Bitmap(avProps.vidProps.nWidth, Math.Abs(avProps.vidProps.nHeight),
        avProps.vidProps.nRowBytes,
        System.Drawing.Imaging.PixelFormat.Format32bppRgb,
        new IntPtr(pbVideo));
    }
    catch (Exception)
    {}
    _vidProps = avProps.vidProps;
    return bmpPicture;
}

public void DrawFrame(bool _draw, MFrame _mFrame, int _panelWidth, int _panelHeight, int _x, int _y)
{
    if (_mFrame != null)
    {  
        M_VID_PROPS vidProps;
        Bitmap bmpPicture = MFrame2Bitmap(ref _mFrame, out vidProps);

        // Calculate mouse position
        if (_draw)
        {
            int x = Math.Abs(vidProps.nWidth * _x / _panelWidth);
            int y = Math.Abs(vidProps.nHeight * (_panelHeight - _y) / _panelHeight);

            _linesToDraw.Last().PenPath.Add(new Point(x, y));
        }

        // Draw lines
        if (bmpPicture != null)
        {
            var graphic = Graphics.FromImage(bmpPicture);
            graphic.SmoothingMode = SmoothingMode.AntiAlias;
            foreach (CurveLine curveLine in _linesToDraw)
            {
                if (curveLine.PenPath.Count > 1)
                graphic.DrawCurve(new Pen(curveLine.PenColor, curveLine.PenSize), curveLine.PenPath.ToArray());
            }

            bmpPicture.Dispose();
            graphic.Dispose();
        }
    }
}