Mix audio from one source with video from another

This article is related to MFormats SDK only.

For example, to mix an audio-only source with a video-only source, you should use MFAudioBufferClass object. You also need to create 2 separated threads - one for video source and another for audio.

The general idea of MFAudioBufferClass is the same that System.Collections.Generic.Queue<T> (FIFO collection) in C#. But instead of <T>, MFAudioBuffer works with individual audio samples. So that, IMFAudioBuffers provides you an opportunity to shift audio relative to video frames with the accuracy of one audio sample.

Audio thread

1. Grab a frame from your microphone with SourceFrameGet method.

2. Fill the audio buffer object with this audio frame using BufferFramePut method. Use string parameters as empty strings.

private static void thread_DoWorkA(CancellationToken token)
{
    while (!token.IsCancellationRequested)
    {
        MFFrame pFrame = null;
        M_AUD_PROPS m_PropsAudio = new M_AUD_PROPS();
        try
        {
        // Grab a frame from a source and put it into a buffer
            ((IMFSource)m_objLive).SourceFrameGet(-1, out pFrame, "");
            m_objAudBuff.BufferFramePut("", pFrame, "");                    
            m_objAudBuff.BufferPropsGet("", out m_PropsAudio, out audioSamples);
            Marshal.ReleaseComObject(pFrame);
        }
        catch
        {
            pFrame = null;
        }
    }
}

Video thread

1. Grab a frame from your video source with SourceFrameGet method.

2. (Just once) Check that the buffer object contains enough audio samples with BufferGet method. It returns you an amount of audio samples with a pointer to audio data. If there are not enough samples - just wait for audio thread to fill the buffer with more data.

3. Fill your video frame with a required amount of audio data using BufferFrameFill method of the audio buffer object.

Use your video frame as a parameter of that method. Use 0 as lSamples parameter - in this case, MFormats SDK adds required amount of audio samples according to the frame rate.

On output, you'll have a frame with video and audio data that you can use for further processing or output.

Also, this method returns you a number of audio samples used for the frame filling process.

4. Remove that amount of samples from the buffer using BufferRemove method (use an empty string as a line).

private static int audioSamples = 0;
private static bool flag = false;
private static void thread_DoWorkV(CancellationToken token)
{
    while (!token.IsCancellationRequested)
    {
        MFFrame pFrame = null;
        try
        {
            ((IMFSource)m_objReader).SourceFrameGet(-1, out pFrame, "");
             // check that the buffer contains enough audio samples
            if (audioSamples > 2000)
            {
                flag = true;
            }
            if (flag)
            {
                int samples = 0;
                // fill a frame with audio data from the buffer
                m_objAudBuff.BufferFrameFill("", pFrame, ref samples, "");
                //remove used samples from the buffer
                m_objAudBuff.BufferRemove("", ref samples);
            }
            m_objPreview.ReceiverFramePut(pFrame, -1, "");
            Marshal.ReleaseComObject(pFrame);
        }
        catch
        {
            pFrame = null;
        }
    }
}