Frame rate control in MFormats SDK

In MFormats SDK, the frame rate - also known as frame frequency or frames per second (FPS) - is what defines the time that a frame is displayed for on the screen. For example, 25 FPS means that every video frame is displayed for exactly 40 ms along with its synced chunks of audio.

Frame rate control in MFormats SDK is implemented in receiver objects. MFPreview, MFRenderer, and MWriter receive samples and render them outside of the application.

These objects are responsible for the display time of a frame (frame rate control). If a frame is sent to the preview, it will be displayed for a pre-defined time. Even if the next frame is received, the ReceiverFramePut method will not return until the display of the previous frame is finished. This logic is implemented in MFormats SDK internally.

Frame rate control with MFPreview

MFPreview is used to render the video to the preview and to play audio to the sound device. Its behavior is a little different depending on whether the stream contains audio or not.

When the frame contains both video and audio, the clock (timer) of the sound card is used to control the rate (the display of the video frames is adjusted to the audio stream).
For video-only frames the system timer is used to control the rate: when the video frame is displayed, the SDK waits for the system timer to trigger the display of the next frame.

Frame rate control with MFRenderer

MFRenderer is used to send frames to a device (such as Blackmagic or AJA). When rendering to a device, the clock of the device is used to control the frame rate (the device itself controls playback according to the way it had been configured). See also the post about playing out to a device.

Usually, devices have a buffer of at least 1 frame and can tell us when the display of the first frame is finished. MFormats SDK waits for the first frame to complete playback, and then only pushes the buffered frame out for display, at the same time sending the next frame to the buffer. At this moment the ReceiverFramePut method is returned. Because of this, there usually is a delay of at least one frame between playback and output (this may vary due to the specifics of the output drivers and hardware).

Frame rate control with MFWriter

MFWriter encodes (compresses) and writes frames to a file or streams them to the network. The MFWriter object controls the frame rate according to its internal speed of processing the frames (usually defined by the speed of the encoder) - it returns the ReceiverPutFrame() method once the processing of the previous frame is finished and MFWriter is ready to receive the next one.

How to use rate control

The ReceiverFramePut method has a _rtMaxWait parameter. Its value is measured in 100 ns as an integer (1 second is "10000000"):

If rate control is not required, this parameter should be set to "0". In this case, the ReceiverPutFrame method doesn't wait: it returns immediately after the processing of this frame is finished (such as when encoding and writing to a file).

If rate control is required, set this parameter to "-1". In this case, ReceiverPutFrame waits until the sink object complete the display of the frame. For example, it will wait for 40 ms if the MFRenderer is configured to play at 25 FPS via a Blackmagic card. When using several sink objects, this value will make this object the master object.

In other cases, it can be set to a maximum value (such as "400000" for 25 FPS). Note, that waiting time is not yet implemented for some objects - so, it's best to use "0" and "-1" values for now.

When there's only one source object and one sink object, the rate is either controlled by the sink object or not controlled at all (the frames are being processed at maximum speed - such as in the fast transcoding use case).

When using several sink objects at the same time (for example, MFRenderer and MFPreview) it is recommended to use just one object for rate control - this object becomes the master object. It is best to use the last object in the cycle to control the rate. If the master sink works faster than the non-master sink than the non-master object will be dropping frames (for example, MFPreview will be dropping frames if you decide that encoding is more important and define MFWriter as the master object).

Summary

Here is a table with common examples of different receiver objects usage:

// preview frame with common object
mySecondPreview.ReceiverFramePut(sourceFrame, 0, "");
// preview frame with master object
myMainPreview.ReceiverFramePut(sourceFrame, -1, "");

// preview frame
myPreview.ReceiverFramePut(sourceFrame, 0, "");
// send frame to output device
myRenderer.ReceiverFramePut(sourceFrame, -1, "");

// preview frame
myPreview.ReceiverFramePut(sourceFrame, 0, "");
// encode the frame
myWriter.ReceiverFramePut(sourceFrame, -1, "");

// encode the frame
myWriter.ReceiverFramePut(sourceFrame, 0, "");
// preview frame
myPreview.ReceiverFramePut(sourceFrame, -1, "");

// preview frame
myPreview.ReceiverFramePut(sourceFrame, 0, "");
// send frame to output device
myRenderer.ReceiverFramePut(sourceFrame, -1, "");

// output frame with common object
mySecondRenderer.ReceiverFramePut(sourceFrame, 0, "");
// output frame with master object
myMainRenderer.ReceiverFramePut(sourceFrame, -1, "");

// encode frame
myWriter.ReceiverFramePut(sourceFrame, 0, "");
// send the frame to output device
myRenderer.ReceiverFramePut(sourceFrame, -1, "");

// preview frame
myPreview.ReceiverFramePut(sourceFrame, 0, "");
// encode the frame
myWriter.ReceiverFramePut(sourceFrame, -1, "");

// encode the frame
myWriter.ReceiverFramePut(sourceFrame, 0, "");
// preview frame
myPreview.ReceiverFramePut(sourceFrame, -1, "");

// encode frame
myWriter.ReceiverFramePut(sourceFrame, 0, "");
// send the frame to output device
myRenderer.ReceiverFramePut(sourceFrame, -1, "");

// encode frame with common object
mySecondWriter.ReceiverFramePut(sourceFrame, 0, "");
// encode frame with master object
myMainWriter.ReceiverFramePut(sourceFrame, -1, "");