In MFormats SDK, the frame rate - also known as frame frequency or frames per second (FPS) - is what defines the time that a frame is displayed for on the screen. For example, 25 FPS means that every video frame is displayed for exactly 40 ms along with its synced chunks of audio.
Frame rate control in MFormats SDK is implemented in receiver objects. MFPreview, MFRenderer, and MWriter receive samples and render them outside of the application.
These objects are responsible for the display time of a frame (frame rate control). If a frame is sent to the preview, it will be displayed for a pre-defined time. Even if the next frame is received, the ReceiverFramePut method will not return until the display of the previous frame is finished. This logic is implemented in MFormats SDK internally.
Frame rate control with MFPreview
MFPreview is used to render the video to the preview and to play audio to the sound device. Its behavior is a little different depending on whether the stream contains audio or not.
When the frame contains both video and audio, the clock (timer) of the sound card is used to control the rate (the display of the video frames is adjusted to the audio stream).
For video-only frames the system timer is used to control the rate: when the video frame is displayed, the SDK waits for the system timer to trigger the display of the next frame.
Frame rate control with MFRenderer
MFRenderer is used to send frames to a device (such as Blackmagic or AJA). When rendering to a device, the clock of the device is used to control the frame rate (the device itself controls playback according to the way it had been configured). See also the post about playing out to a device.
Usually, devices have a buffer of at least 1 frame and can tell us when the display of the first frame is finished. MFormats SDK waits for the first frame to complete playback, and then only pushes the buffered frame out for display, at the same time sending the next frame to the buffer. At this moment the ReceiverFramePut method is returned. Because of this, there usually is a delay of at least one frame between playback and output (this may vary due to the specifics of the output drivers and hardware).
Frame rate control with MFWriter
MFWriter encodes (compresses) and writes frames to a file or streams them to the network. The MFWriter object controls the frame rate according to its internal speed of processing the frames (usually defined by the speed of the encoder) - it returns the ReceiverPutFrame() method once the processing of the previous frame is finished and MFWriter is ready to receive the next one.
How to use rate control
The ReceiverFramePut method has a _rtMaxWait parameter. Its value is measured in 100 ns as an integer (1 second is "10000000"):
If rate control is not required, this parameter should be set to "0". In this case, the ReceiverPutFrame method doesn't wait: it returns immediately after the processing of this frame is finished (such as when encoding and writing to a file).
If rate control is required, set this parameter to "-1". In this case, ReceiverPutFrame waits until the sink object complete the display of the frame. For example, it will wait for 40 ms if the MFRenderer is configured to play at 25 FPS via a Blackmagic card. When using several sink objects, this value will make this object the master object.
In other cases, it can be set to a maximum value (such as "400000" for 25 FPS). Note, that waiting time is not yet implemented for some objects - so, it's best to use "0" and "-1" values for now.
When there's only one source object and one sink object, the rate is either controlled by the sink object or not controlled at all (the frames are being processed at maximum speed - such as in the fast transcoding use case).
When using several sink objects at the same time (for example, MFRenderer and MFPreview) it is recommended to use just one object for rate control - this object becomes the master object. It is best to use the last object in the cycle to control the rate. If the master sink works faster than the non-master sink than the non-master object will be dropping frames (for example, MFPreview will be dropping frames if you decide that encoding is more important and define MFWriter as the master object).
Summary
Here is a table with common examples of different receiver objects usage:
| MFPreview | MFRenderer | MFWriter | |
| MFPreview | Decide what preview should be main and enable rate control for it. Use this "master" object (for which rate control is enabled) the last among your receivers:
// preview frame with common object mySecondPreview.ReceiverFramePut(sourceFrame, 0, ""); // preview frame with master object myMainPreview.ReceiverFramePut(sourceFrame, -1, ""); |
In common case quality of output signal is more important. Use MFRenderer as "master" object (for which rate control is enabled) the last among your receivers:
// preview frame myPreview.ReceiverFramePut(sourceFrame, 0, ""); // send frame to output device myRenderer.ReceiverFramePut(sourceFrame, -1, ""); |
In common case quality of the encoded signal is more important. Use MFWriter as "master" object (for which rate control is enabled) the last among your receivers:
// preview frame myPreview.ReceiverFramePut(sourceFrame, 0, ""); // encode the frame myWriter.ReceiverFramePut(sourceFrame, -1, "");By this way, the source is transcoded on maximal speed. If you need to encode your source in real time you can use a rate control to MFPreview object: // encode the frame myWriter.ReceiverFramePut(sourceFrame, 0, ""); // preview frame myPreview.ReceiverFramePut(sourceFrame, -1, ""); |
| MFRenderer | In common case quality of output signal is more important. Use MFRenderer as "master" object (for which rate control is enabled) the last among your receivers:
// preview frame myPreview.ReceiverFramePut(sourceFrame, 0, ""); // send frame to output device myRenderer.ReceiverFramePut(sourceFrame, -1, ""); |
Decide what device should be main and enable rate control for it. Use this "master" object (for which rate control is enabled) the last among your receivers:// output frame with common object mySecondRenderer.ReceiverFramePut(sourceFrame, 0, ""); // output frame with master object myMainRenderer.ReceiverFramePut(sourceFrame, -1, "");Note please that the output delay depends on hardware so the result may be not synchronous for different devices. |
If you use MFWriter and MFRenderer at once as receivers it is better to use MFRenderer as "master" object because the time to display frame on output is longer than time to encode the frame:
// encode frame myWriter.ReceiverFramePut(sourceFrame, 0, ""); // send the frame to output device myRenderer.ReceiverFramePut(sourceFrame, -1, ""); |
| MFWriter | In common case quality of encoded signal is more important. Use MFWriter as "master" object (for which rate control is enabled) the last among your receivers:
// preview frame myPreview.ReceiverFramePut(sourceFrame, 0, ""); // encode the frame myWriter.ReceiverFramePut(sourceFrame, -1, "");By this way, the source is transcoded on maximal speed. If you need to encode your source in real time you can use a rate control to MFPreview object: // encode the frame myWriter.ReceiverFramePut(sourceFrame, 0, ""); // preview frame myPreview.ReceiverFramePut(sourceFrame, -1, ""); |
If you use MFWriter and MFRenderer at once as receivers it is better to use MFRenderer as "master" object, because the time to display frame on output is longer than time to encode the frame:
// encode frame myWriter.ReceiverFramePut(sourceFrame, 0, ""); // send the frame to output device myRenderer.ReceiverFramePut(sourceFrame, -1, ""); |
Decide what encoder should be main and enable rate control for it. Use this "master" object (for which rate control is enabled) the last among your receivers:
// encode frame with common object mySecondWriter.ReceiverFramePut(sourceFrame, 0, ""); // encode frame with master object myMainWriter.ReceiverFramePut(sourceFrame, -1, "");Note please that different formats requires different speed for encoding so you should use fast drivers for encoding in different formats. If result files have drop frames then enable rate control for all receiver objects. |