Recognizing DVB bitmap subtitles as text

Recognize DVB bitmap subtitles

MFormats' MFReader and the MPlatform's MFile can recognize text from decoded DVB bitmap subtitle compositions using the local FFmpeg ocr filter and Tesseract language models. Recognition runs in the background once per composition. Bitmap subtitle display remains available, and OCR accuracy depends on the images and language models.

MFormats applications receive results on frames in their application-owned frame loop, including when MFReader runs through MServer. MPlatform applications keep their connected MFile playback graph and obtain results from data-bearing frame events. MFile does not emit separate DVB-OCR events. For bitmap display with either SDK, see Displaying DVB subtitles/teletext from transport stream.

Configure models and the source

1. Use the SDK's FFmpeg components with the ocr filter available. The MFile workflow requires its FFmpeg decoding backend.
2. Download the required language files from Tesseract tessdata. For example, English requires eng.traineddata. Selecting a language does not download its model.
3. Put the files in tessdata beside the FFmpeg avfilter DLL loaded by the decoding process, or configure the model folder explicitly. Prefer an absolute ASCII path. A non-ASCII path requires an ASCII Windows short name.
4. Select the required subtitle track and enable OCR before initializing the source. Track indexes are zero-based; choose a track containing DVB bitmap subtitles.

MFormats configuration

For an existing MFReaderClass instance named reader, configure it before ReaderOpen. Here, modelDirectory is the folder containing the language files:

reader.PropsSet("subtitle.ocr.datapath", modelDirectory);
reader.ReaderOpen(sourcePath,
    "subtitle_track=0 subtitle.ocr=true subtitle.ocr.language=eng");

Continue using the normal MFReader frame loop. 

MPlatform configuration

For an existing MFileClass instance named file, use the file:: prefix when setting decoding properties through PropsSet. The equivalent keys in the FileNameSet parameter string are unprefixed. Configure them before FileNameSet so they reach the FFmpeg backend before its reader initializes.

Enable data-bearing asynchronous frame events and attach your OnFrame handler before starting playback. The handler has the signature void OnFrame(string channelId, object pMFrame).

file.PropsSet("object::on_frame.data", "true");
file.PropsSet("object::on_frame.sync", "false");
file.PropsSet("file::subtitle.ocr.datapath", modelDirectory);
file.OnFrameSafe += OnFrame;
file.FileNameSet(sourcePath,
    "decode=ffmpeg subtitle_track=0 subtitle.ocr=true subtitle.ocr.language=eng");
file.FilePlayStart();

Keep the application's normal MFile connections to preview, writer or renderer objects. Playback supplies the frames to the event handler. Backend initialization may occur when playback starts, so a warm-up setting is not an overall FileNameSet deadline.

In both SDKs, use '+' (for example, eng+rus) to select two or more models. Without an override, the decoder uses the subtitle stream language, falling back to eng when the stream language is empty.

Wait explicitly for frame side data

Receiving a frame does not mean its OCR result is ready. MFWaitAsync, pixel readiness and cloning do not wait for OCR. Query IMFFrameGPU on the returned frame and call MFVideoWaitExecution explicitly; this also works on CPU frames.

max_wait does not bound GPU execution or COM transport time. If the side-data wait expires, the call returns 0x800705B4 (ERROR_TIMEOUT). The timeout does not cancel OCR: retain the frame and retry on that same frame. A later frame is not a substitute for the retained result.

MFormats frame processing

The following helper borrows a frame from your existing frame loop. A false return means that the caller should retain it and retry later, subject to the application's own deadline. After a true return, the payloads have been read and the caller can release the frame.

using System;
using System.Runtime.InteropServices;
using MFORMATSLib;

public static class DvbOcrResults
{
    public static bool TryRead(MFFrame frame)
    {
        try
        {
            ((IMFFrameGPU)frame).MFVideoWaitExecution(
                "side_data=true gpu=false max_wait=250");
        }
        catch (COMException error) when
            (error.HResult == unchecked((int)0x800705B4))
        {
            return false;
        }

        for (int index = 0; ; ++index)
        {
            string name = index == 0 ? "DVB-OCR" : "DVB-OCR." + index;
            frame.MFStrGet(name, out string attributes);
            if (string.IsNullOrEmpty(attributes))
                break;

            Console.WriteLine(attributes);
        }
        return true;
    }
}

Keep normal frame ownership and cleanup in the surrounding MFormats application: release each acquired frame when finished and call ReaderClose before releasing the reader. Process retained results, including results attached to the last frame, before closing. If you deliberately abandon a pending frame, its recognized text may be lost to your application.

MPlatform frame processing

With object::on_frame.data=true, asynchronous OnFrameSafe delivery preserves frame data and access to a pending OCR result. A properties-only event copy does not preserve that access. In the callback, make an owned reference clone and transfer it to a COM-initialized worker:

((MPLATFORMLib.IMFrame)pMFrame).FrameClone(
    out MPLATFORMLib.MFrame retainedFrame,
    MPLATFORMLib.eMFrameClone.eMFC_Reference,
    MPLATFORMLib.eMFCC.eMFCC_Default);

Return promptly from the callback. Release its pMFrame parameter in a finally block after transferring the clone. Perform the OCR wait on the worker, using the retained clone:

((MPLATFORMLib.IMFFrameGPU)retainedFrame).MFVideoWaitExecution(
    "side_data=true gpu=false max_wait=250");
((MPLATFORMLib.IMFFrame)retainedFrame).MFStrGet(
    "DVB-OCR", out string attributes);

Handle 0x800705B4 by retaining and retrying that clone, as described above. After a successful wait, enumerate all result names using the same naming rules below. Release the owned clone in the worker's finally block when finished or deliberately discarded. The interface casts refer to the same owned frame. Keep the event delegate for unsubscription, stop production and drain retained work before closing and releasing MFile. Do not wait for OCR or stop SDK objects inside the frame callback.

Read the result attributes

Results use DVB-OCR, followed by DVB-OCR.1, DVB-OCR.2 and so on. Read until the next string is empty. Each payload is a serialized attribute string, rather than plain subtitle text. Parse its named attributes and decode escaped values.

Divide PTS values by 10000000 to convert them to seconds. A successful wait means pending work has finished; it does not guarantee a nonempty result or successful recognition.

Clones that include frame data can retrieve late OCR results by explicitly waiting on the clone before reading its strings. Cloning itself does not wait. Video-only clones do not carry this result dependency.

Warm-up and diagnostics

To warm up OCR before processing subtitles, configure subtitle.ocr.preinit_frame, for example 1920x256, before ReaderOpen in MFormats or FileNameSet in MPlatform. Use file::subtitle.ocr.preinit_frame with MFile's PropsSet. The optional subtitle.ocr.preinit_timeout controls the synchronous backend warm-up wait in milliseconds and defaults to 3000. Use 0 to enqueue warm-up without waiting. This setting does not replace the per-frame side-data wait.

For MFReader, read track-specific diagnostics with PropsGet, replacing N with the subtitle track index:

• stat::subtitle.N.ocr.stream_index identifies the source stream associated with that subtitle track.
• stat::subtitle.N.ocr.dropped counts OCR jobs dropped when the bounded queue is full.
• stat::subtitle.N.ocr.last_error reports the latest OCR diagnostic, such as filter_not_found, traineddata_not_found, datapath_not_ascii or preinit_timeout.

Statistics are published during subtitle processing, so continue processing frames before reading them again. Missing filters or models can leave video playback working while producing no recognized text. Seek and close invalidate pending work from the previous segment; keep application cue state separate across those boundaries.

Use recognized text in an SDI workflow

OCR supplies text and timing metadata. Your application must schedule that text and pass it through a separate caption or ANC encoding step for SDI output. Do not treat timeout_pts as an exact caption end time. For the subsequent CC608 encoding step, see Generating custom CC608 subtitles and inserting them into ANC packets.