How to Insert DVB Subtitles and DVB Teletext On the Fly During Live Output

Overview

This article describes how to insert DVB subtitles or DVB Teletext subtitles into an output stream on the fly during live playback or recording.

In this workflow, subtitles are not loaded from a static subtitle file. Instead, the application generates subtitle data dynamically, serializes it as JSON, attaches the JSON to individual video frames, and passes those frames to the writer. The writer then encodes the subtitle data into the output stream.

The main idea is:

1. Prepare subtitle data in JSON format.
2. Attach the subtitle JSON to the required frame with MFStrSet("subtitle_json", ...). For additional subtitle tracks, use subtitle_json.1, subtitle_json.2, and so on.
3. Pass the frame to MWriter or MFWriter.
4. Configure the writer subtitle stream with subtitle::codec, subtitle::stream_id, and subtitle metadata.
5. The writer encodes the subtitle data into the output stream as DVB subtitles or DVB Teletext subtitles.

This approach is useful for live workflows where subtitles, captions, speaker labels, warnings, score overlays, or other text events are generated while the stream is already running.

Ready-to-use sample

The workflow of this article is implemented in the Subtitle Generation Sample. It plays a file, a capture device, a network URL or an external source (a sender of another SDK object), and its Live Subtitle Editor creates the subtitle events while the output is running: you type the text, the stream ID and the duration, and the sample shows the generated subtitle_json before it attaches it to the next frame of the writer track. Subtitle tracks can also be loaded from ASS and SRT files, see How to Insert ASS and SRT Subtitles as DVB, DVB Teletext. The output (a file, UDP or SRT), the codec of every subtitle track (DVB subtitles or DVB Teletext), the PID, the language and the Teletext page are configured in the output dialog.

The Connect services dialog of the sample is a place for a connection to an external subtitle service (the choices are Whisper AI, REST and WebSockets). The sample does not include an implementation of such a service: the text comes from the editor. To feed the subtitles from your own service, call the same subtitle_json code from the code that receives the text.

Important concepts

There are two different codec-related values in this workflow:

• The JSON field codec describes the format of the subtitle text payload attached to the frame. In the examples below, the payload uses "ass" because the subtitle text is formatted as an ASS event.
• The writer option subtitle::codec defines how the subtitle stream is encoded into the output. For DVB subtitles, use subtitle::codec='dvbsub'. For DVB Teletext subtitles, use subtitle::codec='dvbtxt'.

The subtitle JSON is attached to a frame with the subtitle_json string key:

((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);

The subtitle stream_id in the JSON should match the writer subtitle stream ID configured with subtitle::stream_id.

The JSON stream_id and the frame property name serve different purposes. stream_id identifies the SDK subtitle stream, while the frame property name selects the writer track slot. Use subtitle_json for subtitle::..., subtitle_json.1 for subtitle.1::..., and so on.

Subtitle JSON structure

Each subtitle event is represented as a JSON object.

{
  "stream_id": 12,
  "codec": "ass",
  "text": "1,0,Jordan,Jordan,0,0,0,,Pack my box with five dozen liquor jugs",
  "duration_sec": 1.5
}

The supported fields used by this workflow are:

Subtitle item class

It is convenient to represent each subtitle event as a class and serialize it into JSON when the subtitle should be inserted.

using System.ComponentModel;
using System.Text.Json;
using System.Text.Json.Serialization;

public class SubtitleItem
{
    [JsonPropertyName("stream_id")]
    public int StreamId { get; set; }

    [JsonPropertyName("codec")]
    public string Codec { get; set; } = "ass";

    [JsonIgnore]
    public string Text { get; set; } = string.Empty;

    [JsonIgnore]
    public string Style { get; set; } = string.Empty;

    [EditorBrowsable(EditorBrowsableState.Never)]
    [JsonPropertyName("text")]
    public string SerializedText => Style + Text;

    [JsonPropertyName("duration_sec")]
    public double DurationSec { get; set; }

    [JsonPropertyName("ass_header")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
    public string? AssHeader { get; set; }

    public string ToJson(bool indented = false)
    {
        var options = new JsonSerializerOptions
        {
            WriteIndented = indented,
            DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
        };

        return JsonSerializer.Serialize(this, options);
    }
}

Prepare subtitle events

The first subtitle event can include the ASS header. The following events can reuse the same header and send only the ASS event data.

List<SubtitleItem> subtitles = new List<SubtitleItem>
{
    new SubtitleItem
    {
        StreamId = 12,
        Codec = "ass",
        AssHeader =
@"[Script Info]
Title: Live DVB Subtitle Insertion
ScriptType: v4.00+
WrapStyle: 0
ScaledBorderAndShadow: yes
YCbCr Matrix: TV.709
PlayResX: 1920
PlayResY: 1080

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,Arial,58,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,3,1,2,10,10,40,1
Style: Alex,Arial,58,&H0000A5FF,&H000000FF,&H000000FF,&H80000000,1,0,0,0,100,100,0,0,1,3,1,2,10,10,40,1
Style: Jordan,Arial,58,&H0000FF00,&H000000FF,&H00000000,&H80000000,1,0,0,0,100,100,0,0,1,3,1,2,10,10,40,1

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text",
        Style = "0,0,Alex,Alex,0,0,0,,",
        Text = "The quick brown fox jumps over the lazy dog",
        DurationSec = 1.5
    },
    new SubtitleItem
    {
        StreamId = 12,
        Codec = "ass",
        Style = "1,0,Jordan,Jordan,0,0,0,,",
        Text = "Pack my box with five dozen liquor jugs",
        DurationSec = 1.5
    },
    new SubtitleItem
    {
        StreamId = 12,
        Codec = "ass",
        Style = "2,0,Alex,Alex,0,0,0,,",
        Text = "Sphinx of black quartz, judge my vow",
        DurationSec = 1.5
    }
};

The Style value is combined with Text and serialized into the JSON text field. This allows the application to select ASS styles dynamically for different speakers or subtitle types.

Do not send only plain text when codec is "ass". The text field must include the ASS event payload generated by Style + Text.

Escaping the subtitle text

The text of an ASS event is not plain text. A line break is written as \N, and the characters { and } start and end the ASS override tags, so they have to be escaped when the text comes from a user or from a speech-recognition service. The sample does it like this:

public static string EscapeAssText(string text)
{
    return (text ?? string.Empty)
        .Replace("\r\n", "\\N")
        .Replace("\n", "\\N")
        .Replace("\r", "\\N")
        .Replace("{", "\\{")
        .Replace("}", "\\}");
}

Plain text subtitles

When the text does not need ASS styles, send it as plain text. The JSON then contains only the stream ID, the text and, if the default of 3 seconds does not fit, the duration. There is no ass_header, so the SDK uses the srt payload format:

{
  "stream_id": 12,
  "text": "Plain subtitle text",
  "duration_sec": 2.0
}

The writer track settings (subtitle::codec, subtitle::stream_id) are the same as for ASS subtitles.

Writer configuration

The writer configuration defines how the subtitle packets are encoded into the output stream.

DVB subtitle output

Use subtitle::codec='dvbsub' to encode bitmap DVB subtitles.

format='mpegts'
video::codec='libopenh264'
audio::codec='aac'
muxrate='18.5M'
video::bitrate='15M'
video::maxrate='15M'
subtitle::codec='dvbsub'
subtitle::stream_id=12
subtitle::metadata::language='eng'
program::title='FIRST TITLE'

DVB Teletext subtitle output

Use subtitle::codec='dvbtxt' to encode DVB Teletext subtitles.

format='mpegts'
video::codec='libopenh264'
audio::codec='aac'
muxrate='18.5M'
video::bitrate='15M'
video::maxrate='15M'
subtitle::codec='dvbtxt'
subtitle::stream_id=12
subtitle::metadata::language='eng'
program::title='FIRST TITLE'

The important writer subtitle options are:

MFormats SDK workflow

In MFormats SDK, read frames from MFReader, attach subtitle JSON to the required frames with MFStrSet, and pass the frames to MFWriter.

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using MFORMATSLib;

public static class LiveSubtitleInsertionMFormats
{
    public static void Run(
        string sourcePath,
        string outputUrl,
        List<SubtitleItem> subtitles)
    {
        MFReaderClass reader = null;
        MFWriterClass writer = null;

        try
        {
            string writerConfig =
                "format='mpegts' " +
                "video::codec='libopenh264' " +
                "audio::codec='aac' " +
                "muxrate='18.5M' " +
                "video::bitrate='15M' " +
                "video::maxrate='15M' " +
                "subtitle::codec='dvbsub' " +
                "subtitle::stream_id=12 " +
                "subtitle::metadata::language='eng' " +
                "program::title='FIRST TITLE'";

            reader = new MFReaderClass();
            writer = new MFWriterClass();

            reader.ReaderOpen(sourcePath, "");
            writer.WriterSet(outputUrl, 1, writerConfig);

            int frameIndex = 0;
            int subtitleIndex = 0;

            while (true)
            {
                MFFrame frame = null;

                try
                {
                    reader.SourceFrameGet(-1, out frame, "");

                    if (frame == null)
                        break;

                    // In this example, a new subtitle is inserted every 250 frames.
                    if (frameIndex % 250 == 0 && subtitleIndex < subtitles.Count)
                    {
                        string subtitleJson = subtitles[subtitleIndex].ToJson();

                        ((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);

                        subtitleIndex++;
                    }

                    writer.ReceiverFramePut(frame, -1, "");
                }
                finally
                {
                    if (frame != null)
                        Marshal.ReleaseComObject(frame);
                }

                frameIndex++;
            }

            reader.ReaderClose();
        }
        finally
        {
            if (writer != null)
                Marshal.ReleaseComObject(writer);

            if (reader != null)
                Marshal.ReleaseComObject(reader);
        }
    }
}

The frame index controls when each subtitle appears. The subtitle duration is defined by the duration_sec value in the JSON.

MPlatform SDK workflow

In MPlatform SDK, subtitles can be inserted into frames received from the OnFrameSafe event. Enable frame data events, attach subtitle JSON to the selected frames, and start MWriter from the source object.

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using MPLATFORMLib;

public static class LiveSubtitleInsertionMPlatform
{
    public static void Run(
        string sourcePath,
        string outputUrl,
        List<SubtitleItem> subtitles)
    {
        MFileClass file = null;
        MWriterClass writer = null;

        try
        {
            string writerConfig =
                "format='mpegts' " +
                "video::codec='libopenh264' " +
                "audio::codec='aac' " +
                "muxrate='18.5M' " +
                "video::bitrate='15M' " +
                "video::maxrate='15M' " +
                "subtitle::codec='dvbsub' " +
                "subtitle::stream_id=12 " +
                "subtitle::metadata::language='eng' " +
                "program::title='FIRST TITLE'";

            file = new MFileClass();
            writer = new MWriterClass();

            file.FileNameSet(sourcePath, "");

            file.PropsSet("object::on_frame.sync", "true");
            file.PropsSet("object::on_frame.data", "true");
            file.PropsSet("object::events.use_window", "false");

            int frameIndex = 0;
            int subtitleIndex = 0;

            file.OnFrameSafe += (string channelId, object frameObject) =>
            {
                try
                {
                    if (frameIndex % 250 == 0 &&
                        subtitleIndex < subtitles.Count)
                    {
                        string subtitleJson = subtitles[subtitleIndex].ToJson();

                        ((IMFFrame)frameObject).MFStrSet(
                            "subtitle_json",
                            subtitleJson);

                        subtitleIndex++;
                    }

                    frameIndex++;
                }
                finally
                {
                    if (frameObject != null)
                        Marshal.ReleaseComObject(frameObject);
                }
            };

            writer.WriterNameSet(outputUrl, writerConfig);

            writer.ObjectStart(file);
            file.FilePlayStart();

            // Keep the application running while the live source is being processed.
        }
        finally
        {
            if (writer != null)
                Marshal.ReleaseComObject(writer);

            if (file != null)
                Marshal.ReleaseComObject(file);
        }
    }
}

The object::on_frame.sync, object::on_frame.data, and object::events.use_window properties are used so the application receives frame objects and can attach subtitle data before the writer encodes the output.

Subtitle Timing Model

The subtitle JSON does not contain a separate start-time field. A subtitle starts at the timestamp of the frame where the application attaches subtitle_json.

((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);

The display duration is controlled by the JSON duration_sec value. In other words:

subtitle start time = timestamp of the frame where subtitle_json is attached
subtitle end time   = subtitle start time + duration_sec

For example, if a subtitle JSON object with "duration_sec": 1.5 is attached to frame 250, the subtitle starts at frame 250 and remains active for 1.5 seconds. The exact media time of frame 250 depends on the frame rate. For example, at 25 fps, frame 250 is approximately 10 seconds from the start; at 50 fps, frame 250 is approximately 5 seconds from the start.

The examples above use a simple frame counter only to demonstrate the mechanism:

if (frameIndex % 250 == 0 && subtitleIndex < subtitles.Count)
{
    string subtitleJson = subtitles[subtitleIndex].ToJson();
    ((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);
    subtitleIndex++;
}

This does not mean that subtitles can only be inserted every 250 frames. The value 250 is just a sample trigger. The application can attach subtitle_json to any frame.

In a real live workflow, the insertion condition can be based on frame PTS, wall-clock time, speech-recognition timestamps, SCTE/event metadata, operator actions, automation commands, or any other application-specific trigger.

For example, insertion can be based on the current frame timestamp:

M_TIME frameTime;
((IMFFrame)frame).MFTimeGet(out frameTime);

double frameTimeSec = frameTime.rtStartTime / 10000000.0;

if (frameTimeSec >= subtitleStartTimeSec)
{
    ((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);
}

Or the subtitle can be inserted as soon as a new live subtitle event is available:

if (TryGetNextSubtitle(out string subtitleJson))
{
    ((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);
}

For speech-recognition workflows, the application can use the transcript timestamp, find the matching buffered video frame, and attach the JSON to that specific frame:

if (frameTimeSec >= transcriptStartTimeSec)
{
    string subtitleJson = transcriptSubtitle.ToJson();
    ((IMFFrame)frame).MFStrSet("subtitle_json", subtitleJson);
}

The JSON is attached once at the subtitle start frame. The duration_sec value defines how long the subtitle should remain visible after that start frame. It does not require the application to insert the same JSON on every following frame.

Multiple subtitle streams

To insert several subtitle streams, configure a writer subtitle track for each stream and use a different JSON stream_id for each track.

For example, one subtitle stream can use stream_id=12 and another subtitle stream can use stream_id=13.

subtitle::codec='dvbsub'
subtitle::stream_id=12
subtitle::metadata::language='eng'

subtitle.1::codec='dvbtxt'
subtitle.1::stream_id=13
subtitle.1::metadata::language='deu'

The subtitle JSON for each inserted subtitle must use the matching stream_id. The frame property name must also match the writer track index.

The following example inserts DVB bitmap subtitles on writer track 0 and DVB Teletext subtitles on writer track 1. Both events can be attached to the same video frame because they use different frame property names.

string dvbSubtitleJson = new SubtitleItem
{
    StreamId = 12,
    Codec = "ass",
    AssHeader = assHeader,
    Style = "0,0,Default,Default,0,0,0,,",
    Text = "DVB subtitle text",
    DurationSec = 2.0
}.ToJson();

string teletextSubtitleJson = new SubtitleItem
{
    StreamId = 13,
    Codec = "ass",
    AssHeader = assHeader,
    Style = "0,0,Default,Default,0,0,0,,",
    Text = "DVB Teletext subtitle text",
    DurationSec = 2.0
}.ToJson();

((IMFFrame)frame).MFStrSet("subtitle_json", dvbSubtitleJson);
((IMFFrame)frame).MFStrSet("subtitle_json.1", teletextSubtitleJson);

Do not attach every JSON object with subtitle_json. In that case, the writer receives all events in track 0 and the other configured subtitle tracks do not receive subtitle data.

Notes and limitations

• Inserting subtitles with subtitle_json is supported in 64-bit applications.
• The stream_id and the text are required. A JSON that cannot be parsed, has no text, or has a stream_id that no writer track uses is skipped without an error: the SDK only writes a message to its log.
• One frame carries one event per writer track. If a second event for the same stream is attached to the same frame, it is rejected. Attach it to the next frame, or use the frame property of another track.
• ass_header is remembered for the stream after the first event, so it is enough to send it once and again when the style definition changes.
• The subtitle starts at the time stamp of the frame and lasts duration_sec; there is no separate start time in the JSON.
• Use MFStrSet("subtitle_json", ...) to attach the subtitle JSON to writer track 0. For writer track subtitle.N, use MFStrSet("subtitle_json.N", ...).
• The JSON stream_id should match the writer subtitle::stream_id.
• The frame property index must match the writer track index. For example, subtitle.1::stream_id=13 receives JSON from subtitle_json.1, not from subtitle_json.
• The JSON codec describes the subtitle payload format. In the examples above, "ass" is used for ASS-formatted subtitle event data.
• The writer subtitle::codec controls the output subtitle type: dvbsub for bitmap DVB subtitles or dvbtxt for DVB Teletext subtitles.
• Send ass_header with the first subtitle event, or when the ASS style definition changes.
• duration_sec defines how long the subtitle is displayed after it is inserted into the frame.
• For live workflows, subtitle insertion can be triggered by frame number, time, operator input, automation events, or external metadata.

Summary

To insert DVB subtitles or DVB Teletext subtitles on the fly, generate subtitle events as JSON, attach each JSON event to the required frame with MFStrSet("subtitle_json", ...), and configure the writer subtitle stream with the required output codec and stream ID. For multiple subtitle tracks, use the corresponding frame property name: subtitle_json, subtitle_json.1, subtitle_json.2, and so on.

Use subtitle::codec='dvbsub' for bitmap DVB subtitles and subtitle::codec='dvbtxt' for DVB Teletext subtitles. Make sure that the JSON stream_id matches the writer subtitle::stream_id, and that ASS JSON text contains the complete ASS event payload, including the event fields before the visible text.

See also

• Subtitles in Video SDK: overview and samples
• How to Insert ASS and SRT Subtitles as DVB, DVB Teletext
• Displaying DVB subtitles/teletext from transport stream
• WebVTT subtitles for HLS