szatmary / libcaption Goto Github PK

In digging into this I am just trying to get good CEA compliant captions out of OBS, the devs on Discord mentioned that OBS uses libcaption. That discussion can be found here: https://discord.com/channels/348973006581923840/374636084883095554/937035706507853865
As stated in that topic: Seems a possible fix for getting good CEA compliant captions would be to implement libavutil/cc_fifo.c which as dheitmueller states: "repacks the series of tuples with each frame based on the target framerate, and it's hooked in to each filter which changes the framerate." and then he uses libavfilter/vf_ccrepack.c to "have a filter which does the repacking if the cc_count is simply wrong and the data is intact."
Those libs are here:
https://github.com/LTNGlobal-opensource/FFmpeg-ltn/blob/lted1/libavfilter/vf_ccrepack.c
https://github.com/LTNGlobal-opensource/FFmpeg-ltn/blob/lted1/libavutil/cc_fifo.c

Would this be something libcaption would be able to do? Thanks!

My goal is to update a .srt file through a web form for live streaming captioning. Our output is HLS + Facebook and YouTube. Nginx RTMP converts the stream to HLS, but for my test, I bypassed HLS and tested output of RTMP stream back into Nginx RTMP.

My test was:
Streaming encoder --> Nginx RTMP --> FFMPEG/FLV+SRT --> Nginx RTMP

I tried the example, noticing the paragraph regarding live captioning.

Here is my command:
ffmpeg -i rtmp://streamserver.tld:1936/publish/999 -codec copy -f flv - | flv+srt - ~/libcaption/cc.srt - | ffmpeg -i - -codec copy rtmp://streamserver.tld:1936/rtmp/999_cc

I tried two srt files, because I was unsure of the exact format:

cc.srt:
1
00:00:00,000 --> 00:00:00,000
Test live cc text
\0

cc2.srt:
1
\0
00:00:00,000 --> 00:00:00,000
Test live cc text

Received pretty big output in CLI, including:
Reading flv from -
Reading captons from /home/streambright/libcaption/cc.srt
Writing flv to -
*** buffer overflow detected ***: flv+srt terminated

...

and

...

pipe:: Invalid data found when processing input

Any help on what I am not getting is appreciated. Thanks!

Title explains

I will apologize for the ridiculousness of the test case in advance. Inserting captions every 50 ms is stupid, but the missing caption is easiest to repro from my side this way. This was part of a test for a client to try and identify missing captions and not a normal use case.

Steps I used:

1. Using Wowza Media Server, injected AMF onTextData captions every 50 ms.
2. Used Wowza's onTextData to CEA608 caption conversion
3. Captured first HLS ts segment (attached media_1.ts)
4. Ran ts2srt on ts segment (output in ts2srt.txt)

I noticed caption 199 is missing. It is the last one in the ts file.
I also ran caption-inspector, which did find the 199 caption. (media_1-C1.608 attached)

As a side note, the times from caption inspector and libcaption seem different by 33 ms.

Thanks for creating this library! There are hardly any OSS tools for caption insertion.

I was attempting to use flv+srt to insert captions, but noticed that the resulting flv did not double up control codes, in particular End of Caption. It looks like in other parts of the code this is expected when decoding EIA-608:

I'm happy to submit a PR to fix this, I just want to make sure I'm on the right track. Should I be looking at how caption_frame_from_text is writing in the srt data?

https://github.com/szatmary/libcaption/blob/master/src/caption.c#L359

First off, thanks for this library :)

gst-plugins-rs uses libcaption and has a test scc file for unit testing purposes. It applies to https://www.democracynow.org/shows/2018/12/17?autostart=true, and I find that both the gstreamer wrapper and the scc2srt executable generate empty cues and "offset-by-1" cue timestamps.

cat foo.scc:

Scenarist_SCC V1.0

00:00:00;00 942c 942c

00:00:14;01 9420 9420 94ae 94ae 9454 9454 10ae 10ae 46f2 ef6d 20ce e5f7 20d9 eff2 6b2c 94f2 94f2 10ae 10ae f468 e973 20e9 7320 c4e5 6def e3f2 61e3 7920 ceef f7a1 942f 942f

00:00:17;26 9420 9420 94ae 94ae 9452 9452 97a1 97a1 10ae 10ae d9e5 732c 942c 942c 2049 a76d 2073 7570 70ef f2f4 e96e 6780 94f4 94f4 10ae 10ae c4ef 6e61 ec64 2054 f275 6d70 ae80 942f 942f

scc2srt foo.scc:

01
0:00:00,000 --> 00:00:00,000


02
0:00:00,000 --> 00:00:14,033
From New York,
this is Democracy Now!

03
0:00:14,033 --> 00:00:14,033

Here, I believe:

• The first scc cue should be ignored (simply clearing the screen)

• The second cue should be timestamped as 00:00:14;01 + the number of frames before 0x94 0x2f, as the cue starts by initiating pop-on mode (0x94 0x20, "resume_caption_loading"). This seems to match pretty well with the actual media, as the lady starts saying "From New-York" some time during the 15th second.

• I suspect the last cue isn't drained, probably trivial to fix

From a quick look at the code, I think there are two issues at play here:

• libcaption decides that frames are READY too often, for example when encountering 0x94 0x2c (erase_display_memory) , this leads to the creation of a cue.

• caption_frame_decode updates its timestamp before ignoring duplicate control commands, this explains the offset-by-1 issue, I believe this could be improved further by tracking the current mode, and in pop on mode updating the timestamp at the end of the caption instead of when the last call returned READY, this would however require making assumptions about the framerate (are SCC files always 29.97 fps?)

hello,
I checked your example script. Would this work for a live stream with a text file updated regularly ?
A suggestion: in OBS studio enable the ingestion of a live text file (maybe the code you committed already allows that)
This is of interest with fast transcribers (steno) during live streaming.

I followed similar steps to what's in the add_captions script. I have a file, in.mp4, and an SCC caption file. The resultant FLV has captions, converting to MP4 with ffmpeg does not have captions.

Commands:

> ffmpeg -i in.mp4 -codec copy -f flv in.flv
> flv+scc in.flv captions.scc out.flv
> ffmpeg -i out.flv -codec copy out.mp4

Using ffmpeg 4.0.1, installed via git clone and directions in Readme, running Mac OS 10.12.6.

Any suggestions on how to make the output MP4 have the embedded CCs?

scc has positional data that will get lost if i convert from srt to scc. It would be cool if there was a way to go from 608/708 to scc.

I followed similar steps to what's in the add_captions script. I have a file, in.mp4, and an SCC caption file. The resultant FLV has captions, converting to MP4 with ffmpeg does not have captions.

Commands:

> ffmpeg -i in.mp4 -codec copy -f flv in.flv
> flv+scc in.flv captions.scc out.flv
> ffmpeg -i out.flv -codec copy out.mp4

Using ffmpeg 4.0.1, installed via git clone and directions in Readme, running Mac OS 10.12.6.

Any suggestions on how to make the output MP4 have the embedded CCs?

Does libcaption currently support changing the color, position, display mode or any other advanced features? In caption.c, I see references to eia608_style_white, CAPTION_ROLL_UP and underline, while the other eia608_style_COLORS are defined in eia608.h. But I don't see many examples of how this is being used.

In my own application that is based on flv+srt.c, I modified caption_frame_from_text(frame, data, unsigned int sty) to allow me to pass any style option to caption_frame_write_char() instead of the default eia608_style_white_ that was there previously. This sets the style into cell->sty, but I cannot figure out how cell gets used. And I do not see any changes to the data that is being written into the video file.

I have also tried changing the frame state by setting frame.state.sty, frame.state.mod and frame.state.rup to other appropriate values but did not see differences to the caption data this way either. Are these stubs to be filled out at a later time?

I would like to thank you for all the great work you have put into this library. I was able to embed captions into my encoder just by following your example in flv+srt. You have done a great service for the community! Please let me know if there is anything I can do to help improve it in the future.

Working on adding live subtitles to a live stream. I have this working on another platform using HLS, with subtitles.

Using the example flv+srt binary, I am having no luck creating an output:

Example stanza:

ffmpeg -i rtmp://localhost/show/stream -codec copy -f flv - | flv+srt - /tmp/eng-fifo - | ffmpeg -i - -codec copy rtmp://localhost/subs/stream

I see output that looks like the subtitles are being read:

Loaded new SRT at time 0.000000
T: 0.00 (6.76s):
based on the fact that Aurora might be oversold here and

T: 6.78 (5.23s): 38 q=-1.0 size=    2218kB time=00:00:06.45 bitrate=2815.4kbits/s
bouncing maybe some good news for the sector I don't know what who am

T: 12.01 (2.97s):34 q=-1.0 size=    3773kB time=00:00:11.54 bitrate=2678.0kbits/s
I kidding, it's amazing

However I don't see any output stream, and all I eventually see after quitting ffmpeg is:

[flv @ 0x1f29400] Could not find codec parameters for stream 0 (Video: flv1 (flv), yuv420p): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[flv @ 0x1f29400] Could not find codec parameters for stream 1 (Audio: none, 0 channels): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
pipe:: could not find codec parameters
Input #0, flv, from 'pipe:':
  Duration: N/A, start: 2728370.006000, bitrate: N/A
    Stream #0:0: Video: flv1, yuv420p, 1k tbr, 1k tbn, 1k tbc
    Stream #0:1: Audio: none, 0 channels
[NULL @ 0x1f3d360] Unable to find a suitable output format for 'rtmp://localhost/subs/stream'
rtmp://localhost/subs/stream: Invalid argument

I'm still playing around with parameters, but I am unsure if this is a suitable approach to begin with.

I would like to use libcaption in a C89-compliant project. libcaption code is almost compliant - just a few little changes here and there. Would you be open to a PR to comply with C89 ?

From what I see in the example there is possibly a conversion of data in an RTMP/flv packet being converted to 608. But I see no method to convert to 708. 708 is needed for extended language character sets like Japanese.

Let me know what is possible.

This is a really really cool and useful lib. A cool feature would be the ability to embed 608 captions into a h264 video elementary stream.

I've been playing with captions and subtitles,
but I've just been making wild guesses,
you actually know what you're doing, I'm diggin' it.

This lib is for FFMPEG yeah? How do I implement it? Just trying to play a mpeg2 file and get captions to pass out SDI on decklink, any assist would be great thanks!

I would like to create a brew formula for libcaption to make install easier for users.

Does anyone have a working brew formula which we could contribute to the homebrew project?

Below uses brew to create a basic template.

$ brew create 'https://github.com/szatmary/libcaption/archive/master.zip'

EDITED: Removed old version of libcaption.rb.

Can libcaption be used to add multiple caption tracks to flv?

Is it possible to support SRT captions that contain the ♪ (music note)? When running flv+srt with these captions, I get the "ERR: Unrecognized block". It looks like the code would need to be updated around here to fix this:

If you can give me some direction on just exactly where it's failing, I may be able to put together a pull request.

Thanks,
nabeards

I'm hoping you can help me with a bug I am working through.

From my understanding, CEA-608 should be able to render 64 characters per frame, two rows of 32 columns each. However, with libcaption I am only able to render up to 51 characters at a time. The characters that were cut off are placed into the next sei_message_t. Is this intentional or a bug?

I've been looking at sei_from_caption_frame() and sei_encode_eia608() for some insight into what might be going on.

1. Every time an eia608_control_command is issued the cea708->user_data.cc_count is incremented. Won't this throw off the count if (31 == cea708->user_data.cc_count) when the number of characters is equal to 64?

2. At my 51st character, cc_data = eia608_from_utf8_1(data, DEFAULT_CHANNEL) = 0. This forces the eia608_control_end_of_caption message to be issued (although I never see that in a hexdump of the sei message) and a new header gets issued with the remaining data. I haven't been able to figure out where this cc_data=0 is coming from yet.

I have noticed this behavior working out of both master and develop branches, though at present I am in the develop branch. I was hoping to have a patch for you instead of questions, but I am stumped at the moment.

Can you provide any information on this issue?

Thank you,
Josh

On the latest build as of 5-31-2017 the text metadata is missing after embedding captions.
The captions are definitely in the video stream but the metadata is not updated after I run mediainfo on it.

ffmpeg -i https://cp-open.s3.amazonaws.com/source.mp4 -codec copy -f flv - | flv+srt - caption.srt - | ffmpeg -i - -codec copy source_embeded.mp4

URLS
Source: https://cp-open.s3.amazonaws.com/source.mp4
Source with embedded 608: https://cp-open.s3.amazonaws.com/source_embedded.mp4
SCC: https://cp-open.s3.amazonaws.com/caption.scc
SRT: https://cp-open.s3.amazonaws.com/caption.srt

This is not an issue, I'm just looking for help. Sorry if this is not the correct channel. I'm trying to get an input video coming from an RTMP endpoint and restreaming to another after adding srt file. I'm using a command like this one

ffmpeg -i rtmp://localhost:1935/live/test -codec copy -f flv - | ./flv+srt - ../../sub.srt - | ffmpeg -i - -codec copy -y -f flv rtmp://a.rtmp.youtube.com/live2/aabbcc

The restreaming works fine if I omit the flv+srt part, but it fails if I use the command above. The RTMP never receives the stream and this is printed in console.

Loaded new SRT at time 0.000000=       1kB time=00:00:00.00 bitrate=N/A speed=   0x    
T: 0.07: [CAPTIONS CLEARED]
T: 5.40 (10.00s):41 q=-1.0 size=    1582kB time=00:00:04.86 bitrate=2665.4kbits/s speed=1.36x    
This is an example of
a subtitle.

T: 15.53: [CAPTIONS CLEARED]ize=    4917kB time=00:00:15.08 bitrate=2670.7kbits/s speed=1.09x    
[flv @ 0x7face391d140] Packet mismatch 216678557 6386798 6386798ate=2668.7kbits/s speed=1.07x    
T: 20.40 (99.00s):2 q=-1.0 size=    6607kB time=00:00:20.18 bitrate=2682.2kbits/s speed=1.07x    
This is an example of
a subtitle - 2nd subtitle.
[flv @ 0x7face391d140] Packet mismatch -527437488 5557771 11944569e=2697.8kbits/s speed=1.04x    
[NULL @ 0x7facc2f05f40] missing picture in access unit with size 10178
[extract_extradata @ 0x7facd3006680] No start code is found.
pipe:: could not find codec parameters
Input #0, flv, from 'pipe:':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Audio: aac, 44100 Hz, stereo
  Stream #0:1: Video: h264, none, 1k tbn
[flv @ 0x7facd3006800] dimensions not setB time=00:00:35.98 bitrate=2694.2kbits/s speed=1.04x    
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Error initializing output stream 0:1 -- 
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
    Last message repeated 1 times
av_interleaved_write_frame(): Broken pipe
[flv @ 0x7fcda1304c40] Failed to update header with correct duration.
[flv @ 0x7fcda1304c40] Failed to update header with correct filesize.
Error writing trailer of pipe:: Broken pipe
frame= 1083 fps= 31 q=-1.0 Lsize=   11837kB time=00:00:36.03 bitrate=2691.3kbits/s speed=1.03x    
video:11093kB audio:704kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.343468%
Error closing file pipe:: Broken pipe
Conversion failed!

Do you have any idea about what the cause can be?

Hey There,

Attached is a sample srt file that crashes flv+srt

I used my samples in the tarbal to run git-bisect and I found that it reports 3fb17d5 as the bad commit.

The interesting thing is that I can manually edit the srt file and remove the last caption/cue entry and flv+srt succeeds.

I tried looking through a hex editor for any strange hidden characters but nothing stood out, so I can't explain why that edit allows the process to succeed.

NOTE:
The srt file has been 'anonymized' by replacing every alpha character (a-zA-z) with the letter a.

srt-crash.tar.gz

I'd be happy to provide more testing help as needed.

My test environment is:

$ uname -a
Linux ip-10-0-0-19 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

$ ffmpeg -version
ffmpeg version N-87739-g1fd8010 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.5) 20160609
configuration: --prefix=/home/ubuntu/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/ubuntu/ffmpeg_build/include --extra-ldflags=-L/home/ubuntu/ffmpeg_build/lib --bindir=/home/ubuntu/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree
libavutil      55. 77.101 / 55. 77.101
libavcodec     57.106.104 / 57.106.104
libavformat    57. 82.102 / 57. 82.102
libavdevice    57.  9.101 / 57.  9.101
libavfilter     6.106.100 /  6.106.100
libswscale      4.  7.103 /  4.  7.103
libswresample   2.  8.100 /  2.  8.100
libpostproc    54.  6.100 / 54.  6.100