Indeed, thanks from our side as well for the good job for enhancing the RTMP & FLV specs!

We at Bytedance are interested in adding also the support of H.266/VVC. Is it OK to add that?

in the "Extending NetConnection connect Command" section, there is a new property "fourCcList". the description says it is an array of strings each "representing a supported video codec". the current Enhanced RTMP document is only extending video codecs; however, one would assume (from #2) that audio is coming soon.

with the name "fourCcList", one would presume that in the future, audio and video fourCCs would be intermingled in that array. this might be fine for some applications, but may complicate the semantics if it's important to tell the difference between "is there an audio codec i've never heard of" and "is there a video codec i've never heard of".

since today the audioCodecs and videoCodecs bitmaps are already separate, for symmetry i propose that the arrays of fourCCs for video and audio also be separated, and that they have names indicating which they're for. that way, if the difference is important to an application, it is at least available.

in the VideoTagBody section of version 2023-03-v1.0.0-B.9, there are two PacketTypes defined for coded frames: PacketTypeCodedFrames = 1 and PacketTypeCodedFramesX = 3, where type 3 has an implicit composition time offset of 0 (meaning the presentation time and decode time are the same).

the pseudocode for If FourCC == HEVC explicitly shows the composition time offset SI24 for PacketTypeCodedFrames and defines it to 0 for PacketTypeCodedFramesX. the other defined codecs (VP9 and AV1) use PacketTypeCodedFrames but don't show a composition time offset (presumably because those codecs don't code independent frames out of presentation order so such a field isn't needed).

since the notion of a presentation time different from decode time occurs in at least two codecs today (AVC and HEVC), and in the interest of consistency, separation of layers and concerns (notional header vs payload), and code reuse, i think PacketTypeCodedFrames should always have an SI24 Composition Time Offset field, and for codecs where that's always 0, they MAY use PacketTypeCodedFramesX to save those 3 bytes. any codec should be able to use either packet type 1 or 3 with a consistent parsing and interpretation.

ideally the Enhanced RTMP spec wouldn't itself define any new codec mappings, and instead would define the generic syntax. an addendum, appendix, or separate registry would define mappings for AV1, VP9, and HEVC to start.

the client can indicate support for Enhanced RTMP by including a fourCcList member of the connect command's argument/command object (though see #10 about the name of that member).

however, clients can't tell if servers support Enhanced RTMP. while an unaware server can simply forward Enhanced RTMP messages as they come in, this won't have the desired effect for clients subscribing to a stream after its publish has started. in particular, servers unaware of Enhanced RTMP won't have special treatment of PacketSequenceStart, PacketTypeMetadata, and PacketTypeMPEG2TSSequenceStart messages, remembering them and sending them to new subscribers before the coded frames.

a publishing client should be able to tell that the server will or won't perform the special sequence/metadata processing for subscribers, and subscribing clients should be able to tell that they may not receive the sequence/metadata messages for enhanced messages. this could be accomplished by echoing the fourCcList (and/or others as appropriate) back in the connect transaction's _result Info Object.

Such as opus over rtmp,so that it can be passed to webrtc,no need to transcode

The spec uses AudioPacketType.SequenceEnd in one place, but it has never defined a value for it anywhere.

the "Metadata Frame" section says "When leveraging PacketTypeMetadata to deliver HDR metadata, the metadata should be sent prior to the video sequence that it affects".

if the intention of the PacketTypeMetadata packet is to set metadata that needs to be known before decoding the video sequence, i think this sentence should use a BCP 14 "MUST" instead of lower-case "should".

also, is the qualification of "when leveraging ... to deliver HDR metadata" needed? presumably this message can be used for any metadata that needs to be known before decoding the video sequence. also presumably there's no more than one of these that's expected to be active at a time, and a new message supersedes a previous one.

would this message ever be expected, or be meaningful, after a video sequence is underway? in other words, when encountering one of these messages, would one expect the immediate next video messages to be a sequence start/init message and then a key frame? can a receiver count on that with the force of a BCP 14 MUST?

PacketTypeMPEG2TSSequenceStart is referenced in the pseudocode for AV1. please clarify whether:

1. this packet type has a meaning for any other codecs
2. if this packet type can be used in combination with PacketTypeSequenceStart or if they are mutually exclusive. if they can be used in combination, which one SHOULD/MUST come first?

how and for what is PacketTypeMPEG2TSSequenceStart expected to be used?

Thanks for the good job for enhance the RTMP specs, I'm glad to ask for any plan to publish the document as Internet standard? Isn't possible to propose the document to IETF like HLS spec and make it more popular and used by more company in the world?

the fourCcList currently says it contains strings each "representing a supported video codec". as discussed at #11 (comment) , the description should be clarified to say that this lists codecs that the client is able to receive and process.

the practical historical usage of the videoCodecs and audioCodecs bitmaps sent by the client in the connect command has been to indicate which codecs the client is able to receive. historically it's been irrelevant to the server what codecs a client can send, since the server could handle all traditional RTMP codecs.

This is NOT an issue but a NOTICE

SRS 6.0.42+ media server support this extended RTMP specification, so you can push HEVC via RTMP to SRS very easy:

git clone https://github.com/ossrs/srs.git
cd srs/trunk && ./configure --h265=on && make
./objs/srs -c conf/http.ts.live.conf

Then, you can use OBS 29.1+ to push HEVC via RTMP.
Start OBS with the following settings in the Settings > Stream tab:

• Server: rtmp://localhost/live
• Stream Key: livestream
• Encoder: Please select the HEVC hardware encoder.

Finally, open the player http://localhost:8080/players/srs_player.html?stream=livestream.ts

Or use VLS or ffplay to play http://localhost:8080/live/livestream.ts

I'm a bit confused about the OpusSequenceHeader.
The spec says "read either identification or comment header".

I'm not sure how to interpret that. FFmpeg considers only the identification header as "extradata", and only carries that around.
Is it fine to only send that?

And how is "either" to be interpreted here? If we wanted to send the comment header, we would need to send a second SequenceStart?

i wanted to ask if folks have test/sample media (FLVs) with Enhanced content, and clients & servers to test/interop with. i think that sort of thing belongs in a Discussion rather than an Issue. please consider enabling Discussions on this repo to discuss that and other topics that don't belong in Issues. :)

perhaps eventually links to test media and compatible clients & servers could be on a wiki page or in the README.

Per #2 (comment)

Currently, the only way to deliver lossless audio over RTMP is via linear PCM, which has some drawbacks:

• Limited to 16-bit audio
• Max 2 channels
• Max sample rate of 44.1kHz

Adding FLAC would allow some flexibility for stream producers that want to transcode for multiple bandwidths without losing audio quality. Currently the best solution is to max out the AAC bitrate and hope there isn't too much quality loss on transcodes.

Plus, assuming FLAC would signal its own bit-depth, samplerate, and channel count (I know AAC signal its own samplerate, I can't recall if it also signals channel count) - you could stream incredibly high-quality audio.

As far as solutions shipping today that would benefit from FLAC support:

OBS

Say I want to produce a web radio show. Most web radios are based on Icecast/shoutcast and a lot of the tooling around Icecast isn't great.

OBS is actually a pretty decent tool for producing radio-type content with builtin audio filters, support for multiple audio devices, scripting, hotkeys, and so on.

If OBS could output with FLAC over RTMP - a receiving server could take the single RTMP stream, and transcode to different formats (output to Icecast, HLS, and DASH) with significantly less quality loss.

HLS/DASH repackagers

Similar to above - I know the various nginx RTMP add-ons support converting RTMP into HLS and DASH. Being able to accept a FLAC stream would allow for producing HLS, DASH, and even other RTMP outputs with multiple codecs for multiple devices.

Say I want to create a multivariant playlist in HLS with next-generation codecs like xHE-AAC. Having a single endpoint with a lossless codec would allow me to create a multivariant playlist with perfect sample alignment (since it's all from the same source). Currently I need to have my apps produce multiple streams at multiple bitrates and its hard to get everything aligned perfectly, or I have to just stream with AAC and re-encode/repackage server side and lose quality.

Those are two off the top of my head, but it would really expand the possibilities for using RTMP as an intermediary/transport format. A user producing a web radio could stream in FLAC to a server over RTMP, which transcodes to formats like AAC, AAC-HE, xHE-AAC as HLS, and allow the listener to auto-fallback to codecs based on their bandwidth.

Long story short, the audio producer wouldn't need to worry if the final output/destination supports a particular codec - they would just stream in FLAC and have the ingest handle getting the right codecs out to the receivers. FLAC essentially allows nearly any other codecs to be supported via server-side transcoding.

While the spec does talk a bit about the ordering being used to identify the language of an audio stream, I do think that some way to tag an audio stream with a language would be beneficial.
Adding three bytes for an ISO country code or something somewhere would be super nice and might prevent potential vendor-specific order-magic.

Hi,

I am really glad a new version of RTMP is coming 🙏

In the FLV spec, there is a field in onMetaData called videocodecid where the codec was specified:

videocodecid: Number: Video codec ID used in the file (see E.4.3.1 for available CodecID values)

with CodecID:

Codec Identifier. The following values are defined: 2 = Sorenson H.263
3 = Screen video
4 = On2 VP6
5 = On2 VP6 with alpha channel 6 = Screen video version 2
7 = AVC

Unfortunately, in the current version of enhancements of RTMP spec, there is no mention of onMetaData's videocodecid field.
Is it something missing or is it because onMetaData is not really useful?

Best regards,
Thibault

in traditional RTMP, clients that are capable of receiving any codec (like recorders or forwarders) can just set the videoCodecs and audioCodecs fields to 0xffff to enable all possible code points. however, the number of possible future fourCCs is way too large to list them all. to accommodate clients that can handle any codec, i propose defining a special value to include in the fourCcList codec list (or its replacement(s)) to indicate "any/all codecs".

i further propose this special value to be the one-character string "*" (ASTERISK). being one character long it can never conflict with any real fourCC, and * is already commonly used to mean "wildcard".

In Table 4, first row (IsExHeader | FrameType UB[4]), the Comment column, IsExHeader is set to true in both branches of the first IF. I believe, in the ELSE branch it should be set to false.

@veovera

Any interest in supporting QUIC as a transport? QUIC frame/stream transport can address the Head-of-Line (HoL) blocking issues of TCP.

Is this document stoped?
Or is this document will be updated? I mean modify the FourCC for codec.

U

Several existing implementations used CodecID of 12 to support HEVC before this project began, and there are also hardware implementations based on those concepts, so could this be added to Enhanced RTMP as well?

The timebase of RTMP/FLV is 1/1000, while for MPEGTS or other protocols might be 1/90000. When convert MPEGTS to RTMP/FLV, there will be some small deviation and cause some stuttering.

Is it possible to support other timebase in enhanced RTMP?

See issue SRS#512 and SRS#547