Comments (13)
so, Opus doesn't require a sequence header when the audio messages are in native Opus format. sequence headers are required when the audio messages are in the Ogg format.
WebCodecs uses the presence or absence of the "description" (which would be the payload of a sequence start message) in the AudioDecoderConfig to distinguish between Ogg or native Opus formats, respectively.
if i had my 'drothers, i'd define Enhanced RTMP to use the Opus native format that doesn't require a sequence header at all.
to allow both formats, i'd say that the SequenceStart message is REQUIRED for Opus, but that an empty (0-length) OpusSequenceHeader means coded frames are in "Opus Native Format", and a non-empty one means coded frames are in "Ogg Format".
from enhanced-rtmp.
After reviewing all the feedback, engaging in extensive discussions, and further consideration, it appears that the best way to proceed involves the following steps:
- Sequence Start Requirement: Always send a sequence start for the Opus codec.
- Sequence Start Content: The sequence start is either empty or contains an ID Header.
- Channel Mapping Override: If the ID Header contains Channel Mapping information, it overrides the multichannel configuration set by the RTMP
AudioPacketType.MultichannelConfig
signal. - Opus Packet Handling: The audio data packets on the wire will be native Opus packets, not wrapped into Ogg packets. To manage more than two channels, we will need to send self-delimiting framed Opus packets over the wire. This approach is expected to save bytes compared to delivering full Ogg Opus packets.
- Codec Integration: To forward Opus audio to WebCodecs or any other solution that expects an Ogg Opus stream, you must encapsulate the Opus packets received via RTMP into an Ogg Opus container.
- AudioPacketType.SequenceEnd: We will retain the
AudioPacketType.SequenceEnd
enum and document the silence packet.
These are the high-level details which will be documented in greater detail in the specification.
from enhanced-rtmp.
today's project is adding Enhanced RTMP Audio to my JavaScript. i did the C++ side yesterday. :)
zenomt/rtwebsocket@2bede7d hopefully adds playback support for Enhanced Audio at least for codecs supported by WebCodecs in browsers (Opus, FLAC, plus enhanced mode for AAC and maybe MP3). provisional pending some sample media to test with.
for Opus i went with "if i'm switching to Opus from None or Other and there's no SequenceStart, or there is a SequenceStart but its payload is empty, then use Opus native format; and if there's a SequenceStart with a non-empty payload, then use Ogg format". this is backward-compatible with what my JS senders are sending now, but is brittle if SequenceStart is ever used.
from enhanced-rtmp.
-
OpusSequenceHeader description has been polished based on feedback and further considerations.
-
OpusCodedData description has been polished based on feedback and further considerations.
from enhanced-rtmp.
WebCodecs, at least, doesn't require initialization data for Opus. to date i've been using codec id 15 "Device Specific" for Opus in my JavaScript (*), just sending the coded frames as they come out of the encoder, and for playback i do the normal thing of "oh, i'm receiving audio messages with a different codec, i guess i should flush, create, and initialize a new AudioDecoder".
are we saying that there needs to be an audio "sequence start" for all Enhanced FourCC codecs, whether the codec requires initialization data or not?
(*) today's project is adding Enhanced RTMP Audio to my JavaScript. i did the C++ side yesterday. :)
from enhanced-rtmp.
if i had my 'drothers, i'd define Enhanced RTMP to use the Opus native format that doesn't require a sequence header at all.
Another possible route is for Opus audio data to always be in the Opus format. The Opus sequence header (Identification Header) is optional. The ID header would not be wrapped by an Ogg page. @BtbN Does ffmpeg ever have this hybrid mode? Perhaps when handling Matroska (MKV), WebM or mp4?
from enhanced-rtmp.
I'm not sure, I never looked at that code, and it's not immediately obvious to me.
from enhanced-rtmp.
As far as I can tell the "ID header" is primarily used to signal the channel layout and number of channels, which in ERTMP can be handled by AudioPacketType.MultichannelConfig
.
In FFmpeg the header is treated as the "extradata" for the codec and required to correctly configure the libopus decoder for channel counts above 21. If we omit this header the extradata would have to be reconstructed like is done for MP42.
I don't know how webcodecs would handle configuration for channel counts above 2.
Edit: Also see https://opus-codec.org/docs/opus_api-1.5/group__opus__multistream.html#details
Edit 2: Based on further reading of the specification it seems that the channel map does not necessarily correspond to the channel layout, and would still have to be signaled for multi-channel audio. I seems to me that the easiest way to accomplish this is to simply send the OpusHead
packet as the SequenceStart
, but consider it optional for stereo or mono audio.
Footnotes
from enhanced-rtmp.
if initialization data is ever sent, and it is necessary for some decoding situations (like >2 channels), then to avoid ambiguity (especially around stream publish/unpublish/republish) a SequenceStart should probably be required every time.
it's possible that a WebCodecs AudioDecoder can use the "channels" config property to be told how many channels there are without needing an initialization data blob. if so, and we were to use Opus native format instead of Ogg, we'd need to parse the init data in client code rather than passing it directly to the decoder.
from enhanced-rtmp.
Nice catch, it's fine to only send identification header. We will reword this section to make it clear.
from enhanced-rtmp.
Another possible route is for Opus audio data to always be in the Opus format. The Opus sequence header (Identification Header) is optional.
in this case i'm assuming you mean that an RTMP Audio SequenceStart message is optional, its payload (if any) is not needed to initialize a decoder, and (for WebCodecs at least) its payload-if-any would not be passed to the decoder (since doing so signals a WebCodecs AudioDecoder to expect coded media to be in Ogg format).
from enhanced-rtmp.
in this case i'm assuming you mean that an RTMP Audio SequenceStart message is optional, its payload (if any) is not needed to initialize a decoder
Correct assumption for what is on the wire for RTMP. As far as what to do with the payload that would be solution dependent.
from enhanced-rtmp.
and "optional" here meaning that a forwarder could drop an audio SequenceStart entirely and the coded frames would still be decodable.
from enhanced-rtmp.
Related Issues (20)
- connect response doesn't indicate support for Enhanced RTMP HOT 9
- please clarify meaning and semantics of PacketTypeMPEG2TSSequenceStart HOT 4
- PacketTypeMetadata SHOULD or MUST come before the video sequence it affects? HOT 5
- fourCcList description should explicitly state support for receiving those codecs HOT 1
- fourCcList definition should provide for a wildcard HOT 4
- consider enabling Discussions on this repo HOT 1
- Additional Audio Codec: FLAC HOT 7
- enchanced HOT 1
- Hakiiii
- Support for QUIC? HOT 3
- Hi HOT 1
- Support CodecID 12 for HEVC HOT 6
- Fgvh
- AudioPacketType.SequenceEnd is not defined HOT 11
- Standardized way to signal audio track language? HOT 2
- FLAC sequence start message should include a "fLaC" marker. HOT 1
- latest enhanced-rtmp-v2.pdf has an improperly framed page HOT 1
- Typo in ExAudioTagBody section HOT 1
- No way of retrieving the metadata for multitrack streaming HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from enhanced-rtmp.