PacketTypeMetadata SHOULD or MUST come before the video sequence it affects? about enhanced-rtmp HOT 5 CLOSED

given that the payload of PacketTypeMetadata is a sequence of pairs (key:string, value:any) (with only one defined key so far), i can imagine a potential intent that subsequent PacketTypeMetadata messages would override/supersede only the named metadata items, rather than all of them, to give functionality similar to @setDataFrame. if that's the case, that expands and complicates the semantics of this message, and should be called out and described precisely.

from enhanced-rtmp.

Heya just saw this, thank you so much for all the great feedback here and in other issues. It's like you have done this before;). Stay tuned for multiple responses.

from enhanced-rtmp.

my recommendation on how PacketTypeMetadata should work is:

1. it MUST come before the start of the sequence to which it applies
2. only one of them is in force at a time
3. a new one completely supersedes and replaces a previous one
4. it MUST be followed immediately by a sequence start message (if needed by the codec) and a key frame / IDR -- that is, receipt of PacketTypeMetadata implies a decoder reset including resetting its start/init data

the last point simplifies the logic for live forwarders, particularly ones that want to do a "fast start and catch up" for a new subscriber that joins between in that critical window, since otherwise the old metadata, old start/init, and any frames to be replayed need to be remembered along with the new metadata, and then cleared out if/when the new start/init arrives. the alternative isn't super onerous, but the decoder and forwarder semantics would need to be more complicated than probably ever needed for practical uses of this message.

EDIT: recommendation withdrawn after learning more about this message's intended use.

from enhanced-rtmp.

on the understanding that PacketTypeMetadata messages can arrive at any point and each applies to messages following it (potentially including sequence start/init messages, keyframes, and predicted frames), i implemented the following heuristic in my server (an SFU/live-forwarder):

• keep "Before Sequence Start" and "Latest" metadata messages, each initialized to empty
• on receiving a PacketTypeMetadata message, set "Latest" to that message
• on receiving a Sequence Start message, remember the Sequence Start message; then if there's a non-empty "Latest" metadata message, set the "Before Sequence Start" metadata message to the current Latest, and clear Latest.
• when a client subscribes to a stream, send the "Before Sequence Start" metadata message if not empty, then the Sequence Start message if any, then the "Latest" metadata message if not empty.

this accounts for a PacketTypeMetadata message that needs to be known before processing a Sequence Start, as well as allowing the metadata to be updated after the sequence starts. in cases where there's ever only one PacketTypeMetadata in a stream, this retains the original order of the PacketTypeMetadata vs Sequence Start messages as sent by the publisher.

from enhanced-rtmp.

great discussion here. We have added some language to clarify the current behavior. You will see it in the next drop of the spec. Closing the issue...

from enhanced-rtmp.