inlivedev / sfu Goto Github PK

Just found out that when a race condition of renegotiation happens between the client and SFU after adding new tracks, we always let the SFU do the renegotiation first and not continue the renegotiation on the client side. This makes the new tracks added to the client not trigger the ontrack event on the SFU side. Ideally, when the SFU finish the renegotiation, the SFU let the client knows that they can continue the previous renegotiation.

I will try to create a test to verify this.

Another idea for perfect renegotiation based on conversations on the Pion Slack channel is to separate the publisher and subscriber connection. This will prevent a race condition when both sides add a new track because it will trigger the renegotiation on different connections.

one room on different server

When a client connected to the room then suddenly their internet is drop or disconnected, their connection will be failed after a timeout but the other clients in the room is not receiving track removed event and it make the client video still exists even the client already disconnected.

To reproduce:

1. Join into a room with other clients.
2. After connected, try to turn off wifi or or any internet connection.
3. The client will disconnect
4. Other clients in the room still see the video but in freezing state.

There is a reported issue that the onTrack event won't be triggered when the remote client is using Firefox and sending simulcast track with H264. Similar issue also exist on Chrome mobile.

Based on this conversation the issue may be because:

• The Firefox is not support simulcast with H264, with this we must allow to configure the room with specific codec to use.
• Chrome won't send RID track with lower than 180 resolution, when this happen the SFU will never publish the track because it's waiting the low track to receive before publish it.

When a participant publishes a track to a room, they will call peerConnection.AddTrack(), and this will trigger thepeerConnection.OnTrack() event on the SFU side. We face an issue once more than one type of video is published in a room, for example, once a screensharing stream is published in a room. How do we know which track is from the camera and which is from screensharing? So far, I haven't found a spec from WebRTC that specifies how to tell the difference between video sources. The media track has label property that can be used for the source, but I don't think it's available to use in Pion.

To make it easier to identify the tracks in the room, the user it belong, or the input device of that track, we need to change how we publish and subscribe to the media tracks. This is what I have that I think can help us with that need:

When a participant adds a track:

1. By default, when a participant joins the room, they're not subscribed to any tracks in the room.
2. The participant can subscribe to all available tracks in the room by calling client.SubscribeAllTracks()
3. They can also call room.GetTracks() to get a list of available tracks to subscribe to and subscribe to each track that they like by calling client.SubscribeTrack(MSID) or client.SubscribeTracksByClient(clientid)
4. When a participant adds a track to the peerConnection, the SFU will trigger the peerConnection.OnTrack() event on the pair peerConnection on the SFU side. That event then will make the pair client trigger OnTrackAdded(ssrc), which is only received by the participant who previously added it.
5. Once the publisher receives the OnTrackAdded() event, the publisher must set the metadata of that track to include the source of that track. This can be done by calling client.SetTrackMetadata(SSRC, metadata)
6. Once the track metadata is set, the SFU will publish the track to the room and let the other participants know that a new track is available to subscribe to.

When a participant subscribes to a track

1. The participant will call room.GetTracks() to know what tracks to subscribe to. Each track will contains MSID, clientID, source type(camera, screen, or microphone), and added timestamp
2. The participant then calls client.SubscribeTracks([]MSID) to subscribe to each track they like.
3. The SFU will publish those tracks to the participant
4. If a new track is published by another participant in a room, the client.OnTrackAvailable(trackDetail) will trigger, and the participant can subscribe by calling the same client.SubscribeTracks([]MSID) if they want to.
5. Track detail will have information about clientid, msid, ssrc, kind, source type, codec of original track.

By doing this way, we can manage the publishing and subscription to tracks easier. Either a participant joins as a publisher or only as a subscriber, or as a passive participant.