Code Monkey home page Code Monkey logo

webrtc-hotwire-rails's Introduction

WebRTC + Hotwire + Ruby on Rails

An basic video chat app using the WebRTC Perfect Negotiation pattern, a sprinkling of Hotwire (mainly Turbo Streams & Stimulus), and backed by Ruby on Rails.

How does it work?

The Stimulus room controller handles Enter button click. It gets the user's local audio and video and feeds them into the local video element. It also starts Action Cable subscriptions specific to the current room: one for communicating WebRTC messages: the Signaller; and one for clients to ping others: the RoomSubscription. Once connected, the room channel broadcasts the presence of this new client. Each connected client then "greets" this newcomer by calling the greet action on the room channel and specifying to and from fields.

The greet action broadcasts specifically to the newcomer, a Turbo Stream update which appends a media element representing the remote client. It's important we do this first so that it's ready for incoming streams. The medium controller messages the room controller to notify that it has been added. This begins the WebRTC negotiation to form a connection between the two clients.

The WebRTC negotiation is quite complex, and even though things are required to happen in a particular order, responses are triggered asynchronously, making it tricky to get right. This is where the WebRTC Perfect Negotiation pattern comes in. We won't go into it too much here, as it's covered well elsewhere; but for the purpose of this description, (deep breath), Session Description Protocol (SDP) descriptions and Interactive Connectivity Establishment (ICE) candidates are exchanged over the Signaller. The negotiation is "perfect" as it ensures that the RTCPeerConnection is in the correct state for setting descriptions, and avoids collision errors by having one client be "polite" and other "impolite"—the polite one backs down in the case of a collision. Anyway, the code for this has been mostly lifted from the spec and handled by the Signaller/WebrtcNegotiation. When creating the negotition, the room controller listens for track events on the RTCPeerConnection, so it can start streaming from the remote client.

Once the negotiation has been created, the newcomer client notifies the other client by reciprocating the greeting i.e. calling greet on the room channel. This triggers the same process as above on the other client: appending a media element via a Turbo Stream update, and starting a WebRTC negotiation. This time though, instead of a greeting (which would be very polite, but awkward and endless!), the client starts streaming to the other client, by adding media tracks to the connection. This kicks off the complex exchange of SDP descriptions and ICE candidates, and once either is received by the newcomer, it can start the streaming process from its own stream to the other client. Now all clients are streaming to and from each other.

Finally, when a client disconnects from the room channel, a Turbo Stream update removes the media element from the screen. The media controller broadcasts its removal so the room controller can clean up.

To summarise the flow:

  1. A newcomer broadcasts its presence to others in the room
  2. The connected clients greet this newcomer letting them know their ID
  3. A Turbo Stream update creates a video element on the newcomer's screen for each greeting it receives
  4. The newcomer creates a WebRTC negotiation and sends a greeting back to each of the other clients
  5. Turbo Stream updates create video elements on each of the other clients' screens
  6. WebRTC negotiations are created by each of the other clients, and they start streaming to the newcomer
  7. Reacting to negotiation activity (SDP descriptions and ICE candidate exchanges), the newcomer starts streaming to other clients

Browser Support?

This has only been tested in macOS Firefox/Chrome/Safari and iOS Safari.

TODO

  • Add "Leave" functionality + handle ICE candidate disconnections (rather than just closing the browser window)

License

Copyright © 2021+ Dom Christie and released under the MIT license.

webrtc-hotwire-rails's People

Contributors

dmke avatar domchristie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

webrtc-hotwire-rails's Issues

Mobile Safari 14.1 is unable to negotiate peer connection

Summary

Safari 14.1, on both macOS (observed since 11.3.1) and iOS (since 14.5), fail to negotiate a peer connection, when another peer joins the room, i.e. when Safari is the polite negotiator.

This was not a problem with the previous Safari version (14.0.3 on iOS 14.4.2).

(In an earlier report, I could only reproduce this with Mobile Safari, but not with Desktop Safari. This has now changed, and I can reliably reproduce this issue with a Desktop Safari as well. Of course, I didn't change anything on my system...)

Steps to reproduce

  1. Start the web application.
  2. Use a Mobile Safari 14.1 and create a room
    • copy the URL/UUID from the Rails log
  3. Using Mobile Safari, tap on "Enter".
    • you should see your camera picture
  4. Use any other (desktop) browser and visit the same room.
  5. Using the other browser, click "Enter".
    • you should see your camera picture
  6. After a few seconds, Mobile Safari logs "Negotiation failed after 11 retries".
    • (N.B. there's an off-by-one error, the RETRY_LIMIT is 10 ;-))

I've managed to reproduce this using XCode/Simulator v12.5 (961.1), albeit not as reliable as with an actual iPhone (I get connection errors all over the place, because I currently have only access to a Late 2013 MBP, which seems to not have quite enough resource to simulate an iPhone accessing a camera + running Safari Devtools in the background).

Details

The console log is produced in setDescription() in webrtc_negotiation.js:

async setDescription (description) {
try {
if (this.ignore(description)) return
await this.setRemoteDescription(description)
if (description.type === 'offer') {
await this.setLocalDescription(await this.peerConnection.createAnswer())
}
} catch (error) {
if (this.retryCount <= RETRY_LIMIT) {
this.initiateManualRollback()
this.retryCount++
} else {
console.error(`Negotiation failed after ${this.retryCount} retries`)

The actual error is InvalidStateError: Description type incompatible with current signalling state and is thrown while waiting for the this.peerConnection.setRemoteDescription():

async setRemoteDescription (description) {
this.isSettingRemoteAnswerPending = description.type === 'answer'
await this.peerConnection.setRemoteDescription(description) // SRD rolls back as needed
this.addCandidates()
this.isSettingRemoteAnswerPending = false
}

peerConnection.signalingState is have-local-offer at that time.

Additional information

The Safari release notes state near the end:

Changes in this release of Safari were included in the following Safari Technology Preview releases: 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122.

However, glancing over the TP release notes, nothing stands out to be an obvious cause for the negotiation to fail.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.