Code Monkey home page Code Monkey logo

Comments (5)

ahmed-adly-khalil avatar ahmed-adly-khalil commented on May 20, 2024

@leewardbound would you be able to have a look? Thank you so much

from ion.

leewardbound avatar leewardbound commented on May 20, 2024

hey, so this is a familiar issue, i think we used to have some docs/examples about this but i'm not sure where they are right now --

the issue is related to how webrtc works, basically UDP ports need to be open (but not load balanced or mapped to different host ports). this is tricky on kubernetes. in config.toml, you'll see that we have some UDP port range configured, usually 5000-5200 is the default (which allows about 200 participants i believe?). when a webrtc client (like pion or ion-sfu) sends an offer/answer packet, it includes a list of suggested IPs and ports for the connection to be opened. you need more than just nat1to1={public_ip} in k8s, because a) this doesn't open the proper ports, and b) the public IP will actually depend on which node the pod is scheduled to run on, which you can't anticipate when you are setting up config.toml.

my preferred choice to workaround this is to use a DaemonSet instead of a Deployment (assuring 1 ion-sfu is scheduled on each node), and to use HostNetwork: true so that the pod is able to a) discover its own IP address, and b) open the 5000-5200 UDP ports, without listing them 1-by-1 in the Ports section as would normally be necessary in k8s, which doesn't support opening port ranges.

i've never deployed ion (the whole app) on k8s, just ion-sfu, but here's the config i've used as recently as ion-sfu:1.9, i haven't really updated my ion-sfu projects in a while:

---
apiVersion: v1
kind: Service
metadata:
  name: sfu
  labels:
    name: sfu
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9123"
spec:
  type: ClusterIP
  ports:
    - name: grpc
      port: 50051
      targetPort: grpc
    - name: jsonrpc
      port: 7000
      targetPort: jsonrpc
  selector:
    app: sfu
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sfu
spec:
  selector:
    matchLabels:
      name: sfu
  template:
    metadata:
      labels:
        name: sfu
        app: sfu
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      terminationGracePeriodSeconds: 30
      containers:
      - name: sfu
        image: pionwebrtc/ion-sfu:v1.9.4-allrpc
        command: ["/usr/local/bin/sfu", "-c", "/configs/sfu.toml", "-gaddr", ":50051", "-jaddr", ":7000", "-maddr", ":9123"]
        ports:
          - name: grpc
            containerPort: 50051
          - name: jsonrpc
            containerPort: 7000
          - name: metrics
            containerPort: 9123

from ion.

ahmed-adly-khalil avatar ahmed-adly-khalil commented on May 20, 2024

@leewardbound Thank you so much, i tried deploying ion-sfu beside the rest of ion pods and didn't get much luck, maybe because both sfu app/container inside ion has different deplyoment requirements from the independent ion-sfu.

currently i'm getting the below 2 lines in both signal and sfu pods and trying to wrap my head around, hopefully tonight.

[2022-01-16 21:54:48.548]  INFO nats-discovery.Client: [client.go:91] [client.(*Client).handleNatsMsg] node.delete: dc1.sfu-EhR9Kq 
[2022-01-16 21:54:48.548]  INFO default: [node.go:138] [ion.(*Node).handleNeighborNodes] Service down: rtc node id => [sfu-EhR9Kq]

one observation that the docker version that works locally has the following ports setup

  sfu:
    image: pionwebrtc/ion:latest-sfu
    build:
      dockerfile: ./docker/sfu.Dockerfile
      context: .
    volumes:
      - "./configs/docker/sfu.toml:/configs/sfu.toml"
    ports:
      - "5000:5000/udp"
      - 3478:3478

which means, theoretically at least, it can work with only 3478 and 5000 if configured correctly. That's why i'm trying to get the following service ports to work:

apiVersion: v1
kind: Service
metadata:
  name: sfu
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: tcp
      port: 3478
      targetPort: 3478
    - protocol: UDP
      name: udp
      port: 5000
      targetPort: 5000
  selector:
    app: sfu

I will continue digging and post my findings.

@Sean-Der @adwpc by chance do you have ideas around here?

Thank you everyone so much

from ion.

ahmed-adly-khalil avatar ahmed-adly-khalil commented on May 20, 2024

comparing the sfu logs between k8s on linode and local docker:

k8s on linkdoe:

[2022-01-16 23:39:59.718] [INFO] [peer.go:223] => PeerLocal got answer logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.718] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:3056717505 1 udp 2122262783 2001:1970:5692:c200:e93e:77c8:131f:ca35 57113 typ host generation 0 ufrag 8pjS network-id 2 network-cost 10 
[2022-01-16 23:39:59.718] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.718] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=checking v=1
[2022-01-16 23:39:59.768] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:4081163164 1 udp 1685987071 24.150.74.40 49344 typ srflx raddr 192.168.0.2 rport 49344 generation 0 ufrag 8pjS network-id 1 network-cost 10 
[2022-01-16 23:39:59.768] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.815] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:4172427825 1 tcp 1518283007 2001:1970:5692:c200:e93e:77c8:131f:ca35 9 typ host tcptype active generation 0 ufrag 8pjS network-id 2 network-cost 10 
[2022-01-16 23:39:59.815] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.816] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:345893049 1 tcp 1518214911 192.168.0.2 9 typ host tcptype active generation 0 ufrag 8pjS network-id 1 network-cost 10 
[2022-01-16 23:39:59.817] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:40:00.871] DEBUG default: [service.go:346] [sfu.(*SFUService).Signal] [C=>S] description: offer v=0
o=- 562677169881253040 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS 6d704ff9-279a-41e4-b1dd-f9d0b802c1e9
m=application 54207 UDP/DTLS/SCTP webrtc-datachannel

local docker

 

[2022-01-16 23:38:21.028] [INFO] [peer.go:223] => PeerLocal got answer logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.029] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=checking v=1

[2022-01-16 23:38:21.029] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:3056717505 1 udp 2122262783 2001:1970:5692:c200:e93e:77c8:131f:ca35 59848 typ host generation 0 ufrag 9P/6 network-id 2 network-cost 10 

[2022-01-16 23:38:21.030] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.030] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:1511920713 1 udp 2122194687 192.168.0.2 61493 typ host generation 0 ufrag 9P/6 network-id 1 network-cost 10 

[2022-01-16 23:38:21.031] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.126] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=connected v=1

[2022-01-16 23:38:22.403] DEBUG default: [service.go:346] [sfu.(*SFUService).Signal] [C=>S] description: offer v=0

o=- 8127501359152418552 3 IN IP4 127.0.0.1

the docker version the ICE switch to connected right after checking while on k8s this doesn't happen. I tried also using STUN server with no luck, proceeding with digging ...

from ion.

ahmed-adly-khalil avatar ahmed-adly-khalil commented on May 20, 2024

looking at the following offer and answer, what i can see both has 127.0.0.1 which i anticipate to be the issue, still tring to find ways around that

{
    "type": "offer",
    "sdp": "v=0\r\no=- 2711940938522023088 3 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS 3a8fffb3-cf06-44f8-a10c-144e209991bb\r\nm=application 57934 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 24.150.74.40\r\na=candidate:3194784207 1 udp 2122262783 2001:1970:5692:c200:bc83:7b6d:313e:f371 61453 typ host generation 0 network-id 2 network-cost 10\r\na=candidate:1511920713 1 udp 2122194687 192.168.0.2 57934 typ host generation 0 network-id 1 network-cost 10\r\na=candidate:4081163164 1 udp 1685987071 24.150.74.40 57934 typ srflx raddr 192.168.0.2 rport 57934 generation 0 network-id 1 network-cost 10\r\na=candidate:4042224959 1 tcp 1518283007 2001:1970:5692:c200:bc83:7b6d:313e:f371 9 typ host tcptype active generation 0 network-id 2 network-cost 10\r\na=candidate:345893049 1 tcp 1518214911 192.168.0.2 9 typ host tcptype active generation 0 network-id 1 network-cost 10\r\na=ice-ufrag:0/zJ\r\na=ice-pwd:+9L5W9JtXWOfH+NRU7kzFlHr\r\na=ice-options:trickle\r\na=fingerprint:sha-256 F6:D4:47:2E:60:11:71:0A:47:E4:66:B0:8E:AB:9B:67:03:BD:FB:56:0F:F7:93:A9:8E:8F:50:3B:A6:DB:64:36\r\na=setup:actpass\r\na=mid:0\r\na=sctp-port:5000\r\na=max-message-size:262144\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:0/zJ\r\na=ice-pwd:+9L5W9JtXWOfH+NRU7kzFlHr\r\na=ice-options:trickle\r\na=fingerprint:sha-256 F6:D4:47:2E:60:11:71:0A:47:E4:66:B0:8E:AB:9B:67:03:BD:FB:56:0F:F7:93:A9:8E:8F:50:3B:A6:DB:64:36\r\na=setup:actpass\r\na=mid:1\r\na=extmap:1 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 urn:3gpp:video-orientation\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=sendonly\r\na=msid:3a8fffb3-cf06-44f8-a10c-144e209991bb 2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=ssrc:2659409037 cname:TiUOWZgap2jBYNbL\r\na=ssrc:2659409037 msid:3a8fffb3-cf06-44f8-a10c-144e209991bb 2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\na=ssrc:2659409037 mslabel:3a8fffb3-cf06-44f8-a10c-144e209991bb\r\na=ssrc:2659409037 label:2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\n"
}




{
    "type": "answer",
    "sdp": "v=0\r\no=- 3940562069726591883 1642473100 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS\r\na=ice-lite\r\nm=application 5000 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 45.79.247.128\r\na=candidate:706165708 1 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=candidate:706165708 2 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=candidate:706165708 1 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=ice-ufrag:NGrczxsIXdECNyjl\r\na=ice-pwd:dEenZietQjixEhwubLiIGYBsXjueWNHA\r\na=fingerprint:sha-256 90:EA:F3:AC:B3:3E:B7:30:D4:EB:DC:86:F5:55:0B:CC:F8:1C:E7:2C:B3:5B:2E:B4:F4:EF:3C:89:DF:44:80:A7\r\na=setup:active\r\na=mid:0\r\na=sctp-port:5000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:NGrczxsIXdECNyjl\r\na=ice-pwd:dEenZietQjixEhwubLiIGYBsXjueWNHA\r\na=fingerprint:sha-256 90:EA:F3:AC:B3:3E:B7:30:D4:EB:DC:86:F5:55:0B:CC:F8:1C:E7:2C:B3:5B:2E:B4:F4:EF:3C:89:DF:44:80:A7\r\na=setup:active\r\na=mid:1\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\n"
}

from ion.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.