Connection timed out about amazon-kinesis-video-streams-webrtc-sdk-c HOT 17 CLOSED

@mdonahoe oof that is annoying.

Let me give you some good places to debug, and I think we need more verbose logs around ICE sending packets that would be a lot of help also.

createSocket(): connect() failed with errno Connection timed out looks promising also. I will get you the exact line that happens on.

I am getting on the bus now, but will get more stuff to go off in ~2 hours!

I am also making the cross-compile a lot easier. I hope to land this tonight. You can just set your CC/CXX/AR and the destination host and it will build everything as you expect.

So in the future you should need zero code changes, and will have it running in travis so we are sure it doesn't regress.

from amazon-kinesis-video-streams-webrtc-sdk-c.

@mdonahoe that looks better to me! If both your candidates are in the same LAN I am not sure why ICE is failing.

If you disable trickle (by passing FALSE to createSampleConfiguration) you can do a printf on message.payload here could be useful to confirm gathering worked properly!

You can also printf pSignalingMessage->payload here and look at incoming candidates.

IMO all that state machine logging is pretty useless. We should move that up another level, and give you log message when ICE messages are sent. I will try to get that in tonight.

Sorry for the rough edges, really appreciate the debugging and will get that fixed up!

from amazon-kinesis-video-streams-webrtc-sdk-c.

@mdonahoe Can you try disabling using TURN?

Since the ICEAgent is single threaded if your TURN connection hangs I think it could block everything. I will confirm this

Would you also mind sharing your ICE candidates after you disable TURN. If you are running in the same network I want to make sure they are properly generated.

from amazon-kinesis-video-streams-webrtc-sdk-c.

I think I did it correctly (replacing the second TRUE with FALSE in the createSampleConfiguration(...) call) but I get a different error this time.

[...snip...]
2020-01-15 04:04:11 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.12.1
2020-01-15 04:04:11 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.10.1
2020-01-15 04:04:11 DEBUG   onIceCandidateHandler(): ice candidate gathering finished
2020-01-15 04:04:11 DEBUG   stepStateMachine(): State Machine - Current state: 0x0000000000000002, Next state: 0x0000000000000004
2020-01-15 04:04:11 DEBUG   executeCheckConnectionIceAgentState(): ice candidate pair count 24
2020-01-15 04:04:11 DEBUG   stepIceAgentStateMachine(): Ice agent state changed from ICE_AGENT_STATE_GATHERING to ICE_AGENT_STATE_CHECK_CONNECTION.
2020-01-15 04:04:11 INFO    onConnectionStateChange(): New connection state 2
2020-01-15 04:04:12 DEBUG   lwsWssCallbackRoutine(): Client is writable
2020-01-15 04:04:16 WARN    iceAgentSendPacket(): No valid ice candidate pair available to send data
2020-01-15 04:04:21 DEBUG   stepStateMachine(): State Machine - Current state: 0x0000000000000004, Next state: 0x0000000000000080
2020-01-15 04:04:21 ERROR   executeFailedIceAgentState(): IceAgent failed with 0x5a00000d
2020-01-15 04:04:21 DEBUG   stepIceAgentStateMachine(): Ice agent state changed from ICE_AGENT_STATE_CHECK_CONNECTION to ICE_AGENT_STATE_FAILED.
2020-01-15 04:04:21 INFO    onConnectionStateChange(): New connection state 5
2020-01-15 04:04:21 DEBUG   freeSampleStreamingSession(): Freeing streaming session with peer id: nclvqq5x4wrt6um6yusynn

from amazon-kinesis-video-streams-webrtc-sdk-c.

Ok I made your suggested changes (no trickleIce, added printfs), but no combo I've tried so far has resulted in a successful connection.

For this test, the candidates weren't on the same LAN.

Here are the candidates I see printed.

{"candidate":"candidate:1495340180 1 udp 2122260223 192.168.1.76 52251 typ host generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"1","sdpMLineIndex":1}
{"candidate":"candidate:396552292 1 tcp 1518280447 192.168.1.76 9 typ host tcptype active generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0}
{"candidate":"candidate:2939252032 1 udp 1686052607 76.210.142.73 34753 typ srflx raddr 192.168.1.76 rport 34753 generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0}
{"candidate":"candidate:2939252032 1 udp 1686052607 76.210.142.73 52251 typ srflx raddr 192.168.1.76 rport 52251 generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"1","sdpMLineIndex":1}
{"candidate":"candidate:396552292 1 tcp 1518280447 192.168.1.76 9 typ host tcptype active generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"1","sdpMLineIndex":1}
{"candidate":"candidate:1495340180 1 udp 2122260223 192.168.1.76 34753 typ host generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0}
{"candidate":"candidate:1883879534 1 udp 41885695 54.244.90.62 57075 typ relay raddr 76.210.142.73 rport 34753 generation 0 ufrag R8Mm network-id 1 network-cost 10","sdpMid":"0","sdpMLineIndex":0}
{"candidate":"candidate:1007924187 1 udp 41885439 34.215.13.39 54579 typ relay raddr 76.210.142.73 rport 52251 generation 0 ufrag R8Mm network-id 1 network-cost 10", "sdpMid":"1","sdpMLineIndex":1}
{"candidate":"candidate:1883879534 1 udp 41885695 54.244.90.62 53824 typ relay raddr 76.210.142.73 rport 52251 generation 0 ufrag R8Mm network-id 1 network-cost 10", "sdpMid":"1","sdpMLineIndex":1}
{"candidate":"candidate:1007924187 1 udp 41885439 34.215.13.39 64753 typ relay raddr 76.210.142.73 rport 34753 generation 0 ufrag R8Mm network-id 1 network-cost 10", "sdpMid":"0","sdpMLineIndex":0}

Then the final message payload is this.

{
   "type":"answer",
   "sdp":"v=0\r\no=- 717893088 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS myKvsVideoStream\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 127.0.0.1\r\na=candidate:4 1 udp 658205 50.236.240.214 29068 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:2 1 udp 658881 192.168.10.1 39005 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:1 1 udp 658881 192.168.5.197 39439 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:0 1 udp 658881 192.168.12.1 39462 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=msid:myKvsVideoStream myAudioTrack\r\na=ssrc:1772248001 cname:G0(IHYv2#P_pdYPD\r\na=ssrc:1772248001 msid:myKvsVideoStream myAudioTrack\r\na=ssrc:1772248001 mslabel:myKvsVideoStream\r\na=ssrc:1772248001 label:myAudioTrack\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:TSJO\r\na=ice-pwd:[...snip...]\r\na=fingerprint:sha-256 [...snip...]\r\na=setup:active\r\na=mid:0\r\na=sendrecv\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtcp-fb:111 nack\r\nm=video 9 UDP/TLS/RTP/SAVPF 108 109\r\nc=IN IP4 127.0.0.1\r\na=candidate:4 1 udp 658205 50.236.240.214 29068 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:2 1 udp 658881 192.168.10.1 39005 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:1 1 udp 658881 192.168.5.197 39439 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:0 1 udp 658881 192.168.12.1 39462 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=msid:myKvsVideoStream myVideoTrackRTX\r\na=ssrc-group:FID 1556882299 618215704\r\na=ssrc:1556882299 cname:G0(IHYv2#P_pdYPD\r\na=ssrc:1556882299 msid:myKvsVideoStream myVideoTrack\r\na=ssrc:1556882299 mslabel:myKvsVideoStream\r\na=ssrc:1556882299 label:myVideoTrack\r\na=ssrc:618215704 cname:G0(IHYv2#P_pdYPD\r\na=ssrc:618215704 msid:myKvsVideoStream myVideoTrackRTX\r\na=ssrc:618215704 mslabel:myKvsVideoStreamRTX\r\na=ssrc:618215704 label:myVideoTrackRTX\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:TSJO\r\na=ice-pwd:qQ2DZ@^WKSN%kp6MQTf1^sCo\r\na=fingerprint:sha-256[...snip...]\r\na=setup:active\r\na=mid:1\r\na=sendrecv\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:108 H264/90000\r\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\r\na=rtpmap:109 rtx/90000\r\na=fmtp:109 apt=108\r\na=rtcp-fb:108 nack\r\n"
}

I am concerned that the wrong IP address is being chosen, since the clients aren't on the same LAN...

from amazon-kinesis-video-streams-webrtc-sdk-c.

Any tips on how can I tell which candidate is getting selected and why?

I see several different IP addresses in the final message payload, and I am not sure how to interpret it.

from amazon-kinesis-video-streams-webrtc-sdk-c.

Oh that was my mistake @mdonahoe I thought they were on the same LAN. So I think this is the issue (correct me if I am wrong)

• host candidates and STUN won't work for you, so we need to use TURN.
• TURN via TCP+TLS doesn't work for you either. It seems that it hangs trying to contact our TURN servers.

Do you usually do TURN via UDP or just TCP? Currently we only do TCP/TLS (we just started there)

Does this device have any packet filtering/firewall rules/any network things that might help uncover the issue?

from amazon-kinesis-video-streams-webrtc-sdk-c.

I discovered that one of my networks is behind a symmetric NAT. Given the problem with TURN on my device, I think you were right.

However, attempting to stream between two machines (desktop and arm device), over LAN with TURN disabled, I still see this error on the video sender (arm device)

2020-01-17 02:15:44 WARN onInboundPacket(): decryptSrtcpPacket failed with 0x5b000001

Is that a relevant problem, or just a red herring?

Also, is there a timeline for supporting UDP-based TURN? Alternatively, is there a way to specify different TURN servers?

from amazon-kinesis-video-streams-webrtc-sdk-c.

@mdonahoe

The decryptSrtcpPacket is a nasty problem. We are seeing a similar issue with a Jetson TX2, it is encrypting/decrypting media traffic incorrectly. Things are fine coming out of DTLS though. I will have the developer working on this cut a ticket and we can track there.

UDP-based TURN I would say is in the short term, but I don't have any hard deadlines. It isn't terribly difficult, just not blocking anyone currently. If this is blocking you from KVS would love to escalate/make this happen for you sooner!

createPeerConnection allows you to pass in any TURN servers you want. We tried to make it feel like using the RTCPeerConnection in the browser as much as possible. The code in samples/ is all just boilerplate around the normal APIs.

from amazon-kinesis-video-streams-webrtc-sdk-c.

Hello @mdonahoe ,

I am able to reproduce this issue on ARM64 and it looks like there is an authentication failure while decrypting the SRTCP packet. I have a ticket tracking this error and am working to find out what/where the issue could be.

from amazon-kinesis-video-streams-webrtc-sdk-c.

I'm also testing on a TX2, so it makes sense we'd be seeing the same problem!

Does your TX2 have TURN connection timeouts with TCP+TLS as well?

Is UDP-based TURN something that the SDK supports but AWS cloud doesn't yet? I'm happy to wait for a cloud update and use other TURN servers or TCP+TLS in the meantime.

from amazon-kinesis-video-streams-webrtc-sdk-c.

I do not recall seeing TURN connection timeouts.

The decrypt error is persistent in Ubuntu 16.04, but Ubuntu 18.04 works perfectly fine. Have you tried running the sample on Ubuntu 18.04?

@Sean-Der will have more details on UDP based TURN.

from amazon-kinesis-video-streams-webrtc-sdk-c.

@mdonahoe Unfortunately TCP (or TLS) is the a client restriction.

If you are feeling brave you can flip to UDP here but I doubt things will work.

The code was written to do all four different transport types, we just didn't have the time to finish.

I don't know if TURN works with our 16.04, I know P2P+DataChannels works though! If you upgrade to 18.04 everything works. We have a customer who is actively building with audio+video over TURN. We are working on fixing 16.04, but don't have any estimates yet.

The bug is in libsrtp. When we have an upstream issue will make sure to post it here!

from amazon-kinesis-video-streams-webrtc-sdk-c.

Hello @mdonahoe , I have pushed a change into a branch which allows the samples to run on Ubuntu 16.04.
https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/tree/srtp_stuff

Feel free to use this branch on your Ubuntu 16.04 system and let me know if you have any questions. :)

from amazon-kinesis-video-streams-webrtc-sdk-c.

Thanks @disa6302! Your branch fixes the decrypt error, and I was able to get streaming working with two machines on the same LAN.

I still had to disable TURN to get it working, as my TX2 still hangs on that as before.

@Sean-Der, I tried passing KVS_SOCKET_PROTOCOL_UDP into createTurnConnection(...) instead of TCP, but did not work, as you predicted. I got this error TurnConnection in TURN_STATE_FAILED due to 0x5a000020.

How do I interpret those error messages?

Here's a longer log snippet:

2020-01-24 23:45:42 DEBUG   turnConnectionStepState(): TurnConnection state changed from TURN_STATE_NEW to TURN_STATE_GET_CREDENTIALS
[... skipped 200 lines of ICE stuff ...]
2020-01-24 23:45:47 ERROR   turnConnectionStepState(): operation returned status code: 0x5a000020
2020-01-24 23:45:47 DEBUG   turnConnectionStepState(): TurnConnection state changed from TURN_STATE_GET_CREDENTIALS to TURN_STATE_FAILED
2020-01-24 23:45:47 ERROR   turnConnectionTimerCallback(): operation returned status code: 0x5a000020
2020-01-24 23:45:47 ERROR   timerQueueExecutor(): operation returned status code: 0x5a000020
2020-01-24 23:45:47 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.12.1
2020-01-24 23:45:47 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.14.94
2020-01-24 23:45:47 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.12.1
2020-01-24 23:45:47 DEBUG   fromGatheringIceAgentState(): checking local candidate type srflx, ip 192.168.14.94
2020-01-24 23:45:47 WARN    turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x5a000020

My use case is likely going to require a TURN server (or perhaps an SFU) since I think my TX2s are always going to be attached to LTE modems and presumably behind symmetric NATs. I also suspect that UDP will perform better than TCP over a wireless link.

Is there any AWS issue I can track for progress on UDP TURN support?

And are you planning to launch an SFU or something like it? Please let me know if there is a better channel for asking product questions.

Thanks!

from amazon-kinesis-video-streams-webrtc-sdk-c.

Hi @mdonahoe, My name is Girish and I am the product manager for Kinesis Video Streams. You can send your usecase and details on your organization to me at [email protected] and we would be happy to help.

from amazon-kinesis-video-streams-webrtc-sdk-c.

Thanks @garishsood, I sent an email.

Since my original timeout issue was solved already by updates to master, I am going to close this issue and open a new one just for UDP TURN support. Thanks everyone!

from amazon-kinesis-video-streams-webrtc-sdk-c.