Comments (28)
Also, I find heap usage is growing unexpectedly.
Attaching video which shows, client is receiving the stream but heap keeps growing. I performed test on ARM device.
heap_grow_kvsMaster_1.10.2.mp4
from amazon-kinesis-video-streams-webrtc-sdk-c.
One more observation, heap size becomes stable after holding lot of space. I am wondering why heap keeps growing.
On my Intel + ubuntu(22.04) with 32 RAM, it works fine.
from amazon-kinesis-video-streams-webrtc-sdk-c.
Thank you for the details!
The memory is expected to grow for a few min after which it stablizes. Could you also confirm if there are any sample changes you have made or you ran a clean build post upgrading and then ran the sample?
from amazon-kinesis-video-streams-webrtc-sdk-c.
@disa6302 ,
Thanks for the prompt response.
The sample is intact. I have just used a printf statement to print video file index being sent.
when I run sample on Intel machine with 32 GB RAM, there heap doesn't grow as it grows for embedded device.
NOTE: I am monitoring heap with /proc/pid/smaps file.
Is there any buffer pool being maintained internally? I don't have much memory on embedded device. Is there any configuration which can to be used to avoid this heap growth which happen for few minutes.
from amazon-kinesis-video-streams-webrtc-sdk-c.
Can you please try with cmake flag ENABLE_KVS_THREADPOOL
OFF? https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/blob/master/CMakeLists.txt#L21
from amazon-kinesis-video-streams-webrtc-sdk-c.
I have switched ENABLE_KVS_THREADPOOL to OFF as mentioned, but still lots of thread gets created and I find only one thread is working and other are resting in process address space.
FYI, I have same effect after disabling the thread pool flag as well.
from amazon-kinesis-video-streams-webrtc-sdk-c.
According to your suggestion, there was no significant improvement in turning ENABLE_KVS_THREADPOOL to OFF. The actual memory of the WebRTC process is still growing significantly, and after live view, RSS will not decrease to the level before live view.
from amazon-kinesis-video-streams-webrtc-sdk-c.
@hassanctech ,
Hi, have you tested the above issues? If so, how long will it take to solve them and release a new version of SDK?
from amazon-kinesis-video-streams-webrtc-sdk-c.
@xiaomizhouya I have not been able to reproduce a regression between this version and a previous version, I will report back once I have some new information.
from amazon-kinesis-video-streams-webrtc-sdk-c.
@hassanctech Hi, Can you guys reproduce or make any progress on this issue? This issue is important to us.
from amazon-kinesis-video-streams-webrtc-sdk-c.
@xiaomizhouya We have made significant improvements for memory usage in our develop branch which we hope to release in the next month or so but do not have a date yet. Can you please try this you should see a significant improvement overall in memory: https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/tree/develop
from amazon-kinesis-video-streams-webrtc-sdk-c.
Hi @xiaomizhouya @naseebpanghal, I ran the latest release with valgrind and tested connecting and disconnecting from the JS viewer multiple times like you've suggested. I don't see a leak overall. It could be a possibility that something may not have been freed up until the end of the application:
==807792==
==807792== HEAP SUMMARY:
==807792== in use at exit: 0 bytes in 0 blocks
==807792== total heap usage: 227 allocs, 227 frees, 21,159 bytes allocated
==807792==
==807792== All heap blocks were freed -- no leaks are possible
==807792==
==807792== For lists of detected and suppressed errors, rerun with: -s
==807792== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
from amazon-kinesis-video-streams-webrtc-sdk-c.
@niyatim23
May I ask if you have observed the RSS of the webrtc process? In the testing steps I provided, after live view, the RSS of the webrtc process cannot return to the previous value. Moreover, based on the testing results I provided, after checking the live view three times, the RSS value increased from 5556KB to 14664KB, which seems unreasonable. Can you check the RSS value of the webrtc process during your testing? Will it continue to increase? Not just using the valgrind tool.
from amazon-kinesis-video-streams-webrtc-sdk-c.
@hassanctech I noticed that there have been new changes in the develop branch recently: [Fix MID-NTP macro definition]. Does this solve the memory leak issue? If so, I will conduct further testing on my end
from amazon-kinesis-video-streams-webrtc-sdk-c.
Hi @xiaomizhouya, I don't think that is related to this issue. We are actively investigating this issue, we will get back to you once we have something. Meanwhile, can you check if you can use valgrind for different stages of the application demonstrating the leak?
from amazon-kinesis-video-streams-webrtc-sdk-c.
@niyatim23 @hassanctech @disa6302
I have done some changes to restrict memory usage within RTP rolling buffer, within the application. Below is the patch
-#define DEFAULT_ROLLING_BUFFER_DURATION_IN_SECONDS 3
-#define HIGHEST_EXPECTED_BIT_RATE (10 * 1024 * 1024)
+#define DEFAULT_ROLLING_BUFFER_DURATION_IN_SECONDS 1
+#define HIGHEST_EXPECTED_BIT_RATE (2 * 1024 * 1024)
Although with above changes, I don't find any issue so far.
- But I want to understand what impact it can bring?
- Should I handle anything alongwith these changes?
@xiaomizhouya
You can also try above change your side. I hope that may help you.
from amazon-kinesis-video-streams-webrtc-sdk-c.
The develop branch has multiple changes that help with memory reduction, would suggest you clean build try that out.
These settings are according to your requirements. The buffer defines you changed affect the max rtp packets that can live in memory at a time. The develop branch makes this configurable in the application, so you would not have to change this value in the source
from amazon-kinesis-video-streams-webrtc-sdk-c.
The develop branch makes this configurable in the application, so you would not have to change this value in the source
For above statement, please let me know how to apply configuration during compile time of my application. Which CMakefile has these configurations.
from amazon-kinesis-video-streams-webrtc-sdk-c.
Hi @naseebpanghal, @xiaomizhouya, the heap memory is functioning as expected. The memory increase is due to VM: Stack. The memory drops in the heap below mark the disconnection from the viewer and the increases mark connection
from amazon-kinesis-video-streams-webrtc-sdk-c.
@disa6302 I encountered an error during the compilation process while testing the develop branch, with a prompt that I couldn't find header files such as "kvsrtp/rtp-api.h" and "kvssdp/sdp-data_types. h". Since I didn't automatically compile the develop branch, but rather added the header file path in the makefile, I want to know if the develop branch has added some dependency libraries or if the previously dependent libraries need to be updated?
from amazon-kinesis-video-streams-webrtc-sdk-c.
@naseebpanghal Thank you very much for your suggestion. I tested according to your method and did indeed slow down the rate of memory increase. However, I found another issue. When multiple people simultaneously view the live view, memory cannot return to its previous level when exiting. It seems that memory is still increasing. Have you encountered this situation on your end?
from amazon-kinesis-video-streams-webrtc-sdk-c.
@niyatim23 @disa6302 @hassanctech
-#define DEFAULT_ROLLING_BUFFER_DURATION_IN_SECONDS 3
-#define HIGHEST_EXPECTED_BIT_RATE (10 * 1024 * 1024)
+#define DEFAULT_ROLLING_BUFFER_DURATION_IN_SECONDS 1
+#define HIGHEST_EXPECTED_BIT_RATE (2 * 1024 * 1024)
I have gone through multiple CMakefiles but don't find a way to make above changes configurables. My concern, If I make above changes local, then future kvs releases will overwrite my changes and every time I need correct them.
Suggest anyway, compile time or runtime, i can use above settings without making any change in your source files.
OR
is there any plan to make these Macros configurable through CMakfiles?
@niyatim23
One more thing what is VM: Stack (Is it related to RTP Rolling buffer?)
@xiaomizhouya
Sorry, as of now, I am testing single stream.
from amazon-kinesis-video-streams-webrtc-sdk-c.
@xiaomizhouya ,
For your change, did you clean build? Yes there are some new libraries that have been added. These need to be linked to the webrtc client and signaling libraries if you have modified the cmake files that come with this repo. This is where the linking happens for 3 new libraries:
from amazon-kinesis-video-streams-webrtc-sdk-c.
@xiaomizhouya, @naseebpanghal, I ran v1.7.3 and v1.10.2 on a Linux and MacOS both. Following are my observations
- Linux
- Observed the memory using
htop
and/proc/$PID/smaps
- v1.7.3
-
===== Signaling ===== VmPeak: 238352 kB VmSize: 172816 kB Resident size (kb): 7680 ===== Viewer connection ===== VmPeak: 634780 kB VmSize: 568844 kB Resident size (kb): 14208 ===== Viewer disconnection ===== VmPeak: 634780 kB VmSize: 557868 kB Resident size (kb): 11580 ===== Viewer connection ===== VmPeak: 926592 kB VmSize: 779064 kB Resident size (kb): 14396 ===== Viewer disconnection ===== VmPeak: 926592 kB VmSize: 754476 kB Resident size (kb): 13368
-
- v1.10.2
-
===== Signaling ===== VmPeak: 254460 kB VmSize: 254460 kB Resident size (kb): 7808 ===== Viewer connection ===== VmPeak: 1068332 kB VmSize: 1043744 kB Resident size (kb): 15016 ===== Viewer disconnection ===== VmPeak: 1068332 kB VmSize: 1032728 kB Resident size (kb): 12344 ===== Viewer connection ===== VmPeak: 1270316 kB VmSize: 1188388 kB Resident size (kb): 16440 ===== Viewer disconnection ===== VmPeak: 1270316 kB VmSize: 1163800 kB Resident size (kb): 15264
-
- Observed the memory using
- MacOS
No issues reported by valgrind as pointed out earlier. The memory continues to increase after connection to the viewer but stops after a point. We also see it drop after a disconnection but not back to what it was right after signaling. However, we do see it dropping back to where it started from in heap allocations in the MacOS instruments plots.
This behavior is seen in both v1.7.3 and v1.10.2. Possible reasons for this delay in releasing the VM Stack maybe that the OS may be delaying the release of memory back to the system to avoid the overhead associated with frequent memory allocation and deallocation. Or the virtual memory manager might not immediately release memory back to the system, especially if the system is not under memory pressure. The memory pages can remain in the working set or be swapped out and marked as available for future allocations.
from amazon-kinesis-video-streams-webrtc-sdk-c.
It looks like this issue has not been active for 10 days. If the issue is not resolved, please add an update to the ticket, else it will be automatically resolved in a few days.
from amazon-kinesis-video-streams-webrtc-sdk-c.
I tested the sample code of webrtc 1.10.2 sdk on my device (linux system) and monitored the memory of webrtc using the top command. As shown in the statistical chart, the vertical axis represents the RSS value in kb, the horizontal axis represents the number of points, and the interval between points is 5 seconds (the top command refresh time is 5 seconds). There are a total of 4033 points in the chart, so the testing time is 4033 * 5s/3600s=5.57 hours.
The testing method is to observe the live view for 60 seconds, then stop for 15 seconds, and repeat this process.
Use only one peer to view the live view throughout the entire process.
Is the curve shown in the graph normal?
from amazon-kinesis-video-streams-webrtc-sdk-c.
RSS memory includes heap, stack, code, data, shared libraries, memory-mapped files. It isn't just the heap. We've already established that the heap memory goes back to what it was before any connections after the disconnection and there are no leaks related to it. We have also established that the total memory for a particular peer connection becomes steady after a point which can also be seen in your plot. The VM Stack may not have been claimed back by the OS which may have been causing the increase. I do see some fluctuation around ~2300. What is that drop related to? Do you see a similar pattern on v1.7.3 as seen on v1.10.2?
from amazon-kinesis-video-streams-webrtc-sdk-c.
It looks like this issue has not been active for 10 days. If the issue is not resolved, please add an update to the ticket, else it will be automatically resolved in a few days.
from amazon-kinesis-video-streams-webrtc-sdk-c.
Related Issues (20)
- [Bug]: Network filter does not apply to socket routing HOT 1
- [Bug]: resolveStunIceServerIp Ignores Command Line AWS Region, Uses Environment Variable Instead HOT 1
- [Feature request]: Need to build the kvsdk package with OpenWrt HOT 3
- [Documentation]: README.md should mention the supported video codecs for WebRTC ingestion HOT 2
- [Bug]: kvsWebrtcClientMasterGstSample receive audio not work HOT 1
- [Bug]: KVS SDK RTP retransmitter bug HOT 2
- [Bug]: [2024/05/23 15:19:40:0909] W: [wsicli|5a|WS/h1/default/m-579da5db.kinesisvideo.ap-southeast-]: lws_client_ws_upgrade: got bad HTTP response '403' 2024-05-23 15:19:40.091 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 1 2024-05-23 15:19:40.091 WARN lwsWssCallbackRoutine(): Client connection failed. Connection error string: HS: ws upgrade response not 101 HOT 9
- softphone signaling server HOT 4
- [Bug]: how to ensure the functon "wakeLwsServiceEventLoop " thread safety,because of "lws_callback_on_writable" is not thread-safe HOT 1
- [Bug]: how to ensure the functon "wakeLwsServiceEventLoop " thread safety,because of "lws_callback_on_writable" is not thread-safe
- [Bug]: decryptSrtcpPacket(): Decrypting rtcp packet failed with error code 9 on srtp session HOT 4
- [Bug]: TURN_STATE_GET_CREDENTIALS fails after update from 1.7.2. to 1.10.2 of the SDK HOT 17
- [Bug]: Windows build fails at openssl HOT 1
- [Bug]: Seemingly SessionDescription creation bug HOT 2
- [Feature request]: Reduce the time it takes to connect to live view. HOT 1
- [Bug]: Close peer connection:ERROR fromCheckConnectionIceAgentState(): operation returned status code: 0x5a00000d HOT 4
- [Bug]: Call serializeSessionDescriptionInit and return STATUS_BUFFER_TOO_SMALL HOT 2
- [Bug]: 1.10.2, kvsWebrtcClientMasterGstSample continually increases CPU usage HOT 7
- [Bug]: Wrong SDP answer when the codec doesn't match with viewer HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-kinesis-video-streams-webrtc-sdk-c.