jitsi / jitsi-autoscaler Goto Github PK

I'm using Jitsi autoscaler latest commit (acf86ac 2023/01/13) on ubuntu 22.04 arm64
Oracle cloud: Ampere A1 flex, 2CPU/4Gb mem

The CPU and memory used by the node process keep climbing slowly.
Starting one month ago from almost 0% CPU and 0% memory, it is now about 105% CU and 20% memory
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
455849 ubuntu 20 0 1664176 792724 34056 R 102.7 19.8 23329:20 node

In our test environment there is just one JVB running and reporting statistics to the test autoscaler without any scale-up or down. In our production environment we do scale up and down but we restart the autoscaler every night because of this CPU/memory issue.

I profiled our test autoscaler with chrome://inspect and got this:

46492.8 ms43.13 % | 97726.0 ms90.67 % | (anonymous) status.js:82 |  
46492.8 ms43.13 % | 97726.0 ms90.67 % | ........listOnTimeout internal/timers.js:502 |  
46492.8 ms43.13 % | 97726.0 ms90.67 % | ...............processTimers internal/timers.js:482 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | (anonymous) status.js:96 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | ........(anonymous) status.js:94 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | ...............get stats status.js:93 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | ......................(anonymous) status.js:82 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | .............................listOnTimeout internal/timers.js:502 |  
43874.9 ms40.71 % | 43874.9 ms40.71 % | ..................................processTimers internal/timers.js:482 |  
4877.3 ms4.53 % | 5978.8 ms5.55 % | (anonymous) status.js:124 |  
4877.3 ms4.53 % | 5978.8 ms5.55 % | .......get stats status.js:93 |  
4877.3 ms4.53 % | 5978.8 ms5.55 % | ...............(anonymous) status.js:82 |  
4877.3 ms4.53 % | 5978.8 ms5.55 % | ......................listOnTimeout internal/timers.js:502 |  
4877.3 ms4.53 % | 5978.8 ms5.55 % | .............................processTimers internal/timers.js:502 |  

This suggest that the node process is drowning in timer management, but I'm not sure how to debug more. Does the Jitsi team encounter this issue? Do you have some suggestion how track the root cause?

Hi,

We've been using jibri+pulseaudio in all our streamings & recordings for up to 6 months. So far it has been tested more than 2000 times and every day we have 30+ tests. It simply works better than Alsa.

This made it trivial to scale Jibri on Kubernetes. We're doing it based on CPU consumption so K8s's HPA does the work for us.

We're planning to feed prometheus with Jibri statuses so we can scale it in an even more clean and exact way.

Feel free to share your thoughts and let's team up to get it up and running :-)

PS: you can try kpeiruza/jibri:v12, it has pulseaudio already bundled + a few tweaks at jibri so it launches ffmpeg against pulse instead of Alsa.