Poor performance with many nodes about prometheus-slurm-exporter HOT 2 CLOSED

Our usage of slurm has many nodes, which makes the call to /metrics take >5 minutes.

Ok, I am just curious here. Can you give a me a rough estimate of how many nodes we are talking about?

We have put this exporter in execution in an environment with roundabout 800 nodes (bare metal) and we
never experienced a delay higher than 1 minute or less and every time the problem was on the Slurm
controller itself than on the exporter.

Anyway, I have already tested your PR and I am willing to accept it but I would really be glad if you can
provide me a little bit more context on your environment.

from prometheus-slurm-exporter.

Slurm can be setup with many "virtual nodes", which are actually almost all in a "down" state. This can be useful to bridge the gap between an on-demand instance service and slurm, which expects a more static configuration.

Try creating 20,000 slurm nodes in a "down" state, and the exporter will slow to a crawl on this instance call, mainly because it is making 1 request per node out to slurm. The master branch code took several minutes to come back, whereas the code in this PR takes seconds since it just makes 1 summary call.

from prometheus-slurm-exporter.