[Discussion] WebSocket Benchmarks and useful graphs / charts about bandit HOT 5 CLOSED

Agreed! We have very basic websocket perf testing in our benchmarking suite (you can see the output at e.g.: near the bottom of https://github.com/mtrudel/bandit/actions/runs/7838502587), but they're really intended for us to avoid performance regressions on PRs, not as 'absolute' benchmarks. They also expire from GitHub actions after a few months so they're not great to link to.

I did a run of the WS suite against Bandit and Cowboy's branch heads; you can see it here. It's not ideal (most of the charts only relate to HTTP/1 suites and don't even populate for WS), but it's a start.

If you're curious, the implementation of the benchmarking suite is over here

from bandit.

@mtrudel
Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.

from bandit.

Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.

Sounds great; a PR for this would be very welcome! I personally am not planning to spend any meaningful amounts of time on measurement or project PR for a while (I try to rotate through all of the aspects of the project in turn, and perf just got a big chunk of my time around this time last year).

If you want to take this on, I'd be more than happy to accept the work on a separate PR. In the meantime, I'm going to close this issue for project hygiene reasons. Thanks for the issue!

from bandit.

@mtrudel Sure, no problem. Just so we don't pollute the Issues with discussion, maybe you can open up the discussions section for this repository, so all related talks can be made there.

from bandit.

Yes, I have seen those GitHub actions benchmarking. Well a suggestion would be that, you can maybe combine all of those benchmarks into a Single GitHub Action, which outputs the result artifacts, but not just in a plain CSV fashion, but also as a static website or something, and it deploys that to GitHub pages. Then it would be a good way to link to it, and put in some extras such as graphs and charts.

Sounds great; a PR for this would be very welcome! I personally am not planning to spend any meaningful amounts of time on measurement or project PR for a while (I try to rotate through all of the aspects of the project in turn, and perf just got a big chunk of my time around this time last year).

If you want to take this on, I'd be more than happy to accept the work on a separate PR. In the meantime, I'm going to close this issue for project hygiene reasons. Thanks for the issue!

I have worked on it here: #300

from bandit.