[h2o benchmark] Align with evangelism on benchmark infrastructure about benchmarks HOT 4 CLOSED

@jrbourbeau - Thanks for opening this ticket.

A lot of the benchmarking work with dev advocacy & OSS engineering will overlap (e.g. generating the datasets, creating code to run benchmarks, making sure the Dask queries are optimized), but the actual infrastructure will differ a bit.

For the h2o benchmarks, we'll need to replicate their infrastructure exactly, which is a single r3-8xlarge node. I am hoping to create an issue that shows if we structure the code like this and use your same exact infrastructure, then the Dask queries run 30 times faster (exact numbers to be confirmed later).

For the Databricks benchmarks which I'd also like to address, we'll need to use the environments they list in their blog post.

The GitHub Actions / Coiled infrastructure you'll setup will be great for other dev advocacy content down the road. In the near term, I'll hope to pair with the product engineers on replicating the h2o / Databricks environments exactly and rerunning their benchmarks with properly structured code.

Let me know if this plan sounds alright with you. I get the feeling that there is a widespread impression that "Dask is slow". I'm not seeing that in reality with the benchmarks I am running. I am hoping to dig up the truth here and write content that clears up this misinformation.

from benchmarks.

Happy to help with any questions on getting the infrastructure you need. The #engineering-clusters channel would also be a good place to ask.

from benchmarks.

Cool, joined #engineering-clusters, thanks @shughes-uk!!

from benchmarks.

Closing this as obsolete. We have a separate tracking ticket for a blogpost on h2o benchmarks as well as we are about to invest work to improve the performance of the queries.

Closing in favor of https://github.com/orgs/coiled/projects/12/views/3

from benchmarks.