Document the minimal resources needed to run tests for DataFusion about arrow-datafusion HOT 5 CLOSED

Thank you for bringing this up @SteveLauC. I had the same experience when I first starting working on DataFusion. You can work around memory limitations by running cargo test -- --test-threads=1 which will only run a single test at a time. It will be slower but consume substantially less memory. I think it would be a good idea to document this workaround since many new contributors won't have enough RAM to run all tests at max parallelism on their system.

The precise memory requirements to run cargo test will vary over time and depend on your exact development set up. Running in wsl on Windows for example is a bit more intensive as you need to reserve some memory for Windows. I personally upgraded from 32->64GB and that was plenty running natively on linux.

from arrow-datafusion.

I think the doctests are the main culprit, see #5347

Agree on documenting this as part of contributor/developer guide 👍

from arrow-datafusion.

cargo test -- --test-threads=1

Thanks for showing me this! This indeed makes the memory usage controllable:)

from arrow-datafusion.

Glad that worked for you! I opened #9402 to document this in the contributors guide.

There does not seem to be a way to configure Cargo to default to --test-threads 1 other than setting environment variables, so documentation may be our best bet for now... see rust-lang/cargo#8430

from arrow-datafusion.

I had my computer upgraded to 64GB of ram - the workaround mentioned above I used as well but the time to run the tests was using that was horrendous. Using linux under WSL2 in windows here.

from arrow-datafusion.