Config to run with 64gb RAM? about gtdbtk HOT 7 CLOSED

Thanks Balays. We'll take a look at the mmap-file option for a future GTDB-Tk release.

from gtdbtk.

Hello. Unfortunately, there is no way to reduce the system requirements of GTDB-Tk at the moment. We are hoping this software might be hosted as a web service in the future though there is no immediate plans in this regard. Sorry I can't be of direct help.

from gtdbtk.

Thanks for the reply @dparks1134 . We were just discussing your recent paper this morning in our lab seminar and hoped to be able to use the tool. I'd love to contribute to the software as it is open source but not being able to run it locally makes it hard to do so.

from gtdbtk.

The system requirements are pretty high. This is due to the third party software pplacer which we use to place genomes into a tree. This software is excellent and alternatives we have tried are worse in terms of system requirements so I don't expect this situation to get better.

from gtdbtk.

I see. Thanks.

from gtdbtk.

Hello.
I've had a similar problem, as our system with 96 GB RAM was not enough. Indeed the pplacer part killed the program. So I ran the pplacer sub-command only, with a --pretend flag, which only estimates the memory usage. It said that it needs 102 GB, so only just a little more. Then we increased our swap memory and it worked just fine. Actually it used about 96 (physical) + 25 GB (swap) RAM.
Thanks for this excellent tool btw, it's just what I was looking for! :)

from gtdbtk.

Also, at the pplacer's they know the large memory req. is an issues, so there's this option to get around it: --mmap-file, which creates a file that it uses as address space and thus shrinking the need for physical memory. I've integrated this flag into the GTDBTk classify.py code where it actually runs pplacer (line 101), but unfortunately it didn't work for me. But maybe it just needs a little tweaking. Cheers.

from gtdbtk.