Comments (22)
@peastman we have significantly improved the support for large datasets. Please have a look at the multi-GPU branch. It does not create huge datasets anymore during the preprocessing, and the memory requirements should also be even smaller. For me a preprocessed SPICE is now less than 8Gb. If you need any help, let me know!
from mace.
Thank you for your messsage! I think the several sides for this are :
- Enable parsing of the statistics to avoid running out of memory
- Add multi-processing for the dataloader to speed up the loading and more memory effecient loading and pre-processing
Could you give an estimate of the size of the dataset in you want to load?
from mace.
Hi,
the solutions sound good. At the moment we are thinking about ~35M structures, maybe on average 12 atoms and the corresponding forces and stresses.
best,
Jonathan
from mace.
I just ran into the same problem. I'm trying to fit a MACE model to the SPICE dataset, which is moderately large but not huge. About 1.1 million conformations, an average of around 40 atoms per molecule, with energies and forces. Converted to xyz format it's 3.9 GB. When it tried to load the dataset, it filled up all the memory in my computer (32 GB), then the computer hung and had to be turned off with the power switch.
In my work with TorchMD-Net, I developed a HDF5 based dataset format for dealing with this problem. It transparently pages data into memory as needed. Would something similar be helpful here?
from mace.
The question is whether we should push multi-gpu-multi-node training, assuming that large data sets take a long time to train anyway
from mace.
Multi-GPU is nice if you happen to have them, but it isn't necessary. A single GPU can train models on large datasets. It just takes longer.
In my case it never got as far as using the GPU. It ran out of host memory first.
from mace.
Hi @peastman!
So you did not get any error message but just a crash? I guess HDF5 is a good option for that. In your code, is it interfaced with a torch dataloader ?
from mace.
So you did not get any error message but just a crash?
It went deeply into swap space and started thrashing, which caused the computer to become unresponsive.
In your code, is it interfaced with a torch dataloader ?
It subclasses torch_geometric.data.Dataset.
I created a reduced version of the dataset with less than 10% of the data, so I could analyze where exactly the memory was being used. I noted the total memory used by the process at various points in run_train.py.
Immediately before the call to get_dataset_from_xyz()
: 300 MB
Immediately after it returns: 2.9 GB
After creating train_loader
and valid_loader
: 6.1 GB
After creating the model: 8.2 GB
It looks like there are multiple bottlenecks in memory use. Loading the raw data with get_dataset_from_xyz()
would take over 30 GB for the full dataset. Then when it creates the AtomicData
objects, that more than doubles the memory use. Addressing this would involve two steps:
- Load the raw data in a way that doesn't require everything to be in memory at once.
- Create the
AtomicData
objects as they're needed and then discard them again, rather than building them all in advance.
from mace.
@peastman Have you also considered using LMDB instead of hdf5? I would like to implement a new data loader that can efficiently deal with arbitrarily large datasets, and was wondering if in your experience hdf5 is better / faster? I am currently leaning towards using lmdb, but have not benchmarked them exhaustively yet.
from mace.
I'd never heard of LMDB before. It looks like something very different. LMDB is a database system, while hdf5 is just a file format that's designed to allow efficient read access. It's possible a full database would have advantages, but it's going to be a lot harder for users. There's good support for hdf5 in just about every language, and it takes minimal code to build a file. Users are also much more likely to be familiar with it already.
As for performance, hdf5 is working great for me. I have no trouble handling large datasets and I can get GPU utilization close to 100%.
from mace.
@peastman Could you perhaps try this pull request. I had a go at implementing the on-the-fly data loading and statistics parsing. Let me know if you have any questions, instructions are in the README.
from mace.
It moves the problem to a new place without fixing it. When I run the preprocess_data.py script I get the same behavior as before. The memory used by the process gradually increases until it fills up all available memory. Then the computer hangs and has to be shut down with the power button.
from mace.
Sorry, I was under the assumption that there is 100-s of Gb-s of CPU RAM, and it is only the GPU RAM that is limited. So I moved the whole preprocesssing to the CPU and the GPU should read the preprocessed data from the HDF5 one batch at a time.
I can try to implement a modified low memory version of the preprocessing that processes one config at a time and writes it to disk before going to the next.
from mace.
My computer only has 32 GB of RAM.
from mace.
I have attempted a fix. I really hope it will work. Please use this branch and see the example in the README
https://github.com/davkovacs/mace/tree/on_the_fly_dataloading
from mace.
Thanks! It's running now. The resident size climbed to about 12 GB and then stopped increasing. I'll let you know what happens.
from mace.
Success! I started from a dataset that's about 1 GB in the HDF5 format used by TorchMD-Net. Converting it to xyz format increased it to just under 4 GB. preprocess_data.py ran for over 3.5 hours and produced a pair of files that totaled about 56 GB(!). But training now seems to be working. It's been running for about 12 hours, and the loss is gradually decreasing.
What exactly is the loss function? And is there any way I can tell what epoch it's on?
from mace.
Great to hear!
It should have created a logs
directory which contains a file that logs the validation loss (every 2 epochs by default). There is another folder called results
which contains a file that logs the loss for each batch.
For the precise form of the loss function see Appendix 5.1 of the paper
https://arxiv.org/pdf/2206.07697.pdf
from mace.
I guess that means it hasn't completed two epochs yet. I'll keep watching.
from mace.
That's great to here is the multi-GPU branch operational already?
from mace.
We still have some debugging to do for multi-GPU training, but it works for training on a single GPU.
from mace.
It works great, thanks! I have a training run going right now. I'm also looking forward to multi-GPU support.
from mace.
Related Issues (20)
- MACE-LAMMPS meta issue HOT 1
- Train/Val MAE for MACE on MPTraj HOT 1
- cuda device with a number ignored HOT 1
- Poor energy & force metrics on paper's datasets (carbon nanotube, buckyball catcher) HOT 2
- Can we use pretrain model (mace_mp) and retrain with our configurations? HOT 2
- Multi-GPU evaluation for optimisation of a full protein structure HOT 1
- checkpoint file not read in foundation branch. HOT 7
- Error while using foundation branch HOT 8
- Update install instructions for conda-forge HOT 5
- loss function evaluated as zero. HOT 2
- Setting `dispersion=True` in the MACE-MP-0 foundation model omits data from `calc.results` HOT 1
- LAMMPS-MACE with Kokkos: Illegal memory access encountered with trained MACE but works with MP-MACE-0 HOT 51
- multi-gpu lammps issue HOT 6
- Fine-tunning up-to-date MACE-MP-0 model HOT 7
- Don't modify torch default dtype HOT 1
- typo? in multi-gpu branch run_train.py
- repulsion branch DistributedEnvironment exception produces no output HOT 1
- Where are the Similarity Statement Files for Mace-mp-0? HOT 1
- How to repeat the test-result in your paper
- Possibility to not compute energies on `MACE.forward` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mace.