Identify associations writes out thousands of files about move HOT 4 CLOSED

I worked on this in recent days. I believe that the current implementation is most suitable for this case (See Issue #16 Reduce memory of step 05 and the commit in the comments).
In that implementation, files are saved iteratively for every latent space and every repetition (it could be reduced even into smaller files if needed). Also, it loads the files with np.memmap that doesn't read the entire file into memory and instead creates a memory map to files on disk. It worked without errors on my Laptop.
However, when I tried the former version that appends everything to one big file (here without saving the files and instead saving the parameters of trained models), my PC was already giving memory errors during the processing (before reaching saving the files).
I am aware that the current implementation saves files with huge sizes and it is the most significant disadvantage of this approach. Therefore, as Ricardo said I should print the notification during the training and add this information to the description in README. However, I believe the current implementation is more scalable to larger datasets which would require even more CPU for processing these large data frames.
Let me know what you think about it.

from move.

Ok!

from move.

I know this opens a new front, but I want to put it up for discussion: Maybe a structured way of saving data could be a good idea: I attended a tutorial on SQLAlchemy (previous recording) yesterday evening or we have a look at what others (alphapept) do with HDF5 to handle large data and many files...

from move.

HDF5 could be something we can implement later, when more of the core functionality is updated. Closing for now

from move.