Slow down caused by DBSCAN about bstore HOT 4 OPEN

I just repeated this behavior for five fiducial tracks, each with about 20000 frames on the Olympus computer. It filled up all 16gb of memory. And didn't complete.

Three tracks worked, filling only 12 GB and took 10 or 20 seconds.

Perhaps I can implement ELKI's version of DBSCAN.

from bstore.

@nberliner recommended trying HDBSCAN as a high performance implementation.

from bstore.

I'm not sure about the memory consumption. It apparently is a bit faster than the sklearn DBSCAN implementation (see here). Interestingly, it appears from that comparison as if the sklearn DBSCAN implementation can cluster 200000 points on a laptop with 8GB.

One advantage of HDBSCAN is that it dynamically selects a suitable density for clustering which can vary for each cluster in the field of view. There is only one parameter, the minimum number of clusters, which must be set by the user. I found the description given on the project page very good (see here).

from bstore.

See this discussion: scikit-learn/scikit-learn#5275

Also note discussion on DBSCAN and memory usage here: http://scikit-learn.org/stable/modules/clustering.html#dbscan

It seems that if the neighborhood radius is made too large, then the memory consumption blows up. I noticed this when I recently tried to cluster a dataset that was in units of pixels instead of nanometers. Setting the neighborhood radius to "50" included nearly every point in the radius and ate up all my memory. Resetting it to 0.5 pixels worked without much memory consumption.

from bstore.