`standardize` is slow with large datasets about d47crunch HOT 2 CLOSED

That is probably a lmfit issue. I don't know for sure if parallelization is an option for the Trust Region Reflective method used here, but I doubt it (IIRC it's an iterative process). Same goes for a progress bar, because the algorithm doesn't know in advance how long it will be looking for a local minimum.

Remember that the difficulty of fitting the model increases dramatically with the number of parameters to fit, not so much with the number of analyses. I usually process datasets with 3-6 sessions and 20-30 unknown samples. That is quasi-instantaneous. When processing the more demanding dataset of Anderson et al. (2021), it took something like 10-20 seconds because of the large number of sessions (each batch of a few tens of replicates was treated as a new session). Is this what you're doing? It might be possible to relax the convergence conditions (see here and here, ftol and xtol parameters).

My personal approach is not to process a year's worth of data at once, but usually rather to process together only the sessions related to a given project.

from d47crunch.

Thanks! Yep I didn't realize the slowness came because of all the unique Sample names. After redefining sample to mean something like "period of time for which we calculate one temperature" it is now running within several seconds, depending on the dataset.

FTR:

• "sample" = an amount of presumably homogeneous carbonate material. Each sample should be uniquely identified by a sample name (field Sample in the csv file).
• "analysis" or "replicate" = corresponds to a single acid reaction followed by purification of the evolved CO2 and by a series of dual-inlet IRMS measurements. Each analysis is identified by a unique identifier (field UID in the csv file, but if it's missing a default series of UIDs will be generated).

Originally posted by @mdaeron in #7 (comment)

from d47crunch.