Implementing Barwise TF matrices (Barwise-aligned features). about msaf HOT 3 OPEN

Happy to see that the bar-synced features are getting traction! Currently, the easiest way to implement this on MSAF is to add a completely new set of features here: https://pythonhosted.org/msaf/features.html#

Basically, we would need:

• est_barsync
• ann_barsync

This way, the user could pass timestamp of the annotated downbeats (ann) and MSAF could also provide an algorithm to estimate them (est).

That being said, when I wrote this part of MSAF (around 9 years ago!), computing features was an expensive process. Nowadays, features are typically computed on the fly (since it's so cheap), which saves a lot of disk space (MSAF's feature json files can be quite big).

So... what I would suggest is to remove the temporary storage of features and just compute them on the fly (ie, get rid of those temporary JSON files). This way, we don't need to include these new types of features on the JSON files, and backwards compatibility in future MSAF releases would be much easier. And playing around with different custom features should be much easier.

This would be a major refactor of MSAF, but I think it would be totally worth it.

What do you folks think? @carlthome and/or @ax-le are you up for the challenge? :D

from msaf.

Hi @urinieto and Happy New Year!

Firstly, while it would be interesting to implement barwise synchronized features, the features coined "Barwise TF matrix" mentioned in my first message necessitate a bit more work because they consist of computing a fixed number of samples (defined by a parameter) per bar. In other terms, while barwise synchronized features contain 1 sample per bar, Barwise TF matrices contain n samples per bar (typically 96). This would need additional modifications, to cope with bar discrepancies during the song (right now, this is handled via oversampling the spectrogram and selecting regularly spaced samples in each bar, it may be debated).

Secondly, I could be down for refactoring code, but not right now! ;) Maybe it can wait as this does not seem urgent, but maybe someone would be available sooner (@carlthome ?).

from msaf.

Oh gotcha, I understand. Yeah, I think this begs even further for a potential refactoring of the way MSAF takes care of the features. I would actually love to do that.. but likely not gonna happen after ISMIR 2024 organization 😅 (unless @carlthome is up for the challenge!)

from msaf.