building algorithms/cc about msaf HOT 13 CLOSED

Hi,

I'm glad you're switching to Python! :)

The issue you're reporting should be fixed on the "develop" branch.

Please, checkout that branch (since it's much more stable than the master
one at the moment, sorry!), and you should be able to compile and install
msaf from the root folder by typing:

python setup.py build
(sudo) python setup.py install

Indeed, the problem originates because of the gcc version, which in Linux
requires extra flags to accept C++11.

In the develop branch, and in order for you to play around with msaf, take
a look at the folder "scripts". More specifically, try running the
run_msaf.py once you installed msaf. You can try it in the Sargon dataset
(a new fully operating dataset included in msaf) by typing:

./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d -j 4

This will run msaf on the Sargon dataset, with the foote algorithm for
boundaries and the fmc2d algorithm for labels, using 4 threads (there are
only 4 songs on this dataset).

You should be able to evaluate your results by typing:

./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d -j 4 -e

More options:

./run_msaf.py -h

Let me know if you have any questions. I'm glad to help :)

On Fri, Mar 13, 2015 at 4:50 AM jenmadtu [email protected] wrote:

Hi Guys

I have ventured into the land of Python (ignorantly) on my ubuntu 14.04
machine (compiled python using gcc 4.9) convinced by @urinieto
https://github.com/urinieto

I do seem to have some problems compiling the algorithms/cc hope you can
help

i had some issues with size_t and had to include in
TempoTrackV2.h since apparently newer version of gcc have issues with
size_t. (just FYI)

Now ive run into another issue which is a bit more tricky and hope you can
help out.

i do the following
~/downloads/msaf/algorithms/cc# python setup.py build_ext --force --inplace

running build_ext
building 'cc_segmenter' extension
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC

compile options: '-Idsp/segmentation -I. -Iinclude -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
extra options: '-DUSE_PTHREADS'
x86_64-linux-gnu-gcc: dsp/phasevocoder/PhaseVocoder.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: thread/Thread.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: dsp/rateconversion/Decimator.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: dsp/chromagram/ConstantQ.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: dsp/tonal/TonalEstimator.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from dsp/tonal/TonalEstimator.cpp:16:0:
dsp/tonal/TonalEstimator.h: In member function ‘void ChromaVector::printDebug()’:
dsp/tonal/TonalEstimator.h:35:21: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < size(); i++)
^
dsp/tonal/TonalEstimator.h: In member function ‘void TCSVector::printDebug()’:
dsp/tonal/TonalEstimator.h:71:21: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < size(); i++)
^
x86_64-linux-gnu-gcc: dsp/tempotracking/TempoTrack.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/tempotracking/TempoTrack.cpp: In member function ‘void TempoTrack::initialise(TTParams)’:
dsp/tempotracking/TempoTrack.cpp:74:18: warning: unused variable ‘winPre’ [-Wunused-variable]
unsigned int winPre = Params.WinT.pre;
^
dsp/tempotracking/TempoTrack.cpp:75:18: warning: unused variable ‘winPost’ [-Wunused-variable]
unsigned int winPost = Params.WinT.post;
^
dsp/tempotracking/TempoTrack.cpp: In member function ‘double TempoTrack::tempoMM(double_, double_, int)’:
dsp/tempotracking/TempoTrack.cpp:181:12: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (a=1;a<=numelem;a++)
^
dsp/tempotracking/TempoTrack.cpp: In member function ‘void TempoTrack::createPhaseExtractor(double_, unsigned int, double, unsigned int, unsigned int)’:
dsp/tempotracking/TempoTrack.cpp:590:21: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for( int i = 0; i < scratchLength; i++ )
^
dsp/tempotracking/TempoTrack.cpp:597:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for(int i = 0; i < scratchLength; i ++)
^
dsp/tempotracking/TempoTrack.cpp: In member function ‘int TempoTrack::phaseMM(double_, double_, unsigned int, double)’:
dsp/tempotracking/TempoTrack.cpp:637:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for( int i = 0; i < winLength; i++ )
^
dsp/tempotracking/TempoTrack.cpp:646:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for(int i = 1 + (o - 1); i< winLength; i += (p + 1))
^
dsp/tempotracking/TempoTrack.cpp: In member function ‘std::vector TempoTrack::process(std::vector, std::vector)’:
dsp/tempotracking/TempoTrack.cpp:823:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < TTLoopIndex + 3 && i < TTFrames; ++i) {
^
dsp/tempotracking/TempoTrack.cpp:827:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < TTLoopIndex + 3 && i < TTFrames; ++i) {
^
dsp/tempotracking/TempoTrack.cpp:715:18: warning: unused variable ‘DFCLength’ [-Wunused-variable]
unsigned int DFCLength = m_dataLength + m_winLength;
^
x86_64-linux-gnu-gcc: dsp/transforms/FFT.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: dsp/segmentation/Segmenter.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/segmentation/Segmenter.cpp: In function ‘std::ostream& operator<<(std::ostream&, const Segmentation&)’:
dsp/segmentation/Segmenter.cpp:23:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < s.segments.size(); i++)
^
x86_64-linux-gnu-gcc: dsp/tempotracking/TempoTrackV2.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/tempotracking/TempoTrackV2.cpp: In member function ‘void TempoTrackV2::calculateBeats(const std::vector&, const std::vector&, std::vector&)’:
dsp/tempotracking/TempoTrackV2.cpp:465:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (startpoint >= backlink.size()) startpoint = backlink.size()-1;
^
x86_64-linux-gnu-gcc: maths/CosineDistance.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
maths/CosineDistance.cpp: In member function ‘double CosineDistance::distance(const std::vector&, const std::vector&)’:
maths/CosineDistance.cpp:37:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for(int i=0; i<v1.size(); i++)
^
x86_64-linux-gnu-gcc: dsp/signalconditioning/DFProcess.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/signalconditioning/DFProcess.cpp: In member function ‘void DFProcess::removeDCNormalize(double*, double*)’:
dsp/signalconditioning/DFProcess.cpp:183:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for( unsigned int i = 0; i< m_length; i++)
^
x86_64-linux-gnu-gcc: dsp/mfcc/MFCC.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: maths/KLDivergence.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc: dsp/segmentation/cluster_segmenter.c
x86_64-linux-gnu-gcc: dsp/segmentation/ClusterMeltSegmenter.cpp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘void ClusterMeltSegmenter::extractFeaturesConstQ(const double*, int)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:159:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (!window || window->getSize() != fftsize) {
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘virtual void ClusterMeltSegmenter::segment()’:
dsp/segmentation/ClusterMeltSegmenter.cpp:338:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (features.size() < histogramLength) return;
^
dsp/segmentation/ClusterMeltSegmenter.cpp:345:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < features.size(); i++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp:349:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int j = 0; j < features[0].size(); j++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp:378:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < features.size(); i++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In function ‘int median(std::vector&)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:389:48: error: ‘nth_element’ was not declared in this scope
nth_element(v.begin(), v.begin()+n, v.end());
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘void ClusterMeltSegmenter::makeSegmentation(int, int)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:422:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < annotBounds.size() - 1; i++) {
^
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘void ClusterMeltSegmenter::extractFeaturesConstQ(const double_, int)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:159:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (!window || window->getSize() != fftsize) {
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘virtual void ClusterMeltSegmenter::segment()’:
dsp/segmentation/ClusterMeltSegmenter.cpp:338:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (features.size() < histogramLength) return;
^
dsp/segmentation/ClusterMeltSegmenter.cpp:345:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < features.size(); i++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp:349:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int j = 0; j < features[0].size(); j++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp:378:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < features.size(); i++)
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In function ‘int median(std::vector&)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:389:48: error: ‘nth_element’ was not declared in this scope
nth_element(v.begin(), v.begin()+n, v.end());
^
dsp/segmentation/ClusterMeltSegmenter.cpp: In member function ‘void ClusterMeltSegmenter::makeSegmentation(int*, int)’:
dsp/segmentation/ClusterMeltSegmenter.cpp:422:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < annotBounds.size() - 1; i++) {
^
error: Command "x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Idsp/segmentation -I. -Iinclude -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c dsp/segmentation/ClusterMeltSegmenter.cpp -o build/temp.linux-x86_64-2.7/dsp/segmentation/ClusterMeltSegmenter.o -DUSE_PTHREADS" failed with exit status 1

i can see that i have problems with
dsp/segmentation/ClusterMeltSegmenter.cpp:389:48: error: ‘nth_element’ was
not declared in this scope nth_element(v.begin(), v.begin()+n, v.end());

where nth_element seems to be the bad guy. It might again be an issue
with the new version of the gcc compiler. Any suggestions?

—
Reply to this email directly or view it on GitHub
#3.

from msaf.

Thank you with the devel trick, i indeed could install it now :)

As you suggested i run

./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d
INFO root 2015-03-13 09:48:08,406 run.py:196  Segmenting ../datasets/Sargon/audio/01-Sargon-Mindless.mp3
INFO root 2015-03-13 09:48:08,406 featextract.py:164  Loading audio file 01-Sargon-Mindless.mp3
Traceback (most recent call last):
  File "./run_msaf.py", line 122, in <module>
    main()
  File "./run_msaf.py", line 106, in main
    res = func.process(args.in_path, **params)
  File "/usr/local/lib/python2.7/dist-packages/msaf-0.0.2-py2.7-linux-x86_64.egg/msaf/run.py", line 312, in process
    for file_struct in file_structs[:])
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 653, in __call__
    self.dispatch(function, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 400, in dispatch
    job = ImmediateApply(func, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 138, in __init__
    self.results = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/msaf-0.0.2-py2.7-linux-x86_64.egg/msaf/run.py", line 200, in process_track
    featextract.compute_all_features(file_struct)
  File "/usr/local/lib/python2.7/dist-packages/msaf-0.0.2-py2.7-linux-x86_64.egg/msaf/featextract.py", line 224, in compute_all_features
    features = compute_features_for_audio_file(file_struct.audio_file)
  File "/usr/local/lib/python2.7/dist-packages/msaf-0.0.2-py2.7-linux-x86_64.egg/msaf/featextract.py", line 165, in compute_features_for_audio_file
    audio, sr = librosa.load(audio_file, sr=msaf.Anal.sample_rate)
  File "/usr/local/lib/python2.7/dist-packages/librosa/core.py", line 77, in load
    with audioread.audio_open(os.path.realpath(path)) as input_file:
  File "/usr/local/lib/python2.7/dist-packages/audioread/__init__.py", line 96, in audio_open
    raise NoBackendError()
audioread.NoBackendError

Another trick to make it work maybe? :)

from msaf.

You need ffmpeg. For some reason, Ubuntu removed it from the original sources.

You should be able to install it by doing:

sudo add-apt-repository ppa:jon-severinsson/ffmpeg
sudo apt-get update
sudo apt-get install ffmpeg
sudo apt-get install frei0r-plugins

(as they mentioned here).

Let me know if this works!

from msaf.

well to install ffmpeg i did:

add-apt-repository deb  http://ppa.launchpad.net/jon-severinsson/ffmpeg/ubuntu trusty main
add-apt-repository deb  http://ppa.launchpad.net/jon-severinsson/ffmpeg/ubuntu saucy main
apt-get update
apt-get install ffmpeg
apt-get install frei0r-plugins

that made it work. The next error is in my eyes kinda tricky

./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d
INFO root 2015-03-13 12:51:05,423 run.py:196  Segmenting ../datasets/Sargon/audio/01-Sargon-    Mindless.mp3
INFO root 2015-03-13 12:51:05,424 featextract.py:164  Loading audio file 01-Sargon-Mindless.mp3
INFO root 2015-03-13 12:51:12,488 featextract.py:168  Computing Harmonic Percussive source     separation...
INFO root 2015-03-13 12:51:32,128 featextract.py:70  Computing Spectrogram...
Segmentation fault (core dumped)

from msaf.

Woah, that's weird. Can you try it with just one track? e.g.,

./run_msaf.py ../datasets/Sargon/audio/02-Sargon-Shattered\ World.mp3 -bid foote -lid fmc2d

What versions of librosa, numpy, and scipy do you have?

You might need to use the latest librosa devel branch (sorry, they are about to release the new version, which is the one I've been using with msaf devel).

Let me know if that fixes it. And thanks for your patience and perseverance!

from msaf.

Fyi, on my Ubuntu 14.04, msaf works with these library versions:

print numpy.__version__, scipy.__version__, librosa.__version__
1.9.2 0.15.1 0.4.0rc1

from msaf.

print numpy.__version__, scipy.__version__, librosa.__version__
1.8.2 0.14.0 0.3.1

this might be an issue. although if i do

apt-get install python-numpy python-scipy --upgrade

it says i have the newest versions. I will figure out how i get the versions you mention.

from msaf.

and it fails if i only do one track also

from msaf.

Yeah, msaf won't run on librosa 0.3, you need the newest librosa from the
devel branch.

However, this error message is pretty obscure.

Thanks for reporting this.
On Fri, Mar 13, 2015 at 1:41 PM jenmadtu [email protected] wrote:

and it fails if i only do one track also

—
Reply to this email directly or view it on GitHub
#3 (comment).

from msaf.

Ok i feel like we are getting there. I took another crack at it.

I feel like good things should be reported as well so this worked 👍

./run_msaf.py ../datasets/Sargon/audio/02-Sargon-Shattered\ World.mp3 -bid foote -lid fmc2d
INFO root 2015-03-17 05:33:30,391 featextract.py:164  Loading audio file 02-Sargon-Shattered World.mp3
INFO root 2015-03-17 05:34:02,225 featextract.py:168  Computing Harmonic Percussive source separation...
INFO root 2015-03-17 05:34:24,539 featextract.py:70  Computing Spectrogram...
INFO root 2015-03-17 05:34:33,708 featextract.py:77  Computing MFCCs...
INFO root 2015-03-17 05:34:34,161 featextract.py:81  Computing HPCPs...
INFO root 2015-03-17 05:34:42,637 featextract.py:87  Computing Tonnetz...
INFO root 2015-03-17 05:34:50,696 featextract.py:44  Estimating Beats...
[   0.            0.37151927   27.27183673   33.75020408   41.29668934
   48.34394558   62.39201814   65.90984127   93.76217687  115.06648526
  144.95056689  174.86947846  195.44235828  223.73587302  231.39845805
  247.11836735  263.68580499  280.16036281  299.71156463  324.60335601
  330.86693878] 330.866938776
INFO root 2015-03-17 05:35:00,548 run_msaf.py:110  Estimated times: [   0.            0.37151927   27.27183673   33.75020408   41.29668934
   48.34394558   62.39201814   65.90984127   93.76217687  115.06648526
  144.95056689  174.86947846  195.44235828  223.73587302  231.39845805
  247.11836735  263.68580499  280.16036281  299.71156463  324.60335601
  330.86693878]
INFO root 2015-03-17 05:35:00,550 run_msaf.py:111  Estimated labels: [5, 3, 4, 3, 4, 3, 4, 0, 2, 0, 0, 0, 0, 3, 3, 3, 3, 0, 1, 5]
INFO root 2015-03-17 05:35:00,550 run_msaf.py:118  Done! Took 90.16 seconds.

However this didnt

./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d -e
INFO root 2015-03-17 05:38:41,744 eval.py:260  Evaluating 4 tracks...
ERROR root 2015-03-17 05:38:41,758 input_output.py:139  Could not open JAMS file ../datasets/Sargon/estimations/01-Sargon-Mindless.jams
WARNING root 2015-03-17 05:38:41,759 eval.py:153  No estimations for file: ../datasets/Sargon/estimations/01-Sargon-Mindless.jams
ERROR root 2015-03-17 05:38:41,763 input_output.py:139  Could not open JAMS file ../datasets/Sargon/estimations/03-Sargon-Waiting For Silence.jams
WARNING root 2015-03-17 05:38:41,764 eval.py:153  No estimations for file: ../datasets/Sargon/estimations/03-Sargon-Waiting For Silence.jams
ERROR root 2015-03-17 05:38:41,773 input_output.py:139  Could not open JAMS file ../datasets/Sargon/estimations/04-Sargon-The Curse Of Akkad.jams
WARNING root 2015-03-17 05:38:41,774 eval.py:153  No estimations for file: ../datasets/Sargon/estimations/04-Sargon-The Curse Of Akkad.jams
ERROR root 2015-03-17 05:38:41,778 input_output.py:139  Could not open JAMS file ../datasets/Sargon/estimations/02-Sargon-Shattered World.jams
WARNING root 2015-03-17 05:38:41,778 eval.py:153  No estimations for file: ../datasets/Sargon/estimations/02-Sargon-Shattered World.jams
INFO root 2015-03-17 05:38:41,806 eval.py:274  4 tracks analyzed
INFO root 2015-03-17 05:38:41,819 eval.py:30  Results:
Series([], dtype: float64)
INFO root 2015-03-17 05:38:41,820 run_msaf.py:118  Done! Took 0.08 seconds.

from msaf.

Ok, we're getting there!
I believe this happened because you are evaluating a dataset that hasn't been segmented yet.

Try this to segment the whole dataset:
./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d

And then, to evaluate it, type this:
./run_msaf.py ../datasets/Sargon -bid foote -lid fmc2d -e

So yeah, it's a two-step process. The fact that you had problems with it makes me think it is not so intuitive?

from msaf.

or maybe i should just read the instructions carefully ;) everything seems to work now. I will try out your different segmentation algorithms largescale now :)

from msaf.

Awesome! If you are gonna do some large-scale segmentation, I recommend you use the -j n argument, where n is the number of threads you want to be using in your machine.

Have fun, and let me know if you have any more questions!

from msaf.