tut-arg / dcase2016-baseline-system-python Goto Github PK

Line no 101 in DCASE2016-baseline-system-python/src/evaluation.py is wrong, leading to incorrect evaluation results :
Incorrect Version :
accuracies_per_class = self.accuracies(y_true=system_output, y_pred=annotated_ground_truth, labels=self.class_list)
With correction :
accuracies_per_class = self.accuracies(y_true=annotated_ground_truth, y_pred=system_output, labels=self.class_list)

The on_after_extract() function in "src/dataset.py" at line no 1126 appears to expect a "evaluate.txt" file in the "evaluation_setup" folder which is not the case (it is named 'test.txt') in the dataset :
eval_filename = os.path.join(self.evaluation_setup_path, 'evaluate.txt')

Also, while generating the meta file out of the "evaluate.txt", since there are no labels, it results in an index error while accessing row[1] at line no 1136 of "src/dataset.py" :
meta_data[row[0]] = row[1]

Another suggestion - while initializing the challenge dataset in line no 178 of "task1_scene_classification.py", the data path is not considered. The data path could have been added similar to the development dataset initialization :
challenge_dataset = eval(params['general']['challenge_dataset'])()

Maybe there is a little error in the file /src/features.py when you get the 'window' vector for mfcc feature computation: window = scipy.signal.hamming(mfcc_params['n_fft'], sym=False). But I think we should use mfcc_params['win_length'] when get the hamming window.

The snippet to read a 24 bit PCM wav file in src/files.py , starting at L63, is directly copied from the 'wavio' repository. You can view the original here: https://github.com/WarrenWeckesser/wavio/blob/master/wavio.py.

I think an edit of the code with proper license attribution is warranted.

Hi,
I'm having trouble reproducing the baseline accuracies on my local machine, i.e. multicore Ubuntu with MATLAB 2013b. I'm running latest master on the repo.
Below are the results after calling the script task1_scene_classification:

The above is consistent across runs, due to the re-initialization of the random seed.
This is not too bad of a failure, but since it is sensibly below the reported accuracy of 72.4%, I was worried if something might be wrong it my setup.