knowledgetechnologyuhh / omgemotionchallenge Goto Github PK

I can get 0.78 score in training set while the one is only 0.04 on validation set. What's wrong with my method? Who can give me some advices?

I think there is a mistake in line 82 and line 83 of caculateEvalutionCCC.py

cccArousal.append(ccc(validationArousal,modelArousal)) cccValence.append(ccc(validationArousal, modelArousal))

Which data (test or validation) was used to produce the baseline results in the paper? Thanks!

Hi,

In the 'omg_TrainVideos.csv', corresponding to each utterance clip, there is only one pair of valence-arousal values written. But inside the 'DetailedAnnotation' folder, if we go to that utterance, we can find multiple text files containing various valence-arousal values. So, how is each utterance split? For example, in OMGEmotionChallenge-master/DetailedAnnotation/train/0e02ee3c5_2/utterance_11.mp4/utterance_11.mp4', I can see 0-4 text files.
Kindly help me understand this.

Thanks,
Vandana

Running prepare.py e.g. with the command python prepare_data.py --target-dir "./train_data" --split-file "./omg_TrainVideos.csv" return the error The system cannot find the file specified as files such as https://www.youtube.com/watch?v=CROcsl0lIec are not available

Please have a check for the video named 'c0f84b343_2' in validation set. Because while I was preparing the dataset, I found that this video is no longer available online. Maybe you can delete it from validation set, or you can provide another link for this video.

/newdisk/OMGEmotionChallenge/prepare_data.py(44)prepare_data()
-> print(video+'is extracting...')
(Pdb) video
'c0f84b343_2'
(Pdb) n
c0f84b343_2is extracting...
> /newdisk/OMGEmotionChallenge/prepare_data.py(45)prepare_data()
-> result_dir = os.path.join(os.path.join(target_dir, video))
(Pdb) n
> /newdisk/OMGEmotionChallenge/prepare_data.py(46)prepare_data()
-> if not os.path.exists(result_dir):
(Pdb) n
> /newdisk/OMGEmotionChallenge/prepare_data.py(48)prepare_data()
-> result_filename = os.path.abspath(os.path.join(result_dir, utterance))
(Pdb) n
> /newdisk/OMGEmotionChallenge/prepare_data.py(51)prepare_data()
-> target_file = os.path.abspath(os.path.join(temp_directory, video + ".mp4"))
(Pdb) n
> /newdisk/OMGEmotionChallenge/prepare_data.py(52)prepare_data()
-> if not os.path.exists(target_file):
(Pdb) n
> /newdisk/OMGEmotionChallenge/prepare_data.py(53)prepare_data()
-> dl_youtube(link, target_file)
(Pdb) n
[youtube] iD205b51hZg: Downloading webpage
[youtube] iD205b51hZg: Downloading video info webpage
ERROR: iD205b51hZg: YouTube said: This video is unavailable.

I try to download the youtube videos. However, lots of them have been unavailable. I just download half of them.
Could you provide the whole dataset?

For example, the first utterance in the training set: https://www.youtube.com/watch?v=CROcsl0lIec

Many thanks

could you share the reference labels for the test set?

tks

Please Note. The following video in the test set is not available.

URL: https://www.youtube.com/watch?v=-WbUH7jJ9qQ
video_id: 5ba7890ab_1

Hi,

I found that there are quite a lot of mismatches between the categorical and dimensional annotations. For example, in case of anger (+ve arousal and -ve valence), some instances are labelled with +ve valence. Similarly for other emotions as well. Doesn't this question the validity of the annotations?

Thanks,

I looked at the caculateEvaluationCCC.py and I found something confusing.
Previously, the CCC was calculated for each video and the mean of CCCs can evaluate model performance, as denoted by following codes.

  
    cccArousal = numpy.array(cccArousal)
    cccValence = numpy.array(cccValence)
    print ("CCC Arousals:", cccArousal)
    print ("CCC Valences:", cccValence)

    print ("Mean CCC Arousal:", cccArousal.mean())
    print ("Mean CCC Valence:", cccValence.mean())

Now, the CCC seems to be calculated using all the utterances from validation set, without considering their corresponding videos.

    dataYArousal = dataY["arousal"]
    dataYValence = dataY["valence"]
    dataYPredArousal = dataYPred["arousal"]
    dataYPredValence = dataYPred["valence"]

    arousalCCC, acor = ccc(dataYArousal, dataYPredArousal)
    arousalmse = mse(dataYArousal, dataYPredArousal)
    valenceCCC, vcor = ccc(dataYValence, dataYPredValence)
    valencemse = mse(dataYValence, dataYPredValence)

It's a little strange. Shouldn't CCC be calculated for each video and then be averaged over the validation set? And which method do you use in your baseline model evaluation?

I'll be grateful to your reply.

Hello, is it correct that test split does not have emotion labels like train and validation splits? Cannot find them anywhere. Thanks