aim3-ruc / mmin Goto Github PK

Hello,

I recently downloaded the feature sets for the two datasets provided through the Baidu Cloud link. I'm particularly interested in understanding the nature of the feature extraction process for the three modalities. Could you kindly confirm if these features were obtained through a Word-aligned setting or an Unaligned setting? Clarity on this aspect will significantly aid in my analysis.

Thank you for your time and assistance.

According to the given run scripts, I found that "--share_weight" commond is only used in scripts\MSP_mmin.sh file, may I kindly inquire about the author's rationale for this decision? Thanks!

作者您好，感谢您的开源！您的代码框架对我深受启发，有以下问题希望解答：

Init parameter from checkpoints/mmin_block_5_run2 Traceback (most recent call last): File "train_miss.py", line 102, in <module> model = create_model(opt) # create a model given opt.model and other options File "/data/luowei/MMIN/models/__init__.py", line 65, in create_model instance = model(opt) File "/data/luowei/MMIN/models/mmin_model.py", line 67, in __init__ self.load_pretrained_encoder(opt) File "/data/luowei/MMIN/models/mmin_model.py", line 92, in load_pretrained_encoder self.pretrained_encoder.load_networks_cv(pretrained_path) File "/data/luowei/MMIN/models/base_model.py", line 206, in load_networks_cv assert len(load_filename) == 1, 'In folder: {}, Exists file {}'.format(folder_path, load_filename) AssertionError: In folder: checkpoints/mmin_block_5_run2/1, Exists file []

does it means I need to download the pretrained models?

运行CAP_MMIN.sh时，第143行tst_dataset的acc,uar把137行val_dataset的acc,uar给覆盖了，所以实际上每个训练epoch记录的是测试集上最好的效果。。

重新跑了一遍以后result_total.csv记录的结果如下：
acc uar f1
0.6010 0.6256 0.5926
0.6197 0.6551 0.6247
0.6542 0.6589 0.6573
0.6212 0.6604 0.6259
0.6060 0.5980 0.5992
0.5983 0.6048 0.6024
0.6544 0.6376 0.6166
0.6342 0.6311 0.6135
0.6545 0.6615 0.6506
0.5952 0.6127 0.5926
0.6239 0.6346 0.6175

您好！
我发现train_baseline.py中使用了完整数据集来进行预训练，但这个和实验设置是不相符的。请问这个有合理的解释或实验证明吗，如果使用其他数据来进行预训练是否会对实验结果产生影响？

I see in the file MSP_config that there is a path to the processed feature 'MSP-IMPROV_features_2021' of the MSP-IMPROV set. Can you share it so I can download it directly? Thank you!

模型处理的是部分模态完全缺失的问题还是其中某部分缺失的问题？
训练时的缺失率设置的是多少呢
谢谢解答

Has the code been deleted?

Hi, thank you for your open source. I can't download the IEMOCAP feature with the link https://pan.baidu.com/s/1WmuqNlvcs5XzLKfz5i4iqQ. Can you share in another source?

Hello， could you provide the code and model for extracting the visual features? Thanks a lot.

Hi. I've read the paper of MMIN. In the paper, "Modality Encoder Network" is pretrained and should be fixed during MMIN training process.

In the script of scripts/mmin.sh, it specify the pretrained path to "checkpoints/utt_fusion_AVL_run2". I wonder if it contains the pretrained "Modality Encoder Network".

If so, may you release this model?

If not, may you clarify the script, through which I can train the "Modality Encoder Network" myself?

Thx a lot

这里getitem获取每个样本时随机赋予一种缺失情况，
但是这里missing_index和miss_type都是随机抽取的，可能会不匹配？
例如missing_index = [0,1,1]，但是miss_type = 'azl'

Hi, thank you for your open source. I have some questions during reproduce the paper:

1. Typo in the code
   ./models/utt_fusion_model.py line18, 'lexical' should be 'visual'
   ./data/multimodal_dataset.py line70, 'proveee' should be 'process'

2. Some small questions:

   • why you choose BaseModel(ABC) as the parents class instead of nn.Module?
     some convenient method like model.to(device) and model.parameters() cannot be used.
     (In the code you used for-loop to apply these method to each layer, but it seems too complex)
   • In ./scripts, I did not find the file to train the baseline model with augmented data.
     But there is a ./data_aug.sh which run the train_miss model with augmented data. Is this a miswritting?

Thx!

ef load_pretrained_encoder(self, opt): #加载预训练的编码器网络
print('Init parameter from {}'.format(opt.pretrained_path))
pretrained_path = os.path.join(opt.pretrained_path, str(opt.cvNo))
pretrained_config_path = os.path.join(opt.pretrained_path, 'train_opt.conf')
pretrained_config = self.load_from_opt_record(pretrained_config_path)
pretrained_config.isTrain = False # teacher model should be in test mode
pretrained_config.gpu_ids = opt.gpu_ids # set gpu to the same
self.pretrained_encoder = UttFusionModel(pretrained_config)
self.pretrained_encoder.load_networks_cv(pretrained_path)
self.pretrained_encoder.cuda()
self.pretrained_encoder.eval()

mmin_model里这一部分opt.pretrained_path，指的是哪里，运行显示none，我想设一个默认值不知道指的是哪的路径。
请多指教，感谢！

File "~/MMIN-master/models/utt_fusion_model.py", line 40, in __init__
    self.modality = opt.modality
AttributeError: 'OptConfig' object has no attribute 'modality'

When I want to reproduce the code, this error occured! Could you just show me how to solve this?

您好，为什么代码中有IMPLICIT FUSION BY JOINT AUDIOVISUAL TRAINING FOR EMOTION RECOGNITION IN MONO MODALITY这篇文章的模型，但是论文中实验结果并没有和这篇文章进行比较呢

where is your code?

Hello, thanks for sharing your code.
Could you please tell me following which steps can I preprocess the data from the dataset.