nukezil / iomatch Goto Github PK

Thanks for your excellent work and the code!
I am trying to run the IOMatch, and I found the current code only provides accuracy on the close-set test set.
Could you please tell me how to evaluate on the open-set test set using this code?

Hi! Thanks for your remarkable work and the meticulously crafted open-source code. Your efforts are truly appreciated.

I'm reaching out to inquire about the model parameters for Open-Set SSL baselines. Currently, as we follow your experimental setting, we are suggested to compare the AUROC for both seen and unseen outliers.

[ICCV 2023] SSB: Simple but Strong Baseline for Boosting Performance of Open-Set Semi-Supervised Learning [PDF] [Code]

A concurrent work SSB has shown that OpenMatch and T2T can acchieve fair performance in both inliers classification and outliers detection, as depicted in the following Table A. However, these findings seem to contradict the data presented in the IOMatch report in the following Table B. Upon closer examination, it's evident that there are notable differences in evaluation protocols, as summarized in the following Table C.

Table A: SSB-reported data for OpenMatch and T2T, borrowing data from SSB in Table 8, 15, 16. (CIFAR-10, Inlier/outlier: 6/4, lables per samples: 25)

Table B: IOMatch-reported data OpenMatch and T2T, borrowing data from IOMatch in Table 1, 2, 6. (CIFAR-10, Inlier/outlier: 6/4, lables per samples: 25)

Table C: Comparsion between SSB and IOMatch in evaluation protocols, where "I" indicates inliers, "SO" indicates seen outliers, "UO" indicates unseen outliers, "ALL={CIFAR10, CIFAR100, ImageNet, SVHN, LSUN}"

Nevertheless, we're expected to verify their performance under aligned evaluation protocols. Could you kindly provide the parameters for all your well-trained Open-Set SSL baseline models, including UASD, DS3L, MTCF, T2T, OpenMatch, and SAFE-STUDENT, for CIFAR10, CIFAR100, and IMAGENET30 datasets?

Given that your IOMatch is the first work to propose such a challenging experiment setting under labeled data scarcity, sharing these baseline model parameters will be greatly beneficial for the community to better follow your work.

Many thanks in advance! Looking forward to your reply.

File "IOMatch/semilearn/datasets/cv_datasets/__init__.py", line 1, in <module>
    from .cifar import get_cifar, get_cifar_openset
ImportError: cannot import name 'get_cifar' from 'semilearn.datasets.cv_datasets.cifar' (IOMatch/semilearn/datasets/cv_datasets/cifar.py)

/semilearn/datasets/cv_datasets/cifar.py文件里没有get_cifar函数

Hello, thanks for the interesting work and code. I am trying to run the ImageNet-30, which however is not included in the current version. Could you kindly share the yaml files or some detailed hyper-parameters?

Thanks for your excellent work and providing the code!
I'm trying to run IOMatch and found that you provided the open-set test code for IOMatch. Could you please provide the open-set test code for OpenMatch?

The vision of ruamel.yaml in conda:
ruamel.yaml 0.18.5
ruamel.yaml.clib 0.2.8

The error info is:

Traceback (most recent call last):
  File "train.py", line 284, in <module>
    args = get_config()
  File "train.py", line 136, in get_config
    over_write_args_from_file(args, args.c)
  File "/hy-tmp/IOMatch/semilearn/core/utils/misc.py", line 23, in over_write_args_from_file
    dic = yaml.load(f.read(), Loader=yaml.Loader)
  File "/usr/local/miniconda3/envs/iomatch/lib/python3.8/site-packages/ruamel/yaml/main.py", line 1085, in load
    error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment)
  File "/usr/local/miniconda3/envs/iomatch/lib/python3.8/site-packages/ruamel/yaml/main.py", line 1037, in error_deprecation
    raise AttributeError(s)
AttributeError: 
"load()" has been removed, use

  yaml = YAML(typ='rt')
  yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data,
instead of file "/hy-tmp/IOMatch/semilearn/core/utils/misc.py", line 23

        dic = yaml.load(f.read(), Loader=yaml.Loader)