j-min / dalleval Goto Github PK

View Code? Open in Web Editor NEW

136.0 136.0 6.0 67.76 MB

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (ICCV 2023)

Home Page: https://arxiv.org/abs/2202.04053

License: MIT License

Dockerfile 0.03% Python 25.99% Shell 0.06% Jupyter Notebook 73.45% C++ 0.08% Cuda 0.39%

evaluation text-to-image vision-and-language

dalleval's People

Contributors

Stargazers

Watchers

Forkers

dashstander yuhongmo mbrukman irenesolaiman annisc jiazhi5

dalleval's Issues

TypeError while running score_clip.py

Hi Team,

Thank you for sharing your work.

I am trying to run experiments related to biases, and I am facing an error while running score_clip.py as shown below:

python score_clip.py --test
Namespace(image_dir='./test_images', out='./bias_scores_clip.json', test=True)
Loaded 1 neutral prompts
Loaded 2 classifier prompts
Loaded CLIP at cpu
  0%|                                                                                        | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "score_clip.py", line 83, in <module>
    classifier_probs = np.append(classifier_probs_gender[0])
  File "<__array_function__ internals>", line 198, in append
TypeError: append() missing 1 required positional argument: 'values'

I am not sure if classfier_probs was to be defined earlier, and then passed to the append call

Evaluation of Text2Img models with DETR

Great work on building this benchmark for text to image models!

In this page, the evaluation of text to image models with DETR is still "coming soon". Is there any plan to release the procedure for text to image model evaluation?

Recommend Projects