Unable to run mii-sd.py for txt2img benchmark about deepspeed-mii HOT 7 CLOSED

I think you're doing things correct. Let me try to follow these steps in a fresh environment and see if I can reproduce.

from deepspeed-mii.

@aponte411 I'm not able to reproduce the protobuf error installing with a clean environment. pip installs protobuf==3.20.3 for me. But regarding the other issue after you have set the environment variable... What version of transformers do you have? I just checked and it looks like the latest transformers version is breaking our SD example. I would recommend version 4.24.0 until we can find a solution.

from deepspeed-mii.

@aponte411 try this PR on the DeepSpeed side, should fix the config problem you are seeing:
microsoft/DeepSpeed#2589

from deepspeed-mii.

@mrwyattii works when installing from master :)

from deepspeed-mii.

What's the version of protobuf you have in this environment? pip list | grep proto

Also, you can avoid this by setting the following environment variable (but as the error message notes, it could add latency to the message passing): export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python.

from deepspeed-mii.

@mrwyattii when I install using pip install deepspeed[sd] deepspeed-mii it installs protobuf==4.21.11. Using this version, I ran into the TypeError: Descriptors cannot be created directly. I tried downgrading but that may have messed it up. I tried again setting that env variable, but keeping this version 4.21.11, and that seems to have resolved the protobuf issue but now the AttributeError from this issue seems to pop back up.

I checked and I am using deepspeed==0.7.6 and deepspeed-mii==0.0.4. Am I missing something?

from deepspeed-mii.

Looks like the protobuf issue was due to my local setup. Creating a new venv with a fresh install fixes it! @mrwyattii I'm using transformers==4.24.0 but I am still seeing the DSUNet config issue.

from deepspeed-mii.