Unexpected Error about pipedream HOT 8 OPEN

What stage is this happening on? Usually this happens when we try to do something like an inplace addition of tensors. It usually can be resolved by cloning inputs (or choosing to set inplace=False for the relevant operator).

from pipedream.

It happened in the first stage, rank 0 and rank 1. I didn't change your code, and I tried to find where the inplace operation is but failed. What confused me is that vgg model runs successfully in hybrid mode, it only has two stages, while the ResNet has 3 stages.

from pipedream.

I see. This is a different model specification, so this is possible.

It's probably something to do with the inline ReLUs (https://github.com/msr-fiddle/pipedream/blob/master/runtime/image_classification/models/resnet50/gpus%3D4/stage0.py#L68), but I'm not sure exactly what.

One thing that should definitely fix this is replacing all instances of inline=True in the constructor to inline=False, but this will probably increase the memory footprint. You can also try adding some .clone()s.

I haven't run ResNet-50 in hybrid mode for a while since DP often outperforms the hybrid setup by often a lot; text around Table 1 in our paper talks about why this is the case.

from pipedream.

This may happen when you run pipedream with a recent version of PyTorch. I run with a version of Python 3.7 + PyTorch release 1.5.0 and have the same issue.

from pipedream.

PyTorch 3.8 or Python 3.8?

from pipedream.

Sorry, it might be close to the PyTorch latest release 1.5.0, or actually I compiled a recent commit from master branch of PyTorch github repo. Cannot match a PyTorch release version to it.

from pipedream.

@deepakn94 @SimonZsx @kanonjz
I also got this issue, but I tried with an older version of PyTorch

pip install torch===1.2.0 torchvision===0.4.0 -f https://download.pytorch.org/whl/torch_stable.html

This worked fine for me. I am also not quite sure why it is not working with the latest (I initially tried with 1.5.0).

@deepakn94 Could this be due to updates with the distributed autograd, etc from PyTorch end?
By the way, I didn't install the patch. Will this be a trouble for getting the expected results?

from pipedream.

See this #52 issue. The reason is that PyTorch added some new version consistency checking in the latest PyTorch version.

For the patch, it is only for profiling AFAIK. The runtime does not use the pre-hooking features provided by the patch.

from pipedream.