Error while building project - Variable not defined: 'Shape_ExtentTensorType' about torch-mlir HOT 17 CLOSED

Are you using the llvm hash in the git submodule?
https://buildkite.com/iree

from torch-mlir.

I don't think so? Could you tell me how I can do that?

I think the following message while building (that I ignored) can have relevance here too:
realpath: /home/nithin/mlir-npcomp/build/iree/bindings/python: No such file or directory

from torch-mlir.

'git submodule init'
'git submodule update'
This will result in an llvm tree in external/llvm-project.

Or you can just checkout the git hash that is referenced (currently c89e46e) in your preferred llvm location
You will need the mlir-hlo submodule, however.

from torch-mlir.

I did initialize submodule inside mlir-npcomp and the above two errors persist.

from torch-mlir.

The above errors look like a mismatched LLVM version but it is hard to tell without further details.

I just realized that the instructions in the README are slightly out of date with respect to the LLVM dependency (they reference what we had before switching to a submodule). I'm going to fix them in a little bit, but if you could confirm the following it would help.

This section should be removed, and make sure that the LLVM_SRC_DIR variable is not set:

export LLVM_SRC_DIR=/path/to/llvm-project

# Check out last known good commit.
LLVM_COMMIT="$(cat ./build_tools/llvm_version.txt)"
(cd $LLVM_SRC_DIR && git checkout $LLVM_COMMIT)

Then git submodule update --init.

Then proceed with the build:

./build_tools/install_mlir.sh
./build_tools/cmake_configure.sh

# Build and run tests
# ./build_tools/test_all.sh runs all of these commands.
cd build
ninja
ninja check-npcomp

from torch-mlir.

Thank you, your suggestion solved the error however I am still unable to build successfully because of the following error:

./frontends/pytorch/csrc/aten_mlir_type_default.cpp:10:10: fatal error: 'ATen/CPUGenerator.h' file not found #include <ATen/CPUGenerator.h> ^~~~~~~~~~~~~~~~~~~~~

The file ATen/CPUGenerator.h isn't present in the source-compiled pytorch. However, I was able to find the file in a conda installation of pytorch.

from torch-mlir.

The PyTorch integration is very much not working out of the box yet. There are a couple of PRs pending that Stephen and I are working through. Also, the current dispatch approach is pegged to a strategy for an older version of PyTorch and we have work to do to upgrade.

from torch-mlir.

Could you please tell me which pytorch version mlir-npcomp is based on?

from torch-mlir.

It is one of the 1.3 versions, but it is presently very version sensitive. The only version that it builds against is what is packaged/built as part of the docker image defined here: https://github.com/llvm/mlir-npcomp/pull/33/files

We're still working through getting this in a shape for general use and there aren't even presubmit builds for it yet.

from torch-mlir.

Oh alright, got it. Thank you very much for all the information and prompt replies Stella.

from torch-mlir.

You're welcome - we are actively working on this. It just isn't there yet and is suffering a lot from "works on my machine"...

from torch-mlir.

@stellaraccident @stephenneuendorffer do you know what exact pytorch 1.3 SHA works on your system ? The Dockerfile just mounts an external pytorch directory but I couldn't locate the version.

from torch-mlir.

nvm - I think I found the SHA from the implicit 19.10-py3 Docker version.

from torch-mlir.

Great! if you use the docker image, you'll also have to rebuild pytorch as well. I think some of the cmake bits are not installed properly out of the box. Let me know if you have questions.

from torch-mlir.

Sounds good will keep you posted. Meanwhile have you tried with the latest Pytorch by any chance? We have some dependency on latest pytorch and have to use it for our runtime. Thx

from torch-mlir.

We're pretty sure it won't work. Parts of the pytorch integration are generated from export information that is generated by pytorch. This export information no longer has the same format, and there is a new mechanism for exporting information about the ATen interface. The plan is to rebuild things around this new source of export information, which should be more stable.

from torch-mlir.

The project has now graduated slightly past "it works on my machine" and our build/docs are more robust (though still not perfect). I haven't seen these issue crop up for anybody in months, so closing this bug.

from torch-mlir.