I can't solve this error: "libtorch_python.so: undefined symbol" about stonne HOT 17 CLOSED

Hi @Kiminje ,

Try to run python setup clean in stonne_connection and then run install again.

After running step 2, have you tried to import torch from a Python script? If so, does it work?

from stonne.

Since same error occurred, I can't run clean my setup.py in stonne-connection.

• you mean that I should build using command 'make local' on pytorch-frontend for using script? (I understand about your 'script' meaning folder name 'scripts')
  I clean my previous build in pytorch-frontend and rebuild using 'make local' just now.
  Thanks for your fast reply.

from stonne.

You could try it, but it should run using the script. Try to clean the build directory in pytorch-frontend and in pytorch-frontend/stonne_connection and try the three steps again.

After installing the second step, try to open a Python console and run "import torch" command. Let me know if you can import the package. If so, go to step 3.

from stonne.

I have following error statement after installing second step.
Python 3.7.10 (default, Feb 26 2021, 18:47:35)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import torch
Traceback (most recent call last):
File "", line 1, in
File "/home/inje/stonne/pytorch-frontend/torch/init.py", line 384, in
from .tensor import Tensor
File "/home/inje/stonne/pytorch-frontend/torch/tensor.py", line 9, in
from torch._C import _add_docstr
ImportError: cannot import name '_add_docstr' from 'torch._C' (unknown location)

from stonne.

mm, actually, this has more to do with the Pytorch installation. It could seem weird, but have you tried to import the module from other path? If your working directory is pytorch-frontend when you open the python console you get that error. So maybe that is what is happening? Try to do so (import torch from a Python console) from other path and let me know if the problem persists.

from stonne.

Yes, I understood now that should not run python in the compiled folder.
Here is my full command line.

from stonne.

Apparently, the installation of pytorch was not successful then. Otherwise, you would be capable of importing it. How does the step 2 finish? Does it say it was installed correctly? I suppose it doesn't. So, what is the error?

from stonne.

This is a part of log when build in pytorch-frontend. And I also attached error.txt after processing step 2.
error.txt

-- ******** Summary ********
-- CMake version : 3.19.6
-- CMake command : /home/inje/anaconda3/envs/test/bin/cmake
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 7.5.0
-- CXX flags : -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -Wnon-virtual-dtor
-- Build type : Release
-- Compile definitions : ONNX_ML=1;ONNXIFI_ENABLE_EXT=1
-- CMAKE_PREFIX_PATH : /home/inje/anaconda3/envs/test/lib/python3.7/site-packages;/usr/local/cuda-10.2
-- CMAKE_INSTALL_PREFIX : /home/inje/stonne/pytorch-frontend/torch
-- CMAKE_MODULE_PATH : /home/inje/stonne/pytorch-frontend/cmake/Modules;/home/inje/stonne/pytorch-frontend/cmake/public/../Modules_CUDA_fix

-- ONNX version : 1.4.1
-- ONNX NAMESPACE : onnx_torch
-- ONNX_BUILD_TESTS : OFF
-- ONNX_BUILD_BENCHMARKS : OFF
-- ONNX_USE_LITE_PROTO : OFF
-- ONNXIFI_DUMMY_BACKEND : OFF

-- Protobuf compiler :
-- Protobuf includes :
-- Protobuf libraries :
-- BUILD_ONNX_PYTHON : OFF
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
-- Adding -DNDEBUG to compile flags
-- MAGMA not found. Compiling without MAGMA support
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Performing Test HAVE_GCC_GET_CPUID
-- Performing Test HAVE_GCC_GET_CPUID - Success
-- Performing Test NO_GCC_EBX_FPIC_BUG
-- Performing Test NO_GCC_EBX_FPIC_BUG - Success
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- AVX compiler support found
-- AVX2 compiler support found
-- MKL_THREADING = OMP
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
-- Library mkl_intel_lp64: /home/inje/anaconda3/envs/test/lib/libmkl_intel_lp64.so
-- Library mkl_gnu_thread: /home/inje/anaconda3/envs/test/lib/libmkl_gnu_thread.so
-- Library mkl_core: /home/inje/anaconda3/envs/test/lib/libmkl_core.so
CMake Warning (dev) at /home/inje/anaconda3/envs/test/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
The package name passed to find_package_handle_standard_args (OpenMP_C)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)
cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindLAPACK.cmake:22 (FIND_PACKAGE)
cmake/Dependencies.cmake:1627 (find_package)
CMakeLists.txt:480 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/inje/anaconda3/envs/test/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
The package name passed to find_package_handle_standard_args (OpenMP_CXX)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)
cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindLAPACK.cmake:22 (FIND_PACKAGE)
cmake/Dependencies.cmake:1627 (find_package)
CMakeLists.txt:480 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Library gomp: -fopenmp
-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so
-- Library m: /usr/lib/x86_64-linux-gnu/libm.so
-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so
CMake Warning at cmake/Modules/FindMKL.cmake:385 (MESSAGE):
MKL libraries files are found, but MKL header files are not. You can get
them by conda install mkl-include if using conda (if it is missing, run
conda upgrade -n root conda first), and pip install mkl-devel if using
pip. If build fails with header files available in the system, please make
sure that CMake will search the directory containing them, e.g., by setting
CMAKE_INCLUDE_PATH.
Call Stack (most recent call first):
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindLAPACK.cmake:22 (FIND_PACKAGE)
cmake/Dependencies.cmake:1627 (find_package)
CMakeLists.txt:480 (include)

-- MKL library not found
-- Checking for [Accelerate]
-- Library Accelerate: BLAS_Accelerate_LIBRARY-NOTFOUND
-- Checking for [vecLib]
-- Library vecLib: BLAS_vecLib_LIBRARY-NOTFOUND
-- Checking for [openblas]
-- Library openblas: BLAS_openblas_LIBRARY-NOTFOUND
-- Checking for [openblas - pthread]
-- Library openblas: BLAS_openblas_LIBRARY-NOTFOUND
-- Checking for [goto2 - gfortran]
-- Library goto2: BLAS_goto2_LIBRARY-NOTFOUND
-- Checking for [goto2 - gfortran - pthread]
-- Library goto2: BLAS_goto2_LIBRARY-NOTFOUND
-- Checking for [acml - gfortran]
-- Library acml: BLAS_acml_LIBRARY-NOTFOUND
-- Checking for [blis]
-- Library blis: BLAS_blis_LIBRARY-NOTFOUND
-- Checking for [ptf77blas - atlas - gfortran]
-- Library ptf77blas: BLAS_ptf77blas_LIBRARY-NOTFOUND
-- Checking for [blas]
-- Library blas: BLAS_blas_LIBRARY-NOTFOUND
-- Cannot find a library with BLAS API. Not using BLAS.
-- LAPACK requires BLAS
-- Cannot find a library with LAPACK API. Not using LAPACK.
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
-- MKL_THREADING = OMP
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
-- Library mkl_intel_lp64: /home/inje/anaconda3/envs/test/lib/libmkl_intel_lp64.so
-- Library mkl_gnu_thread: /home/inje/anaconda3/envs/test/lib/libmkl_gnu_thread.so
-- Library mkl_core: /home/inje/anaconda3/envs/test/lib/libmkl_core.so
CMake Warning (dev) at /home/inje/anaconda3/envs/test/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
The package name passed to find_package_handle_standard_args (OpenMP_C)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)
cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindMKLDNN.cmake:23 (FIND_PACKAGE)
cmake/public/mkldnn.cmake:3 (find_package)
cmake/Dependencies.cmake:1671 (include)
CMakeLists.txt:480 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/inje/anaconda3/envs/test/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
The package name passed to find_package_handle_standard_args (OpenMP_CXX)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)
cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindMKLDNN.cmake:23 (FIND_PACKAGE)
cmake/public/mkldnn.cmake:3 (find_package)
cmake/Dependencies.cmake:1671 (include)
CMakeLists.txt:480 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Library gomp: -fopenmp
-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so
-- Library m: /usr/lib/x86_64-linux-gnu/libm.so
-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so
CMake Warning at cmake/Modules/FindMKL.cmake:385 (MESSAGE):
MKL libraries files are found, but MKL header files are not. You can get
them by conda install mkl-include if using conda (if it is missing, run
conda upgrade -n root conda first), and pip install mkl-devel if using
pip. If build fails with header files available in the system, please make
sure that CMake will search the directory containing them, e.g., by setting
CMAKE_INCLUDE_PATH.
Call Stack (most recent call first):
cmake/Modules/FindBLAS.cmake:99 (FIND_PACKAGE)
cmake/Modules/FindMKLDNN.cmake:23 (FIND_PACKAGE)
cmake/public/mkldnn.cmake:3 (find_package)
cmake/Dependencies.cmake:1671 (include)
CMakeLists.txt:480 (include)

from stonne.

I also try to find where the fixed path('paco') exists in the directory using grep, but I can't..

from stonne.

• did you install all the dependencies for Pytorch?
• Did you remove the build directory on pytorch-frontend before running the installation? (Sorry I think the build directory is on the repo and I should remove it. I still have to polish it)

from stonne.

I just installed packages(numpy, pyyaml, cmake) before so it could become problems.
so I installed all the dependencies(from https://pypi.org/project/torch/) common part. And then I remove and reinstall stonne repository, second step for building is going on now.

from stonne.

Perfect, then I think you should not have any problem to install the third step, right?

from stonne.

Sorry, I tried step 2 in non-CUDA computer, but neither occurred same error before.
(test) inje@franck:~$ python
Python 3.7.10 (default, Feb 26 2021, 18:47:35)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import torch
Traceback (most recent call last):
File "", line 1, in
File "/home/inje/anaconda3/envs/test/lib/python3.7/site-packages/torch/init.py", line 189, in
from torch._C import *
ImportError: /home/inje/anaconda3/envs/test/lib/python3.7/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZN5torch3jit7testing9FileCheck10check_nextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

If okay, Would you tell me your system info & installed dependencies list? mine is below.
(Ubuntu 18.04, GCC 7.3, anaconda python==3.7,
installed packages after build step 2
_libgcc_mutex 0.1 main
blas 1.0 mkl
brotlipy 0.7.0 py37h27cfd23_1003
bzip2 1.0.8 h7b6447c_0
ca-certificates 2021.5.25 h06a4308_1
certifi 2020.12.5 py37h06a4308_0
cffi 1.14.5 py37h261ae71_0
chardet 4.0.0 py37h06a4308_1003
cmake 3.19.6 h973ab73_0
cryptography 3.4.7 py37hd23ed53_0
dataclasses 0.8 pyh6d0b6a4_7
expat 2.4.1 h2531618_2
future 0.18.2 py37_1
idna 2.10 pyhd3eb1b0_0
intel-openmp 2021.2.0 h06a4308_610
krb5 1.18.2 h173b8e3_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.71.1 h20c2e04_1
libedit 3.1.20210216 h27cfd23_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libuv 1.40.0 h7b6447c_0
lz4-c 1.9.3 h2531618_0
mkl 2021.2.0 h06a4308_296
mkl-include 2021.2.0 h06a4308_296
mkl-service 2.3.0 py37h27cfd23_1
mkl_fft 1.3.0 py37h42c9631_2
mkl_random 1.2.1 py37ha9443f7_2
ncurses 6.2 he6710b0_1
ninja 1.10.2 hff7bd54_1
numpy 1.20.2 py37h2d18471_0
numpy-base 1.20.2 py37hfae3a4d_0
openssl 1.1.1k h27cfd23_0
pip 21.1.1 py37h06a4308_0
pycparser 2.20 py_2
pyopenssl 20.0.1 pyhd3eb1b0_1
pysocks 1.7.1 py37_1
python 3.7.10 hdb3f193_0
pyyaml 5.4.1 py37h27cfd23_1
readline 8.1 h27cfd23_0
requests 2.25.1 pyhd3eb1b0_0
rhash 1.4.1 h3c74f83_1
setuptools 52.0.0 py37h06a4308_0
six 1.15.0 py37h06a4308_0
sqlite 3.35.4 hdfb4753_0
tk 8.6.10 hbc83047_0
torch 1.7.0a0+6197569 pypi_0 pypi
typing_extensions 3.7.4.3 pyha847dfd_0
urllib3 1.26.4 pyhd3eb1b0_0
wheel 0.36.2 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 haebb681_0

from stonne.

Umm.. I fixed it sir. my LD_LIBRARY_PATH is quite wrong.
i fix them right. (/home/inje/stonne/pytorch-frontend->/home/inje/stonne/pytorch-frontend/torch/lib)
Thank you so much for your help.

from stonne.

You are welcome! Does it work now?

from stonne.

yes! all of packages installed successfully except torchvision.
As you mentioned in installation guide, I try to install torchvision from source but it is failed.
Error shown in below. Is there some ways to install using pip?
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/inje/anaconda3/envs/test/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1521, in _run_ninja_build
env=env)
File "/home/inje/anaconda3/envs/test/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:
File "/home/inje/anaconda3/envs/test/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1237, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "/home/inje/anaconda3/envs/test/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1537, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

from stonne.

I solve this error.(install torchvision after build step 2, then uninstall only torch and reconfigure step 2)
I can run my benchmarks sir. Thank you!(very very)
p.s where is prebuilt-benchmarks?(bert, ssd-mobilenet, ssd-resnets...) I can't find them in stonne/benchmarks.

from stonne.