Comments (13)
solomonik
commented on August 14, 2024
Hi @bigerl, are you working with the master or python branch of CTF? The master branch has essentially no working python support, while the python branch has fairly mature support by now and a different build process. We are planning to merge that to master and have a release with python within two weeks, but are still tweaking things to make sure the build process handles new options (shared lib build and python build) and optional dependencies appropriately, and functionality works fine.
If you are on the python branch already, I'd be happy to help debug your build process there.
from ctf.
bigerl
commented on August 14, 2024
Hi @solomonik,
yes, I'm using the python branch. I'm looking forward for the release. Maybe I can replace my own sparse tensor einsum implementation then with this one. It seems quite mature. And thank you for helping.
from ctf.
solomonik
commented on August 14, 2024
@bigerl thanks for the detailed error report. It looks like you need to specify a path to the MPI libs. You can try to do so by adding the MPI library directory to LD_LIBRARY_PATH and PYTHON_PATH (keep in mind a shared mpi library is necessary). Alternatively (or additionally) you can try setting lib_dirs in setup.py. Please let me know if these work or run into issues, I haven't tried getting python working with OpenMPI yet. Suggestions for how the build process could have been improved to fix issues you encountered are also much appreciated since now is a good window to shape the CTF python build system.
from ctf.
bigerl
commented on August 14, 2024
thanks. I had to add the paths for headers and shared libraries to the setup.py's include_dirs and library_dirs. Then make pylibsucceeded.
But make python fails now with an error I don't know how it comes:
cd src_python && LDFLAGS="-L../lib_shared" python setup.py build_ext --inplace && cd ..
running build_ext
cd src_python && pip -vv install . --upgrade
Processing /home/me/Downloads/ctf/src_python
Running setup.py (path:/tmp/pip-ZoWrPF-build/setup.py) egg_info for package from file:///home/me/Downloads/ctf/src_python
Running command python setup.py egg_info
running egg_info
creating pip-egg-info/CTF.egg-info
writing pip-egg-info/CTF.egg-info/PKG-INFO
writing top-level names to pip-egg-info/CTF.egg-info/top_level.txt
writing dependency_links to pip-egg-info/CTF.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/CTF.egg-info/SOURCES.txt'
reading manifest file 'pip-egg-info/CTF.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/CTF.egg-info/SOURCES.txt'
Source in /tmp/pip-ZoWrPF-build has version 1.5.0, which satisfies requirement CTF==1.5.0 from file:///home/me/Downloads/ctf/src_python
Installing collected packages: CTF
Running setup.py install for CTF ... Running command /usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-ZoWrPF-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mU2j0C-record/install-record.txt --single-version-externally-managed --compile
running install
running build
running build_py
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/ctf
copying ctf/__init__.py -> build/lib.linux-x86_64-2.7/ctf
running build_ext
building 'ctf.core' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Ictf -I. -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/usr/include/mpich-x86_64 -I/usr/include/python2.7 -c ctf/core.cpp -o build/temp.linux-x86_64-2.7/ctf/core.o -std=c++11 -O0 -g
In file included from /usr/include/bits/libc-header-start.h:33:0,
from /usr/include/limits.h:26,
from /usr/lib/gcc/x86_64-redhat-linux/7/include/limits.h:194,
from /usr/lib/gcc/x86_64-redhat-linux/7/include/syslimits.h:7,
from /usr/lib/gcc/x86_64-redhat-linux/7/include/limits.h:34,
from /usr/include/python2.7/Python.h:19,
from ctf/core.cpp:45:
/usr/include/features.h:373:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
# warning _FORTIFY_SOURCE requires compiling with optimization (-O)
^~~~~~~
In file included from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ctf/core.cpp:587:
/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^~~~~~~
ctf/core.cpp:591:10: fatal error: ctf.hpp: No such file or directory
#include "ctf.hpp"
^~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
error
Cleaning up...
Removing source in /tmp/pip-ZoWrPF-build
Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-ZoWrPF-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mU2j0C-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-ZoWrPF-build/
Exception information:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 365, in run
strip_file_prefix=options.strip_file_prefix,
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 882, in install
spinner=spinner,
File "/usr/lib/python2.7/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-ZoWrPF-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mU2j0C-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-ZoWrPF-build/
make: *** [Makefile:105: python] Error 1
I think the important lines are:
ctf/core.cpp:591:10: fatal error: ctf.hpp: No such file or directory
#include "ctf.hpp"
^~~~~~~~~
compilation terminated.
Have you any suggestions?
edit:
I tried with python2+pip2 and python3+pip3 as well as with pip --user. Same result, always.
from ctf.
solomonik
commented on August 14, 2024
It looks like you need to run sudo make install or make install with choice of INSTALL_DIR in config.mk that is in your PATH. Sorry this is not well documented, we just changed the build system structure this weekend on the branch. After make python with a global CTF install works you should be to easily use CTF python though.
from ctf.
bigerl
commented on August 14, 2024
ok, finally got it working.
I set a INSTALL_DIR = /ctflib in config.mk.
Then I had to add use sudo su to run make install because sudo make install did not find mpicxx despite its directory was in sudo echo $PATH. Couldn't find out why.
make install in su failed first due to non-existing folders, but adding:
install: $(BDIR)/lib/libctf.a $(BDIR)/lib_shared/libctf.so
mkdir -p $(INSTALL_DIR)/lib/
mkdir -p $(INSTALL_DIR)/include/
...
resolved that.
To run non-su make python afterwards I had to add "/ctflib/include" to include_dirs and "/ctflib/lib" to lib_dirs in setup.py additionally. As well as adding --user to pip install:
python: pylib
cd src_python && pip install . --upgrade --user
I'll check now if it actually works. :)
from ctf.
bigerl
commented on August 14, 2024
I can't see how to use the package.
Could you maybe provide a minimal working example of for example a einsum of randomly filled tensors?
from ctf.
solomonik
commented on August 14, 2024
Hi @bigerl glad you where able to get it to build. INSTALL_DIR=/ctflib is a bit strange, typical is INSTALL_DIR=/usr/local/, which should have been there if you executed the latest version of ./configure successfully. On linux systems this is the typical way to install software from source.
For an example, this jupyter notebook has exactly what you are looking for with einsum
https://github.com/solomonik/ctf/blob/python/doc/CTF%20introductory%20demo.ipynb
It should look like this
http://solomonik.cs.illinois.edu/demos/CTF_introductory_demo.html
And hmm, I guess if you had to user --user in pip install, the you are not running python out of a virtualenv (which I would recommend, mostly have tested CTF with python 3.5). I will look into whether adding --user by default makes sense. Likely we'll also move toward trying to at least partially generate the setup.py from configure.
from ctf.
bigerl
commented on August 14, 2024
Thanks. That's exacly what I was looking for.
But I ran into another problem:
When I run now import ctf it doesn't find the linked library:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/__init__.py", line 1, in <module>
from ctf.core import *
ImportError: libctf.so: cannot open shared object file: No such file or directory
I have the same problem running the whole build process with python versions 2.7.14, 3.5.4 and 3.6.3. I now used virtualenv so I didn't use --user this time and I set INSTALL_DIR to the default /usr/local.
Do you have a tip where I have to add an path or where I should copy the libctf.so?
from ctf.
solomonik
commented on August 14, 2024
If you run make install, libctf.so should end up in the INSTALL_DIR, so we just need to make sure python finds that. That requires some combination of PATH, PYTHONPATH, LD_LIBRARY_PATH being set appropriately to point to the directory /usr/local/lib (check to make sure that's where libctf.so is). I believe LD_LIBRARY_PATH is likely what needs to be set, but mentioning them all in case I am wrong.
from ctf.
bigerl
commented on August 14, 2024
next step:
It was actually looking into /usr/local/lib but not for libctf.so but for ctf.so. But after I created a copy of the library with the appropriate name I got the next error when running import ctf:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /usr/local/lib/ctf.so: undefined symbol: cscal_
or with python -vv
# trying /home/me/Downloads/ctf/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/Downloads/ctf/ctf.abi3.so
# trying /home/me/Downloads/ctf/ctf.so
# trying /home/me/Downloads/ctf/ctf.py
# trying /home/me/Downloads/ctf/ctf.pyc
# trying /home/me/Downloads/ctf/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/Downloads/ctf/ctf.abi3.so
# trying /home/me/Downloads/ctf/ctf.so
# trying /home/me/Downloads/ctf/ctf.py
# trying /home/me/Downloads/ctf/ctf.pyc
# trying /usr/local/lib/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /usr/local/lib/ctf.abi3.so
# trying /usr/local/lib/ctf.so
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 938, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: /usr/local/lib/ctf.so: undefined symbol: cscal_
comment: I was working from the folder /home/me/Downloads/ctf/.
edit:
and here also the complete python -vv trace when /usr/local/lib/ctf.so doesn't exist.
>>> import ctf
# trying /home/me/Downloads/ctf/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/Downloads/ctf/ctf.abi3.so
# trying /home/me/Downloads/ctf/ctf.so
# trying /home/me/Downloads/ctf/ctf.py
# trying /home/me/Downloads/ctf/ctf.pyc
# trying /home/me/Downloads/ctf/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/Downloads/ctf/ctf.abi3.so
# trying /home/me/Downloads/ctf/ctf.so
# trying /home/me/Downloads/ctf/ctf.py
# trying /home/me/Downloads/ctf/ctf.pyc
# trying /usr/local/lib/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /usr/local/lib/ctf.abi3.so
# trying /usr/local/lib/ctf.so
# trying /usr/local/lib/ctf.py
# trying /usr/local/lib/ctf.pyc
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/ctf.abi3.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/ctf.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/ctf.py
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/ctf.pyc
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/plat-linux/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/plat-linux/ctf.abi3.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/plat-linux/ctf.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/plat-linux/ctf.py
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/plat-linux/ctf.pyc
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/lib-dynload/ctf.cpython-35m-x86_64-linux-gnu.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/lib-dynload/ctf.abi3.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/lib-dynload/ctf.so
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/lib-dynload/ctf.py
# trying /home/me/.pyenv/versions/3.5.4/lib/python3.5/lib-dynload/ctf.pyc
# /home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/__pycache__/__init__.cpython-35.pyc matches /home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/__init__.py
# code object from '/home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/__pycache__/__init__.cpython-35.pyc'
# trying /home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/core.cpython-35m-x86_64-linux-gnu.so
# destroy ctf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 697, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/me/.pyenv/versions/ctf354venv/lib/python3.5/site-packages/ctf/__init__.py", line 1, in <module>
from ctf.core import *
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 938, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: libctf.so: cannot open shared object file: No such file or directory
from ctf.
solomonik
commented on August 14, 2024
@bigerl the first error suggests that you also need to add shared BLAS libraries (libblas.so) to your link path, as ctflib.so
from ctf.
solomonik
commented on August 14, 2024
The build system in the latest release should make this process significantly smoother, so I am closing this issue. Please let me know if you still encounter any issues in building or execution.
from ctf.
Related Issues (20)
- Efficiency comparisons with einsum and opt-einsum and how to utilize symmetry? HOT 4
- ctf-einsum.py:78: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. HOT 1
- issues using sparse file io to load tensors HOT 4
- Compile error with GCC 11 HOT 10
- Wrong results when slicing a symmetric sparse tensor in python lib
- understanding performance overheads in CTF HOT 27
- ModuleNotFoundError: No module named 'ctf.core' HOT 1
- Comparing Fortran and CTF performance on symmetries in tensor contractions HOT 2
- segfault executing sparse inner product HOT 18
- oom/memory corruption running an SDDMM (using TTTP specialized routine) HOT 1
- unexpected performance for SpMV operation HOT 1
- [question] setting all nonzeros to a value HOT 2
- Warnings
- make test failure in SVD test HOT 3
- compiling issue when including ctf.hpp
- test_suite failure on Apple HOT 1
- compile issues with undefined references to mkl commands (that appear in the relevant folders) HOT 9
- Scale with Endomorphism
- Set values with low memory footprint
- ctf header error during installation on perlmutter HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctf.