Make issues for Python module (macOS) about ai-toolbox HOT 7 CLOSED

This post suggests to just rename the .dylib into a .so, and that not being able to load .dylibs directly is just a Python problem. This issue also refers to something similar.

This solves the problem. Thank you very much!

In summary, no need to install new compiler. Apple g++ can do the work. All dependencies can be installed directly from homebrew. Changing to set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY") in CMakeLists.txt is indeed needed for Boost 1.68. --- For Python3.7, change COMPONENTS python3 REQUIRED to COMPONENTS python37 REQUIRED (line 167 of CMakeList.txt) and set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON37_LIBRARY") (line 168 of CMakeList.txt)

Only one caveat for me to make this work --- if I install lp_solve through homebrew, I need to create a folder /usr/local/include/lpsolve and move (or copy to be safe) the created headers (lp_*.h) from /usr/local/include to /usr/local/include/lpsolve. It seems that homebrew install behaves differently from apt install on Ubuntu.

I left my working bash file here for anyone who might be interested

#!/bin/bash

mkdir build
cd build

PYTHON_VERSION=`python -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)";`
PYTHON_LIBRARY=/usr/local/Frameworks/Python.framework/Versions/$PYTHON_VERSION/lib/libpython$PYTHON_VERSION.dylib
PYTHON_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Versions/$PYTHON_VERSION/Headers/
##only build MDP to save compiling time
#cmake -DPYTHON_LIBRARY=$PYTHON_LIBRARY \
    #-DPYTHON_INCLUDE_DIR=$PYTHON_INCLUDE_DIR \
    #-DMAKE_MDP=1 -DPYTHON_VERSION=2 -DMAKE_PYTHON=1 ..
cmake -DPYTHON_LIBRARY=$PYTHON_LIBRARY \
    -DPYTHON_INCLUDE_DIR=$PYTHON_INCLUDE_DIR ..

from ai-toolbox.

Only one caveat for me to make this work --- if I install lp_solve through homebrew, I need to create a folder /usr/local/include/lpsolve and move (or copy to be safe) the created headers (lp_*.h) from /usr/local/include to /usr/local/include/lpsolve

I witnessed the same; with current brew, the lp_*.h headers install directly into /opt/homebrew/include rather than into /opt/homebrew/include/lpsolve, where they would have been found by the FindLpSolve.cmake module. The quick workaround I did was just making /opt/homebrew/include/lpsolve a soft link into /opt/homebrew/include/. I think a proper fix would require removing the assumption in the cmake module as well as in a bit of code that lp_lib.h (for instance) can always be included as <lpsolve/lp_lib.h>: that's a good convention for how packages should provide their includes, but apparently lpsolve doesn't force that convention (the headers themselves don't make that assumption which I guess(?) is why the brew formula was set up this way).

(If the brew formula changed to place the headers under lpsolve, that would also work (but would again break if in the future some other package provider didn't do that)).

from ai-toolbox.

Thanks for the well reviewed question.

Some notes:

• In the cmake command, -MAKE_LIB=1 -MAKE_PYTHON=1 should be -DMAKE_LIB=1 -DMAKE_PYTHON=1 or they are not going to be read. In this case it doesn't matter though, it's just for clarification. You may also want to set -DPYTHON_VERSION=2, but it's also probably not necessary.
• I have no idea why make (gmake) is being called during the cmake call. In particular, there's some mentions in there of a foo dependency (Scanning dependencies of target foo), and I have no idea what those are supposed to be.
• It's also very weird that you end up with a AIToolbox.dylib file if the linking process fails.. usually nothing should be produced at all. I don't know how this could have happened.

An important thing is that you mention changing set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY"). I'm not sure whether this is 100% correct: please make sure that it is. You can test this by simply adding a:

message(${Boost_PYTHON27_LIBRARY}) 

and checking that the found library is printed. If nothing is printed, it means that the library won't be linked anymore, which can result in the errors you are seeing. If that's the case, please revert to Boost_PYTHON_LIBRARY (and again, check with message that the library filename is in there) - if not, ignore this step.

I have found by Googling some mentions of a similar problem: here, here, and here.

It seems another possible way this issue can happen is by some version mismatch between the Python used to compile the boost you installed via homebrew and the Python you are trying to link now in the library. The links suggest using Python 3 and reinstalling boost while explicitly removing Python 2 support - maybe doing the opposite in your case may also work. Unfortunately I'm not familiar with how homebrew works, so I can't be sure. Maybe you could also try to see how the boost you installed was compiled and check if the various versions are the same.

Let me know if you find anything out, hopefully we can fix this problem :)

from ai-toolbox.

Ah, and as a last note: when trying something out in CMake in order to make stuff compile, it's usually better to always start from a clean slate, or CMake files might mess new runs up. This means deleting the build folder and re-creating it for every attempt. This is annoying but it really prevents some weird classes of errors (at least when changing important parameters such as which libraries to link and so on).

Since in your case this would require re-compiling everything every time, you might want to only specify the library flags -DMAKE_MDP=1 and -DMAKE_PYTHON=1. This means that you only try to compile the MDP part with Python (no tests, no examples, ...), which should take less time than trying to compile everything.

Once you are able to produce a viable .dylib, you can remove the options from the CMake call and get everything.

from ai-toolbox.

Thanks for the quick reply. Regarding your comments,

• In the cmake command, -MAKE_LIB=1 -MAKE_PYTHON=1 should be -DMAKE_LIB=1 -DMAKE_PYTHON=1 or they are not going to be read. In this case it doesn't matter though, it's just for clarification. You may also want to set -DPYTHON_VERSION=2, but it's also probably not necessary.

Thanks, these changes have been made. As expected, this mistake itself won't solve the error in binding Python.

• I have no idea why make (gmake) is being called during the cmake call. In particular, there's some mentions in there of a foo dependency (Scanning dependencies of target foo), and I have no idea what those are supposed to be.

I think make(gmake) is called for testing "IPO / LTO not supported", for which a simple and temporary file foo.cpp is created. However, I am not 100% sure.

• It's also very weird that you end up with a AIToolbox.dylib file if the linking process fails.. usually nothing should be produced at all. I don't know how this could have happened.

Sorry for the confusion. The AIToolbox.dylib is only created when g++ shipped with macOS (clang-1000.10.44.4) is used. Neverthless, this AIToolbox.dylib cannot be imported into Python. When gcc-7 from homebrew is used, AIToolbox.dylib cannot be created. The make process ends at around 47% with messages shown in the original post.

An important thing is that you mention changing set(BOOST_PYTHON_LIBRARY_NAME "Boost_PYTHON27_LIBRARY"). I'm not sure whether this is 100% correct: please make sure that it is. You can test this by simply adding a:

message(${Boost_PYTHON27_LIBRARY}) 

and checking that the found library is printed. If nothing is printed, it means that the library won't be linked anymore, which can result in the errors you are seeing. If that's the case, please revert to Boost_PYTHON_LIBRARY (and again, check with message that the library filename is in there) - if not, ignore this step.

I did the test. ${Boost_PYTHON27_LIBRARY} is the correct one. I think this may be related to the boost I am using (1.68).

I have found by Googling some mentions of a similar problem: here, here, and here.

It seems another possible way this issue can happen is by some version mismatch between the Python used to compile the boost you installed via homebrew and the Python you are trying to link now in the library. The links suggest using Python 3 and reinstalling boost while explicitly removing Python 2 support - maybe doing the opposite in your case may also work. Unfortunately I'm not familiar with how homebrew works, so I can't be sure. Maybe you could also try to see how the boost you installed was compiled and check if the various versions are the same.

I'll try the solutions there to see if any luck. I'll keep you posted on the results.

from ai-toolbox.

Sorry for the confusion. The AIToolbox.dylib is only created when g++ shipped with macOS (clang-1000.10.44.4) is used. Neverthless, this AIToolbox.dylib cannot be imported into Python. When gcc-7 from homebrew is used, AIToolbox.dylib cannot be created. The make process ends at around 47% with messages shown in the original post.

Ahhhhh, sorry, it was my bad for misunderstanding. I then have the strong suspicion that if you managed to compile it with the default g++, then you should be able to make it work. I wouldn't try to switch gcc, as this may also cause problems (if you compile something with a compiler version - in this case could be boost - and something else with another version - in this case the ilbrary - weird stuff might also happen since C++ does not have an ABI).

This post suggests to just rename the .dylib into a .so, and that not being able to load .dylibs directly is just a Python problem. This issue also refers to something similar.

Let me know if this solves it!

from ai-toolbox.

Awesome! And thanks a lot for leaving scripts and instructions!!

I'll close this issue then, if you need help for anything else feel free to open more issues :)

from ai-toolbox.