Sparrow Build Failure due to unlinked yaml-cpp about sparrow HOT 4 OPEN

Based on your output in qcscine/utilities#7 which was:

 -- Found package yaml-cpp at /home/njh80/modulesSource/miniforge3/envs/chemoton/share/cmake/yaml-cpp

you have a similar problem as with the Core dependency. Only that here the problem is not a different version, but that Sparrow cannot find the dependency with which Utils has been built.
It should work if you add the directory where libyaml.so is located (probably /home/njh80/modulesSource/miniforge3/envs/chemoton/lib to your LD_LIBRARY_PATH, but that could get you back to problems like the one in qcscine/utilities#7

Due to the dependencies between our modules, it is a bit difficult to mix in dependencies between different locations without setting up the environment carefully. In general, we recommend: only install boost, mongo-c, and mongo-cxx separately and then our bootstrap should be able to download and compile all dependencies and configure everything into one install directory.

So my hotfix recommendation would be: remove the yaml-cpp location from all your paths, delete the utils build directory (/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/utils) and start the bootstrap again. If the cmake of utils shows the message:

-- yaml-cpp was not found in your PATH, so it was downloaded.

and then first compiles the yaml-cpp library (noticeable by having more warnings printed), then you should have it properly installed within the puffin bootstrap and Sparrow should link fine.

Simply adding the yaml location to the LD_LIBRARY_PATH would be the faster solution if it works straightaway, because you don't need to compile utils again, but you need to be careful to always add this to your path if you want to use our software and this dependency will not be added to the puffin.sh file generated after the bootstrap

from sparrow.

Thanks again @steinmig! Looks like the LD_LIBRARY_PATH did the trick here. I'm not sure how yaml-cpp was installed into the virtual environment originally, maybe Conda installed it with gcc?

I managed to get a bit further with the bootstrapping up to Molassembler, but then there was a different problem. It's probably a similar problem but I thought I'd just add it to this thread (I can setup a fresh query if preferred).

This time Nauty isn't downloading. I've got a copy of nauty (2.8.6) elsewhere but it is not in the PATH, LD_LIBRARY_PATH, or CPATH. The identical error occurs even when I include my nauty on the PATH (I assume it is looking for 2.7.0).

Preparing Molassembler...

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Check for working C compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/cc
-- Check for working C compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/c++
-- Check for working CXX compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Scine::UtilsOS found locally at /rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/install/lib/cmake/ScineUtilsOS
-- Enabling GCC specific warning flags
-- Found MKL: /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/include
-- Found Intel MKL libraries:
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/include
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_avx2.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_vml_avx2.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_core.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_intel_lp64.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_gnu_thread.so
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found MKL for use with Eigen3
CMake Error at dev/cmake/ColorMessages.cmake:39 (message):
Could not download file: "Unsupported protocol" Please download the
file at https://scine.ethz.ch/static/download/nauty27r1.tar.gz and place it
in your build tree at
/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/molassembler/build/nauty27r1.tar.gz
Call Stack (most recent call first):
dev/cmake/DownloadFileHelper.cmake:18 (cmessage)
cmake/ImportNauty.cmake:23 (download_file)
CMakeLists.txt:77 (import_nauty)

-- Configuring incomplete, errors occurred!

I can do wget to the https fine in the folder but every time I rebootstrap Puffin I need to clear out the existing software to avoid the error:

File "/home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/scine_puffin/programs/program.py", line 121, in scine_module_install
repository.remotes.origin.pull()
File "/home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/git/remote.py", line 1068, in pull
res = self._get_fetch_info_from_stderr(proc, progress, kill_after_timeout=kill_after_timeout)
File "/home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/git/remote.py", line 858, in _get_fetch_info_from_stderr
proc.wait(stderr=stderr_text)
File "/home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/git/cmd.py", line 657, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v -- origin

Maybe editing the bootstrap.py so that it can clear and continue interrupted processes might help? For example:

core = Utils(config.programs()["core"])
if not os.path.isfile(install_dir + "/lib/libcore.so"):
shutil.rmtree(core_build_dir, ignore_errors=True)
core.install(core_build_dir, install_dir, config["resources"]["cores"])

from sparrow.

According to https://stackoverflow.com/questions/29816529/unsupported-protocol-while-download-tar-gz-package and similar search results, the problem is that your cmake installation does not support SSL.

A possible quick fix would be:

• Download the file and place it as instructed
• Instead of starting the bootstrap, go to /rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/molassembler/build/ and execute:
  • cmake ..
  • make -j4 install
    This might compile and install Molassembler together with the library correctly. Then you could try to relaunch the boostrap, which should hopefully detect that Molassembler is configured and installed, or detects now the installed nauty library and does not require the download. The later might require you to set:
    export LD_LIBRARY_PATH=/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/install/lib:/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/install/lib64:$LD_LIBRARY_PATH

Again, this is just a hot fix and might complicate things, if you are updating in the future. The cleaner version is to load/install a cmake version with SSL support

from sparrow.

Hi @steinmig, sorry for not getting back sooner. I've been having a go and essentially the way bootstrap of Puffin is coded it doesn't try to check if modules are already there first. It just starts the build in the same folder and evidently errors when it realises the things are already there, so that would need a change to the puffin bootstrap method.

I've got around this by editing the puffin module - I'd recommend it as a general fix. However, the module still causes some trouble. I've also had to manually pip install . in each python module because the puffin pip call was installing in the wrong place.

The CMake version should have SSL support but I didn't install this and I'm finding an install of a newer CMake a tad difficult.

Any ideas why the created in situ modules aren't happy? My understanding is that the file ImportNauty.cmake is meant to build a file nautyTargets.cmake but that this is being ignored.
CMakeCache.txt
ImportNauty.txt

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Check for working C compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/cc
-- Check for working C compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/c++
-- Check for working CXX compiler: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Scine::UtilsOS found locally at /rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/install/lib/cmake/ScineUtilsOS
-- Enabling GCC specific warning flags
-- Found MKL: /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/include
-- Found Intel MKL libraries:
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/include
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_avx2.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_vml_avx2.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_core.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_intel_lp64.so
-- /usr/local/Cluster-Apps/intel/2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64/libmkl_gnu_thread.so
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found MKL for use with Eigen3
CMake Error at build/nauty/nautyConfig.cmake:1 (include):
include could not find load file:

/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/molassembler/build/nauty/nautyTargets.cmake

Call Stack (most recent call first):
cmake/ImportNauty.cmake:12 (find_package)
CMakeLists.txt:77 (import_nauty)

CMake Error at build/RingDecomposerLib/RingDecomposerLibConfig.cmake:1 (include):
include could not find load file:

/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/molassembler/build/RingDecomposerLib/RingDecomposerLibTargets.cmake

Call Stack (most recent call first):
cmake/ImportRingDecomposerLib.cmake:11 (find_package)
CMakeLists.txt:83 (import_rdl)

-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- pybind11 v2.10.4
-- Found PythonInterp: /home/njh80/modulesSource/miniforge3/envs/chemoton/bin/python3 (found suitable version "3.8.18", minimum required is "3.6")
-- Found PythonLibs: /home/njh80/modulesSource/miniforge3/envs/chemoton/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Pybind11 was not found in your PATH, so it was downloaded.
-- Found PY_pytest: /home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/pytest
-- Found PY_scine_utilities: /home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/scine_utilities/scine_utilities.cpython-38-x86_64-linux-gnu.so
-- Could NOT find PY_pybind11_stubgen (missing: PY_PYBIND11_STUBGEN)
-- Not generating typechecking stubs for python package as pybind11-stubgen was not found
-- Found PY_pip: /home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/pip
-- Found PY_sphinx: /home/njh80/modulesSource/miniforge3/envs/chemoton/lib/python3.8/site-packages/sphinx
-- Configuring incomplete, errors occurred!
See also "/rds-d7/user/njh80/hpc-work/chemoton-conda/puffin/software/build/molassembler/build/CMakeFiles/CMakeOutput.log".

from sparrow.