Fail of the build process if hwloc is not found about libmsr HOT 13 CLOSED

Hi @danielecesarini, thanks for the feedback! Looks like I missed a couple things to enable downloading hwloc from source. I pushed the changes to my libmsr fork here. Can you give this branch a try and let me know if this resolves the build issue? Thank you!

from libmsr.

It's a bit better, but there are still some bugs.

-- The C compiler identification is Intel 17.0.4.20170411
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Downloading and building Hwloc from source
-- [+] Adding external project: hwloc
-- ExternalDownload Hwloc
-- HWLOC_INCLUDE_DIRS = HWLOC_INCLUDE_DIRS-NOTFOUND
-- HWLOC_LIBRARY = HWLOC_LIBRARY-NOTFOUND
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components: doxygen dot
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
HWLOC_INCLUDE_DIRS
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
used as include directory in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
HWLOC_LIBRARY
linked by target "msr-static" in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src
linked by target "msr" in directory /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src

-- Configuring incomplete, errors occurred!

from libmsr.

I just pushedv another change to grab the paths to HWLOC_INCLUDE_DIRS and HWLOC_LIBRARY when building hwloc from source. Note, the hwloc download will happen when you run 'make', and just paths will be set in the cmake stage.

from libmsr.

It doesn't find hwloc.h during the make stage because the libmsr library starts to compile before the hwloc stages (download & compilation).

-- The C compiler identification is Intel 17.0.4.20170411
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Downloading and building Hwloc from source
-- [+] Adding external project: hwloc
-- ExternalProject Hwloc
-- HWLOC_INCLUDE_DIRS = /galileo/home/userexternal/dcesarin/workspace_hpc/git/builds/libmsr/src/thirdparty_builtin/libhwloc-prefix/include
-- HWLOC_LIBRARY = /galileo/home/userexternal/dcesarin/workspace_hpc/git/builds/libmsr/src/thirdparty_builtin/libhwloc-prefix/lib/libhwloc.so
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components: doxygen dot
-- Configuring done
-- Generating done
-- Build files have been written to: /galileo/home/userexternal/dcesarin/workspace_hpc/git/builds/libmsr
[dcesarin@node166 libmsr]$ make
Scanning dependencies of target msr-static
[ 1%] Building C object src/CMakeFiles/msr-static.dir/cpuid.c.o
[ 3%] Building C object src/CMakeFiles/msr-static.dir/csr_core.c.o
[ 4%] Building C object src/CMakeFiles/msr-static.dir/csr_imc.c.o
[ 6%] Building C object src/CMakeFiles/msr-static.dir/memhdlr.c.o
[ 7%] Building C object src/CMakeFiles/msr-static.dir/libmsr_error.c.o
[ 9%] Building C object src/CMakeFiles/msr-static.dir/msr_clocks.c.o
[ 10%] Building C object src/CMakeFiles/msr-static.dir/msr_core.c.o
/galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src/msr_core.c:39:19: fatal error: hwloc.h: No such file or directory
#include <hwloc.h>
^
compilation terminated.
make[2]: *** [src/CMakeFiles/msr-static.dir/msr_core.c.o] Error 1
make[1]: *** [src/CMakeFiles/msr-static.dir/all] Error 2
make: *** [all] Error 2

from libmsr.

Interesting. On my system when I run make, the first step (before compiling the object files) is to download and build hwloc from source. The configure step identifies where the library and includes will be located. Sometime Can you delete CMakeCache.txt and src/thirdparty_builtin and try cmake . && make VERBOSE=1 again? If this fails, can you send me your CMakeCache.txt and the output of make?

from libmsr.

I always run cmake and make in an empty folder.

Yep!
CMakeCache.txt
output_cmake_make.txt

from libmsr.

Thanks! Hmm I wonder if it's because libmsr fails when doing an out-of-source build. We have a PR to support this, which is high on our to-do list.

Can you try running cmake . && make from the libmsr top-level directory? The beginning of my make shows the git clone steps for hwloc.

labasan1@impetus:~/for_impetus/libmsr (slabasan-fix-hwloc-download)]$ make
[  2%] Creating directories for 'libhwloc'
[  4%] Performing download step (git clone) for 'libhwloc'
Initialized empty Git repository in /g/g19/labasan1/for_impetus/libmsr/src/thirdparty_builtin/hwloc/.git/
remote: Enumerating objects: 377, done.
remote: Counting objects: 100% (377/377), done.
remote: Compressing objects: 100% (142/142), done.
^Cmake[2]: *** [src/thirdparty_builtin/libhwloc-prefix/src/libhwloc-stamp/libhwloc-download] Interrupt

from libmsr.

Same issue if I run cmake . && make from the libmsr top-level directory. I wait news about that, thanks for the support!

[dcesarin@node166 libmsr]$ cmake . && make
-- The C compiler identification is Intel 17.0.4.20170411
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc
-- Check for working C compiler: /cineca/prod/compilers/intel/pe-xe-2017/binary/bin/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Downloading and building Hwloc from source
--  [+] Adding external project: hwloc
-- ExternalProject Hwloc
--   HWLOC_INCLUDE_DIRS = /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src/thirdparty_builtin/libhwloc-prefix/include
--   HWLOC_LIBRARY = /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src/thirdparty_builtin/libhwloc-prefix/lib/libhwloc.so
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components:  doxygen dot 
-- Configuring done
-- Generating done
-- Build files have been written to: /galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr
Scanning dependencies of target msr-static
[  1%] Building C object src/CMakeFiles/msr-static.dir/cpuid.c.o
[  3%] Building C object src/CMakeFiles/msr-static.dir/csr_core.c.o
[  4%] Building C object src/CMakeFiles/msr-static.dir/csr_imc.c.o
[  6%] Building C object src/CMakeFiles/msr-static.dir/memhdlr.c.o
[  7%] Building C object src/CMakeFiles/msr-static.dir/libmsr_error.c.o
[  9%] Building C object src/CMakeFiles/msr-static.dir/msr_clocks.c.o
[ 10%] Building C object src/CMakeFiles/msr-static.dir/msr_core.c.o
/galileo/home/userexternal/dcesarin/workspace_hpc/git/libmsr/src/msr_core.c:39:19: fatal error: hwloc.h: No such file or directory
 #include <hwloc.h>
                   ^
compilation terminated.
make[2]: *** [src/CMakeFiles/msr-static.dir/msr_core.c.o] Error 1
make[1]: *** [src/CMakeFiles/msr-static.dir/all] Error 2
make: *** [all] Error 2

from libmsr.

@danielecesarini I am not able to reproduce your error on my system. Are you available for a meeting to share your screen and debug interactively? I'm PST, so we can do the meeting in the morning my time (end of business day for you), or vice versa.

from libmsr.

Hi @slabasan,

I have fixed the bug, take a look on my pull request on your fork.

https://github.com/danielecesarini/libmsr/tree/dcesarin-fix-hwloc-dep

from libmsr.

@danielecesarini Excellent! I'll get this merged in soon. What kernel and compiler were you using? I want to make sure we test against your configuration moving forward.

from libmsr.

I tested on a Centos 7 with kernel 3.10.0-327.36.3.el7.x86_64 and I used both GCC 6.1.0 and ICC 17.0.4 compilers.

from libmsr.

Sorry @slabasan, I have found another bug.

We have to exclude the dependency from hwloc when the autodiscovery stage of libmsr get success to find a hwloc library in the local system otherwise it raises a warning.

from libmsr.