ecmwf / eckit Goto Github PK

View Code? Open in Web Editor NEW

21.0 22.0 23.0 11.89 MB

A C++ toolkit that supports development of tools and applications at ECMWF.

Home Page: https://confluence.ecmwf.int/display/eckit

License: Apache License 2.0

CMake 2.19% Shell 0.08% C++ 92.27% C 4.49% Lex 0.23% Yacc 0.61% Python 0.12%

eckit's Introduction

eckit

Project home: https://software.ecmwf.int/wiki/display/ECKIT

ecKit is a cross-platform C++ toolkit that supports development of tools and applications at ECMWF. It is based on code developed over the last 20 years within the MARS software and was re-factored out to be reused by other applications. It provides a an abstraction layer on top of the operating system, so it is easier to port code to new architectures. It is developed taking into account the robustness requirements of running production systems at ECMWF. The main focus is UNIX/POSIX systems, and it has been thoroughly tested on AIX, Linux and Mac OSX. Historically, the code base pre-dates and in some way can be seen as a leaner substitute for some 'Boost' libraries.

It features facilities to easily handle, in a cross-platform way:

multi-threading
json and yaml parsing
object serialization and persistence
object serialization supporting compression, to and from file and network
configuration of user options and resources
file-system abstraction
regular expressions
socket networking
http protocol
type-to-type conversions
asynchronous IO
asynchronous processing
exception handling and stack dumping
MPI object-oriented wrapper
linear algebra abstraction with multiple backends (BLAS, MKL, Armadillo, Eigen3)
advanced container classes
- space partition trees
- file-mapped arrays

Requirements

Tested compilers include:

GCC 4.9.1, 5.3.0, 6.3.0, 7.2.0
Intel 15.0.2, 16.0.3, 17, 18
CCE 8.4.5, 8.5.8, 8.6.2
Apple LLVM 9.1.0 (clang-902.0.39.1)

Required dependencies:

CMake --- For use and installation see http://www.cmake.org/
ecbuild --- ECMWF library of CMake macros
Perl5 --- For some code generation

Installation

Eckit employs an out-of-source build/install based on CMake.

Make sure ecbuild is installed and the ecbuild executable script is found ( which ecbuild ).

Now proceed with installation as follows:

# Environment --- Edit as needed
srcdir=$(pwd)
builddir=build
installdir=$HOME/.local

# 1. Create the build directory:
mkdir $builddir
cd $builddir

# 2. Run CMake
ecbuild --prefix=$installdir -- $srcdir

# 3. Compile / Install
make -j10
make install

# 4. Check installation
$installdir/bin/eckit-version

eckit's People

Contributors

Stargazers

Watchers

eckit's Issues

Segmentation fault caused by buffer overflow in ChannelBuffer

What happened?

On LUMI, with huge amount of logging thanks to MULTIO_DEBUG=1, and the fact the filesystem there is sometimes slow, the default buffer size of 1024 may overflow and cause a crash.

What are the steps to reproduce the bug?

Run an HPC experiment on LUMI, using RAPS and IFS+FESOMv2 or IFS+NEMOv4. Set MULTIO_DEBUG=1. There is a good chance (but no certainly) that the error will occur.

We have not been able to reproduce it in other contexts, but if the buffer-size is set to a small value, the expectation is that it should be reproducible more readily.

Version

1.24.4

Platform (OS and architecture)

LUMI/23.03 & cpeCray/23.03

Relevant log output

No response

Accompanying data

No response

Organisation

ECMWF

Eckit fails to build on macOS using Intel compilers

I get an error when using Intel compilers on macOS. It's related to handle_strerror_r. It attempts to pass a type int to char *. So, I think the first #ifdef _GNU_SOURCE is being used which accepts a char *, but should be using the second function definition with #elif (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || _POSIX_VERSION >= 200112L || _XOPEN_VERSION >= 600) which does take an int.

Maybe a preprocessor check like __INTEL_COMPILER and APPLE? I guess _GNU_SOURCE is defined but Intel by the function template doesn't match.

Using Intel 2021.4.0 compilers, and macOS 11.6.5

eckit/src/eckit/log/Log.cc

Lines 36 to 65 in 9b8fc27

    
           #if defined(_GNU_SOURCE) 
        
           /* To use with GNU libc strerror_r */ 
        
           static void handle_strerror_r(std::ostream& s, int e, char[], char* p) { 
        
               if (p) { 
        
                   s << " (" << p << ")"; 
        
               } 
        
               else { 
        
                   s << " (errno = " << e << ") "; 
        
               } 
        
           } 
        
           #elif (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || _POSIX_VERSION >= 200112L || _XOPEN_VERSION >= 600) 
        
           /* To use with XSI-compliant strerror_r 
        
            * 
        
            * Linux defines _POSIX_C_SOURCE and _XOPEN_SOURCE 
        
            * BSD defines _POSIX_VERSION and _XOPEN_VERSION 
        
            * glibc defines _GNU_SOURCE and implements a non-XSI compliant strerror_r 
        
            */ 
        
           static void handle_strerror_r(std::ostream& s, int e, char es[], int hs) { 
        
               if (hs == 0) { 
        
                   s << " (" << es << ") "; 
        
               } 
        
               else { 
        
                   s << " (errno = " << e << ") "; 
        
               } 
        
           }

   eckit.dir/log/SavedStatus.cc.o -c /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src/eckit/log/SavedStatus.cc
  >> 679    /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src/eckit/log/Log.cc(367): error: argument of type "int" is incompatible with parameter of type "char *"
     680          handle_strerror_r(s, e, estr, strerror_r(e, estr, sizeof(estr)));
     681                                        ^
     682
     683    [ 27%] Building CXX object src/eckit/CMakeFiles/eckit.dir/log/Seconds.cc.o
     684    cd /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-build-xru6mio/src/eckit && /Users/KIG/Desktop/hpc-stack/install/intel-2021.4/mpich/3.4.2/bin/mpic++ -Deckit_EXPORTS -I/Users/KIG/Desktop/spack-stack/cache/build_s
            tage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-build-xru6mio/src -I/Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src -I/Users/KIG/Desktop/spack-stack/envs/all.default/install/intel/2021
            .4.0/lz4-1.9.3-cn5gcbi/include -I/opt/local/include -Wall -Wextra -Wno-unused-parameter -Wno-unused-variable -Wno-sign-compare -O2 -g -DNDEBUG -std=gnu++11 -MD -MT src/eckit/CMakeFiles/eckit.dir/log/Seconds.cc.o -MF CMakeFiles/eckit.dir/log/Seconds.cc.o.d -o CMakeFiles/eckit.di
            r/log/Seconds.cc.o -c /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src/eckit/log/Seconds.cc
     685    [ 27%] Building CXX object src/eckit/CMakeFiles/eckit.dir/log/Statistics.cc.o
     ...

     689    remark #11074: Inlining inhibited by limit max-size
     690    remark #11074: Inlining inhibited by limit max-total-size
     691    remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
     692    [ 28%] Building CXX object src/eckit/CMakeFiles/eckit.dir/log/SysLog.cc.o
     693    cd /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-build-xru6mio/src/eckit && /Users/KIG/Desktop/hpc-stack/install/intel-2021.4/mpich/3.4.2/bin/mpic++ -Deckit_EXPORTS -I/Users/KIG/Desktop/spack-stack/cache/build_s
            tage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-build-xru6mio/src -I/Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src -I/Users/KIG/Desktop/spack-stack/envs/all.default/install/intel/2021
            .4.0/lz4-1.9.3-cn5gcbi/include -I/opt/local/include -Wall -Wextra -Wno-unused-parameter -Wno-unused-variable -Wno-sign-compare -O2 -g -DNDEBUG -std=gnu++11 -MD -MT src/eckit/CMakeFiles/eckit.dir/log/SysLog.cc.o -MF CMakeFiles/eckit.dir/log/SysLog.cc.o.d -o CMakeFiles/eckit.dir/
            log/SysLog.cc.o -c /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src/eckit/log/SysLog.cc
     694    compilation aborted for /Users/KIG/Desktop/spack-stack/cache/build_stage/spack-stage-eckit-1.18.0-xru6mio4qaly46azyjnghicrlbtfukpz/spack-src/src/eckit/log/Log.cc (code 2)
  >> 695    make[2]: *** [src/eckit/CMakeFiles/eckit.dir/log/Log.cc.o] Error 2
     696    make[2]: *** Waiting for unfinished jobs....
     697    remark #11074: Inlining inhibited by limit max-total-size
     698    remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
     699    remark #11074: Inlining inhibited by limit max-total-size
     700    remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
     701    remark #11074: Inlining inhibited by limit max-total-size
     702    remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
     703    remark #11074: Inlining inhibited by limit max-total-size
     704    remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
  >> 705    make[1]: *** [src/eckit/CMakeFiles/eckit.dir/all] Error 2
  >> 706    make: *** [all] Error 2

Vector product in Point3.cc

I would like to add a vector product method to Point3.cc :

Something like :

    static Point3 cross (const Point3 & p1, const Point3 & p2) 
    {   
      return Point3 
      (   
        p1[YY] * p2[ZZ] - p1[ZZ] * p2[YY],
        p1[ZZ] * p2[XX] - p1[XX] * p2[ZZ],
        p1[XX] * p2[YY] - p1[YY] * p2[XX] 
      );  
    }

May I submit a branch on github ?

literal input of string array in configuration?

JEDI uses eckit::Configuration.get() methods to read our yaml configuration files. One of our data assimilation applications requires a list of state variables which includes nitric oxide (NO):

    state variables: [u,v,T,DELP,ps,sphum,ice_wat,liq_wat,o3mr,
                      no2, no,o3,no3,h2o2,n2o5,
                      hno3,hono,pna,so2,sulf,
[...]

Well, when the configuration get() method parses this string vector, it is converting no to false. This is causing an error in our application.

Is there currently a way to suppress this conversion? In other words, read the string literally, without modification? If not, do you have any suggestions on how to address this? We could of course change the variable name but this is not ideal because the variable names for the application (FV3 LAM) are standardized. Thanks for any ideas you may have.

We're using eckit 1.16.0

ISSUE ON INSTALLING eckit-1.20.0.tar.gz

Hi!
I'm trying to install the eckit (required packages by other program) version 1.20.0 but encouter the following problem.
I have ecbuild correctly installed (I think > darivo@Rivo:~/eckit-1.20.0/build$ which ecbuild give /opt/cdo-install/bin/ecbuild
But when I enter this /eckit-1.20.0/build$ ecbuild --prefix=$installdir ../ to install eckit, it requires root permission. So I try with sudo but unfortinately it says ecbuild not found.
:/eckit-1.20.0/build$ sudo ecbuild --prefix=$installdir ../
sudo: ecbuild: command not found
A help is most wellcome! Thanks inadvance.
For info, I downloaded tarball from here (github)

Need of MPI_Sendrecv_replace interfaces

We (Benjamin Ménétrier and myself) are trying to optimize a 4D localization in JEDI-OOPS, based on an open-end shift. We miss an eckit interface to MPI_Sendrecv_replace. In order to implement open-end shifts, we also need a method to expose the MPI_PROC_NULL constant.
Thank you.

Intermittant failures in eckit_test_container_sharedmemarray

I am getting intermittant failures in the test eckit_test_container_sharedmemarray:

4/140 Test #3: eckit_test_container_sharedmemarray ........Subprocess aborted***Exception: 0.52 sec
Running 4 tests:
Running case 0: test_eckit_sharedmemarray_construction ...
semget failed: No space left on device
�[31mTest "test_eckit_sharedmemarray_construction" failed with unhandled eckit::Exception: Failed system call: semget (Invalid argument) @ �[0m
Stack trace: backtrace [1] stack has 15 addresses
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::BackTrace::dumpabi:cxx11)0x48
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Exception::Exception())0x93
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::FailedSystemCall::FailedSystemCall(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))0x29
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Semaphore::Semaphore(eckit::PathName const&, int))0x2ef
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/usr/lib64/libc.so.6+__libc_start_main)0xf5
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)

end of backtrace dump ...
Completed case 0: test_eckit_sharedmemarray_construction
Running case 1: test_eckit_sharedmemarray_checkvalues ...
semget failed: No space left on device
�[31mTest "test_eckit_sharedmemarray_checkvalues" failed with unhandled eckit::Exception: Failed system call: semget (No space left on device) @ �[0m
Stack trace: backtrace [2] stack has 15 addresses
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::BackTrace::dumpabi:cxx11)0x48
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Exception::Exception())0x93
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::FailedSystemCall::FailedSystemCall(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))0x29
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Semaphore::Semaphore(eckit::PathName const&, int))0x2ef
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/usr/lib64/libc.so.6+__libc_start_main)0xf5
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)

end of backtrace dump ...
Completed case 1: test_eckit_sharedmemarray_checkvalues
Running case 2: test_eckit_sharedmemarray_add_more ...
semget failed: No space left on device
�[31mTest "test_eckit_sharedmemarray_add_more" failed with unhandled eckit::Exception: Failed system call: semget (No space left on device) @ �[0m
Stack trace: backtrace [3] stack has 15 addresses
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::BackTrace::dumpabi:cxx11)0x48
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Exception::Exception())0x93
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::FailedSystemCall::FailedSystemCall(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))0x29
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Semaphore::Semaphore(eckit::PathName const&, int))0x2ef
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/usr/lib64/libc.so.6+__libc_start_main)0xf5
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)

end of backtrace dump ...
Completed case 2: test_eckit_sharedmemarray_add_more
Running case 3: test_eckit_sharedmemarray_checkvalues_2 ...
semget failed: No space left on device
�[31mTest "test_eckit_sharedmemarray_checkvalues_2" failed with unhandled eckit::Exception: Failed system call: semget (No space left on device) @ �[0m
Stack trace: backtrace [4] stack has 15 addresses
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::BackTrace::dumpabi:cxx11)0x48
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Exception::Exception())0x93
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::FailedSystemCall::FailedSystemCall(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))0x29
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/lib/libeckit.so+eckit::Semaphore::Semaphore(eckit::PathName const&, int))0x2ef
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)
(/usr/lib64/libc.so.6+__libc_start_main)0xf5
(/home/h01/frwd/cylc-run/EckitFailure/share/build-mo-spice_gnu_debug/eckit/tests/container/eckit_test_container_sharedmemarray)

end of backtrace dump ...
Completed case 3: test_eckit_sharedmemarray_checkvalues_2
FAILED: test_eckit_sharedmemarray_construction
FAILED: test_eckit_sharedmemarray_checkvalues
FAILED: test_eckit_sharedmemarray_add_more
FAILED: test_eckit_sharedmemarray_checkvalues_2
4 tests failed out of 4.
terminate called after throwing an instance of 'eckit::FailedSystemCall'
what(): Failed system call: ::shm_unlink("/baz_hosts") in (/home/h01/frwd/cylc-run/EckitFailure/share/mo-bundle/eckit/tests/container/test_sharedmemarray.cc +107 main) (No such file or directory)

It happens on different platforms.

eckit installation of Cray environments

eckit build is successful on cray environment. However, when I try to build FDB, I get an error message related to eckit file missing.
It looks like there is no cmd folder in my eckit installation directory. Do you know why cmd folder is not found even though while installing eckit no error found.

See the error message below:

/project/devaraju/fdb/src/fdb5/remote/fdb-monitor.cc:16:10: fatal error: eckit/cmd/CmdApplication.h: No such file or directory
16 | #include "eckit/cmd/CmdApplication.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

How can logging-levels be set for eckit/log/log.h?

I am a user of infero. Infero uses #include "eckit/log/Log.h" for its logging.
Unfortunately I could not find a way to descrease verbosity of that logging-instance therefore the logfiles contain a lot of noise.
Browsing through the files in log did not reveal any possibility to set a specific log-level.

My question:

How can I set the logger provided by eckit to a certain level? I.e level warning or level debug?

MPI problems with srun

Many clusters use slurm to manage MPI processes across multiple nodes. Often one can use mpirun or mpiexec within a slurm batch script but sometimes cluster admins may configure the system to use srun instead. We're finding problems in the eckit mpi initialization for jobs run with srun.

The problem is with the size() and rank() methods of eckit::mpi::Comm. These return values of 1 and 0 for all MPI tasks when executables are executed with srun. Strangely, this does not trigger test failures in eckit_test_mpi - the tests pass, presumably because the mpi calls don't return error codes when run with a single node (comm size 1) - for example, node 0 can happily send and receive to itself. Meanwhile, other tests, such as eckit_test_mpi_addcom hang.

However, you can make a small change to eckit/test/mpi/eckit_test_mpi.cc adding a third check to the test_rank_size case as follows:

CASE("test_rank_size") {
    EXPECT_NO_THROW(mpi::comm().size());
    EXPECT_NO_THROW(mpi::comm().rank());
    EXPECT(mpi::comm().size() > 1);
}

Then, after compilation, one can test it with each of the following two commands:

mpiexec -n 4 eckit/tests/mpi/eckit_test_mpi 
srun -n 4 eckit/tests/mpi/eckit_test_mpi

In the first case (mpiexec) all tests pass. In the second case, there is one (and only one!) test failure, namely test_rank_size. If one investigates this further with a debugger or print statements, one finds that indeed, all tasks believe that their rank is 0 and that the comm size is 1. I should add that the result is the same if one passes "world" to comm(): eckit::mpi::comm("world").

Some mpi jobs still appear to run fine (and many mpi tests pass). However, any jobs that query eckit::mpi::Comm to get their rank and then use this information for some purpose (such as data partitioning) will likely fail.

We believe that it is a problem with eckit because a simple "hello world" test program such as this runs fine with both mpirun/mpiexec and srun:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char **argv)
{
  char nodename[128];
  int commsize;
  int resultlen;
  int iam;
  int ret;

  ret = MPI_Init (&argc, &argv);
  ret = MPI_Comm_size (MPI_COMM_WORLD, &commsize);
  ret = MPI_Comm_rank (MPI_COMM_WORLD, &iam);
  ret = MPI_Get_processor_name (nodename, &resultlen);

  printf ("Hello from rank %d of %d running on %s\n", iam, commsize, nodename);
  ret = MPI_Finalize ();
}

We have seen this issue on two different platforms (Amazon Web Services and the S4 cluster at the Univ. of Wisconsin) and with multiple compiler/mpi combinations: intel+intel mpi version 17.0.1 and 17.0.6 and gnu 7.4.0 / openmpi-3.1.2.

Has anyone run into this problem before?

normalise_angle stalls when passed very large numbers

What happened?

The function eckit::geometry::normalise_angle will stall when passed very large numbers.

This is because the normalization is accomplished using a loop which repeatedly subtracts 360 from the input angle until reaching a value in the range [minimum, minimum+360). Thus the loop runs a/360 times. For very large inputs, it effectively becomes an infinite loop.

What are the steps to reproduce the bug?

For example, the following program will never exit:

#include "eckit/geometry/CoordinateHelpers.h"

int main(int argc, char** argv) {
  double a=1e36;
  a=eckit::geometry::normalise_angle(a,0);
}

Version

I found the bug in v. 1.24.4, but the bug was first introduced in commit f8d15f2.

Platform (OS and architecture)

I was testing on MacOS 12.6.1 running on an M1 processor, but this bug should be reproducible on all platforms.

Relevant log output

No response

Accompanying data

No response

Organisation

US Naval Research Laboratory

Compile failure in test_densemap.cc

[ 66%] Building CXX object tests/container/CMakeFiles/eckit_test_container_densemap.dir/test_densemap.cc.o
In file included from /home/david/src/oops-stuff/eckit/tests/container/test_densemap.cc:14:0:
/home/david/src/oops-stuff/eckit/src/eckit/container/DenseMap.h: In instantiation of ‘class eckit::DenseMap<unsigned int, std::__cxx11::basic_string >’:
/home/david/src/oops-stuff/eckit/tests/container/test_densemap.cc:114:37: required from here
/home/david/src/oops-stuff/eckit/src/eckit/container/DenseMap.h:144:11: error: ‘const V& eckit::DenseMap<K, V>::operator[](const size_t&) const [with K = unsigned int; V = std::cxx11::basic_string; size_t = unsigned int]’ cannot be overloaded
const V& operator[] (const size_t& i ) const { ASSERT(i < values.size()); return values[ i ]; }
^~~~~~~~
/home/david/src/oops-stuff/eckit/src/eckit/container/DenseMap.h:141:11: error: with ‘const V& eckit::DenseMap<K, V>::operator[](const K&) const [with K = unsigned int; V = std::_cxx11::basic_string]’
const V& operator[] (const K& k ) const { return values[ find(k)->idx() ]; }
^~~~~~~~
/home/david/src/oops-stuff/eckit/src/eckit/container/DenseMap.h:145:5: error: ‘V& eckit::DenseMap<K, V>::operator[](const size_t&) [with K = unsigned int; V = std::cxx11::basic_string; size_t = unsigned int]’ cannot be overloaded
V& operator[] (const size_t& i ) { ASSERT(i < keys.size()); return values[ i ]; }
^~~~~~~~
/home/david/src/oops-stuff/eckit/src/eckit/container/DenseMap.h:142:5: error: with ‘V& eckit::DenseMap<K, V>::operator[](const K&) [with K = unsigned int; V = std::_cxx11::basic_string]’
V& operator[] (const K& k ) { return values[ find(k)->idx() ]; }
^~~~~~~~

david@david-System-Product-Name ~/src/oops-stuff $ /home/david/installs/gcc/7.1.0/v1/bin/g++ -v
Using built-in specs.
COLLECT_GCC=/home/david/installs/gcc/7.1.0/v1/bin/g++
COLLECT_LTO_WRAPPER=/home/david/installs/gcc/7.1.0/v1/libexec/gcc/i686-pc-linux-gnu/7.1.0/lto-wrapper
Target: i686-pc-linux-gnu
Configured with: /home/david/src/downloads/gcc-7.1.0/configure --prefix=/home/david/installs/gcc/7.1.0/v1 --enable-languages=all
Thread model: posix
gcc version 7.1.0 (GCC)

david@david-System-Product-Name ~/src/oops-stuff $ uname -a
Linux david-System-Product-Name 3.13.0-149-generic #199-Ubuntu SMP Thu May 17 10:12:57 UTC 2018 i686 athlon i686 GNU/Linux

I think this is because in DenseMap.h the operator[] has overloads for K (from the template) and specifically size_t. test_densemap.cc creates "m" for the test_map_int_string test by passing unsigned in the template for K which it seems is the same type as size_t on this platform. These overloads clash and the compiler cannot resolve them. One fix seems to be to pass "int" instead of "unsigned" in the template for K in test_densemap.cc.

Compile failure in test_configuration.cc

[ 63%] Building CXX object tests/config/CMakeFiles/eckit_test_config_configuration.dir/test_configuration.cc.o
cd /home/david/src/oops-stuff/eckit/eckit-build/tests/config && /home/david/installs/gcc/7.1.0/v1/bin/g++ -D_FILE_OFFSET_BITS=64 -I/home/david/src/oops-stuff/eckit/src -I/home/david/src/oops-stuff/eckit/eckit-build/src -I/home/david/installs/oops-stuff/eigen/include/eigen3 -L/home/david/installs/gcc/7.1.0/v1/lib -lgfortran -g -pipe -O2 -g -std=gnu++11 -o CMakeFiles/eckit_test_config_configuration.dir/test_configuration.cc.o -c /home/david/src/oops-stuff/eckit/tests/config/test_configuration.cc
/home/david/src/oops-stuff/eckit/tests/config/test_configuration.cc: In function ‘void eckit::test::test_46(std::__cxx11::string&, int&, int)’:
/home/david/src/oops-stuff/eckit/tests/config/test_configuration.cc:57:60: error: conversion from ‘std::vector’ to non-scalar type ‘std::vector’ requested
std::vector<size_t> value_arr_size_t = make_vector(6ul,7ul);
~~~~~~~~~~~^~~~~~~~~

david@david-System-Product-Name ~/src/oops-stuff $ uname -a
Linux david-System-Product-Name 3.13.0-149-generic #199-Ubuntu SMP Thu May 17 10:12:57 UTC 2018 i686 athlon i686 GNU/Linux

I think the problem here is that the LHS is a vector of size_t's but unsigned long is not size_t on this platform so the assignment is invalid. Seems fixable by making the RHS

make_vector(std::size_t{6},std::size_t{7});

Definition of eckit::offset:dump(DumpLoad&) missing

What happened?

I am using the develop branch and have encountered an issue that I hope you might be able to provide clarity on.

I attempted to use the eckit::offset::dump(DumpLoad&) member function in my program. Upon doing so, I observed a linker error. Upon closer examination, I found that while the declaration of the said member function is present in the library, its corresponding definition appears to be absent.

May I kindly inquire if this omission is intentional? Your insights on this matter will greatly benefit me as it will aid in my understanding and utilization of the eckit library.

What are the steps to reproduce the bug?

Write and link a program that tries to use eckit::offset:dump member function.

Version

develop branch

Platform (OS and architecture)

Arch OS x86_64

Relevant log output

No response

Accompanying data

No response

Organisation

FORTH

More verbose output from failed comparisons

In many unit testing frameworks, such as Boost.Test and Catch2, failed comparison assertions (such as EXPECT(value1 == value2)) produce a diagnostic message containing the values of the two expressions being compared. For example, a test such as

int a = 1;
int b = 2;
EXPECT(a == b);

might produce a message such as

error: check a == b has failed [1 != 2]

This can be quite helpful when debugging unit test failures, and @srherbener and I are wondering whether this feature could be added to eckit. We can think of two ways of doing that:

Define new macros such as EXPECT_EQUAL(actual, expected) that, on failure, will throw an exception with an error message including the values of the macro parameters. In general, a separate macro would be needed for each comparison operator (EXPECT_EQUAL, EXPECT_GREATER, EXPECT_GREATER_OR_EQUAL etc.) and for the special floating-point comparisons (EXPECT_APPROXIMATELY_EQUAL etc.).
Continue to use a single macro (EXPECT), but make the functions is_equal, is_greater_or_equal, is_approximately_equal etc. defined in eckit/types/FloatCompare.h print their arguments e.g. to the Log::info() stream if they're about to return false. For backward compatibility, this behaviour could be disabled by default and activated by passing true to a new optional parameter of these functions, say bool verbose=false. Alternatively, a new family of verbose wrappers to these function (verbose_is_equal, verbose_is_greater_or_equal etc.) could be defined; each wrapper would call the original function and print the arguments if that function returned false.

We'd be happy to submit a pull request implementing this feature in one of these ways if you think it could be a useful addition to eckit.

See https://github.com/JCSDA/eckit/pull/9 for the original discussion about this feature and an example implementation of the EXPECT_EQUAL macro.

Any documents for Eckit?

This is a question to Eckid developers.
Eckit has a much larger user/develop community when it is used by the JEDI project.
When it is an wonderful lib with so many functions and is regarded as "a leaner substitute for some 'boost' libraries", I am wondering where there are some documentation for Eckit? Any suggestions to users/developers on how to learn to use Eckit ( even not have to dig inside of Eckit)?
Thanks a lot.

Travis-ci build is failing on AMD64

Hi,

I am trying to enable ppc64le build on travis-ci, But its failing on AMD64 itself with the error below:

"$ cmake -DCMAKE_MODULE_PATH=${ECBUILD_MODULE_PATH} ${ECKIT_CMAKE_OPTIONS} ${ECKIT_SOURCE_DIR}
CMake Error at CMakeLists.txt:11 (cmake_minimum_required):
CMake 3.12 or higher is required. You are running version 3.10.2
-- Configuring incomplete, errors occurred!
The command "cmake -DCMAKE_MODULE_PATH=${ECBUILD_MODULE_PATH} ${ECKIT_CMAKE_OPTIONS} ${ECKIT_SOURCE_DIR}" exited with 1.
0.01s$ make -j4
make: *** No targets specified and no makefile found. Stop.
The command "make -j4" exited with 2.
0.00s$ bin/eckit-version
/home/travis/.travis/functions: line 109: bin/eckit-version: No such file or directory
The command "bin/eckit-version" exited with 127.
0.01s$ ctest

No test configuration file found!"

Full error log can be tracked here : https://travis-ci.com/github/sanjaymsh/eckit/builds/188376660
Please have a look on this.
Thanks !!

Compilation failure for eckit_test_mpi_objserialisation.cc => solved

Hello,

I had an error message when compiling eckit:

[ 83%] Building CXX object tests/mpi/CMakeFiles/eckit_test_mpi_objserialisation.dir/eckit_test_mpi_objserialisation.cc.o
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:167:10: error: ‘unique_ptr’ in namespace ‘std’ does not name a template type
std::unique_ptr next_;
^~~~~~~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc: In constructor ‘eckit::test::Obj::Obj(eckit::Stream&)’:
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:87:13: error: ‘next_’ was not declared in this scope
next_.reset(new Obj(s));
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:87:13: note: suggested alternative: ‘next’
next_.reset(new Obj(s));
^~~~~
next
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc: In member function ‘void eckit::test::Obj::encode(eckit::Stream&) const’:
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:98:12: error: ‘next_’ was not declared in this scope
if(next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc: In member function ‘void eckit::test::Obj::add(eckit::test::Obj*)’:
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:106:13: error: ‘next_’ was not declared in this scope
if(!next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:110:9: error: ‘next_’ was not declared in this scope
next_->add(o);
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc: In lambda function:
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:123:20: error: ‘next_’ was not declared in this scope
if(not next_ && not rhs.next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:123:37: error: ‘const class eckit::test::Obj’ has no member named ‘next_’
if(not next_ && not rhs.next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:126:16: error: ‘next_’ was not declared in this scope
if(next_ && rhs.next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:126:29: error: ‘const class eckit::test::Obj’ has no member named ‘next_’
if(next_ && rhs.next_) {
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:127:39: error: ‘const class eckit::test::Obj’ has no member named ‘next_’
return *next_ == *rhs.next_;
^~~~~
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc: In member function ‘void eckit::test::Obj::print(std::ostream&) const’:
${code}/eckit/tests/mpi/eckit_test_mpi_objserialisation.cc:150:12: error: ‘next_’ was not declared in this scope
if(next_)
^~~~~

Adding the include "#include " at the top of eckit/tests/mpi/eckit_test_mpi_objserialisation.cc solved the issue.

Benjamin

ctest failures when setting multiple MPI_ARGS

For a particular platform I am running on I need to run mpiexec with -launcher ssh options. To do this I set -DMPI_ARGS="-launcher;ssh" when running cmake. This allows e.g. eckit_test_mpi to succeed but ECKIT-221.sh and ECKIT-166.sh fails with:

    Start 118: ECKIT-166.sh

118/118 Test #118: ECKIT-166.sh ...............................***Failed 0.01 sec

cd /var/tmp/frwd-8923939/eckit-build/regressions
exe=./ECKIT-166.x
'[' 1 == 1 ']'
'[' /home/h01/frwd/extraction/lfric-bundle/Scriptify/installs/mpich/bin/mpiexec '!=' MPIEXEC-NOTFOUND ']'
/home/h01/frwd/extraction/lfric-bundle/Scriptify/installs/mpich/bin/mpiexec -launcher
[mpiexec@expspicesrv030] HYDU_set_str (utils/args/args.c:198): cannot assign NULL object
[mpiexec@expspicesrv030] launcher_fn (ui/mpich/utils.c:797): error setting launcher
[mpiexec@expspicesrv030] match_arg (utils/args/args.c:156): match handler returned error
[mpiexec@expspicesrv030] HYDU_parse_array (utils/args/args.c:178): argument matching returned error
[mpiexec@expspicesrv030] parse_args (ui/mpich/utils.c:1642): error parsing input array
[mpiexec@expspicesrv030] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1694): unable to parse user arguments
[mpiexec@expspicesrv030] main (ui/mpich/mpiexec.c:148): error parsing parameters

On the other hand running cmake with -DMPI_ARGS="-launcher ssh" results in e.g. test_mpi_parallel failing like this:

Start 56: eckit_test_mpi_parallel

56: Test command: /home/h01/frwd/extraction/lfric-bundle/Scriptify/installs/mpich/bin/mpiexec "-launcher ssh" "-n" "4" "/var/tmp/frwd-8923939/eckit-build/tests/mpi/eckit_test_mpi_parallel"
56: Environment variables:
56: OMP_NUM_THREADS=1
56: Test timeout computed to be: 1500
56: [mpiexec@expspicesrv030] match_arg (utils/args/args.c:163): unrecognized argument launcher ssh
56: [mpiexec@expspicesrv030] HYDU_parse_array (utils/args/args.c:178): argument matching returned error
56: [mpiexec@expspicesrv030] parse_args (ui/mpich/utils.c:1642): error parsing input array
56: [mpiexec@expspicesrv030] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1694): unable to parse user arguments
56: [mpiexec@expspicesrv030] main (ui/mpich/mpiexec.c:148): error parsing parameters
1/1 Test #56: eckit_test_mpi_parallel ..........***Failed 0.03 sec
[mpiexec@expspicesrv030] match_arg (utils/args/args.c:163): unrecognized argument launcher ssh
[mpiexec@expspicesrv030] HYDU_parse_array (utils/args/args.c:178): argument matching returned error
[mpiexec@expspicesrv030] parse_args (ui/mpich/utils.c:1642): error parsing input array
[mpiexec@expspicesrv030] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1694): unable to parse user arguments
[mpiexec@expspicesrv030] main (ui/mpich/mpiexec.c:148): error parsing parameters

whereas the ECKIT* tests pass.

Build failure in atlas related to codec header

What happened?

I tried to build atlas develop with eckit develop and got this failure:

In file included from /home/h01/frwd/cylc-run/mi-be984/share/installs/eckit-gnu/include/eckit/codec/codec.h:30,
from /home/h01/frwd/cylc-run/mi-be984/work/1/git_clone_atlas/atlas/atlas_io/eckit_codec_adaptor/src/atlas_io/atlas-io.h:13,
from /home/h01/frwd/cylc-run/mi-be984/work/1/git_clone_atlas/atlas/atlas_io/tests/test_io_encoding.cc:14:
/home/h01/frwd/cylc-run/mi-be984/share/installs/eckit-gnu/include/eckit/codec/detail/StaticAssert.h:15:10: fatal error: eckit/codec/eckit_codec_config.h: No such file or directory
15 | #include "eckit/codec/eckit_codec_config.h"

What are the steps to reproduce the bug?

Build atlas and eckit develops.

Version

develop

Platform (OS and architecture)

not relevant

Relevant log output

No response

Accompanying data

No response

Organisation

Meto

test_thread_mutex Build failure for Intel 19.0.5

The most recent release, 1.10.1 fails to build with version Intel 19.0.5 of the intel compiler suite:

[ 90%] Linking CXX executable eckit_test_thread_mutex
ld: CMakeFiles/eckit_test_thread_mutex.dir/test_mutex.cc.o(.text+0x1a5): unresolvable R_X86_64_TPOFF32 relocation against symbol `_ZSt15__once_callable@@GLIBCXX_3.4.11'
ld: final link failed: Nonrepresentable section on output
tests/thread/CMakeFiles/eckit_test_thread_mutex.dir/build.make:100: recipe for target 'tests/thread/eckit_test_thread_mutex' failed
make[2]: *** [tests/thread/eckit_test_thread_mutex] Error 1
CMakeFiles/Makefile2:5573: recipe for target 'tests/thread/CMakeFiles/eckit_test_thread_mutex.dir/all' failed
make[1]: *** [tests/thread/CMakeFiles/eckit_test_thread_mutex.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2

This is using the most recent release of ecbuild, namely 2020.04.0 (also occurs for earlier ecbuild releases).

Has anyone else noticed this? A simple workaround is to comment out that test. Everything else works.

an initialized argv issue in Parallel.cc

Dear Mr. or Mrs.,
When I ran a JEDI application, built with debug compiling option, which includes eckit , I got an error message pointing to one line in parallel.cc (line 247 in my version)
^
MPI_CALL(MPI_Init_thread(&argc, &argv, required, &provided));
V
It said argv was not initialized, which should be initialized when
^^
if (eckit::Main::ready())
...
argv = eckit::Main::instance().argv();
.....
V
So, seems the eckit is not ready yet. So, any clues/suggestions what I can do for this.
Your help is appreciated

Encountered error during compilation with nvhpc22.1

What happened?

When I compile the eckit with nvhpc22.1 compiler. I met errors like follows:
CMakeFiles/eckit_info.dir/eckit-info.cc.o: In function EckitInfo::EckitInfo(int, char**)': /PathofEckit/eckit/src/tools/eckit-info.cc:38: undefined reference to operator delete(void*, unsigned long)'
CMakeFiles/eckit_info.dir/eckit-info.cc.o: In function EckitInfo::~EckitInfo()': /PathofEckit/eckit/src/tools/eckit-info.cc:45: undefined reference to operator delete(void*, unsigned long)'
CMakeFiles/eckit_info.dir/eckit-info.cc.o: In function eckit::option::SimpleOption<bool>::~SimpleOption()': /PathofEckit/eckit/src/tools/eckit/src/eckit/option/SimpleOption.cc:57: undefined reference to operator delete(void*, unsigned long)'
../../lib/libeckit.so: undefined reference to `operator delete[](void*, unsigned long)'

What are the steps to reproduce the bug?

using nvhpc 22.1 compiler

Version

the newest version

Platform (OS and architecture)

Linux ln01 3.10.0-1160.31.1.el7.x86_64

Relevant log output

No response

Accompanying data

No response

Organisation

No response

Better adhere to BUILD_SHARED_LIBS instead of enforcing shared build?

I am trying to statically link against libeckit, but, unfortunately, it is hard-coded to be built shared-only:

eckit/src/eckit/CMakeLists.txt

Line 909 in 20c6b9c

TYPE SHARED

Is it possible to lift this to follow the BUILD_SHARED_LIBS option (as in https://github.com/ecmwf/ecbuild/blob/77ad783236c72e66c449a8b58000b751807c80f3/cmake/ecbuild_add_library.cmake#L645 when the build TYPE is not explicitely given)? When removing the enforcing line, eckit (and my executable) build fine... Thank you!

Build issue with eigen (MappedSparseMatrix)

What happened?

MappedSparseMatrix was removed from eigen 3.4.x

Therefore any build against it fails:

./eckit/linalg/sparse/LinearAlgebraEigen.cc:29:22: error: 'MappedSparseMatrix' in namespace 'Eigen' does not name a template type; did you mean 'SparseMatrix'?
   29 | using spm_t = Eigen::MappedSparseMatrix<Scalar, Eigen::RowMajor, Index>;
      |                      ^~~~~~~~~~~~~~~~~~
      |                      SparseMatrix
./eckit/linalg/sparse/LinearAlgebraEigen.cc: In member function 'virtual void eckit::linalg::sparse::LinearAlgebraEigen::spmv(const eckit::linalg::SparseMatrix&, const eckit::linalg::Vector&, eckit::linalg::Vector&) const':
./eckit/linalg/sparse/LinearAlgebraEigen.cc:45:5: error: 'spm_t' was not declared in this scope
   45 |     spm_t Ai(A.rows(), A.cols(), A.nonZeros(), const_cast<Index*>(A.outer()), const_cast<Index*>(A.inner()), const_cast<Scalar*>(A.data()));
      |     ^~~~~
./eckit/linalg/sparse/LinearAlgebraEigen.cc:49:10: error: 'Ai' was not declared in this scope; did you mean 'yi'?
   49 |     yi = Ai * xi;
      |          ^~
      |          yi
./eckit/linalg/sparse/LinearAlgebraEigen.cc: In member function 'virtual void eckit::linalg::sparse::LinearAlgebraEigen::spmm(const eckit::linalg::SparseMatrix&, const eckit::linalg::Matrix&, eckit::linalg::Matrix&) const':
./eckit/linalg/sparse/LinearAlgebraEigen.cc:62:5: error: 'spm_t' was not declared in this scope
   62 |     spm_t Ai(A.rows(), A.cols(), A.nonZeros(), const_cast<Index*>(A.outer()), const_cast<Index*>(A.inner()), const_cast<Scalar*>(A.data()));
      |     ^~~~~
./eckit/linalg/sparse/LinearAlgebraEigen.cc:66:10: error: 'Ai' was not declared in this scope; did you mean 'Ci'?
   66 |     Ci = Ai * Bi;
      |          ^~
      |          Ci

When I however build against a locally cloned eigen (3.2.1 or patched 3.3.9) at eckit/maths/eigen3 the build works, but unluckily it ~~segfaults~~(?) SIGABRTs when running:

./eckit-info

Exception: Bad operator: ELF (String) method 'contains' not implemented  (./eckit/value/Content.cc +124 contains)
terminate called after throwing an instance of 'eckit::BadOperator'
  what():  Bad operator: ELF (String) method 'contains' not implemented

What are the steps to reproduce the bug?

  cmake \
    -DCMAKE_INSTALL_PREFIX=/usr \
    -DCMAKE_BUILD_TYPE=production \
..
make

Version

git master

Platform (OS and architecture)

ArchLinux

Relevant log output

No response

Accompanying data

No response

Organisation

earth friends

Error with ecbuild (CRITICAL - Please define a version for eckit)

Hello Sir/Madam:

I'm trying to build eckit on MacOS, here is the Error message I got:

> ecbuild ..
Found CMake version 3.20.2

cmake -DCMAKE_MODULE_PATH=/usr/local/share/ecbuild/cmake ..

-- ecbuild   3.6.2	/usr/local/share/ecbuild/cmake
-- cmake     3.20.2	/usr/local/Cellar/cmake/3.20.2/bin/cmake
-- ---------------------------------------------------------
-- Added C flag [-pipe]
-- Added CXX flag [-pipe]
-- ---------------------------------------------------------
CMake Error at /usr/local/share/ecbuild/cmake/ecbuild_log.cmake:190 (message):
  CRITICAL - Please define a version for eckit

  	project( eckit VERSION x.x.x LANGUAGES C CXX Fortran )
Call Stack (most recent call first):
  /usr/local/share/ecbuild/cmake/ecbuild_declare_project.cmake:103 (ecbuild_critical)
  CMakeLists.txt:13 (ecbuild_declare_project)


-- Configuring incomplete, errors occurred!
See also "/Users/dangch/Documents/jedi/projects/eckit/build/CMakeFiles/CMakeOutput.log".
See also "/Users/dangch/Documents/jedi/projects/eckit/build/CMakeFiles/CMakeError.log".

Here is my system configuration:

HOMEBREW_VERSION: 3.1.7
ORIGIN: https://github.com/Homebrew/brew
HEAD: 92cf9047191393a2d624c64fc28d9595309ba273
Last commit: 5 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 1837d0c34a85a96692f3b55619df90562a51f096
Core tap last commit: 59 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 16
Homebrew Ruby: 2.6.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3_2/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 12.0.5 build 1205
Git: 2.25.0 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 11.3.1-x86_64
CLT: 12.5.0.0.1.1617976050
Xcode: 12.5

ecbuild version 3.6.2
cmake version 3.20.2

Could you please instruct me on how to set the eckit version? I'm using the code branch 0.24.1, while I'm not sure if this is the appropriate version for macOS 11.

Nested SECTIONs in tests do not appear to work

What happened?

The test case test_polygon uses nested SECTION scopes, and it appears that lines of code in inner SECTION scopes are not run. I think test_polygon may be the only eckit test using this nesting.

What are the steps to reproduce the bug?

In file test_polygon.cc below line 240 (inside nested SECTION "Contains edge"), add new line of code EXPECT(false);. Expect test ctest -R eckit_test_geometry_polygon to fail, but it passes.
To show expected behavior, move this new line of code to above line 240 (in outer SECTION "Simple rectangular polygon"). Expect test to fail, and it does.

Version

v1.20.2

Platform (OS and architecture)

macOS 12.6.4; intel i7

Relevant log output

No response

Accompanying data

No response

Organisation

No response

Shared object not getting created

I'm running into a problem when compiling in which build/lib/libeckit.so is not being built, so it can't be found and used during 'make install'. All of the other shared objects in that directory are successfully built, and there are no other errors. Any ideas? Log files and build script attached.
make.log
cmake.log
install.txt

Not sure what installdir, builddir and srcdir to use

Hi,

I'm really new to linux and this is essentially part of the first large program I'm trying to install. I was hoping you could tell me how to set my source, build and install directories properly. I don't know what is meant by $(pwd) or $HOME/local. Should I just leave these directories as is or should I write my own locations?
srcdir=$(pwd)
builddir=build
installdir=$HOME/local

Sorry if this is really obvious.

eckit::Comm constructor in a container

I'm trying to run an eckit application (JEDI) in a singularity container on an HPC system. In order to run on multiple nodes (864 MPI tasks, 20 tasks/node), I'm using what Singularity describes as the hybrid model, meaning that I'm using the native (system) MPI but I also have MPI installed in the container. The mpi versions are compatible - I'm using Intel MPI version 17.0.6 outside the container and Intel MPI version 17.0.1 inside the container. The code in the container is compiled with Intel 17.0.1 compilers (C++, C, and Fortran). slurm is the parallel job scheduler.

I don't know if you're familiar with singularity, but, for what it's worth, this is my run command:

srun --ntasks=864 --cpu_bind=cores --distribution=block:block --verbose singularity exec -e --home=$PWD $JEDICON/jedi-intel17-impi-hpc-dev.sif ${JEDIBIN}/fv3jedi_var.x Config/3dvar_bump.yaml

The -e option to singularity exec cleans the environment, so the container does not see any of the environment variables on the host system. I find that this is required in order to successfully run an mpi "hello world" program. If I leave it out, then I get errors from singularity that it cannot read the slurm configuration file.

So here is the problem. JEDI uses eckit to do its MPI initialization. When I include the -e option (required for hello world as described above), then the eckit::Comm object is not properly constructed. It mistakenly instantiates the Serial subclass instead of Parallel and the application fails because all 864 MPI processes think that they are rank 0. But, if I omit the -e option (so all environment variables on the host are available to the application), then I get the same slurm configuration error as with hello_world. But, at least it calls the Parallel constructor.

So, in summary, I believe eckit is somehow reading environment variables to determine whether or not the application is parallel - i.e. whether to instantiate the Parallel or Serial subclass of eckit::Comm. My question is: which variables? I know of at least one place where eckit accesses the environment variable SLURM_NTASKS to determine whether or not the job is parallel:

eckit/src/eckit/mpi/Comm.cc

Line 46 in 341babd

::getenv("SLURM_NTASKS"))) { // slurm srun

But this is not enough. When I use the -e (clean environment option) and explicitly export SLURM_NTASKS to the container, I still get the Serial instantiation of eckit::Comm. I see here that another requirement for instantiating the Parallel class is that the CMake variable ECKIT_HAVE_MPI is set:

eckit/src/eckit/mpi/Comm.cc

Line 26 in 341babd

#ifdef ECKIT_HAVE_MPI

But this is indeed set - the code was compiled with MPI enabled, as seen for example, here in the ecbuild.log file:

2020-03-14T14:12:06 - fckit - DEBUG - ecbuild_evaluate_dynamic_condition(_fckit_test_mpi_condition): checking condition 'HAVE_ECKIT AND ECKIT_HAVE_MPI' -> TRUE

What other information from the runtime environment does eckit need to determine that the application is parallel? In other words - what more can I do to force eckit to instantiate the Parallel class? Thanks for any insight you can provide.

Intermittant eckit_test_mpi_parallel failures

I am getting intermittant failures in eckit_test_mpi_parallel with this sort of output:

Running case 19: test_probe ...
Completed case 18: test_waitAny
Running case 19: test_probe ...
Completed case 18: test_waitAny
Running case 19: test_probe ...
Completed case 18: test_waitAny
Running case 19: test_probe ...
Completed case 19: test_probe
Running case 20: test_iProbe ...
Completed case 19: test_probe
Running case 20: test_iProbe ...
Completed case 19: test_probe
Running case 20: test_iProbe ...
�[31mTest "test_probe" failed: Condition failed: i == data[i] @ (/home/d03/frwd/cylc-run/EckitFailures/share/mo-bundle/eckit/tests/mpi/eckit_test_mpi.cc +776 test_746)�[0m
Completed case 19: test_probe
Running case 20: test_iProbe ...
Completed case 20: test_iProbe
0 tests failed out of 21.
Completed case 20: test_iProbe
0 tests failed out of 21.
Completed case 20: test_iProbe
0 tests failed out of 21.
�[31mTest "test_iProbe" failed: Condition failed: i == data[i] @ (/home/d03/frwd/cylc-run/EckitFailures/share/mo-bundle/eckit/tests/mpi/eckit_test_mpi.cc +821 test_786)�[0m
Completed case 20: test_iProbe
FAILED: test_probe
FAILED: test_iProbe
2 tests failed out of 21.

The output varies slightly when it fails but it is in the test_probe/test_iProbe tests. The platform where this fails is a Cray using either gcc 6.3.0 or Intel 2018.

	#if defined(_GNU_SOURCE)

	/* To use with GNU libc strerror_r */

	static void handle_strerror_r(std::ostream& s, int e, char[], char* p) {
	if (p) {
	s << " (" << p << ")";
	}
	else {
	s << " (errno = " << e << ") ";
	}
	}

	#elif (_POSIX_C_SOURCE >= 200112L \|\| _XOPEN_SOURCE >= 600 \|\| _POSIX_VERSION >= 200112L \|\| _XOPEN_VERSION >= 600)

	/* To use with XSI-compliant strerror_r
	*
	* Linux defines _POSIX_C_SOURCE and _XOPEN_SOURCE
	* BSD defines _POSIX_VERSION and _XOPEN_VERSION
	* glibc defines _GNU_SOURCE and implements a non-XSI compliant strerror_r
	*/

	static void handle_strerror_r(std::ostream& s, int e, char es[], int hs) {
	if (hs == 0) {
	s << " (" << es << ") ";
	}
	else {
	s << " (errno = " << e << ") ";
	}
	}

ecmwf / eckit Goto Github PK

eckit's Introduction

eckit

Requirements

Installation

eckit's People

Contributors

Stargazers

Watchers

Forkers

eckit's Issues

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

Recommend Projects

Recommend Topics

Recommend Org