xsdk-project / xsdk-examples Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 9.0 721 KB

Example codes demonstrating the use of various XSDK packages in combination.

C++ 57.92% C 20.43% Makefile 0.60% CMake 11.87% Shell 0.22% Python 8.96%

xsdk-examples's People

Contributors

Stargazers

Watchers

Forkers

osborn9 nbeams tcojean pghysels luszczek shuds13 jpmeng mkstoyanov masterleinad

xsdk-examples's Issues

New example: Deal.II+(Trilinos or Petsc)

Description: TBD

[email protected]%[email protected] build failure at ANL

This is the same build box as #16 [but now its xsdk-examples issue so creating a new one]

./bin/spack install -j32 --fail-fast xsdk-examples % [email protected]

make[2]: Leaving directory '/tmp/balay/spack-stage/spack-stage-xsdk-examples-0.3.0-u77eynim4bzxz5c35zr5qjslqzvgkzik/spack-build-u77eyni'
make[1]: *** [CMakeFiles/Makefile2:251: mfem/hypre-superlu/CMakeFiles/convdiff.dir/all] Error 2

[and subsequent errors]

spack-build-out.txt

Add dealii-preCICE Laplace problem

I found out about the xSDK examples today. If there is interest in adding further cases, this one could be a good option:

https://github.com/dealii/code-gallery/tree/master/coupled_laplace_problem

It is a very simple, actually even a bit artificial example how to use deal.II with preCICE. One could configure a PETSc-based RBF mapping in preCICE. And I guess in deal.II, once could probably use another xSDK dependency as a linear solver.

Different to the existing examples would be that one needs to build two executables (we want to couple them via preCICE) and that a preCICE configuration file is required.

Please note that this is not a good candidate for performance testing. It is just the simplest thing one can construct with deal.II and preCICE.

Just putting this out there. I am happy to provide more information and help contributing if this sounds interesting.

@fsimonis @davidscn @kronbichler

trilinos example build failure

ref: [amrex, mfem, plasma, sundials off]

balay@xsdk:/data/balay/spack>nice ./bin/spack install --keep-stage --dont-restage --test=root --fail-fast -j32 [email protected]%[email protected]+cuda cuda_arch=70 ^[email protected] ^openmpi ^[email protected]+trilinos+cuda cuda
_arch=70 ^dealii~hdf5

spack-build-out.txt

[email protected]%[email protected] build failure at UTK

In file included from /data/balay/spack.x/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/trilinos-13.2.0-ua4l7jnldcvlt4kpwl2p5j3u5waqvttt/include/Teuchos_ScalarTraits.hpp:59,
                 from /data/balay/spack.x/spack-stage/spack-stage-xsdk-examples-0.3.0-wrytqibsuv7qu53l6tidpzencsx566q5/spack-src/trilinos/SimpleSolve_WithParameters.cpp:53:
/data/balay/spack.x/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/trilinos-13.2.0-ua4l7jnldcvlt4kpwl2p5j3u5waqvttt/include/Kokkos_Complex.hpp:739:8: error: redefinition of default argument for 'bool <anonymous>'

Perhaps trilinos example needs an update?

spack-build-out.txt

Ref: ./bin/spack install --fail-fast -j24 xsdk-examples%[email protected]

strumpack run takes a long time

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                       
62724 balay     30  10  852292  23972  18884 R 100.3  0.0  17:06.36 sparse                                                                                                                                        
62721 balay     30  10  853508  25248  19004 R 100.0  0.0  17:06.22 sparse                                                                                                                                        
62722 balay     30  10  852168  24292  19208 R 100.0  0.0  17:06.63 sparse                                                                                                                                        
62723 balay     30  10  852316  24360  19276 R 100.0  0.0  17:06.11 sparse

balay@xsdk:/data/balay/spack>cat /data/balay/spack/spack-stage/spack-stage-xsdk-examples-0.4.0-jwviqi4ibrcjiewwlhbghrkm3vu64i2h/spack-build-jwviqi4/Testing/Temporary/LastTestsFailed.log 
1:dealii-petsc_trilinos
8:STRUMPACK-sparse

ref: [amrex, mfem, plasma, sundials, trilinos disabled]


 1573  nice ./bin/spack install --keep-stage --dont-restage --test=root --fail-fast -j32 [email protected]%[email protected]+cuda cuda_arch=70 ^[email protected] ^openmpi ^[email protected]+trilinos+cuda cuda_arch=70 ^dealii~hdf5

xsdk-examples%[email protected]+cuda with balos1/updates branch build failure on Lassen

I am using the xsdk-examples spack package from https://gitlab.com/xsdk-project/spack-xsdk/-/tree/xsdk-examples but pointed 0.3.0 to the branch balos1/updates instead of master.

The error appears to come from Trilinos while building the SimpleSolve_WithParameters example in xsdk-examples:

==> Installing xsdk-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe
==> No binary for xsdk-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe found: installing from source
==> No patches needed for xsdk-examples
==> xsdk-examples: Executing phase: 'cmake'
==> xsdk-examples: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

9 errors found in build log:
     191                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/mfem/hypr
            e-superlu/convdiff.cpp:3:
     192    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /mfem-4.3.0-kwauho6km5nc4bya3varft3ax4a46smi/include/mfem/mesh/mesh
            .hpp:780:4: note: declared here
     193        Mesh(int nx, int ny, Element::Type type, bool generate_edges = 
            false,
     194        ^~~~
     195    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     196                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 197    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:739:8: error: redefinition of default argument for 'bool 
            <anonymous>'
     198     struct promote {
     199            ^~~~~~~
     200    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     201                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     202                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     203    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:60:48: note: original definition appeared her
            e
     204     template <class T, bool = std::is_integral<T>::value>
     205                                                    ^~~~~
     206    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     207                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 208    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:743:8: error: redefinition of 'struct Kokkos::Impl::promo
            te<T, false>'
     209     struct promote<T, false> {};
     210            ^~~~~~~~~~~~~~~~~
     211    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     212                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     213                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     214    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:65:8: note: previous definition of 'struct Ko
            kkos::Impl::promote<T, false>'
     215     struct promote<T, false> {};
     216            ^~~~~~~~~~~~~~~~~
     217    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     218                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 219    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:745:8: error: redefinition of 'struct Kokkos::Impl::promo
            te<long double>'
     220     struct promote<long double> {
     221            ^~~~~~~~~~~~~~~~~~~~
     222    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     223                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     224                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     225    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:67:8: note: previous definition of 'struct Ko
            kkos::Impl::promote<long double>'
     226     struct promote<long double> {
     227            ^~~~~~~~~~~~~~~~~~~~
     228    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     229                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 230    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:749:8: error: redefinition of 'struct Kokkos::Impl::promo
            te<double>'
     231     struct promote<double> {
     232            ^~~~~~~~~~~~~~~
     233    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     234                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     235                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     236    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:71:8: note: previous definition of 'struct Ko
            kkos::Impl::promote<double>'
     237     struct promote<double> {
     238            ^~~~~~~~~~~~~~~
     239    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     240                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 241    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:753:8: error: redefinition of 'struct Kokkos::Impl::promo
            te<float>'
     242     struct promote<float> {
     243            ^~~~~~~~~~~~~~
     244    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     245                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     246                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     247    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:75:8: note: previous definition of 'struct Ko
            kkos::Impl::promote<float>'
     248     struct promote<float> {
     249            ^~~~~~~~~~~~~~
     250    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     251                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
  >> 252    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6ulywu/include/Kokkos_Co
            mplex.hpp:759:8: error: redefinition of 'struct Kokkos::Impl::promo
            te_2<T, U>'
     253     struct promote_2 {
     254            ^~~~~~~~~
     255    In file included from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Kokkos_Complex.hpp:48,
     256                     from /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rh
            el7-power9le/gcc-8.3.1/trilinos-13.2.0-dcnlyou3mhrqyx5zfebo27risf6u
            lywu/include/Teuchos_ScalarTraits.hpp:59,
     257                     from /var/tmp/balos1/spack-stage/spack-stage-xsdk-
            examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-src/trilinos/
            SimpleSolve_WithParameters.cpp:53:
     258    /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1
            /kokkos-3.5.00-x53mwrg5ijywgrqkajbrp5yoryzfdc2g/include/Kokkos_Math
            ematicalFunctions.hpp:81:8: note: previous definition of 'struct Ko
            kkos::Impl::promote_2<T, U>'

     ...

     264    [ 87%] Linking CXX executable transient-heat
     265    cd /tmp/balos1/spack-stage/spack-stage-xsdk-examples-0.3.0-5s4a72yy
            uj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a72y/mfem/sundials && /usr/
            WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/cmak
            e-3.22.2-j2qffuwtvdlbrhgpwke6jpq7ta4pt4vc/bin/cmake -E cmake_link_s
            cript CMakeFiles/transient-heat.dir/link.txt --verbose=1
     266    /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-gcc-8.3
            .1/bin/mpig++ -O2 -g -DNDEBUG CMakeFiles/transient-heat.dir/transie
            nt-heat.cpp.o -o transient-heat  -Wl,-rpath,/usr/WS2/balos1/xsdk/sp
            ack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/hypre-2.23.0-jyf72isbe
            y2hd2upl3djaiuiraj6umqf/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/li
            nux-rhel7-power9le/gcc-8.3.1/zlib-1.2.11-jw6upiryftnfsrzsvkosyej7ka
            ndkhmk/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9l
            e/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64:/
            usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/
            superlu-dist-7.1.1-b3pdazr6epojfqles7h3rqxiai5e554u/lib:/usr/WS2/ba
            los1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/openblas-0
            .3.19-cdjo724zsefio4j4zfiwhubqmj3nwym3/lib:/usr/WS2/balos1/xsdk/spa
            ck/opt/spack/linux-rhel7-power9le/gcc-8.3.1/petsc-3.16.1-oiwwqkupvg
            y422jjpwbph2u7jn4ldkje/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/lin
            ux-rhel7-power9le/gcc-8.3.1/metis-5.1.0-jblc7o42mrqnsm5enybflrsyhzm
            3y3dz/lib:/usr/tce/packages/cuda/cuda-11.2.0/lib64:::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::: /usr/WS2/balos1/xsdk/spack/opt/spa
            ck/linux-rhel7-power9le/gcc-8.3.1/hypre-2.23.0-jyf72isbey2hd2upl3dj
            aiuiraj6umqf/lib/libHYPRE.so /usr/WS2/balos1/xsdk/spack/opt/spack/l
            inux-rhel7-power9le/gcc-8.3.1/mfem-4.3.0-kwauho6km5nc4bya3varft3ax4
            a46smi/lib/libmfem.a /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhe
            l7-power9le/gcc-8.3.1/zlib-1.2.11-jw6upiryftnfsrzsvkosyej7kandkhmk/
            lib/libz.so /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9
            le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/
            libsundials_arkode.so.4.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/li
            nux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce
            5i22iqm5m/lib64/libsundials_cvodes.so.5.8.0 /usr/WS2/balos1/xsdk/sp
            ack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2f
            o5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_nvecserial.so.5.8.0 /us
            r/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/su
            ndials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_nve
            cparallel.so.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7
            -power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m
            /lib64/libsundials_nvecmpiplusx.so.5.8.0 /usr/WS2/balos1/xsdk/spack
            /opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5j
            txyzuokba5gce5i22iqm5m/lib64/libsundials_nveccuda.so.5.8.0 /usr/WS2
            /balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundial
            s-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_kinsol.s
            o.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/g
            cc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libs
            undials_sunnonlinsolpetscsnes.so.5.8.0 /usr/WS2/balos1/xsdk/spack/o
            pt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtx
            yzuokba5gce5i22iqm5m/lib64/libsundials_nvecpetsc.so.5.8.0 /usr/WS2/
            balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials
            -5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_sunlinsol
            superludist.so.3.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhe
            l7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm
            5m/lib64/libsundials_sunmatrixslunrloc.so.3.8.0 /usr/WS2/balos1/xsd
            k/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/superlu-dist-7.1.1
            -b3pdazr6epojfqles7h3rqxiai5e554u/lib/libsuperlu_dist.so /usr/WS2/b
            alos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/openblas-
            0.3.19-cdjo724zsefio4j4zfiwhubqmj3nwym3/lib/libopenblas.so -lm /usr
            /WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/pet
            sc-3.16.1-oiwwqkupvgy422jjpwbph2u7jn4ldkje/lib/libpetsc.so /usr/WS2
            /balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/hypre-2
            .23.0-jyf72isbey2hd2upl3djaiuiraj6umqf/lib/libHYPRE.so /usr/WS2/bal
            os1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/metis-5.1.0
            -jblc7o42mrqnsm5enybflrsyhzm3y3dz/lib/libmetis.so /usr/tce/packages
            /cuda/cuda-11.2.0/lib64/libcudart.so /usr/tce/packages/cuda/cuda-11
            .2.0/lib64/libcusparse.so
     267    [ 93%] Linking CXX executable advection
     268    cd /tmp/balos1/spack-stage/spack-stage-xsdk-examples-0.3.0-5s4a72yy
            uj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a72y/mfem/sundials && /usr/
            WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/cmak
            e-3.22.2-j2qffuwtvdlbrhgpwke6jpq7ta4pt4vc/bin/cmake -E cmake_link_s
            cript CMakeFiles/advection.dir/link.txt --verbose=1
     269    /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-gcc-8.3
            .1/bin/mpig++ -O2 -g -DNDEBUG CMakeFiles/advection.dir/advection.cp
            p.o -o advection  -Wl,-rpath,/usr/WS2/balos1/xsdk/spack/opt/spack/l
            inux-rhel7-power9le/gcc-8.3.1/hypre-2.23.0-jyf72isbey2hd2upl3djaiui
            raj6umqf/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power
            9le/gcc-8.3.1/zlib-1.2.11-jw6upiryftnfsrzsvkosyej7kandkhmk/lib:/usr
            /WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sun
            dials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64:/usr/WS2/balos1/
            xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/petsc-3.16.1-oi
            wwqkupvgy422jjpwbph2u7jn4ldkje/lib:/usr/WS2/balos1/xsdk/spack/opt/s
            pack/linux-rhel7-power9le/gcc-8.3.1/superlu-dist-7.1.1-b3pdazr6epoj
            fqles7h3rqxiai5e554u/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/linux
            -rhel7-power9le/gcc-8.3.1/openblas-0.3.19-cdjo724zsefio4j4zfiwhubqm
            j3nwym3/lib:/usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9
            le/gcc-8.3.1/metis-5.1.0-jblc7o42mrqnsm5enybflrsyhzm3y3dz/lib:/usr/
            tce/packages/cuda/cuda-11.2.0/lib64::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            ::::::::::::::::: /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-
            power9le/gcc-8.3.1/hypre-2.23.0-jyf72isbey2hd2upl3djaiuiraj6umqf/li
            b/libHYPRE.so /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-powe
            r9le/gcc-8.3.1/mfem-4.3.0-kwauho6km5nc4bya3varft3ax4a46smi/lib/libm
            fem.a /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc
            -8.3.1/zlib-1.2.11-jw6upiryftnfsrzsvkosyej7kandkhmk/lib/libz.so -lm
             /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.
            1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials
            _arkode.so.4.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-p
            ower9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/l
            ib64/libsundials_cvodes.so.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spa
            ck/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokb
            a5gce5i22iqm5m/lib64/libsundials_nvecserial.so.5.8.0 /usr/WS2/balos
            1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.
            0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_nvecparallel.s
            o.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/g
            cc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libs
            undials_nvecmpiplusx.so.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/
            linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5g
            ce5i22iqm5m/lib64/libsundials_nveccuda.so.5.8.0 /usr/WS2/balos1/xsd
            k/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7m
            iu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_kinsol.so.5.8.0 /us
            r/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/su
            ndials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_sun
            nonlinsolpetscsnes.so.5.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/li
            nux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce
            5i22iqm5m/lib64/libsundials_nvecpetsc.so.5.8.0 /usr/WS2/balos1/xsdk
            /spack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/petsc-3.16.1-oiwwqk
            upvgy422jjpwbph2u7jn4ldkje/lib/libpetsc.so /usr/WS2/balos1/xsdk/spa
            ck/opt/spack/linux-rhel7-power9le/gcc-8.3.1/sundials-5.8.0-p7miu2fo
            5jtxyzuokba5gce5i22iqm5m/lib64/libsundials_sunlinsolsuperludist.so.
            3.8.0 /usr/WS2/balos1/xsdk/spack/opt/spack/linux-rhel7-power9le/gcc
            -8.3.1/sundials-5.8.0-p7miu2fo5jtxyzuokba5gce5i22iqm5m/lib64/libsun
            dials_sunmatrixslunrloc.so.3.8.0 /usr/WS2/balos1/xsdk/spack/opt/spa
            ck/linux-rhel7-power9le/gcc-8.3.1/superlu-dist-7.1.1-b3pdazr6epojfq
            les7h3rqxiai5e554u/lib/libsuperlu_dist.so /usr/WS2/balos1/xsdk/spac
            k/opt/spack/linux-rhel7-power9le/gcc-8.3.1/openblas-0.3.19-cdjo724z
            sefio4j4zfiwhubqmj3nwym3/lib/libopenblas.so /usr/WS2/balos1/xsdk/sp
            ack/opt/spack/linux-rhel7-power9le/gcc-8.3.1/hypre-2.23.0-jyf72isbe
            y2hd2upl3djaiuiraj6umqf/lib/libHYPRE.so /usr/WS2/balos1/xsdk/spack/
            opt/spack/linux-rhel7-power9le/gcc-8.3.1/metis-5.1.0-jblc7o42mrqnsm
            5enybflrsyhzm3y3dz/lib/libmetis.so /usr/tce/packages/cuda/cuda-11.2
            .0/lib64/libcudart.so /usr/tce/packages/cuda/cuda-11.2.0/lib64/libc
            usparse.so
  >> 270    make[2]: *** [trilinos/CMakeFiles/SimpleSolve_WithParameters.dir/bu
            ild.make:79: trilinos/CMakeFiles/SimpleSolve_WithParameters.dir/Sim
            pleSolve_WithParameters.cpp.o] Error 1
     271    make[2]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-xsd
            k-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a7
            2y'
  >> 272    make[1]: *** [CMakeFiles/Makefile2:392: trilinos/CMakeFiles/SimpleS
            olve_WithParameters.dir/all] Error 2
     273    make[1]: *** Waiting for unfinished jobs....
     274    make[2]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-xsd
            k-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a7
            2y'
     275    [ 93%] Built target convdiff
     276    make[2]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-xsd
            k-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a7
            2y'
     277    make[2]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-xsd
            k-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a7
            2y'
     278    [ 93%] Built target transient-heat
     279    [ 93%] Built target advection
     280    make[1]: Leaving directory '/tmp/balos1/spack-stage/spack-stage-xsd
            k-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-5s4a7
            2y'
  >> 281    make: *** [Makefile:139: all] Error 2

See build log for details:
  /var/tmp/balos1/spack-stage/spack-stage-xsdk-examples-0.3.0-5s4a72yyuj5yrcxiy3qkrbeohbop4boe/spack-build-out.txt

Relevant files (I am using a spack environment):
concretized.txt
spack.lock.txt
spack.yaml.txt

I was able to build xsdk-examples against the installed [email protected] via CMake directly without issue, so this seems spack specific.

New example: Tasmanian+heFFTe

Description: TBD

Use Slate's `slateConfig.cmake` instead of `FindSlate.cmake`

Follow up from #48 to consider after the v0.4.0 release.

After some discussions in #46, we should consider updating this PR to use slateConfig.cmake instead of the cmake/FindSlate.cmake module. If we do that, there will be no need to explicitly link the PLASMA example with OpenMP -- that dependency will come from slateConfig.cmake.

Any thoughts?

Originally posted by @v-dobrev in #48 (comment)

Also consider this suggestion: #48 (comment)

New example: PUMI/Omega_h+MFEM

Description: Adaptive mesh refinement for a diffusion problem using MFEM+PUMI

Extend PETSc example

Example location: https://github.com/xsdk-project/xsdk-examples/blob/master/petsc/ex19.c
Description: add HIP support

New example: HiOp+MFEM

Description: TBD

Extend the HYPRE+SuperLU_DIST example

Example location: https://github.com/xsdk-project/xsdk-examples/blob/master/hypre/ij_laplacian.c
Description: TBD

hypre example failure with [email protected]~cuda build

spack-build-out.txt

ref:

 1044  nice ./bin/spack install  --test=root --fail-fast -j32 [email protected]%[email protected] ^openmpi ^[email protected]+trilinos

adolc/adolc_fatalerror.h: No such file or directory

Building xsdk-examples on linux (Cels GCE at Argonne) with gcc 9.4.0.

I got the following error. Repeated without dealii, all else built.

./bin/spack install --fail-fast -j 32 xsdk-examples%[email protected]
...
==> Error: Terminating after first install failure: ProcessError:
Command exited with status 2:
    'make' '-j32' 'install'
shudson@compute-386-07:/scratch/shudson/spack$
0 bash  1 bash
                                   compute-386-07  04/22  3:12pm
     1596
     1597    creating build
     1598    creating build/temp.linux-x86_64-3.8
     1599    compile options:
'-I/usr/lib/python3/dist-packages/numpy/core/include
-I/usr/include/python3.8 -I/home/shudson
             /adolc_base/include -c'
     1600    extra options: '-std=c++11 -fPIC -w'
     1601    gcc: adolc-python_wrap.cpp
  >> 1602    adolc-python_wrap.cpp:3230:10: fatal error:
adolc/adolc_fatalerror.h: No such file or directory
     1603     3230 | #include <adolc/adolc_fatalerror.h>
     1604          |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
     1605    compilation terminated.
  >> 1606    error: Command
"/scratch/shudson/spack/lib/spack/env/gcc/gcc -Wno-unused-result
-Wsign-compare -DNDEBUG -g -fw
             rapv -O2 -Wall -g -fstack-protector-strong -Wformat
-Werror=format-security -g -fwrapv -O2 -g -fstack-protecto
             r-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packag
             es/numpy/core/include -I/usr/include/python3.8
-I/home/shudson/adolc_base/include -c adolc-python_wrap.cpp -o
             build/temp.linux-x86_64-3.8/adolc-python_wrap.o
-std=c++11 -fPIC -w" failed with exit status 1
  >> 1607    make[3]: *** [Makefile:490: install] Error 1
     1608    make[3]: Leaving directory
'/tmp/shudson/spack-stage/spack-stage-adol-c-2.7.2-ugz6itltxyvocmqbfud4czvtobqkppbr
             /spack-src/ADOL-C/swig'
  >> 1609    make[2]: *** [Makefile:554: install-recursive] Error 1
     1610    make[2]: Leaving directory
'/tmp/shudson/spack-stage/spack-stage-adol-c-2.7.2-ugz6itltxyvocmqbfud4czvtobqkppbr
             /spack-src/ADOL-C'
  >> 1611    make[1]: *** [Makefile:711: install] Error 2
     1612    make[1]: Leaving directory
'/tmp/shudson/spack-stage/spack-stage-adol-c-2.7.2-ugz6itltxyvocmqbfud4czvtobqkppbr
             /spack-src/ADOL-C'
  >> 1613    make: *** [Makefile:537: install-recursive] Error 1

sundials example failure with [email protected]

spack-build-out.txt

Ref:

balay@xsdk:/data/balay/spack> nice ./bin/spack install  --test=root --fail-fast -j32 [email protected]%[email protected]+cuda cuda_arch=70 ^[email protected] ^openmpi ^[email protected]+trilinos

AMReX + SUNDIALS test takes a long time to run

Compared to other tests the AMReX + SUNDIALS test takes a very long time:

      Start  1: AMREX-amrex_sundials_advection_diffusion
 1/20 Test  #1: AMREX-amrex_sundials_advection_diffusion ...   Passed  620.17 sec
      Start  2: HYPRE-ij_laplacian
 2/20 Test  #2: HYPRE-ij_laplacian .........................   Passed    0.46 sec
      Start  3: MFEM-mfem_ex22_gko
 3/20 Test  #3: MFEM-mfem_ex22_gko .........................   Passed    1.76 sec
      Start  4: MFEM-magnetic-diffusion--cpu
 4/20 Test  #4: MFEM-magnetic-diffusion--cpu ...............   Passed    0.44 sec
      Start  5: MFEM-convdiff--hypre-boomeramg
 5/20 Test  #5: MFEM-convdiff--hypre-boomeramg .............   Passed    0.33 sec
      Start  6: MFEM-convdiff--superlu
 6/20 Test  #6: MFEM-convdiff--superlu .....................   Passed    1.23 sec
      Start  7: MFEM-obstacle
 7/20 Test  #7: MFEM-obstacle ..............................   Passed   19.39 sec
      Start  8: MFEM-transient-heat
 8/20 Test  #8: MFEM-transient-heat ........................   Passed    1.04 sec
      Start  9: MFEM-advection--cpu
 9/20 Test  #9: MFEM-advection--cpu ........................   Passed    0.82 sec
      Start 10: MFEM-diffusion-eigen--strumpack
10/20 Test #10: MFEM-diffusion-eigen--strumpack ............   Passed    6.21 sec
      Start 11: MFEM-diffusion-eigen--superlu
11/20 Test #11: MFEM-diffusion-eigen--superlu ..............   Passed    0.49 sec
      Start 12: MFEM-diffusion-eigen--hypre-boomeramg
12/20 Test #12: MFEM-diffusion-eigen--hypre-boomeramg ......   Passed    0.26 sec
      Start 13: MFEM-HIOP-adv
13/20 Test #13: MFEM-HIOP-adv ..............................   Passed    3.71 sec
      Start 14: PETSc-ex19_1
14/20 Test #14: PETSc-ex19_1 ...............................   Passed    0.57 sec
      Start 15: PETSc-ex19_hypre
15/20 Test #15: PETSc-ex19_hypre ...........................   Passed    0.27 sec
      Start 16: PETSc-ex19_superlu_dist
16/20 Test #16: PETSc-ex19_superlu_dist ....................   Passed    0.25 sec
      Start 17: SUNDIALS-cv_petsc_ex7_1
17/20 Test #17: SUNDIALS-cv_petsc_ex7_1 ....................   Passed    0.21 sec
      Start 18: SUNDIALS-cv_petsc_ex7_2
18/20 Test #18: SUNDIALS-cv_petsc_ex7_2 ....................   Passed    0.21 sec
      Start 19: SUNDIALS-ark_brusselator1D_FEM_sludist
19/20 Test #19: SUNDIALS-ark_brusselator1D_FEM_sludist .....   Passed    0.74 sec
      Start 20: STRUMPACK-sparse
20/20 Test #20: STRUMPACK-sparse ...........................   Passed    0.47 sec

It will be best if we can get the runtime under 1 min or even shorter.

AMReX + SUNDIALS (and maybe hypre?) GPU-enabled example.

Extend the MFEM+HYPRE magnetic diffusion example

Example location: https://github.com/xsdk-project/xsdk-examples/blob/master/mfem/hypre/magnetic-diffusion.cpp
Description: add HIP support when using xSDK v0.8.0 including in Spack

Enable GPU support in PETSc + hypre example

New example: Deal.II+SUNDIALS

Description: TBD

Testing of release v0.4.0

Hello @xsdk-project/ecp-xsdk,

Let's use this issue to coordinate the testing of the v0.4.0 release.

Summary

I propose we perform the testing using the following branches:

Use branch testing-v0.4.0 in this repo for testing the xSDK examples PRs before merging them in master
- use this branch for testing with CMake
- when PR branches for v0.4.0 are updated, they should be re-merged in this branch
- this common branch allows us to test multiple PR branches working together
- after testing, individual PRs for v0.4.0 are merged directly into master
- this branch will NOT be merged in master -- it can be deleted after the release
Use branch packages/xsdk-examples-v0.4.0 in the main Spack repo (i.e. PR spack/spack#37295) for testing with Spack
- this Spack branch temporarily points v0.4.0 to the above branch: testing-v0.4.0 in this repo
- before we merge the Spack PR, we'll update v0.4.0 to point to the actual release

Instructions for testing with CMake

Mac

Install xSDK packages via Spack on Mac:

./bin/spack install --no-cache --fresh \
    [email protected]%apple-clang~alquimia~arborx+precice \
        ^dealii~oce~hdf5 \
        ^mfem+ginkgo+hiop+pumi+strumpack

Notes about this spec:

The compiler configuration where this was tested is [email protected] with [email protected] on Apple silicon Mac.
Building with alquimia fails, so it is disabled.
Disable arborx because it has no xSDK examples and it brings in kokkos as dependency which conflicts with the built-in kokkos in trilinos.
precice is disabled by default, however we can enable it and test the deal.II + preCICE example.
Disable the oce variant of deal.II because building oce takes time and we don't need it.
Disable the hdf5 variant of deal.II because it has linking problems, see #36 (comment).
Enable the following MFEM variants which are needed by the respective xSDK examples: +ginkgo+hiop+pumi+strumpack.
This spec does not build amrex because it is not enabled in the xsdk package when using the apple-clang compiler. Therefore, the AMReX + SUNDIALS xSDK example will not be tested.
This spec does not build plasma because it is enabled in the xsdk package only for gcc v6 or newer. Therefore, the PLASMA xSDK example will not be tested.

Using the xSDK packages installed via Spack on Mac:

SPACK_DIR=<spack-dir>
cd $SPACK_DIR
./bin/spack view add xsdk-0.8.0 -i [email protected]

cd <xsdk-examples-dir>
mkdir build
cd build
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0 \
  -DCMAKE_VERBOSE_MAKEFILE=ON \
  -DCMAKE_Fortran_COMPILER=mpifort \
  -DENABLE_CUDA=OFF \
  -DENABLE_HIP=OFF \
  -DENABLE_AMREX=OFF \
  -DENABLE_DEAL_II=ON \
  -DENABLE_GINKGO=ON \
  -DENABLE_HEFFTE=ON \
  -DENABLE_HIOP=ON \
  -DENABLE_HYPRE=ON \
  -DENABLE_MAGMA=OFF \
  -DENABLE_MFEM=ON \
  -DENABLE_PETSC=ON \
  -DENABLE_PLASMA=OFF \
  -DENABLE_PRECICE=ON \
  -DENABLE_PUMI=ON \
  -DENABLE_STRUMPACK=ON \
  -DENABLE_SUNDIALS=ON \
  -DENABLE_SUPERLU=ON \
  -DENABLE_TASMANIAN=ON \
  -DENABLE_TRILINOS=ON

make -j
ctest --output-on-failure

CPU build on Tioga (LLNL)

Install xSDK packages (CPU-only build) via Spack on Tioga:

Tioga compilers.yaml

compilers:
- compiler:
    spec: [email protected]
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    operating_system: rhel8
    target: any
    modules:
    # TCE Standard Environment
    - StdEnv
    # PrgEnv-cray loads: craype, cray-mpich, cray-libsci (default versions)
    # craype loads: (none)
    # cray-mpich loads: (none)
    # cray-libsci loads: (none)
    - PrgEnv-cray
    # for consistency load specific versions in case the defaults change
    - craype/2.7.20
    - cray-mpich/8.1.25
    - cray-libsci/23.02.1.1
    # cce/15.0.1 loads: (none)
    - cce/15.0.1
    - craype-x86-trento
    extra_rpaths: []
- compiler:
    spec: [email protected]
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: rhel8
    target: any
    modules:
    - StdEnv
    # PrgEnv-gnu loads: gcc, craype, cray-mpich, cray-libsci (default versions)
    # gcc loads: PrgEnv-gnu
    # craype loads: (none)
    # cray-mpich loads: (none)
    # cray-libsci loads: (none)
    - PrgEnv-gnu
    # for consistency load specific versions in case the defaults change
    - gcc/12.2.0
    - craype/2.7.20
    - cray-mpich/8.1.25
    - cray-libsci/23.02.1.1
    - craype-x86-trento
    extra_rpaths: []

Tioga packages.yaml

packages:
  autoconf:
    externals:
    - spec: [email protected]
      prefix: /usr
  automake:
    externals:
    - spec: [email protected]
      prefix: /usr
  bison:
    externals:
    - spec: [email protected]
      prefix: /usr
  bzip2:
    externals:
    - spec: [email protected]
      prefix: /usr
  cmake:
    externals:
    - spec: [email protected]
      modules:
      - cmake/3.24.2
    - spec: [email protected]
      modules:
      - cmake/3.22.4
    - spec: [email protected]
      modules:
      - cmake/3.19.2
    buildable: false
  cpio:
    externals:
    - spec: [email protected]
      prefix: /usr
  diffutils:
    externals:
    - spec: [email protected]
      prefix: /usr
  expat:
    externals:
    - spec: [email protected]
      prefix: /usr
  file:
    externals:
    - spec: [email protected]
      prefix: /usr
  findutils:
    externals:
    - spec: [email protected]
      prefix: /usr
  flex:
    externals:
    - spec: [email protected]+lex
      prefix: /usr
  fontconfig:
    externals:
    - spec: [email protected]
      prefix: /usr
  freetype:
    externals:
    - spec: [email protected]
      prefix: /usr
  gawk:
    externals:
    - spec: [email protected]
      prefix: /usr
  gettext:
    externals:
    - spec: [email protected]
      prefix: /usr
  ghostscript:
    externals:
    - spec: [email protected]
      prefix: /usr
  git:
    externals:
    - spec: [email protected]
      prefix: /usr
  gmake:
    externals:
    - spec: [email protected]
      prefix: /usr
  gnutls:
    externals:
    - spec: [email protected]
      prefix: /usr
  groff:
    externals:
    - spec: [email protected]
      prefix: /usr
  hwloc:
    externals:
    - spec: [email protected]
      prefix: /usr
  libpng:
    externals:
    - spec: [email protected]
      prefix: /usr
  libtool:
    externals:
    - spec: [email protected]
      prefix: /usr
  libx11:
    externals:
    - spec: [email protected]
      prefix: /usr
  libxcb:
    externals:
    - spec: [email protected]
      prefix: /usr
  libxml2:
    externals:
    - spec: [email protected]
      prefix: /usr
  libxext:
    externals:
    - spec: [email protected]
      prefix: /usr
  libxt:
    externals:
    - spec: [email protected]
      prefix: /usr
  m4:
    externals:
    - spec: [email protected]
      prefix: /usr
  ncurses:
    externals:
    - spec: [email protected]
      prefix: /usr
  opengl:
    externals:
    - spec: [email protected]
      prefix: /usr
  openglu:
    externals:
    - spec: [email protected]
      prefix: /usr
  openssh:
    externals:
    - spec: [email protected]
      prefix: /usr
  openssl:
    externals:
    - spec: [email protected]
      prefix: /usr
  perl:
    externals:
    - spec: [email protected]
      prefix: /usr
  rsync:
    externals:
    - spec: [email protected]
      prefix: /usr
  ruby:
    externals:
    - spec: [email protected]
      prefix: /usr
  sed:
    externals:
    - spec: [email protected]
      prefix: /usr
  tar:
    externals:
    - spec: [email protected]
      prefix: /usr
  texinfo:
    externals:
    - spec: [email protected]
      prefix: /usr
  xproto:
    externals:
    # see /usr/share/pkgconfig/xproto.pc
    - spec: [email protected]
      prefix: /usr
  xrandr:
    externals:
    # xrandr --version
    - spec: [email protected]
      prefix: /usr
  xxd-standalone:
    externals:
    # vim-common version
    - spec: [email protected]
      prefix: /usr
  xz:
    externals:
    - spec: [email protected]
      prefix: /usr

  # ROCm 5.4.3
  hip:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3
      extra_attributes:
        compilers:
          c: /opt/rocm-5.4.3/llvm/bin/clang
          c++: /opt/rocm-5.4.3/llvm/bin/clang++
          hip: /opt/rocm-5.4.3/hip/bin/hipcc
    buildable: false
  hsa-rocr-dev:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3
      extra_attributes:
        compilers:
          c: /opt/rocm-5.4.3/llvm/bin/clang
          cxx: /opt/rocm-5.4.3/llvm/bin/clang++
    buildable: false
  llvm-amdgpu:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/llvm
      extra_attributes:
        compilers:
          c: /opt/rocm-5.4.3/llvm/bin/clang
          cxx: /opt/rocm-5.4.3/llvm/bin/clang++
    buildable: false
  hipcub:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/hipcub
  hipblas:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/hipblas
  hipsolver:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/hipsolver
  rocblas:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocblas
  rocsolver:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocsolver
  hipsparse:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/hipsparse
  rocsparse:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocsparse
  rocthrust:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocthrust
  rocrand:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocrand
  rocprim:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocprim
  rocfft:
    externals:
    - spec: [email protected]
      prefix: /opt/rocm-5.4.3/rocfft

  cray-mpich:
    externals:
    - spec: "[email protected]"
      modules:
      - cray-mpich/8.1.25
      - craype-network-ofi
    buildable: false
  cray-libsci:
    buildable: false
    # version: [23.02.1.1]
    externals:
    # - spec: [email protected]
    #   modules:
    #   - cray-libsci/23.02.1.1
    - spec: "[email protected]+mpi"
      modules:
      - cray-libsci/23.02.1.1
      - cray-mpich/8.1.25
      - craype-network-ofi

  all:
    providers:
      mpi: [cray-mpich]
      lapack: [cray-libsci]
      blas: [cray-libsci]
      # scalapack: [netlib-scalapack]
      scalapack: [cray-libsci+mpi]
      libglx: [opengl]
      glu: [openglu]
      xxd: [xxd-standalone]
    compiler: [[email protected]]

./bin/spack install -j 96 --no-cache --fresh \
    [email protected]%gcc~arborx \
        ^dealii~oce~threads~hdf5 cxxstd=14 \
        ^mfem+ginkgo+hiop+pumi+strumpack \
        ^[email protected]

Using the xSDK packages (CPU-only build) installed via Spack on Tioga:

SPACK_DIR=<spack-dir>
cd $SPACK_DIR
./bin/spack view add xsdk-0.8.0 -i [email protected]~rocm

ml PrgEnv-gnu

cd <xsdk-examples-dir>
mkdir build
cd build
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DMPIEXEC_EXECUTABLE="flux;run" \
  -DMPIEXEC_NUMPROC_FLAG="-n" \
  -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0 \
  -DCMAKE_VERBOSE_MAKEFILE=ON \
  -DENABLE_CUDA=OFF \
  -DENABLE_HIP=OFF \
  -DENABLE_AMREX=ON \
  -DENABLE_DEAL_II=ON \
  -DENABLE_GINKGO=ON \
  -DENABLE_HEFFTE=ON \
  -DENABLE_HIOP=ON \
  -DENABLE_HYPRE=ON \
  -DENABLE_MAGMA=OFF \
  -DENABLE_MFEM=ON \
  -DENABLE_PETSC=ON \
  -DENABLE_PLASMA=ON \
  -DENABLE_PUMI=ON \
  -DENABLE_STRUMPACK=ON \
  -DENABLE_SUNDIALS=ON \
  -DENABLE_SUPERLU=ON \
  -DENABLE_TASMANIAN=ON \
  -DENABLE_TRILINOS=ON

make -j
ctest --output-on-failure

Note: the test STRUMPACK-sparse may fail, however it actually works when run with the command

cd <xsdk-examples-dir>/build/strumpack
flux run -n 4 env OMP_NUM_THREADS=4 ./sparse 24 --sp_compression hodlr --hodlr_butterfly_levels 10

HIP build on Tioga (LLNL)

Install xSDK packages (`+rocm` build) via Spack on Tioga:

Tioga compilers.yaml and packages.yaml: same as in the section "CPU build on Tioga (LLNL)" above.

./bin/spack install -j 96 --no-cache --fresh \
    [email protected]%gcc~arborx~phist +rocm amdgpu_target=gfx90a \
        ^dealii~oce~threads~hdf5 cxxstd=14 \
        ^mfem+ginkgo+hiop+pumi+strumpack \
        ^strumpack~slate \
        ^[email protected]

Using the xSDK packages (`+rocm` build) installed via Spack on Tioga:

SPACK_DIR=<spack-dir>
cd $SPACK_DIR
./bin/spack view add xsdk-0.8.0-rocm -i [email protected]+rocm

ml PrgEnv-gnu
ml rocm/5.4.3

cd <xsdk-examples-dir>
mkdir build
cd build
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_HIP_ARCHITECTURES="gfx90a" \
  -DMPIEXEC_EXECUTABLE="flux;run" \
  -DMPIEXEC_NUMPROC_FLAG="-n" \
  -DCMAKE_PREFIX_PATH=$SPACK_DIR/xsdk-0.8.0-rocm \
  -DCMAKE_VERBOSE_MAKEFILE=ON \
  -DENABLE_CUDA=OFF \
  -DENABLE_HIP=ON \
  -DENABLE_AMREX=ON \
  -DENABLE_DEAL_II=ON \
  -DENABLE_GINKGO=ON \
  -DENABLE_HEFFTE=ON \
  -DENABLE_HIOP=ON \
  -DENABLE_HYPRE=ON \
  -DENABLE_MAGMA=ON \
  -DENABLE_MFEM=ON \
  -DENABLE_PETSC=ON \
  -DENABLE_PLASMA=ON \
  -DENABLE_PUMI=ON \
  -DENABLE_STRUMPACK=ON \
  -DENABLE_SUNDIALS=ON \
  -DENABLE_SUPERLU=ON \
  -DENABLE_TASMANIAN=ON \
  -DENABLE_TRILINOS=ON

make -j
ctest --output-on-failure

Machines and configurations to test with CMake

On machines A, and B test the following CMake configuration: (TBD)
On machines B, and C test the following CMake configuration: (TBD)

Instructions for testing with Spack

(TBD)

Spack machine configurations

Machine A: (TBD)
Machine B: (TBD)

Machines and configurations to test with Spack

On machines A, and B test the following Spack spec: (TBD)
On machines B, and C test the following Spack spec: (TBD)
ubuntu-22.04 with nvidia-gpu, use spack xsdk-examples-v0.4.0 branch ( starting with empty ~/.spack)

./bin/spack compiler find
./bin/spack install --test=root --fail-fast -j32  [email protected]%[email protected]+cuda cuda_arch=80 ^[email protected] ^openmpi ^[email protected]+trilinos

xsdk.icl.utk.edu with some pre-configured ~/.spack

./bin/spack install --fail-fast -j32 [email protected]%[email protected]+cuda cuda_arch=70 ^[email protected] ^openmpi ^[email protected]+trilinos

Please, feel free to add more bullets to the Summary above. Also, please help with filling in the Instructions sections.

AMReX + SUNDIALS example runtime error with HIP

Originally posted by @v-dobrev in #45 (comment)

When trying to run this example with HIP enabled I get the following error from ctest -V:

...
test 1
      Start  1: AMREX-amrex_sundials_advection_diffusion

1: Test command: /dev/shm/dobrev1/xsdk-examples/build/amrex/sundials/amrex_sundials_advection_diffusion
1: Working Directory: /dev/shm/dobrev1/xsdk-examples/build/amrex/sundials
1: Test timeout computed to be: 1500
1: Initializing HIP...
1: HIP initialized.
1: amrex::Abort::0::GPU last error detected in file /dev/shm/dobrev1/spack/var/spack/stage/spack-stage-amrex-22.09-huuzz4saz73j72hmwd6wupsmfpx62owg/spack-src/Src/Base/AMReX_GpuLaunchFunctsG.H line 809: shared object initialization failed !!!
1: SIGABRT
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: /usr/bin/addr2line: Dwarf Error: Invalid or unhandled FORM value: 0x25.
1: See Backtrace.0 file for details
1: MPICH ERROR [Rank 0] [job id 364801714345214976] [Fri May 12 21:03:56 2023] [tioga23] - Abort(6) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0
1: 
 1/26 Test  #1: AMREX-amrex_sundials_advection_diffusion ...***Failed    0.92 sec
...

Does anyone have any suggestions?

The Spack spec for AMReX is:

[+]  [email protected]%[email protected]~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools+rocm~shared+sundials~sycl~tiny_profile amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo dimensions=3 generator=make precision=double arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]+wrappers build_system=generic arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]~guile build_system=autotools patches=ca60bd9,fe5b60d arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]~cuda~ipo+rocm build_system=cmake build_type=Release generator=make patches=ca523f1 arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]+image~ipo+shared build_system=cmake build_type=Release generator=make patches=71e6851 arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]~ipo~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake build_type=Release generator=ninja patches=a08bbe1 arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]~ipo amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]+hiprand~ipo amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=a35e689 arch=linux-rhel8-zen3
[+]      ^[email protected]%[email protected]+ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math+ginkgo+hypre~int64~ipo~klu~kokkos~kokkos-kernels~lapack+magma~monitoring+mpi~openmp+petsc~profiling~pthread~raja+rocm+shared+static+superlu-dist~superlu-mt~sycl+trilinos amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo cstd=99 cxxstd=14 generator=make logging-level=0 logging-mpi=OFF precision=double arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~cuda~develtools~full_optimizations~hwloc~ipo+mpi~oneapi~openmp+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo generator=make patches=ba0956e arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=c447537 arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~ipo amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp+rocm+shared+superlu-dist~sycl~umpire~unified-memory amdgpu_target=gfx90a build_system=autotools arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]+mpi~openmp+shared build_system=generic arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~ipo~test amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~cuda+fortran~ipo+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~X+batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123+rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=gfx90a build_system=generic clanguage=C arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~cxx+fortran+hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make patches=0b5dd6f arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~gdb~int64~ipo~real64+shared build_system=cmake build_type=RelWithDebInfo generator=make patches=4991da9,93a7903,b1225da arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~gdb~int64~ipo+shared build_system=cmake build_type=RelWithDebInfo generator=make patches=4f89253,50ed208,704b84f arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]+libbsd build_system=autotools arch=linux-rhel8-zen3
[+]                      ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]                          ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools patches=9acdb4e arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~obsolete_api build_system=autotools arch=linux-rhel8-zen3
[+]                      ^[email protected]%[email protected]+cpanm+open+shared+threads build_system=generic patches=8cf4302 arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~docs~shared build_system=generic certs=mozilla arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected] build_system=autotools arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~ipo+tensile amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=81591d9 arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~ipo+optimal amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=8067bfb arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]+optimize+pic+shared build_system=makefile arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~cuda~int64~ipo~openmp~rocm+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rhel8-zen3
[+]          ^[email protected]%[email protected]~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest+hdf5+hypre+ifpack+ifpack2~intrepid+intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps+nox~openmp~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec+shards+shared~shylu~stk~stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist~teko~tempus+thyra+tpetra~trilinoscouplings~wrapper~x11+zoltan+zoltan2 build_system=cmake build_type=RelWithDebInfo cxxstd=14 generator=make gotype=int arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]+atomic+chrono~clanglibcpp~container~context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic cxxstd=14 patches=a440f96 visibility=hidden arch=linux-rhel8-zen3
[+]                  ^[email protected]%[email protected]~programs build_system=makefile libs=shared,static arch=linux-rhel8-zen3
[+]              ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel8-zen3

Note that hypre is built with +rocm -- could that be a problem?

Enable GPU support in MFEM + hypre example.

plasma example build error [with [email protected]]

spack-build-out.txt

Ref:

balay@xsdk:/data/balay/spack> nice ./bin/spack install  --test=root --fail-fast -j32 [email protected]%[email protected]+cuda cuda_arch=70 ^[email protected] ^openmpi ^[email protected]+trilinos

[email protected]%[email protected]+cuda build failure at UTK

/usr/bin/ld: CMakeFiles/advection.dir/advection.cpp.o: undefined reference to symbol 'cusparseDestroy@@libcusparse.so.11'
/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/lib64/libcusparse.so.11: error adding symbols: DSO missing from command line

spack-build-out.txt

Ref: ./bin/spack install --fail-fast -j24 xsdk-examples+cuda ^xsdk cuda_arch=70 ^[email protected] ^openmpi ^mfem+shared

Don't remember why I had to use mfem+shared - I'll have to retry that

xsdk-project / xsdk-examples Goto Github PK

xsdk-examples's People

Contributors

Stargazers

Watchers

Forkers

xsdk-examples's Issues

Summary

Instructions for testing with CMake

Mac

Install xSDK packages via Spack on Mac:

Using the xSDK packages installed via Spack on Mac:

CPU build on Tioga (LLNL)

Install xSDK packages (CPU-only build) via Spack on Tioga:

Using the xSDK packages (CPU-only build) installed via Spack on Tioga:

HIP build on Tioga (LLNL)

Install xSDK packages (+rocm build) via Spack on Tioga:

Using the xSDK packages (+rocm build) installed via Spack on Tioga:

Machines and configurations to test with CMake

Instructions for testing with Spack

Spack machine configurations

Machines and configurations to test with Spack

Recommend Projects

Recommend Topics

Recommend Org

Install xSDK packages (`+rocm` build) via Spack on Tioga:

Using the xSDK packages (`+rocm` build) installed via Spack on Tioga: