ecp-copa / exampm Goto Github PK
View Code? Open in Web Editor NEWMaterial point method proxy application based on Cabana.
License: BSD 3-Clause "New" or "Revised" License
Material point method proxy application based on Cabana.
License: BSD 3-Clause "New" or "Revised" License
Hi! I'm new to using this app, and was wondering if you have an example for running with mpirun (or similar?) I'm looking at the docs here: https://github.com/ECP-copa/ExaMPM/wiki/Run
Thank you! And apologies if this is an overly simple question (e.g., just put mpirun in front of that :P )
106 cd /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/sp
ack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples -I/tmp/
junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-buil
d/src -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-cent
os7-haswell/gcc-8.2.0/openmpi-3.1.6-jnys4eardy2dlj25lklmxcxuarflps3i/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-0.3.0-qzqsu3nor3rltt3ecq63m6pgx
xn7u7eb/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cajita-0.1-lsqwyf355hijcigqrs6xwn24aarxxmd6/include -isystem /home/junghans/spack.git/opt/spack/lin
ux-centos7-haswell/gcc-8.2.0/silo-4.10.2-ga3tsj5twrcwtk5zgeq6xapshedtoghj/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /
tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/dam_break.cpp
107 In file included from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ProblemManager.hpp:16,
108 from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_Solver.hpp:15,
109 from /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/examples/free_fall.cpp:1:
110 /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp: In function 'void ExaMPM::ParticleCommunication::redistrib
ute(const LocalGridType&, int, ParticleContainer&, std::integral_constant<long unsigned int, CoordIndex>, bool)':
>> 111 /tmp/junghans/spack-stage/spack-stage-exampm-master-wbxxt67xxailmsn7po3u3kbwvlxyyhgw/spack-src/src/ExaMPM_ParticleCommunication.hpp:224:13: error: 'Distributor' is not a member of 'Cabana'
112 Cabana::Distributor<device_type> distributor( local_grid.globalGrid().comm(),
113 ^~~~~~~~~~~
No idea, but what version of cabaa is needed to build this?
2020-08-03T02:19:08.1445642Z [ 66%] Building CXX object examples/CMakeFiles/FreeFall.dir/free_fall.cpp.o
2020-08-03T02:19:08.1455217Z cd /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/kokkos-3.1.01-4qzdxlx5fux2ydikb2ez5e42qtp66uhy/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/openmpi-3.1.6-5gsagjujszo4s347uw7vkyqn2teqzprv/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/cabana-master-xps3be3gaxfbk4izrjvn2hwxz233wfee/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/silo-4.10.2-va4cvfzv3ydoyjxjshzdlby4hihpmikt/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -mrtm -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/dam_break.cpp
2020-08-03T02:19:08.1462512Z cd /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src -I/tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-build/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/kokkos-3.1.01-4qzdxlx5fux2ydikb2ez5e42qtp66uhy/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/openmpi-3.1.6-5gsagjujszo4s347uw7vkyqn2teqzprv/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/cabana-master-xps3be3gaxfbk4izrjvn2hwxz233wfee/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-9.3.0/silo-4.10.2-va4cvfzv3ydoyjxjshzdlby4hihpmikt/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -mrtm -pthread -o CMakeFiles/FreeFall.dir/free_fall.cpp.o -c /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/free_fall.cpp
2020-08-03T02:19:09.5645222Z In file included from /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src/ExaMPM_Solver.hpp:15,
2020-08-03T02:19:09.5804759Z from /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/examples/free_fall.cpp:1:
2020-08-03T02:19:09.5805597Z /tmp/runner/spack-stage/spack-stage-exampm-master-n5v37glhooiigpsoumjmjzgpb55kvlw2/spack-src/src/ExaMPM_ProblemManager.hpp:78:49: error: wrong number of template arguments (2, should be 1)
2020-08-03T02:19:09.5806127Z 78 | using halo = Cajita::Halo<double,MemorySpace>;
2020-08-03T02:19:09.5806375Z | ^
from https://github.com/proxyapps/proxy-ci/runs/938692143?check_suite_focus=true
From Mark on slack:
I've been using ExaMPM (DamBreak) for performance and profiling on one Crusher node and continue to see what look like communications errors leading to spurious new velocities at processor boundaries which then lead to numerical blow-up and crashes (almost always in g2p->scatter->packBuffer
). This only occurs for problems over about 50^3 cells and 4 or more MPI ranks (I test with srun -N1 -n8 -S16 --exclusive -t30:00 --cpus-per-task=1 --threads-per-core=1 --gpus-per-task=1 --gpu-bind=closest ./DamBreak 0.01 2 3 0.00004 10.0 2500 hip
) and does not go away with wider halos, evenly-divisible ny, different Y boundary conditions (periodic, slip, noslip), or different versions of Cabana (0.5.0, head). It seems to be suppressed somewhat with fewer particles per cell and by using AMD_SANITIZE_KERNEL
and AMD_SANITIZE_COPY
vars, but never goes away. For a while, it seemed to always happen 2 or 3 time steps after a Silo write, but still occurs without any Silo writes. It can occur anywhere between steps 5000 and 100000.
From https://github.com/proxyapps/proxy-ci/runs/1051635986
[ 95%] Building CXX object cajita/unit_test/CMakeFiles/Cajita_SplineEvaluation_test_SERIAL.dir/SERIAL/tstSplineEvaluation_SERIAL.cpp.o
cd /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test && /home/runner/spack/lib/spack/env/gcc/g++ -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/gtest -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test/../../core/unit_test -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src -I/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/src -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-9.3.0/kokkos-3.2.00-g6y3qy5fm6vw66eqwhoemkbkg7cbcqar/include -isystem /home/runner/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-9.3.0/openmpi-3.1.6-c4biojltyxh4hvwvjlc47vmh3vkoiwfq/include -DGTEST_HAS_PTHREAD=0 -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -std=c++14 -o CMakeFiles/Cajita_SplineEvaluation_test_SERIAL.dir/SERIAL/tstSplineEvaluation_SERIAL.cpp.o -c /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL/tstSplineEvaluation_SERIAL.cpp
In file included from /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/unit_test/tstInterpolation.hpp:17,
from /tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-build/cajita/unit_test/SERIAL/tstInterpolation_SERIAL.cpp:2:
/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src/Cajita_Interpolation.hpp:243:51: error: type/value mismatch at argument 4 in template parameter list for 'template<class DataType, class Layout, class DeviceType, class Op, class Duplication, class Contribution> class Kokkos::Experimental::ScatterView'
243 | DataType, Layout, ExecSpace, Op, duplication, contribution>>
| ^~~~~~~~~~~~
/tmp/runner/spack-stage/spack-stage-cabana-master-str47o5vqa3jvlwfstww7aqiqd6qzois/spack-src/cajita/src/Cajita_Interpolation.hpp:243:51: note: expected a type, got 'Op'
Hi,
My aim is to write a SPH-DEM solver and simulate millions of particles using Cabana, and in the process I am learning to understand the internals and basics of the API.
I am successful at installing the ExaMPM software, however, the DamBreak example does not dump any output files while running.
Here is the output:
/DamBreak 0.05 2 0 0.001 1.0 50 serial
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
For unit testing set OMP_PROC_BIND=false
Time 0.000000 / 1.000000
Time 0.050000 / 1.000000
...
Can someone please help with this issue?
Many thanks.
spack install exampm
leads to:
[ 66%] Building CXX object examples/CMakeFiles/DamBreak.dir/dam_break.cpp.o
cd /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/spack-stage-exa
mpm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples -I/tmp/junghans/spack-stage/spack
-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/src -isystem /home/junghans/spack.git
/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/openmpi-3.1.6-22nn2lya7jfgf7eo
4vw42n7aq6t7dz2q/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-master-ueinsnbtht7zrp5rmwof3wrt4jkd3pq7/include -isystem /home/junghans/spack.git/opt/spack/linu
x-centos7-haswell/gcc-8.2.0/silo-4.10.2-bk5whiftxlqnmvjcigaz5466nmm6vhaf/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/FreeFall.dir/free_fall.cpp.o -c /tmp/junghans/s
pack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/free_fall.cpp
cd /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples && /home/junghans/spack.git/lib/spack/env/gcc/g++ -I/tmp/junghans/spack-stage/spack-stage-exa
mpm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/examples -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples -I/tmp/junghans/spack-stage/spack
-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src -I/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-build/src -isystem /home/junghans/spack.git
/opt/spack/linux-centos7-haswell/gcc-8.2.0/kokkos-3.1.01-ewfgo3wepuf7d37jgtinll7qk6zigof7/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/openmpi-3.1.6-22nn2lya7jfgf7eo
4vw42n7aq6t7dz2q/include -isystem /home/junghans/spack.git/opt/spack/linux-centos7-haswell/gcc-8.2.0/cabana-master-ueinsnbtht7zrp5rmwof3wrt4jkd3pq7/include -isystem /home/junghans/spack.git/opt/spack/linu
x-centos7-haswell/gcc-8.2.0/silo-4.10.2-bk5whiftxlqnmvjcigaz5466nmm6vhaf/include -O2 -g -DNDEBUG -march=core-avx2 -mtune=core-avx2 -pthread -o CMakeFiles/DamBreak.dir/dam_break.cpp.o -c /tmp/junghans/s
pack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/dam_break.cpp
In file included from /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:16,
from /tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/examples/free_fall.cpp:1:
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp: In instantiation of 'void ExaMPM::TimeIntegrator::fieldSolve(const ExecutionSp
ace&, const ProblemManagerType&, double, double, const ExaMPM::BoundaryCondition&) [with ProblemManagerType = ExaMPM::ProblemManager<Kokkos::HostSpace>; ExecutionSpace = Kokkos::Serial]':
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp:373:15: required from 'void ExaMPM::TimeIntegrator::step(const ExecutionSpace
&, const ProblemManagerType&, double, double, const ExaMPM::BoundaryCondition&) [with ProblemManagerType = ExaMPM::ProblemManager<Kokkos::HostSpace>; ExecutionSpace = Kokkos::Serial]'
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:97:33: required from 'void ExaMPM::Solver<MemorySpace, ExecutionSpace>::solve(double,
int) [with MemorySpace = Kokkos::HostSpace; ExecutionSpace = Kokkos::Serial]'
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_Solver.hpp:79:10: required from here
/tmp/junghans/spack-stage/spack-stage-exampm-master-rqaueribiehmdob6zd7sxbj56rls54wu/spack-src/src/ExaMPM_TimeIntegrator.hpp:134:10: error: no matching function for call to 'Cajita::LocalGrid<Cajita::UniformMesh<double> >::indexSpace(Cajita::Ghost, Cajita::Node, Cajita::Global)'
auto global_nodes =
^~~~~~~~~~~~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.