Comments (19)
- Modify our testing scripts, passing stuff like
-DCMAKE_CXX_COMPILER:STRING=mpicxx
.
That is what you need to do if you don't get compilers from Trilinos after calling find_package(Trilinos ...)
.
- Add a call to
find_package (MPI REQUIRED)
, and then link albanyLib against MPI. I thought Trilinos already linked to MPI, but I guess it simply relies on being built with MPI wrappers.
Even if you override the Trilinos lib install dir to be <prefix>/lib
, that does not work with CUDA builds (because you can't call find_package(CUDAToolkit)
until after you define the compilers).
@bartlettroscoe am I correct saying that trilinos does not link against MPI (in a CMake sense), but assumes mpi wrappers are used as compilers?
Trilinos just uses the MPI compiler wrappers. If you are building shared libraries, then the shared Trilinos libraries should be linked to the shared MPI libraries (because that is what the mpicxx
wrapper will do when building shared libs). But if you are using a static build of Trilinos, then the static Trilinos libraries are not linked to anything (because static libs are just dumb globs of object files).
Btw, I checked my local build of trilinos, and I found
set(Trilinos_MPI_LIBRARIES "") set(Trilinos_MPI_LIBRARY_DIRS "") set(Trilinos_MPI_INCLUDE_DIRS "")
I don't the above vars were ever set when using MPI complier wrappers. Those should have been removed years ago. (We have lost the history for who put those in from the CVS days.)
set(Trilinos_MPI_EXEC "/usr/lib64/openmpi/bin/mpiexec") set(Trilinos_MPI_EXEC_PRE_NUMPROCS_FLAGS "") set(Trilinos_MPI_EXEC_MAX_NUMPROCS "4") set(Trilinos_MPI_EXEC_POST_NUMPROCS_FLAGS "") set(Trilinos_MPI_EXEC_NUMPROCS_FLAG "-np")
Yes, those do get set and you can count on those. (I know EMPIRE used to use those or still does.)
so I could not even just link
${Trilinos_MPI_LIBRARIES}
to Albany, and be done with it. Is it expected that those vars be empty? Note: I used mpicxx/mpicc/mpifort as CXX/C/F compilers when building trilinos.
When doing MPI builds, you should use MPI compiler wrappers when building Albany.
NOTE: There has been some discussion of switching TriBITS and Trilinos to use the standard FindMPI.cmake
module with find_package(MPI)
, see:
but that will be another major break in Trilinos backward compatibility.
from albany.
It's interesting that Trilinos did not go the
find_package
way for MPI. It seems to mo that it would be the most cmake-y way, although the compiler wrappers is probably the easiest way.
@bartgol, way back in 2008 when Trilinos first adopted CMake, the standard FindMPI.cmake
module did not work on a few critical platforms that Trilinos had to support (because the MPI compiler wrappers on those systems did not support a --show
argument or whatever it was called to spit out the compiler and link options). It is unclear when those systems disappeared or when/if a version of CMake was released where FindMPI.cmake
works on all modern systems.
This is just part of the legacy code in TriBITS that needs to be addressed as part of:
(and it is not clear if and when that will ever happen).
As you can see, even small changes in backward compatibility can have large shock waves with something like Trilinos.
from albany.
Thanks for the history background! :)
from albany.
@ikalash I will try to fix the build scripts, though I'm not super expert on how our nightlies work (there are a lot of scripts in our dashboards folder). I will ping you on the PR, so you may take a second look and possibly fix what I did wrong.
Sounds good. You should find configure scripts for most of the nightlies here: https://github.com/sandialabs/Albany/tree/master/doc/dashboards . There are module files too when relevant. I am happy to test some fixes as well as they come up. I was intending to test your PR before it was merged, but did not get the chance - sorry.
from albany.
@bartlettroscoe disruptive changes are welcome if they contribute to improve the quality of the package config/build system. I think CMake was adopted by many project without paying as much attention to software quality as it was payed to actual source code. Patches and hacks piled up through the years, all of them designed with a cmake2.0 eye. True switch to cmake3.0 practices is still far away in certain projects, and is sometimes viewed as a burden. Imho, it makes things simpler, cleaner, and faster to tweak (once the project is fully in cmake3.0-style). Alas, it takes effort (in certain projects) to convince ppl that we should switch, because "their scripts won't work anymore".
That said, in Albany we're happy to follow more modern approaches, mine was just a "general" rant. :)
from albany.
Perlmutter scripts are updated in the dashboard and the builds are working. I'm adding some new features for the Perlmutter tests today and then I'm going to push all the changes to the repo together.
from albany.
Yes, I think these are definitely b/c of that PR. What I think is happening is that we're using serial compilers, rather than mpicxx/mpicc/mpifort.
Two possible solutions:
- Modify our testing scripts, passing stuff like
-DCMAKE_CXX_COMPILER:STRING=mpicxx
. - Add a call to
find_package (MPI REQUIRED)
, and then link albanyLib against MPI. I thought Trilinos already linked to MPI, but I guess it simply relies on being built with MPI wrappers.
@bartlettroscoe am I correct saying that trilinos does not link against MPI (in a CMake sense), but assumes mpi wrappers are used as compilers? Btw, I checked my local build of trilinos, and I found
set(Trilinos_MPI_LIBRARIES "")
set(Trilinos_MPI_LIBRARY_DIRS "")
set(Trilinos_MPI_INCLUDE_DIRS "")
set(Trilinos_MPI_EXEC "/usr/lib64/openmpi/bin/mpiexec")
set(Trilinos_MPI_EXEC_PRE_NUMPROCS_FLAGS "")
set(Trilinos_MPI_EXEC_MAX_NUMPROCS "4")
set(Trilinos_MPI_EXEC_POST_NUMPROCS_FLAGS "")
set(Trilinos_MPI_EXEC_NUMPROCS_FLAG "-np")
so I could not even just link ${Trilinos_MPI_LIBRARIES}
to Albany, and be done with it. Is it expected that those vars be empty? Note: I used mpicxx/mpicc/mpifort as CXX/C/F compilers when building trilinos.
from albany.
Thanks! We'll set the compilers to be the mpi wrappers, and be done with it.
It's interesting that Trilinos did not go the find_package
way for MPI. It seems to mo that it would be the most cmake-y way, although the compiler wrappers is probably the easiest way.
from albany.
@ikalash I will try to fix the build scripts, though I'm not super expert on how our nightlies work (there are a lot of scripts in our dashboards folder). I will ping you on the PR, so you may take a second look and possibly fix what I did wrong.
from albany.
Quick question about your changes to the cmake files @bartgol : I assume they are backward compatible with
-D Trilinos_USE_GNUINSTALLDIRS=OFF
Is that right? In other words, if I have the above option in my Trilinos configure script, Albany should still build, correct?
from albany.
Quick question about your changes to the cmake files @bartgol : I assume they are backward compatible with
-D Trilinos_USE_GNUINSTALLDIRS=OFF
Is that right? In other words, if I have the above option in my Trilinos configure script, Albany should still build, correct?
I think so? I do make one assumption on the trilinos install dir, and that is the bin
subfolders (for all the seacas executables), but I think that subdir will be there with virtually every meaningful installation way, including whatever happens with GNUINSTALLDIRS=OFF.
That said, we should remove that as soon as these feailures are fixed. In fact, we should probably remove it from some builds, to ensure everything works as planned.
from albany.
Quick question about your changes to the cmake files @bartgol : I assume they are backward compatible with
-D Trilinos_USE_GNUINSTALLDIRS=OFF
Is that right? In other words, if I have the above option in my Trilinos configure script, Albany should still build, correct?
I think so? I do make one assumption on the trilinos install dir, and that is the
bin
subfolders (for all the seacas executables), but I think that subdir will be there with virtually every meaningful installation way, including whatever happens with GNUINSTALLDIRS=OFF.That said, we should remove that as soon as these feailures are fixed. In fact, we should probably remove it from some builds, to ensure everything works as planned.
Ok, I will start changing this in the nightlies once you get the builds fixed. Let me know if that sounds reasonable.
from albany.
FYI: Responding to above, configuring older versions of Trilinos with either:
-D Trilinos_USE_GNUINSTALLDIRS=OFF
(because that was the old default for that var) or
-D CMAKE_INSTALL_LIBDIR:STRING=lib
(because that var is ignored when not using GNUInstallDirs.cmake) will be backward compatible and yield the same behavior (i.e. having the Trilinos lib install dir named <trilinosInstallDir>/lib
instead of <trilinosInstallDir>/lib64
).
SIDENOTE: It is amazing how much of a shockwave this one seemingly small non-backward compatible change to adopt one modern CMake practice has made on even just the local the Trilinos user community. This does not bode well for all of the changes to switch to modern CMake listed in:
from albany.
@jewatkins @ikalash I'm trying to update our nightly scripts, but I don't understand what scripts run. I do see lots of scripts in, say, doc/dashboards/weaver.sandia.gov
, and I have a hard time figuring out how they are invoked. There seem to be a lot of hacky bash stuff happening, which I can't detangle. I thought I just had to modify do-cmake-albany
, but I don't understand if that's correct. E.g., that file has the line
TRILINSTALLDIR=
Is some other script going to modify this line before we run this script? Also, do-cmake-weaver-trilinos
only sets the CXX compiler. How are the C/Fortran mpi compilers detected?
from albany.
@bartgol : this gets set automatically to the right dir in the .cmake script that gets created from this do-cmake script during the running of the nightly tests. A user using just the do-cmake script is supposed to set that path themself.
from albany.
I believe this should be fixed starting tomorrow everywhere except perlmutter. On perlmutter, it looks like there are some issues pointing to the write mpicc / mpicxx compiler: https://my.cdash.org/build/2414115/configure . @mcarlson801 can you please have a look?
from albany.
Thanks for fixing all the testing scripts!
from albany.
Sure. Max still needs to fix the perlmutter ones. We can discuss your proposition to clean up the testing scripts at the next Albany meeting.
from albany.
I forgot to say, I haven't tried removing -D Trilinos_USE_GNUINSTALLDIRS=OFF
yet from the nightly configure scripts, but I will do this next. Note that there are some failures now in Albany tests, namely #988 , #989 , #990 .
from albany.
Related Issues (20)
- Nightly test failures HOT 39
- corePDEs_SideSetLaplacian_3D failing on weaver after epetra removal HOT 6
- Clean up nightlies to turn off Epetra HOT 4
- PyAlbany MatrixOperations test failing in spack nightly HOT 18
- Attaway sems modules are broken, causing those tests to fail
- Failing PyAlbany test in spack nightly build HOT 4
- Failing nightly builds due to Omegah HOT 2
- Rework nightlies on CEE for new RHEL8 OS HOT 1
- Can not finish albany spack install since trilinos failed to be installed HOT 6
- Failing attaway nightlies due to Netcdf/Pnetcdf HOT 28
- Weaver failing nightlies HOT 10
- Failing landIce_FO_GIS_AdjointSensitivity_Thickness test in nightly spack build HOT 12
- Failing corePDEs_SteadyHeatConstrainedOpt2D_Conductivity_Dist_Param_Restart test in some nightlies HOT 3
- Failing corePDEs_SideSetLaplacian_3D in CUDA weaver build HOT 2
- Failing landIce_FO_MMS_FO tests on some platforms HOT 8
- Failing corePDEs_SideSetLaplacian_3D test in weaver build HOT 2
- Get CEE clang builds back up on cee-compute001 and cee-compute005
- CALI build failure in nightlies HOT 3
- landIce_FO_GIS_AdjointSensitivity_Thickness failing again on several platforms HOT 6
- New milestone or Version after 200 closed issues since 2020 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from albany.