Hi,

I noticed there is some inconsistency for outputs of potential temperature and moist potential temperature in MPAS atmosphere. This happens when the output frequency is different from restart frequency.

A simple experiment:

set output frequency to "X" days, set restart frequency to "Y" days
Symptom: theta and theta_m fields are identical in output files (wrong), but different in restart files (correct). theta and theta_m in restart files are differed by a factor of 1+Rv/Rd*qv as expected.

set output frequency to "X" days, set restart frequency to "X" days
Symptom: theta and theta_m fields are different in both output and restart files (all correct)

Could anyone take a look of the issue?

Thanks,
Yuan

(I was told to log these PIO issues here, if this is not the right forum can you please direct me to the right one? -- thanks)

The variable INCLUDEPATH is used by the PIO cmake but is not documented in the --help output.
Ordinarily I would expect this variable to be used in the form dir1:dir2:.... which is how some of the modules on our system are setting it.
The PIO build, however, is expanding the $INCLUDEPATH directly in some of the build commands

/shared/apps/rhel-6.2/tools/cponder/OpenMPI-1.8.4/Intel-15.0_CUDA-6.5/bin/mpifort -c -DSPMD -DHAVE_MPI -DUSEMPIIO -DSYSLINUX -D_NETCDF -I/shared/apps/rhel-6.2/tools/cponder/NetCDF-Fortran-4.4.1/Intel-15.0_OpenMPI-1.8.4_CUDA-6.5/include -D_NETCDF4 -D_PNETCDF -fPIC -march=corei7-avx -mtune=corei7-avx -g -I/shared/apps/rhel-6.2/tools/cponder/NetCDF-Fortran-4.4.1/Intel-15.0_OpenMPI-1.8.4_CUDA-6.5/include /shared/apps/cuda/CUDA-v6.5.14/include/CL:/shared/apps/cuda/CUDA-v6.5.14/include -I/shared/apps/rhel-6.2/tools/cponder/PNetCDF-1.6.0/OpenMPI-1.8.4_Intel-15.0_CUDA-6.5/include pio_kinds.F90

and giving me the error

ifort: error #10236: File not found: '/shared/apps/cuda/CUDA-v6.5.14/include/CL:/shared/apps/cuda/CUDA-v6.5.14/include'

Evidently it's expected here to use a form like

-I DIR1 -I DIR2 ...

I can get around the problem by unsetting the variable, which is OK for my purposes. It may be the case that the -I DIR format really is the standard convention, but I think at the very least that you ought to document that INCLUDEPATH is a settable variable in the cmake build.

Background

Previously, there have been some incompatibilities between different versions of netcdf (and its bindings), parallel-netcdf, and PIO. I've reconciled this by "tagging" the versions to known numbers that work at https://github.com/pwolfram/homebrew-mpas, namely with the following installation recipe.

Installation

Run ./install.sh to install brew and taps needed for this installation.

• netcdf bindings for MPAS-Tools: brew install pwolfram/mpas/netcdf --enable-fortran --enable-cxx-compat
  • netcdf 4.3.3.1
  • netcdf-cxx 4.2
  • netcdf-fortran 4.4.2
• parallel-netcdf 1.4.1: brew install pwolfram/mpas/parallel-netcdf
• pio 1.7.2: brew install pwolfram/mpas/pio

Compile MPAS, e.g.,

export NETCDF=/usr/local/Cellar/netcdf/4.3.3.1/
export PNETCDF=/usr/local/Cellar/parallel-netcdf/1.4.1/
export PIO=/usr/local/Cellar/pio/1.7.2/
make clean gfotran CORE=ocean

Question

What is the preferred library stack, especially for PIO2? In particular, the problem with the
homebrew recipe in that there is an incompatibility with openmpi > 1.6.

Any recommendations on the current best-practices library stack is greatly appreciated. Furthermore, I would recommend that we place this information on the README.md for the project so that new users will have fewer challenges building.

cc @hguo, @tpeterka, @vanroekel

To whom it may concern:

I believe there to be an issue with MPAS-A atmosphere core executables built using PGI 12.5. There currently exist issues with MPAS-O, using PGI 12.5, and PGI 12.5 is thus unsupported by MPAS-O (as per Doug Jacobsen). The PGI bug reports can be found here: http://www.pgroup.com/support/release_tprs_2012.htm. The specific issue for MPAS-O is:

"Wrong answers when "transpose(derc)" is passed to matmul
Corrected the handling of matmul when the first argument is a transpose of a non-square matrix"

MPAS-A would (seemingly inexplicably and randomly) fail. My initial inclination was that either I was running into CFL issues and/or my vertical velocities were too large thus causing the model to segmentation fault. I reduced my timestep, for a 60-km MPAS-A mesh, from the default 180.0 seconds and test again. In some of these instances, the experiment would run to completion or it would simply run longer into it's respective forecast before again failing. I also changed the config_zd and config_xnutr values from the default 22000.0 and 0.2, respectively to 5000.0 and 1.0, respectively. Again, in some instances these experiments would run either to completion and/or just longer.

After speaking with Doug Jacobsen regarding some issues with MPAS-O, I decided to build MPAS-A with PGI 13.2-0 (which is the latest PGI compiler available on the NOAA Jet machines). So far, all experiments which had failed previously have completed successfully -- using the default timestep (180 seconds) and config_zd and config_xnutr values. Whether the MPAS-O and MPAS-A issues are due to the same PGI bug is currently unknown to me.

It should be noted that I DID NOT rebuild PIO and PNETCDF with the latest available PGI compiler.

Regards,

Henry Winterbottom

Hi,

I am trying to understand the dynamical core of the MPAS model and trying to run the test cases. I could run supercell and Mountain wave (cases 5,6) easily but having trouble with the 1-4 test cases where I am encountering with floating-point error and core dumps. Below are the errors. Is this error something to do with the flags or single/double precision? The have tried with 1-4 cases with the case 5 and 6 executables as both of them differ.

Test-1 error

grid metrics setup complete
MAX U wind before REBALANCING ----> 34.9849780606604
MAX U wind after REBALANCING ----> 34.9990526152960
forrtl: error (73): floating divide by zero
Image PC Routine Line Source
init_atmosphere_m 000000000123B615 Unknown Unknown Unknown
init_atmosphere_m 0000000001239237 Unknown Unknown Unknown
init_atmosphere_m 00000000011E7294 Unknown Unknown Unknown
init_atmosphere_m 00000000011E70A6 Unknown Unknown Unknown
init_atmosphere_m 000000000117A0E6 Unknown Unknown Unknown
init_atmosphere_m 000000000117E4B6 Unknown Unknown Unknown
libpthread.so.0 00002AB2D30D8790 Unknown Unknown Unknown
init_atmosphere_m 00000000007B2CCE init_atm_cases_mp 1009 mpas_init_atm_cases.F
init_atmosphere_m 0000000000764B55 init_atm_cases_mp 121 mpas_init_atm_cases.F
init_atmosphere_m 0000000000763499 init_atm_core_mp_ 81 mpas_init_atm_core.F
init_atmosphere_m 0000000000412ABC mpas_subdriver_mp 326 mpas_subdriver.F
init_atmosphere_m 000000000040E7DA MAIN__ 16 mpas.F
init_atmosphere_m 000000000040E75E Unknown Unknown Unknown
libc.so.6 00002AB2D3588D5D Unknown Unknown Unknown
init_atmosphere_m 000000000040E669 Unknown Unknown Unknown

Test-4 error

WARNING: File
squall line - super cell test case
squall line test case
calling test case setup
forrtl: error (65): floating invalid
Image PC Routine Line Source
init_atmosphere_m 000000000123B615 Unknown Unknown Unknown
init_atmosphere_m 0000000001239237 Unknown Unknown Unknown
init_atmosphere_m 00000000011E7294 Unknown Unknown Unknown
init_atmosphere_m 00000000011E70A6 Unknown Unknown Unknown
init_atmosphere_m 000000000117A0E6 Unknown Unknown Unknown
init_atmosphere_m 000000000117DFA7 Unknown Unknown Unknown
libpthread.so.0 00002B33654FC790 Unknown Unknown Unknown
init_atmosphere_m 000000000122C8FB Unknown Unknown Unknown
init_atmosphere_m 000000000122C387 Unknown Unknown Unknown
init_atmosphere_m 000000000122C357 Unknown Unknown Unknown
init_atmosphere_m 000000000092DDAD atm_advection_mp_ 543 mpas_atm_advection.F
init_atmosphere_m 000000000092D17B atm_advection_mp_ 437 mpas_atm_advection.F
init_atmosphere_m 0000000000918B10 atm_advection_mp_ 147 mpas_atm_advection.F
init_atmosphere_m 00000000007D7D5E init_atm_cases_mp 1398 mpas_init_atm_cases.F
init_atmosphere_m 000000000076596B init_atm_cases_mp 143 mpas_init_atm_cases.F
init_atmosphere_m 0000000000763499 init_atm_core_mp_ 81 mpas_init_atm_core.F
init_atmosphere_m 0000000000412ABC mpas_subdriver_mp 326 mpas_subdriver.F
init_atmosphere_m 000000000040E7DA MAIN__ 16 mpas.F
init_atmosphere_m 000000000040E75E Unknown Unknown Unknown
libc.so.6 00002B33659ACD5D Unknown Unknown Unknown
init_atmosphere_m 000000000040E669 Unknown Unknown Unknown

Regards,
Phani.

Hi, everyone
After compiling the MPAS init_atmosphere and atmosphere cores, i try to run example 'supercell'.
make a new directory
mkdir supercell
down load the package
wget http://www2.mmm.ucar.edu/projects/mpas/test_cases/v5.1/supercell.tar.gz tar zxvf supercell.tar.gz cd supercell
link to init_atmosphere_model
ln -s $MPAS/MPAS-Model/init_atmosphere_model
process the static field
./init_atmosphere_model
and get an error
Segmentation fault(core dumped)

file log.init_atmosphere.0000.out writes

Beginning MPAS-init_atmosphere Output Log File for task 0 of 1
Opened at 2018/10/21 18:32:34

Using default double-precision reals

Reading namelist from file namelist.init_atmosphere
*** Encountered an issue while attempting to read namelist record data_sources
The following values will be used for variables in this record:

     config_geog_data_path = /glade/p/work/wrfhelp/WPS_GEOG/
     config_met_prefix = CFSR
     config_sfc_prefix = SST
     config_fg_interval = 86400
     config_landuse_data = USGS
     config_topo_data = GTOPO30
     config_use_spechumd = F

*** Encountered an issue while attempting to read namelist record vertical_grid
The following values will be used for variables in this record:

     config_ztop = 30000.0000000000
     config_nsmterrain = 1
     config_smooth_surfaces = T
     config_dzmin = 0.300000000000000
     config_nsm = 30
     config_tc_vertical_grid = T

*** Encountered an issue while attempting to read namelist record interpolation_control
The following values will be used for variables in this record:

     config_extrap_airtemp = linear

*** Encountered an issue while attempting to read namelist record preproc_stages
The following values will be used for variables in this record:

     config_static_interp = T
     config_native_gwd_static = T
     config_gwd_cell_scaling = 1.00000000000000
     config_vertical_grid = T
     config_met_interp = T
     config_input_sst = F
     config_frac_seaice = T

*** Encountered an issue while attempting to read namelist record io
The following values will be used for variables in this record:

     config_pio_num_iotasks = 0
     config_pio_stride = 1

*** Encountered an issue while attempting to read namelist record decomposition
The following values will be used for variables in this record:

     config_block_decomp_file_prefix = x1.40962.graph.info.part.
     config_number_of_blocks = 0
     config_explicit_proc_decomp = F
     config_proc_decomp_file_prefix = graph.info.part.

Reading streams configuration from file streams.init_atmosphere
Found mesh stream with filename template supercell_grid.nc
Using default io_type for mesh stream
** Attempting to bootstrap MPAS framework using stream: input
Bootstrapping framework with mesh fields from input file 'supercell_grid.nc'
WARNING: Attribute file_id not found in supercell_grid.nc
WARNING: Using '' for the previous file_id.
WARNING: Attribute parent_id not found in supercell_grid.nc
WARNING: Setting parent_id to ''
WARNING: Attribute mesh_spec not found in supercell_grid.nc
WARNING: Setting mesh_spec to '0.0'
WARNING: is_periodic attribute not found in supercell_grid.nc
WARNING: Setting is_periodic to .false.

Parsing run-time I/O configuration from streams.init_atmosphere ...

----- found immutable stream "input" in streams.init_atmosphere -----
filename template: supercell_grid.nc
filename interval: none
direction: input
reference time: initial_time
record interval: -
input alarm: initial_only

----- found immutable stream "output" in streams.init_atmosphere -----
filename template: supercell_init.nc
filename interval: none
direction: output
reference time: initial_time
record interval: -
output alarm: initial_only
package: initial_conds

----- found immutable stream "surface" in streams.init_atmosphere -----
filename template: not_needed_for_supercell
filename interval: none
direction: output
reference time: initial_time
record interval: -
output alarm: 86400
package: sfc_update

----- done parsing run-time I/O from streams.init_atmosphere -----

** Validating streams

Reading dimensions from input streams ...

----- reading dimensions from stream 'input' using file supercell_grid.nc
nCells = 28080
nEdges = 84240
nVertices = 56160
TWO = 2
maxEdges = 6
maxEdges2 = 12
vertexDegree = 3

----- done reading dimensions from input streams -----

Processing decomposed dimensions ...

----- done processing decomposed dimensions -----

Assigning remaining dimensions from definitions in Registry.xml ...
THREE = 3
FIFTEEN = 15
TWENTYONE = 21
R3 = 3
nVertLevels = 40 (config_nvertlevels)
nSoilLevels = 4 (config_nsoillevels)
nFGLevels = 38 (config_nfglevels)
nFGSoilLevels = 4 (config_nfgsoillevels)
nVertLevelsP1 = 41
nMonths = 12 (config_months)

----- done assigning dimensions from Registry.xml -----

and gdb the core says
#0 0x000055963e735c8e in box_rearrange_create ()

How could i process a static field?

Packages:

• mpas_mesh_tools (MPAS-Dev/MPAS-Tools#248):
  • mesh conversion tools (MPAS-Dev/MPAS-Tools#243)
  • mark_horns_for_culling.py (?) (MPAS-Dev/MPAS-Tools#249)
  • planar hex (MPAS-Dev/MPAS-Tools#246)
  • 4 scripts from landice mesh tools (MPAS-Dev/MPAS-Tools#249):
    • create_landice_grid_from_generic_MPAS_grid.py
    • define_cullMask.py
    • interpolate_to_mpasli_grid.py
    • mark_domain_boundaries_dirichlet.py
  • planar mesh transformation tools
    • translate_planar_grid.py
    • ...
  • ...
• geometric_features (MPAS-Dev/geometric_features#102)
• metis
• jigsaw (https://github.com/xylar/staged-recipes/tree/jigsaw/recipes/jigsaw)
• jigsaw-geo-matlab?
• ...

This is in reference to my recent Okubo Weiss 3.3 pull request.

In Registry_okubo_weiss.xml, the output_interval parameter for the "okuboWeissOutput" stream is not being carried to the streams.ocean_forward.defaults after compilation.

I tried putting it in different positions, but in each case it was not being copied to the defaults file after compilation.

There is a bug in "coupled" mode here:
https://github.com/MPAS-Dev/MPAS-Model/blob/e3sm/develop/src/core_ocean/shared/mpas_ocn_surface_land_ice_fluxes.F#L810-L813
where T0 is accessed before it is computed.

Matus Martini and I discovered a bug at line 2488 of mpas_atm_time_integration.F. This line loops from k=1,nVertLevels to calculate tend_w. However, ru_edge_w(k) is only given values for k=2,nVertLevels.

We believe the appropriate fix is just to change line 2488 to loop from k=2,nVertLevels since the w-tendency is always zero at the bottom anyway, and it is given that value higher up in the subroutine.

-Bill

William I. Gustafson Jr., Ph.D.
Scientist
ATMOSPHERIC SCIENCES AND GLOBAL CHANGE DIVISION

Pacific Northwest National Laboratory
P.O. 999, MSIN K9-30
Richland, WA 99352
Tel: 509-372-6110
[email protected]
http://www.pnnl.gov/atmospheric/staff/staff_info.asp?staff_num=5716
http://www.researcherid.com/rid/A-7732-2008

When running init_atmosphere.exe, the following is dumped into the log.0000.err file:

Reading namelist.input

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source

init_nhyd_atmos_m 0000000000562CA3 mpas_block_creato 1057 mpas_block_c
reator.f90
init_nhyd_atmos_m 000000000053DDE9 mpas_io_input_mp_ 242 mpas_io_inpu
t.f90
init_nhyd_atmos_m 00000000004074CC mpas_subdriver_mp 38 mpas_subdriv
er.f90
init_nhyd_atmos_m 0000000000407396 MAIN__ 14 mpas.f90
init_nhyd_atmos_m 000000000040733C Unknown Unknown Unknown
libc.so.6 00002B133807CCDD Unknown Unknown Unknown
init_nhyd_atmos_m 0000000000407239 Unknown Unknown Unknown

My namelist.input is as follows:

&nhyd_model
config_init_case = 7
config_theta_adv_order = 3
config_start_time = '2012-08-26_06:00:00'
config_stop_time = '2012-08-26_15:00:00'
/
&dcmip
config_dcmip_case = '2-0-0'
config_planet_scale = 1.0
config_rotation_rate_scale = 1.0
/
&dimensions
config_nvertlevels = 41
config_nsoillevels = 4
config_nfglevels = 38
config_nfgsoillevels = 4
/
&data_sources
config_geog_data_path = './geog/'
config_met_prefix = 'FILE'
config_sfc_prefix = 'FILE'
config_fg_interval = 3600
/
&vertical_grid
config_ztop = 45000.0
config_nsmterrain = 2
config_smooth_surfaces = .false.
/
&preproc_stages
config_static_interp = .true.
config_vertical_grid = .false.
config_met_interp = .false.
config_input_sst = .false.
/
&io
config_input_name = 'grid.nc'
config_output_name = 'static.nc'
/
&restart
/
&decomposition
/

When setting up a Compass test suite, the maximum number of MPI tasks is reported as part of the final output after the suite has been set up. However, the driver script (config_driver.xml that produces run_test.py) may only include a subset of the configurations available in a given test case (e.g. forwrad_short but not forward). In many cases, it might be desirable to have a larger number of MPI tasks be used for longer runs but only a few for regression testing.

manage_regression_suite.py should, therefore, report the maximum MPI tasks for only those configurations that are part of config_driver.xml.

@mark-petersen ocn_init_alarms_TEMPLATE line 193 in mpas_ocn_TEMPLATE.F in analysis members is not used any longer.

The MOC online analysis member currently does not take into account the bolus velocity component when computing the streamfunction, in cases when the GM parameterization is on. This should be added to the AM, in order to have the total MOC result.

V6.1 tag (b777c06 on ocean/develop) has runtime failure on grizzly with intel debug for these tests:

  Periodic_Planar_20km_-_LIGHT_particle_region_reset_test
  Periodic_Planar_20km_-_LIGHT_particle_time_reset_test
  ZISO_20km_-_Smoke_Test

All tests pass for gnu optimized and intel optimized.

Hi all,

I have compiled the core_sw and successfully generated the sw_model. However, there is a crucial problem when running the ./sw_model with the x1.2562.. mesh, as following,

" CRITICAL ERROR: Dimension 'nTracers' required by field 'tracers' was neither read nor defined. "

The cofigration of namelist.sw may be right.

How should i solve this problem? Any ideas will be appreciated.

THANKS，
MEI

In setting up MPAS on my laptop (Ubuntu 18.04), I decided to avoid the hassle of compiling NetCDF libraries myself and used the default versions that come with the Ubuntu package manager. This worked well except one hiccup. The library files are placed in:

/usr/lib/x86_64-linux-gnu/

I was able to get MPAS-O to build by setting

export MPAS_EXTERNAL_LIBS='-L/usr/lib/x86_64-linux-gnu/ -lnetcdff'

but this seems to somewhat defeat purpose of all the fancy automation in the Makefile and it's also not very intuitive for users.

Might it be worth supporting separate $NETCDF_INC and $NETCDF_LIB variables, perhaps in addition to the current $NETCDF?

Also partly wondering if others have run into this problem (or something similar).

We don't run RK4 very often. Running EC60to30 on 576 cores on grizzly with gnu optimized, it runs for 7 days and then dies with memory errors.

Previously, the SOMA ocean test case used an ad-hoc approach to generating the unstructured mesh. This workflow needs updated following #74.

I'm running the ISOMIP+ test cases and recently turned on timeSeriesStatsMonthly. I run MPAS-Ocean repeatedly for one month, then update the forcing with a python script. Initially, I was also writing out restart files for timeSeriesStatsMonthly. It appears that xtime_startMonthly never gets updated in these circumstances (it is always the start of the simulation) and that none of the variables being written out get reset at the beginning of a month either. Yesterday, I re-ran without timeSeriesStatsMonthly restart files (which are not needed when the restart interval is a multiple of the compute interval) and xtime_startMonthly is now updated each month as expected.

I suspect, but haven't proven, that there is something bigger that is wrong with how alarms work on restart, and that this may related to E3SM-Project/E3SM#2499. My suspicion is that certain alarms are not triggered immediately on restart (or init) that would be triggered if the simulation continued without a restart. In the case of timeSeriesStatsMonthly, I think the alarm is not being triggered to zero out averaging variables and xtime_startMonthly is not being reset. It is not yet clear to me if this should happen immediately before restart files are written or immediately after they are read back in, but one of these would be the expected behavior.

I have a workaround and the same is true of E3SM as currently configured but this bug should be fixed, particularly if it indicates a bigger problem with alarms and how they interact with restart/init.

When I run a Compass test case with MPICH 3.3a2 installed on my laptop (running Ubuntu 18.04), I get the error (the first line repeated once for each MPI task):

[proxy:0:0@eleven] HYDU_create_process (utils/launch/launch.c:74): execvp error on file ocean_model (No such file or directory)
Traceback (most recent call last):
  File "./run.py", line 37, in <module>
    'namelist.ocean', '-s', 'streams.ocean'])
  File "/home/xylar/miniconda3/envs/mpas/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpirun', '-n', '4', 'ocean_model', '-n', 'namelist.ocean', '-s', 'streams.ocean']' returned non-zero exit status 255

The solution seems to be changing from ocean_model to ./ocean_model.

I was able to get this result by changing this line:
https://github.com/MPAS-Dev/MPAS-Model/blob/master/testing_and_setup/compass/setup_testcase.py#L1086
as follows

-                        grandchild.text = executable_link
+                        grandchild.text = './{}'.format(executable_link)

Before I make a pull request, I thought I'd get feedback on whether this is an issue others have run into and also whether this fix seems appropriate.

Version of MPAS:
2.0. Git repository pulled within past day.

Description of issue:
Log output shows NaNs after the first timestep:

Begin timestep 0000-01-01_00:00:03
...
global min, max w NaN NaN

Plotted fields at initial time look reasonable, but subsequent times fail to plot.

Program continues to run after NaNs are printed.

If compiled with DEBUG=true, the NaNs go away. Did not check netCDF.

Platform used (System Specs, as well as name, and operating system if possible):
Head node: RHEL 6.0, 16 x Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 16 GB RAM
Compute node: CentOS 6.3, 8 x Intel(R) Xeon(R) CPU X5560 @ 2.80GHz, 24 GB RAM

Compiler used (With version):
mpiifort/ifort 13.0.1
Had to modify Makefile, under ifort, to use mpiifort rather than mpif90,
and mpiicc rather than mpicc.

NetCDF version used:
4.1.3

Parallel NetCDF version used:
1.4.1

Parallel I/O version used:
1.7.1

Number of processors used:
Tried running separately on head and compute nodes.
Tried running witout MPI, or with 1 or 8 processors specified.

Test case used (if applicable):
Tried both supercell and jw_baroclinic_wave.

Is this issue reproducible in debug mode (make ... DEBUG=TRUE):
No. NaNs are not present in the log output when compiled with DEBUG=true, or with -O0.

If possible, please provide a backtrace for this issue:

I'm building the current ocean/develop on Edison, details below. When I run make intel-nersc, I'm running into an error:

(cd inc; /global/homes/x/xylar/mpas_work/model/edison/temp/src/tools/registry/parse < ../Registry_processed.xml )

Please verify that both the operating system and the processor support Intel(R) F16C instructions.

Makefile:32: recipe for target 'gen_includes' failed

The issue is that the compiler is configured for ivybridge, whereas the login nodes are sandybridge (see http://www.nersc.gov/users/computational-systems/edison/updates-and-status/open-issues/compiling-serial-codes-to-run-on-login-nodes-in-the-intel-programming-environment/)

I am able to build successfully if I make the following edits to the Makefile:

-       "FC_SERIAL = ftn" \
-       "CC_SERIAL = cc" \
-       "CXX_SERIAL = CC" \
+       "FC_SERIAL = ifort" \
+       "CC_SERIAL = icc" \
+       "CXX_SERIAL = icpc" \

However, editing the Makefile isn't a very practical solution, since I would need to do this every time I create a new branch on Edison.

How are other folks handling MPAS builds on Edison?

Details

Steps to set up the environment:

module unload PrgEnv-intel
module unload PrgEnv-cray
module unload PrgEnv-gnu
module unload intel
module unload cce
module unload gcc
module unload cray-parallel-netcdf
module unload cray-parallel-hdf5
module unload cray-libsci
module unload cray-netcdf
module unload cray-hdf5
module unload cray-netcdf-hdf5parallel
module unload papi
module unload cray-petsc
module unload esmf

module load PrgEnv-intel/6.0.4
module unload intel
module load intel/18.0.1.163
module unload cray-libsci

module unload cray-netcdf-hdf5parallel
module load cray-netcdf-hdf5parallel/4.4.1.1.3
module load cray-hdf5-parallel/1.10.1.1
module load cray-parallel-netcdf/1.8.1.3

export CORE=ocean
export NETCDF=${NETCDF_DIR}
export PNETCDF=${PARALLEL_NETCDF_DIR}
export PIO=/global/homes/x/xylar/mpas_work/model/${NERSC_HOST}/pio/pio2.3.1
export USE_PIO2=true

The result:

> module list
Currently Loaded Modulefiles:
  1) modules/3.2.10.6
  2) nsg/1.2.0
  3) cray-mpich/7.6.2
  4) git/2.15.1
  5) altd/2.0
  6) darshan/3.1.4
  7) craype-ivybridge
  8) craype-network-aries
  9) craype/2.5.12
 10) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari
 11) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari
 12) pmi/5.0.12
 13) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari
 14) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari
 15) xpmem/2.2.4-6.0.5.1_8.18__g35d5e73.ari
 16) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari
 17) dvs/2.7_2.2.64-6.0.5.2_15.3__g97b34c4
 18) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari
 19) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari
 20) atp/2.1.1
 21) PrgEnv-intel/6.0.4
 22) intel/18.0.1.163
 23) cray-netcdf-hdf5parallel/4.4.1.1.3
 24) cray-hdf5-parallel/1.10.1.1
 25) cray-parallel-netcdf/1.8.1.3

Hi all,

I'm not sure whether this should be classified as an MPAS-A bug or a CESM one, as I think the RRTMG code is pulled from there, but in the 'generate_stochastic_clouds_sw' routine inside 'src/core_atmosphere/physics/physics_wrf/module_ra_rrtmg_sw.F', at line 1701, there's the following test:

if (pmid(i,1).lt.pmid(i,2)) then
    stop 'MCICA_SUBCOL: KISSVEC SEED GENERATOR REQUIRES ... <cut>..'
endif

Following this, the bottom four levels of pmid [ pmid(i,1-4) ] are used to initialize the RNG stream, but there doesn't seem to be a need to check if pmid(i,1) > pmid(i,2), and this has caused a crash on a set of tests being run at KISTI using a 163K cell domain (60km, uniform). I've advised they simply comment out the test for the time being, but I'm not very familiar with this part of the code so I thought I'd pass this along as a bug for the experts to weigh in on.

Thanks,

• Brian

From @maltrud: The value of the climatological SSS field that is used for SSS restoring is not being updated correctly. i ran a E3SM G-case for 15 months and while the first month agreed with the read-in climatology file, subsequent months didn't change much--certainly no seasonal cycle.

i am currently testing a version of the code from before the following commit to see if that may be the cause.

commit d2a7f49
Author: Mark Petersen [email protected]
Date: Thu Nov 16 10:09:34 2017 -0700

Only a single forcing group linked list should exist per core.

These were put in by Adrian in PR 1333, but then removed
inadvertenly in PR 1427

To whom it may concern:

I am experimenting using the MPAS ocean core (compiled using PGI) for the 60-km global ocean test case found here: http://mpas-dev.github.io/ocean/release_1.0/release_1.0.html .

When compiled in DEBUG mode, the following is reported in some of the log files:

Caught error: Floating point exception (signal 8)

which suggests either a NaN or INF value may be present.

Has anyone set up a similar experiment, using PGI and the 60-km QU ocean and either had success or similar runtime issues?

Thanks in advance.

Henry Winterbottom

The compilation process has been completed, and produced init_atmosphere_model and atmosphere_model file. The init_atmosphere_model file run and produced static.nc and init.nc.
When used the mesh of x1.2562.grid.nc, the output is right. My problem is no results when using mesh of x1.655362.grid.nc and x1.40962.init.nc.

Additionally please enter this information if applicable:

• Compiler and version
  ifort and icc, 2017
• MPI implementation and version
  mpich 3.2
• NetCDF version
  netcdf 4.4.1.1
• Parallel NetCDF version
  parallel netcdf 1.8.1
• Parallel IO version
  ParallelIO 2.3.0
• The make line used to build the executable
  make ifort CORE=atmosphere USE_PIO2=true
• The line used to run the executable
  /XXX/MPAS/mpas_io/bin/mpiexec.hydra -genv I_MPI_DEVICE rdma /XXX/modelcore/MPAS/MPAS-Release-5.2/atmosphere_model
• Test case used
  CFSR:2010-10-23_00
  x1.40962.init.nc

Here is the compiling information:

export LIBSRC=XXX/MPAS_sources
export LIBBASE=XXX

Compilers

export SERIAL_FC=ifort
export SERIAL_F77=ifort
export SERIAL_CC=icc
export SERIAL_CXX=icpc
export MPI_FC=mpifort
export MPI_F77=mpifort
export MPI_CC=mpicc
export MPI_CXX=mpicxx
########################################

MPICH

########################################
tar xzvf ${LIBSRC}/mpich-3.2.tar.gz
cd mpich-3.2
export CC=$SERIAL_CC
export CXX=$SERIAL_CXX
export F77=$SERIAL_F77
export FC=$SERIAL_FC
unset F90 # This seems to be set by default on NCAR's Cheyenne and is problematic
unset F90FLAGS
./configure --prefix=${LIBBASE} --disable-shared
make
make check
make install
export PATH=${LIBBASE}/bin:$PATH
cd ..
rm -rf mpich-3.2
########################################

cmake

########################################
tar xzvf ${LIBSRC}/cmake-3.4.0-rc3.tar.gz
cd cmake-3.4.0-rc3
./bootstrap --prefix=${LIBBASE}
gmake
gmake install
export PATH=${LIBBASE}/bin:$PATH
cd ..
rm -rf cmake-3.4.0-rc3
########################################

zlib

########################################
tar xzvf ${LIBSRC}/zlib-1.2.8.tar.gz
cd** zlib-1.2.8
./configure --prefix=${LIBBASE} --static
make
make install
cd ..
rm -rf zlib-1.2.8
########################################

HDF5

########################################
tar xjvf ${LIBSRC}/hdf5-1.8.19.tar.bz2
cd hdf5-1.8.19
export FC=$MPI_FC
export CC=$MPI_CC
export CXX=$MPI_CXX
./configure --prefix=${LIBBASE} --enable-parallel --with-zlib=${LIBBASE} --disable-shared
make
make check
make install
cd ..
rm -rf hdf5-1.8.19
########################################

Parallel-netCDF

########################################
tar xzvf ${LIBSRC}/parallel-netcdf-1.8.1.tar.gz
cd parallel-netcdf-1.8.1
export CC=$SERIAL_CC
export CXX=$SERIAL_CXX
export F77=$SERIAL_F77
export FC=$SERIAL_FC
export MPICC=$MPI_CC
export MPICXX=$MPI_CXX
export MPIF77=$MPI_F77
export MPIF90=$MPI_FC

Will also need gcc in path

./configure --prefix=${LIBBASE}
make
make testing
make install
export PNETCDF=${LIBBASE}
cd ..
rm -rf parallel-netcdf-1.8.1
########################################

netCDF (C library)

########################################
tar xzvf ${LIBSRC}/netcdf-4.4.1.1.tar.gz
cd netcdf-4.4.1.1
export CPPFLAGS="-I${LIBBASE}/include"
export LDFLAGS="-L${LIBBASE}/lib"
export LIBS="-lhdf5_hl -lhdf5 -lz -ldl"
export CC=$MPI_CC
./configure --prefix=${LIBBASE} --disable-dap --enable-netcdf4 --enable-pnetcdf --enable-parallel-tests --disable-shared
make
make check
make install
export NETCDF=${LIBBASE}
cd ..
rm -rf netcdf-4.4.1.1

########################################

netCDF (Fortran interface library)

########################################
tar xzvf ${LIBSRC}/netcdf-fortran-4.4.4.tar.gz
cd netcdf-fortran-4.4.4
export FC=$MPI_FC
export F77=$MPI_F77
export LIBS="-lnetcdf ${LIBS}"
./configure --prefix=${LIBBASE} --enable-parallel-tests --disable-shared
make
make check
make install
cd ..
rm -rf netcdf-fortran-4.4.4

########################################

PIO

########################################
tar -xvf ${LIBSRC}/ParallelIO.tar
cd ParallelIO
export PIOSRC=pwd
cd ..
mkdir pio
cd pio
export CC=mpicc
export FC=mpifort
cmake -DNetCDF_C_PATH=$NETCDF -DNetCDF_Fortran_PATH=$NETCDF -DPnetCDF_PATH=$PNETCDF -DCMAKE_INSTALL_PREFIX=$LIBBASE -DPIO_ENABLE_TIMING=OFF $PIOSRC
make
make install
cd ..
rm -rf pio ParallelIO
export PIO=$LIBBASE
########################################

Other environment vars needed by MPAS

########################################
export MPAS_EXTERNAL_LIBS="-L${LIBBASE}/lib -lhdf5_hl -lhdf5 -ldl -lz"
export MPAS_EXTERNAL_INCLUDES="-I${LIBBASE}/include"

Version of MPAS (From source, in README.md, core's Registry.xml file, and output file):
MPAS-O Release 2.0

Description of issue:
Segmentation Fault, Address not mapped when running test cases(overflow, baroclinic). The only test case that could run without error is overflow_10km_40layer.
Verisions of I/O libraries are those that are recommended in the user's guide in section 3.2

Platform used (System Specs, as well as name, and operating system if possible):
Marmot Cluster at CMU
Hardware Spec:
128 Nodes, Dual Socket, Single Core AMD Opteron
Opteron 242, 64 bit, 1 MB L2, 1GHz HT CPU
PassMark 1510 (dual cpu) PassMark
Passive Heatsink cooling Alpha U81C Heatsink

    8 GB RAM per core
    SMART 2GB, RDIMM, PC3200, CL3 Memory 

Hard Drives
    1x Western Digital Black SATA 7200rpm 2 TB WD Caviar Black 

Power supply
    Lemacs 500w 100-240v ATX Power Supply 

PDU
    13x 10 socket Ice-Box 3000 

Supermicro H8DCE Motherboard
    nVidia Pro 2200/2050 chipset
    Dual-port Gigabit LAN, 8x SATA, 2xUSB
    AMI BIOS 

Connections
    2x Gigabit ethernet
    Mellanox MHES 18-XTC XSC XS 

Switches
    288 port Voltaire Grid Detector ISR9288 Infiniband switch
    152 port Extreme Networks BlackDiamond 6808 Ethernet switch
    6x 24 port Dell PowerConnect 5224 Ethernet switch 

Operating System:
Standard 64-bit ubuntu 12 image

Compiler used (With version):
gfortran 4.6.3

NetCDF version used: 4.1.3

Parallel NetCDF version used: 1.3.1

Parallel I/O version used: 1.4.1

Number of processors used: 1 - 20

Test case used (if applicable):
overflow_10km_40layer(no error)
overflow_1km_100layer(segmentation fault, address not mapped, there are also warnings about variables such as bottomDepth,etc)
baroclinic_channel_10000m_20levs(segmentation fault, address not mapped)
baroclinic_channel_4000m_20levs(segmentation fault, address not mapped)
baroclinic_channel_1000m_20levs(segmentation fault, address not mapped)

Is this issue reproducible in debug mode (make ... DEBUG=TRUE): Yes

If possible, please provide a backtrace for this issue:
This message is from log.0000.err of baroclinic_channel_10000m test case
[h0:03366] *** Process received signal ***
[h0:03366] Signal: Segmentation fault (11)
[h0:03366] Signal code: Address not mapped (1)
[h0:03366] Failing at address: 0xffffffffb42d3010
[h0:03366] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f08bcc0acb0]
[h0:03366] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x91a8b) [0x7f08bc8cda8b]
[h0:03366] [ 2] /usr/local/lib/openmpi/mca_io_romio.so(ADIOI_NFS_WriteStrided+0xb23) [0x7f08b4572b03]
[h0:03366] [ 3] /usr/local/lib/openmpi/mca_io_romio.so(ADIOI_GEN_WriteStridedColl+0xcea) [0x7f08b457d19a]
[h0:03366] [ 4] /usr/local/lib/openmpi/mca_io_romio.so(MPIOI_File_write_all+0x1d3) [0x7f08b4589b03]
[h0:03366] [ 5] /usr/local/lib/openmpi/mca_io_romio.so(mca_io_romio_dist_MPI_File_write_all+0x29) [0x7f08b4589bc9]
[h0:03366] [ 6] /usr/local/lib/libmpi.so.1(MPI_File_write_all+0xf6) [0x7f08bd8d3056]
[h0:03366] [ 7] ocean_model() [0x669e15]
[h0:03366] [ 8] ocean_model() [0x66a2b4]
[h0:03366] [ 9] ocean_model(ncmpii_wait+0x294) [0x66ad84]
[h0:03366] [10] ocean_model(ncmpi_wait_all+0x40) [0x66b530]
[h0:03366] [11] ocean_model(__piodarray_MOD_darray_write_complete+0x1b8) [0x6106e8]
[h0:03366] [12] ocean_model(__piolib_mod_MOD_closefile+0x118) [0x5adba8]
[h0:03366] [13] ocean_model(__mpas_io_streams_MOD_mpas_closestream+0x46) [0x5902e6]
[h0:03366] [14] ocean_model(__mpas_io_output_MOD_mpas_output_state_finalize+0x15) [0x581ba5]
[h0:03366] [15] ocean_model(__mpas_subdriver_MOD_mpas_finalize+0x19) [0x42eb89]
[h0:03366] [16] ocean_model(main+0x37) [0x42ea77]
[h0:03366] [17] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f08bc85d76d]
[h0:03366] [18] ocean_model() [0x42eaa9]
[h0:03366] *** End of error message **

_Log of baroclinic_channel_1000m_*
[h0:01639] *** Process received signal ***
[h0:01639] Signal: Segmentation fault (11)
[h0:01639] Signal code: Address not mapped (1)
[h0:01639] Failing at address: 0x7f76e64c0010
[h0:01639] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f767bd6fcb0]
[h0:01639] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x91a8b) [0x7f767ba32a8b]
[h0:01639] [ 2] /usr/local/lib/openmpi/mca_io_romio.so(ADIOI_NFS_WriteStrided+0xb23) [0x7f7674e37b03]
[h0:01639] [ 3] /usr/local/lib/openmpi/mca_io_romio.so(ADIOI_GEN_WriteStridedColl+0xcea) [0x7f7674e4219a]
[h0:01639] [ 4] /usr/local/lib/openmpi/mca_io_romio.so(MPIOI_File_write_all+0x1d3) [0x7f7674e4eb03]
[h0:01639] [ 5] /usr/local/lib/openmpi/mca_io_romio.so(mca_io_romio_dist_MPI_File_write_all+0x29) [0x7f7674e4ebc9]
[h0:01639] [ 6] /usr/local/lib/libmpi.so.1(MPI_File_write_all+0xf6) [0x7f767ca38056]
[h0:01639] [ 7] ocean_model() [0x647a05]
[h0:01639] [ 8] ocean_model() [0x647ea4]
[h0:01639] [ 9] ocean_model(ncmpii_wait+0x294) [0x648974]
[h0:01639] [10] ocean_model(ncmpi_wait_all+0x40) [0x649120]
[h0:01639] [11] ocean_model(__piodarray_MOD_darray_write_complete+0x1b8) [0x5ee2d8]
[h0:01639] [12] ocean_model(__piodarray_MOD_add_data_to_buffer_double+0x1f8) [0x5ee7e8]
[h0:01639] [13] ocean_model(__piodarray_MOD_write_darray_nf_double+0xda9) [0x5f2769]
[h0:01639] [14] ocean_model(__piodarray_MOD_write_darray_1d_double+0x110) [0x5fc550]
[h0:01639] [15] ocean_model(__piodarray_MOD_write_darray_2d_double+0x177) [0x5fcb37]
[h0:01639] [16] ocean_model(__mpas_io_MOD_mpas_io_put_var_generic+0x11c6) [0x53d476]
[h0:01639] [17] ocean_model(__mpas_io_MOD_mpas_io_put_var_real2d+0x105) [0x53e3e5]
[h0:01639] [18] ocean_model(__mpas_io_streams_MOD_mpas_writestream+0x6b6a) [0x57508a]
[h0:01639] [19] ocean_model(__mpas_io_output_MOD_mpas_output_state_for_domain+0x1a5c) [0x5611fc]
[h0:01639] [20] ocean_model(__mpas_core_MOD_ocn_write_output_frame+0x390) [0x432580]
[h0:01639] [21] ocean_model(__mpas_core_MOD_mpas_core_run+0x61d) [0x432bfd]
[h0:01639] [22] ocean_model(main+0x30) [0x42ea70]
[h0:01639] [23] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f767b9c276d]
[h0:01639] [24] ocean_model() [0x42eaa9]
[h0:01639] *** End of error message ***

_Log of overflow_1km_
Abort: NaN detected
[h0:10035] *** Process received signal ***
[h0:10035] Signal: Segmentation fault (11)
[h0:10035] Signal code: Address not mapped (1)
[h0:10035] Failing at address: 0x50
[h0:10035] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f686040fcb0]
[h0:10035] [ 1] /usr/local/lib/libmpi.so.1(PMPI_Abort+0x2f) [0x7f68610b72cf]
[h0:10035] [ 2] /usr/local/lib/libmpi_f77.so.1(MPI_ABORT+0x26) [0x7f686140ea06]
[h0:10035] [ 3] ocean_model(__mpas_dmpar_MOD_mpas_dmpar_abort+0x19) [0x537e69]
[h0:10035] [ 4] ocean_model(__ocn_time_integration_MOD_ocn_timestep+0x1ac) [0x44fadc]
[h0:10035] [ 5] ocean_model(__mpas_core_MOD_mpas_core_run+0x38a) [0x43296a]
[h0:10035] [ 6] ocean_model(main+0x30) [0x42ea70]
[h0:10035] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f686006276d]
[h0:10035] [ 8] ocean_model() [0x42eaa9]
[h0:10035] *** End of error message ***

When the source file and the destination file are the same, gird_rotate overwrites the source file. Upon running, cp will fail as seen below; however, immediately afterwards the source file will be opened with nf_open()

$ grid_rotate x1.10242.grid.nc x1.10242.grid.nc
> cp: ‘x1.10242.grid.nc’ and ‘x1.10242.grid.nc’ are the same file

The following code is culprit:

! Copy original file to output file
copyCmd = "cp " // trim(filename) // " " // trim(newFilename)
call system(copyCmd)

! Make sure the output file is writeable
copyCmd = "chmod u+w " // trim(newFilename)
call system(copyCmd)

ierr = nf_open(newFilename, NF_WRITE, ncid)
if(ierr /= 0) then
    write(0,*) "Error: could not open file " // trim(filename) // " for writing"
     return
 end if

@mgduda and I discussed this and realized it might be desirable to be able to overwrite a large file, say if its a very large mesh.

Is this feature desirable? The ability to overwrite a grid file instead of making a copy? How about overwriting the destination file? Would it be nice to have a command line argument to force overwriting a file? e.g.

grid_rotate x1.10242.grid.nc x1.10242.nc --overwrite
or
grid_rotate x1.10242.grid.nc pnw.grid.nc --overwrite

Or would not being able to overwrite the source file be desirable?

Cleanup of naming to match E3SM project name

Pat Worley reported this in E3SM Issue #2467. Though it is not critical, it would be good to do at some point.

Is there any reason why config_hmix_scaleWithMesh should be defaulting to false? Given most of our use cases, I would argue it should be set to true because we are largely running multiscale simulations with the exception of idealized "test cases". We implicitly assumed this and if it were actually true as I believe it should be, our debugging would have been a lot easier.

I’m interested in recommending we default it to true for future folks who will be setting up multiscale simulations.

Thoughts on this proposal?

cc @sbrus89 and @mark-petersen

Hi,

I was trying to run MPAS with 1024 processes using 128 machines.  During init state for domain, I got this error: 

box_rearrange::compute_dest:: INFO io 349 start= 1 1895877 count= 40 5448
box_rearrange::compute_dest:: INFO io 350 start= 1 1901325 count= 40 5448
box_rearrange::compute_dest:: INFO io 351 start= 1 1906773 count= 40 5448
box_rearrange::compute_dest:: INFO io 352 start= 1 1912221 count= 40 5448
box_rearrange.F90.in 752 1005 0 5509783 0 5469764 40 5475212
box_rearrange.F90.in 752 1006 0 5509783 0 5475212 40 5480660
box_rearrange::compute_dest:: ERROR: no destination found for compdof= 220391321
box_rearrange::compute_dest:: INFO: gsize= 40 5578724
box_rearrange::compute_dest:: INFO: nioproc 1024 ioproc 555 ioindex -1

The error output is actually way more but I believe these few lines show the actual error.
I think this is related to the PIO library? I am currently using PIO1.7.2

Thanks,
Lei

As discussed this week, compass error messages are currently suppressed and should instead be output to the terminal.

cc @xylar and @mark-petersen

I'm getting the following linkage errors in the MPAS-A 3.3 build:

undefined reference to `netcdf_mp_nf90_inquire_'
undefined reference to `netcdf_mp_nf90_inquire_attribute_'
 undefined reference to `netcdf_mp_nf90_inq_attname_'
undefined reference to `netcdf_mp_nf90_inq_varid_'
undefined reference to `netcdf_mp_nf90_inquire_variable_'
<etc.>

When I try the compile-command on the command-line, I can get it to link by adding

 -L$NETCDF_F/lib -lnetcdff

I've tried adding these settings to the build environment

 export LDFLAGS="-L$NETCDF_F/lib -lnetcdff -L$NETCDF/lib -lnetcdf"
 export LIBS="-L$NETCDF_F/lib -lnetcdff"
 export FFLAGS="-L$NETCDF_F/lib -lnetcdff"
 export FCFLAGS="-L$NETCDF_F/lib -lnetcdff"

but they never show up on the problematic compile line

mpif90 -O3 -o atmosphere_model driver/*.o -L. -ldycore -lops -lframework -L/shared/apps/rhel-6.2/tools/cponder/PIO-1.9.15/Intel-15.0/lib -lpio -L/shared/apps/rhel-6.2/tools/cponder/PNetCDF-1.6.0/OpenMPI-1.8.4_Intel-15.0_CUDA-6.5/lib -lpnetcdf -L/shared/apps/rhel-6.2/tools/cponder/NetCDF-4.3.3-rc3/Intel-15.0_HDF5-1.8.14_OpenMPI-1.8.4_CUDA-6.5/lib -lnetcdf -I./external/esmf_time_f90 -L./external/esmf_time_f90 -lesmf_time

How do I pass the NetCDF-Fortran info into the build?

As noted in #123, cleaning up forcing logic may be worth considering for restarts (https://github.com/MPAS-Dev/MPAS-Model/pull/123/files#diff-9ab8e70ff214bd7d2173e45491e8ac94R638), e.g.,:

        if (ocn_shortwave_forcing_active(domain) .or. &
                ocn_ecosys_forcing_active(domain) .or. &
                ocn_salinity_restoring_forcing_active(domain) .or. &
                ocn_time_varying_forcing_active(domain)) then

cc @xylar and @sbrus89

Python 2 is going away and we need to start supporting python 3 before we run into trouble.

Hi, I just noticed that the equation calculating the virtual potential temperature has an error in it.

Formula used in current MPAS atmosphere: theta_m = ( 1 + R_v/R_d*qv ) * theta
Correct formula: theta_m = ( 1 + ( R_v / R_d - 1 ) * qv ) * theta

In above equation: theta_m is virtual potential temperature, theta is potential temperature,
R_v is gas constant for vapor, R_d is gas constant for dry air, and qv is relative humidity.

This error spreads everywhere, whenever virtual potential temperature is calculated. There are also hardwired value 1.61 in mpas_init_atm_cases.F. It should be 0.61 instead.

Could anyone take a look? Thanks.

I turn on the TAU by typing the command that is :make gfortran CORE=init_atmosphere TAU=true. And after run the ./init_atmosphere_model, I get the profile.0.0.0. Then I print it by pprof and I get some information. However, the information is all about MPI_XXXX calls. But I want profiling other different function calls of the program besides the MPI_XXXX calls. Is there anything wrong ?
The profile information is below:

On the ocean/develop branch, solution artifacts appear at sub-domain boundaries in certain fields (density, kineticEnergyCell, salinity, temperature, relativeVorticityCell) as shown here

Reproducibility information:

• commit: 575586e
• Modules: 1) python/anaconda-2.7-climate 2) gcc/5.3.0 3) openmpi/1.10.5 4) netcdf/4.4.1 5) parallel-netcdf/1.5.0 6) pio/1.7.2
• make line: make clean gfortran CORE=ocean AUTOCLEAN=true
• run line: srun -n 4 ocean_model -n namelist.ocean -s streams.ocean
• test case: -o ocean -c soma -r 32km -t default
• uname -a: Linux gr-fe1.lanl.gov 3.10.0-862.14.4.1chaos.ch6.x86_64 #1 SMP Wed Sep 26 12:27:08 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux

(I was told to log these PIO issues here, if this is not the right forum can you please direct me to the right one? -- thanks)

I'm building PIO 1.7.1 with the following components

NetCDF 4.3.3-rc2 (the 4.3.2 had problems)
NetCDF-Fortran 4.4.1
PNetCDF 1.6.0
OpenMPI 1.8.4

and running into a parameter mismatch with calls to the nf90_create function.
This happens with the following compilers

Intel 15.0
GCC 4.8.4
PGI 15.1

so I don't think this is an issue specific to any compiler.
I'm getting this compilation error regarding the comm & info arguments to the nf90_create function

ionf_mod.F90(82): error #6627: This is an actual argument keyword name, and not a dummy argument name.   [COMM]
               comm=File%iosystem%io_comm, info=File%iosystem%info)
---------------^
ionf_mod.F90(82): error #6627: This is an actual argument keyword name, and not a dummy argument name.   [INFO]
               comm=File%iosystem%io_comm, info=File%iosystem%info)
-------------------------------------------^

for example; here is one of the code snippets

65 #ifdef _NETCDF
66 #ifdef _NETCDF4
      ......
77 #ifdef _MPISERIAL
78           ierr = nf90_create(fname, nmode , File%fh)
79 #else
80           nmode = ior(nmode,NF90_MPIIO)
81           ierr = nf90_create(fname, nmode, File%fh, &
82                comm=File%iosystem%io_comm, info=File%iosystem%info)
83 #endif

I'm using the flags

FLAGS+=" --enable-pnetcdf"
FLAGS+=" --enable-netcdf4"
FLAGS+=" --enable-netcdf"

The PIO website says to use the first one, and the second two should either be on by default or auto-detected:

http://www.cesm.ucar.edu/models/pio/install.html

What I'm NOT enabling is the --enable-mpiserial and the instructions do not say to do this.
I don't see declarations of nf90_create under either NetCDF-4.3.3-rc3 or PNetCDF-1.6.0; here is what I found under NetCDF-Fortran-4.4.1 in fortran/netcdf4_file.f90:

77 function nf90_create(path, cmode, ncid, initialsize, chunksize, cache_size, &
78      cache_nelems, cache_preemption, comm, info)
79   implicit none
80   character (len = *), intent(in) :: path
81   integer, intent(in) :: cmode
82   integer, intent(out) :: ncid
83   integer, optional, intent(in) :: initialsize
84   integer, optional, intent(inout) :: chunksize
85   integer, optional, intent(in) :: cache_size, cache_nelems
86   integer, optional, intent(in) :: cache_preemption
87   integer, optional, intent(in) :: comm, info

The parameter-list here doesn't line up with what's in the ionf_mod.F90 file.

The SOMA variable mesh workflow needs updating following the JIGSAW-COMPASS workflow implemented in #74.

Mats Bentsen reported this bug to me:

The namelists for the overflow test cases (10 km and 1 km) include the following for hmix:

&hmix
config_hmix_ScaleWithMesh = .false.
config_visc_vorticity_term = .true.
config_apvm_scale_factor = 0.0

The second entry is not a part of the Registry, i.e. our namelists are not compatible with the source code.

The error message is as bellow:

mpif90 -DLANL_FORMULATION -D_MPI -DUNDERSCORE -Dmpas -real-size 64 -O3 -convert big_endian -FR -c mpas_atmphys_lsm_noahinit.F -I/home/zp/local/netcdf/include -I/home/zp/local/pio/include -I/home/zp/local/parallel-netcdf/include -I/home/zp/local/netcdf/include -I/home/zp/local/pio/include -I/home/zp/local/parallel-netcdf/include -I../../framework -I../../operators -I./physics_wrf -I../../external/esmf_time_f90
mpas_atmphys_lsm_noahinit.F(23): error #6581: Unresolved rename. [GRAV]
use mpas_atmphys_constants, grav => g
-----------------------------^
mpas_atmphys_lsm_noahinit.F(146): error #6498: The use-name for this local-name is not defined. [GRAV]
fk = (((hlice/(grav_(-psisat))) * &
-------------------------------^
compilation aborted for mpas_atmphys_lsm_noahinit.F (code 1)
make[4]: *_* [mpas_atmphys_lsm_noahinit.o] Error 1
make[4]: Leaving directory /home/zp/MPAS-Release-2.0/src/core_atmosphere/physics' make[3]: *** [physcore] Error 2 make[3]: Leaving directory/home/zp/MPAS-Release-2.0/src/core_atmosphere'
make[2]: *** [dycore] Error 2
make[2]: Leaving directory /home/zp/MPAS-Release-2.0/src' make[1]: *** [mpas_main] Error 2 make[1]: Leaving directory/home/zp/MPAS-Release-2.0'
make: *** [ifort] Error 2

my ifort compiler version is 14.0.0.
BTW,I can build MPAS-A sucessfully with gfortran(4.4.7),but when i turned to ifort,the error above happened.Does anyone know how to fix it?

In a stream with clobber mode set to overwrite, I see evidence that netdf file global attributes are being written every time output is written. I would think they should only be written once on init, or perhaps not at given it's an existing file.

I discovered this accidentally by doing a run, adding a new namelist option to the code and recompiling, then doing a restart writing to the same file with clobber mode set to overwrite. I get a PIO error in subroutine MPAS_io_put_att_real0d in mpas_io.F here:
https://github.com/MPAS-Dev/MPAS-Model/blob/master/src/framework/mpas_io.F#L4383
where it tries to write the global attribute specifying the value of the new namelist option to the existing output file (which understandably does not work). This error occurs on every timestep when writing to the file occurs, not just once, demonstrating that the stream manager is attempting to write global attributes on every write of the stream.

This would be useful to have to coalesce eddy connected component data in the ocean Okubo-Weiss analysis member.

The cross processor communication is currently implemented with a dmpar max reduce to gather the eddy statistics, but it would be cleaner with an all gather. This is because with a max reduce, the input array has to be as big as the output array, and properly initialized to negative values to get correct values after the max reduce. With an allgather, each processor just has to have a segment of the final array which is then joined together with the gather.

Looking at mpas_atm_time_integration.F at line 2586:

theta_turb_flux = 0.25*(kdiff(k,cell1)+kdiff(k,cell2)+kdiff(k-1,cell1)+kdiff(k,cell2)) &

I think the vertical index in the last term should be kdiff(k-1,cell2)) and not kdiff(k,cell2)). Is it supposed to be an average of four values (two neighboring cells in horizontal and vertical)? Would you please take a look at this.

Thank you,
Matus

Matus Martini
Postdoctoral Research Associate
Atmospheric Sciences and Global Change Division
Pacific Northwest National Laboratory
Tel: 509-372-4082

Aloha. I'm attempting to build MPAS dependencies using Intel compilers and have gotten through HDF5, netCDF4, netCDF4-Fortran, parallel-netcdf and am now attempting to build ParallelIO-pio2_2_1 using the following script.

CC=mpicc FC=mpif90 cmake -DPnetCDF_PATH=/home/hellyj/src/parallel-netcdf-1.8.1/Build
-DNetCDF_C_PATH=/home/hellyj/src/netcdf-4.4-parallel
-DNetCDF_Fortran_PATH=/home/hellyj/src/netcdf-4.4-parallel
/home/hellyj/src/ParallelIO-pio2_2_1

This is the result.

./cmake-command.bash
-- MPI Fortran module verified and enabled.
-- Using internal GPTL C library for timing
-- MPIIO failed verification and therefore disabled.
-- MPI Fortran module verified and enabled.
-- Using internal GPTL Fortran library for timing
-- Could NOT find MPE_C (missing: MPE_C_LIBRARY MPE_C_INCLUDE_DIR)
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hellyj/src/ParallelIO-pio2_2_1

It complains about MPIIO but I don't know what that implies or how to correct it. When I try to compile with

make

I get the following:

[hellyj@comet-ln3 ParallelIO-pio2_2_1]$ make
[ 3%] Building Fortran object src/gptl/CMakeFiles/gptl.dir/perf_utils.F90.o
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(15): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [MPI]
use mpi

------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(135): error #6404: This name does not have a type, and must have an explicit type. [MPI_COMM_WORLD]
call mpi_abort(MPI_COMM_WORLD,0,ierr)
------------------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(190): error #6404: This name does not have a type, and must have an explicit type. [MPI_SUCCESS]
if (rcode /= MPI_SUCCESS) then
----------------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(182): error #6279: A specification expression object must be a dummy argument, a COMMON block object, or an object accessible through host or use association. [MPI_MAX_ERROR_STRING]
character(MPI_MAX_ERROR_STRING) :: lstring
-------------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(182): error #6404: This name does not have a type, and must have an explicit type. [MPI_MAX_ERROR_STRING]
character(MPI_MAX_ERROR_STRING) :: lstring
-------------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(275): error #6404: This name does not have a type, and must have an explicit type. [MPI_INTEGER]
call MPI_BCAST(vec,lsize,MPI_INTEGER,0,comm,ierr)
----------------------------^
/home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90(307): error #6404: This name does not have a type, and must have an explicit type. [MPI_LOGICAL]
call MPI_BCAST(vec,lsize,MPI_LOGICAL,0,comm,ierr)
----------------------------^
compilation aborted for /home/hellyj/src/ParallelIO-pio2_2_1/src/gptl/perf_utils.F90 (code 1)
make[3]: *** [src/gptl/CMakeFiles/gptl.dir/perf_utils.F90.o] Error 1
make[2]: *** [src/gptl/CMakeFiles/gptl.dir/perf_utils.F90.o.provides] Error 2
make[1]: *** [src/gptl/CMakeFiles/gptl.dir/all] Error 2
make: *** [all] Error 2

So it looks like MPIIO might be implicated although I can't tell. Any suggestions would be helpful.

Mahalo.
J.

In Registry_okubo_weiss.xml, I had to create a scratch variable in a separately named group that I called "okuboWeissScratch."

The two places that I tried it, that did not work:

• in the existing "scratch" group -- this failed at run time in mpas_grid_types.F when it tried to allocate the scratch variable. This was a hard error/crash for MPAS.
• in the "amOkuboWeiss" package -- this failed at compile time as the compilation told me that I wasn't allowed (I don't remember the exact message)

I don't know if either one of these are bugs or features.

When compiling with Intel debug, warnings are printed every time global stats is called:

ocean_model        0000000002E1CBF7  ocn_global_stats_         903  mpas_ocn_global_stats.F
...
forrtl: warning (406): fort: (1): In call to OCN_COMPUTE_FIELD_AREA_WEIGHTED_LOCAL_STATS_SURFACE, an array temporary was created for argument #4

This occurs when you pass in an implicit array like this: activeTracersSurfaceFlux(2,1:nCellsSolve) into a subroutine, because Fortran makes a temporary array. The solution is to make a temporary array ourselves, and pass in a full array.

Error occurs in MPAS V6.

Question:

It seems like an odd choice to use 6371229.0 m as the radius of the earth in MPAS. That number seems to be based on Hayford's 1910 estimate of the Earth. It's average radius is 6371229.31 m, by my calculation.

Here's a few sphere alternatives, all based on the more recent WGS84 ellipsoid:

average of the 3 radii: 6371008.7714 m
radius of sphere of equal area: 6371007.1809 m
radius of sphere of equal volume: 6371000.7900 m

These were taken from page 3-7 of the WGS84 definition at:
http://earth-info.nga.mil/GandG/publications/tr8350.2/wgs84fin.pdf

I might be missing something, of course, but thought I would speak up sooner rather than later.Either way, I am curious about the choice.