From @amdasilva

... there were several aspects that were left kind of unfinished. For example, hardwiring wavelengths:"

!   Get wavelength index for Mie Table
!   Channel values are 4.7e-7 5.5e-7 6.7e-7 8.7e-7 [meter]. Their indices are 1,2,3,4 respectively.
    if ((wavelength .ge. 4.69e-7) .and. (wavelength .le. 4.71e-7)) then
       mieTable_index = 1.
    else if ((wavelength .ge. 5.49e-7) .and. (wavelength .le. 5.51e-7)) then
       mieTable_index = 2.
    else if ((wavelength .ge. 6.69e-7) .and. (wavelength .le. 6.71e-7)) then
       mieTable_index = 3.
    else if ((wavelength .ge. 8.68e-7) .and. (wavelength .le. 8.71e-7)) then
       mieTable_index = 4.
    else
       print*,trim(Iam),' : wavelength of ',wavelength,' is an invalid value.'
       return
    end if

Further guidance is needed from @amdasilva for his preferred approach for this. Presumably the bounds would be derived from frequencies obtained in a resource file and a loop used to compute the table index. (And also to compute the size of the array in that dimension.)

Resolution-dependent scaling factors used to modulate sea salt emissions are selected based on standard NASA cubic grid resolutions (C48, C90, C180, C360, C720). These resolutions are hardcoded in function Chem_UtilResVal(), included in the process library.

Since GOCART may now be driven using external grids (as in the case of UFS), the function above could be extended to include other global grids (e.g. NOAA C48, C96, C192, C384, C768, C1152, C3072) or refactored to use grid point distance rather than the number of grid point per tile to select the proper scaling coefficient.

@lizziel of Harvard recently made a PR on MAPL (GEOS-ESM/MAPL#1377) that was checking for file existence in ExtData. It found a bug in the Dust ExtData RC file a la this line:

As you can see, this:

ExtData/chemistry/MERRAero/v0.0.0/sfcdR_MERRA-AA-r2.aer_Nx.2003_2014.2008clm.nc4

should be:

ExtData/chemistry/MERRAero/v0.0.0/sfc/dR_MERRA-AA-r2.aer_Nx.2003_2014.2008clm.nc4

with a slash after sfc.

I'll make a hotfix PR soon.

This is mainly a place for discussion, etc. from @rmontuoro and @gmao-esherman. Namely, I recently added the Changelog enforcer on GOCART since, well, if you have a Changelog, you should use it!

But, when I recently "created" a good-ish Changelog in main, I didn't really fill out much in the Unreleased section as I don't know GOCART well enough. But, comparing develop to main shows there are a lot of updates not yet in main:

main...develop

I don't think this needs to be too detailed, but at least cover the major differences/fixes/etc. that distinguish GOCART develop from GOCART v1.0.1

So, if @rmontuoro and @gmao-esherman can maybe list things they think should be added from past commits here, I can work on making a Changelog update to develop so that we only have a single PR we need to get it up-to-date.

Thanks!

Input OH is provided as #/cm3. However, SulfateUpdateOxidants() assumes input OH to be in mol/mol and converts mixing rations again to #/cm3:

Input OH units should be checked and converted only if needed.

Wavelength-dependent Aerosol Optical Depths (AODs) are computed incorrectly due to aerosol_monochromatic_optics_wavelength being read as a list of integers as opposed to reals by the GOCART2G gridded component.

This array of wavelengths (channels) are provided in units of m and therefore read as 0s:

aerosol_monochromatic_optics_wavelength: 4.7e-7 5.5e-7 6.7e-9 8.7e-7

These wavelengths are further multiplied by 1.e-09 internally before computing AODs using the diagnostic Mie lookup table.

I suggest entering the aerosol_monochromatic_optics_wavelength in units of nm for consistency with other settings and updating the internal code appropriately. The configuration label may also be updated as aerosol_monochromatic_optics_wavelength_in_nm to reflect this change and to improve consistency. For example:

wavelengths_for_profile_aop_in_nm: 470 550 670 870
wavelengths_for_vertically_integrated_aop_in_nm: 470 550 670 870
aerosol_monochromatic_optics_wavelength_in_nm: 470 550 670 870

There might be a bug in GOCART2G, though maybe it's just "bad variable name" instead of bug. The code is a bit confusing to me at times. 😄

To wit, I was looking at trying to improve the performance of GOCART2G (as @wmputman has said he sees it slower). I was trying to convert some MieQuery calls from "in a 4-d loop" to "in a band loop" when I saw this (showing DU, but it's in all):

The issue is that a keyword is not being used for bext. If we now look at the signature of Chem_MieQuery → Chem_MieQueryInt:

   impure elemental subroutine Chem_MieQueryByInt ( this, idx, channel, q_mass, rh,     &
                                   tau, ssa, gasym, bext, bsca, bbck,  &
                                   reff, p11, p22, gf, rhop, rhod, &
                                   vol, area, refr, refi, rc )


! !INPUT PARAMETERS:


   type(Chem_Mie), target, intent(in ) :: this
   integer,                intent(in ) :: idx     ! variable index on Chem_Mie
   real,                   intent(in ) :: channel ! channel number 
   real,                   intent(in ) :: q_mass  ! aerosol mass [kg/m2],
   real,                   intent(in ) :: rh      ! relative himidity


! !OUTPUT PARAMETERS:


   real,    optional,      intent(out) :: tau   ! aerol extinction optical depth
   real,    optional,      intent(out) :: ssa   ! single scattering albedo
   real,    optional,      intent(out) :: gasym ! asymmetry parameter
   real,    optional,      intent(out) :: bext  ! mass extinction efficiency [m2 (kg dry mass)-1]
   real,    optional,      intent(out) :: bsca  ! mass scattering efficiency [m2 (kg dry mass)-1]
   real,    optional,      intent(out) :: bbck  ! mass backscatter efficiency [m2 (kg dry mass)-1]
...

we can see that the first optional argument is tau, not bext. I believe per Fortran this means we are actually doing:

        call Chem_MieQuery(mie_table, l, real(offset+1.), q(:,:,:,l), rh, tau=bext, gasym=gasym, ssa=bssa)

We are filling bext with tau not bext. @tclune can confirm if I'm correct.

Now bext does seem related to tau:

      if(present(tau  )) tau   = bextIn * q_mass
      if(present(bext )) bext  = bextIn

by q_mass.

So my question for @amdasilva or @pcolarco is: do you want tau here or bext? If tau, then it's not a bug, just a bad name and all is well. But if this is supposed to be bext it might be incorrect code.

follow convention

GOCART2G process library procedures must return an error status to allow the calling application to gracefully handle errors.

They also should not include stop statements, which may cause the application to hang.

GOCART's develop branch was capable of being built standalone prior to the reintroduction of GOCART legacy. This is a nice feature for CI testing. Due to this issue the CI testing was limited to just testing if the GOCART2G target builds, which ignores all of the UFS related code. Hence, GOCART-legacy should be updated, so that GOCART's develop branch can start building standalone.

I think the cause of the issue is related to a comment made by @rmontuoro noting that GOCART legacy's cmake should express an explicit dependency on MAPL. There are also some dependencies on Shared in GEOSchem_GridComp that need to be brought in as well. These dependencies were originally part of the GOCART repository, but where removed by earlier commits. I suggest bringing them back in until GOCART legacy is removed.

This is a question for @amdasilva just from my own thoughts.

Currently in, say, CA2G we have:

CA2G_GridComp/
├── AMIP
│   ├── CA2G_GridComp_ExtData.rc
│   ├── CA2G_instance_CA.bc.rc -> ../CA2G_instance_CA.bc.rc
│   ├── CA2G_instance_CA.br.rc -> ../CA2G_instance_CA.br.rc
│   └── CA2G_instance_CA.oc.rc -> ../CA2G_instance_CA.oc.rc
├── AMIP.20C
│   ├── CA2G_GridComp_ExtData.rc
│   ├── CA2G_instance_CA.bc.rc -> ../CA2G_instance_CA.bc.rc
│   ├── CA2G_instance_CA.br.rc -> ../CA2G_instance_CA.br.rc
│   └── CA2G_instance_CA.oc.rc -> ../CA2G_instance_CA.oc.rc
├── CA2G_GridCompMod.F90
├── CA2G_GridComp_ExtData.rc
├── CA2G_StateSpecs.rc
├── CA2G_instance_CA.bc.rc
├── CA2G_instance_CA.br.rc
├── CA2G_instance_CA.oc.rc
└── CMakeLists.txt

where the OPS emissions files are the ones in the main directory. What I'm wondering is might there be a benefit to having instead:

CA2G_GridComp
├── AMIP
│   ├── CA2G_GridComp_ExtData.rc
│   ├── CA2G_instance_CA.bc.rc -> ../OPS/CA2G_instance_CA.bc.rc
│   ├── CA2G_instance_CA.br.rc -> ../OPS/CA2G_instance_CA.br.rc
│   └── CA2G_instance_CA.oc.rc -> ../OPS/CA2G_instance_CA.oc.rc
├── AMIP.20C
│   ├── CA2G_GridComp_ExtData.rc
│   ├── CA2G_instance_CA.bc.rc -> ../OPS/CA2G_instance_CA.bc.rc
│   ├── CA2G_instance_CA.br.rc -> ../OPS/CA2G_instance_CA.br.rc
│   └── CA2G_instance_CA.oc.rc -> ../OPS/CA2G_instance_CA.oc.rc
├── CA2G_GridCompMod.F90
├── CA2G_StateSpecs.rc
├── CMakeLists.txt
└── OPS
    ├── CA2G_GridComp_ExtData.rc
    ├── CA2G_instance_CA.bc.rc
    ├── CA2G_instance_CA.br.rc
    └── CA2G_instance_CA.oc.rc

We still use symlinks to prevent code duplication, but now it's a bit more obvious that some files belong to the OPS emissions without knowledge that the "main directory" ones are those.

I only say this because then in the $EXPDIR/RC directory, a user would have an OPS, AMIP, and AMIP.20C directory where you'd know if you want to affect an OPS emissions run, you change the files in OPS. At the moment, you have to know that $EXPDIR/RC/CA2G_GridComp_ExtData.rc is the one for OPS and $EXPDIR/RC/AMIP/CA2G_GridComp_ExtData.rc is the one for AMIP.

Obviously, we'd need to make scripting changes, etc (as well as concurrent changes in Chem), but it might make things a bit clearer?

Not sure. Thought I'd ask.

As noticed in testing by @sdrabenh and myself, GOCART2G does not seem to currently layout regress. (It does start-stop regress, so that means we are not looking at a missing field in an internal state.)

When run at C48 with AMIP emissions, I see failures in both NI and SU restarts when I compare 4x24 to 4x30:

Comparing NC4 ni_internal_checkpoint using nccmp...
Failure!
Checking for data differences
Variable Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
NH3      /     11953  -2.8532e-30 2.28618e-29 -1.10441e-29 7.88861e-30 1.89327e-29 -2.38701e-34 1.25437e-31
NH4a     /      4754  4.16565e-30 8.98964e-29 -2.52435e-29 1.89327e-29 4.41762e-29  8.76241e-34 5.61762e-31
NO3an1   /      1847 -1.88273e-31  2.1484e-31 -1.97215e-31 6.16298e-33 2.03378e-31 -1.01934e-34 4.59288e-33

Comparing NC4 su_internal_checkpoint using nccmp...
Failure!
Checking for data differences
Variable Group Count          Sum      AbsSum          Min        Max       Range         Mean      StdDev
MSA      /     31294 -3.56464e-18 5.73955e-18 -1.73472e-18 2.1684e-19 1.95156e-18 -1.13908e-22 1.28398e-20

So not that big. But if we compare 4x24 to 2x48 every single restart is different:

$ cmpscratch scratch.4x24/ scratch.2x48/
Comparing NC4 achem_internal_checkpoint using nccmp...
Failure!

Comparing NC4 aiau_import_checkpoint using nccmp...
Failure!

Comparing NC4 cabc_internal_checkpoint using nccmp...
Failure!

Comparing NC4 cabr_internal_checkpoint using nccmp...
Failure!

Comparing NC4 caoc_internal_checkpoint using nccmp...
Failure!

Comparing NC4 catch_internal_checkpoint using nccmp...
Failure!

Comparing NC4 du_internal_checkpoint using nccmp...
Failure!

Comparing NC4 fvcore_internal_checkpoint using nccmp...
Failure!
...

Not good.

In the past, this sort of thing usually means there is a global sum, but I took a look and can't find any global reductions.

I talked with @bena-nasa and he had a possible thought: areas. AreaMean calls had caused issues in the past, but none of those. But, In DistributePointEmission in GOCART2G_Process.F90 there is a calculation that involves the area:

@bena-nasa had some thoughts that maybe this area might not be guaranteed to always "conserve" under layout changes. (I'm not wording/explaining this correctly I'm sure; he or @tclune can explain better than I.)

So a question for @pcolarco, @adarmenov, @amdasilva, etc. who knows this code: Is there a way to disable or bypass the point emissions code? So we could do some tests to see if this code is bypassed, if layout regression is restored? (Comment out calls? Turn things off in ExtData?, etc...)

From @amdasilva

The Mie interface is extremely inefficient and has slowed down radiation a great deal as noted by Bill. Making subroutine calls inside a deep do-loop is a big part of it.

   do n = 1, nbins
      do i = 1, i2
        do j = 1, j2
          do k = 1, km
            call Chem_MieQuery(self%diag_MieTable(instance), n, mieTable_index, q_4d(i,j,k,n), rh(i,j,k), tau(i,j,k), __RC__)
            tau_s(i,j,k) = tau_s(i,j,k) + tau(i,j,k)
          end do
        end do
      end do
    end do

@tclune thinks that the array interface has already been overloaded for this.

During the development of GOCART2G, we updated the rest of the GCM to use DSOs and thereby eliminated compile time dependencies between components. GOCART2G has not been updated in that manner.

Should this be done before v2.0.0 or after?

HEMCO_GridComp won't build under gfortran listing errors such as:

/Users/wjamieso/GEOS/GCM_HEMCO_repo/GEOSgcm/src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@HEMCO_GridComp/src/Interfaces/GEOS/HEMCO_GridCompMod.F90:1416:87:

 1416 |           CALL GetSUNCOS( Clock, HcoState, TMP, N, __RC__ )
      |                                                                                       1
Error: Unterminated character constant beginning at (1)

I think there is something slightly off with the CMake in GOCART2G, probably involving the automatic code generator.

First, build and install GEOSgcm. Now, doing nothing to the source code, go to your build directory and re-run make -jN install. When you do, you'll see what I think is the ACG re-running:

...
[  8%] Generating automatic code for CA2G_StateSpecs.rc
[  8%] Built target NCEP_bacio_r8i8
Consolidate compiler generated dependencies of target NCEP_bacio_r8i4
[  8%] Built target Process_Library
[  8%] Generating automatic code for NI2G_StateSpecs.rc
[  8%] Built target acg_phony_CA2G_GridComp
[  8%] Generating automatic code for SU2G_StateSpecs.rc
...

which I'm guessing generates code with a newer timestamp and so CMake has no choice but to re-compile:

...
[ 80%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp/CMakeFiles/DU2G_GridComp.dir/DU2G_GridCompMod.F90.o
[ 80%] Built target ec_prs2eta.x
[ 80%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SS2G_GridComp/CMakeFiles/SS2G_GridComp.dir/SS2G_GridCompMod.F90.o
[ 80%] Built target convert_eta.x
...

which then triggers a library re-link and a rebuild of all of GOCART2G's dependents:

...
[100%] Linking Fortran static library ../../../../../../../../../../lib/libNI2G_GridComp.a
[100%] Built target NI2G_GridComp
[100%] Linking Fortran static library ../../../../../../../../../../lib/libCA2G_GridComp.a
[100%] Built target CA2G_GridComp
Scanning dependencies of target GOCART2G_GridComp
[100%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/CMakeFiles/GOCART2G_GridComp.dir/GOCART2G_GridCompMod.F90.o
[100%] Linking Fortran static library ../../../../../../../../../lib/libGOCART2G_GridComp.a
[100%] Built target GOCART2G_GridComp
Scanning dependencies of target Aerosol_GridComp
[100%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/CMakeFiles/GEOSchem_GridComp.dir/GEOS_ChemGridComp.F90.o
[100%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@GOCART/ESMF/Aerosol_GridComp/CMakeFiles/Aerosol_GridComp.dir/Aerosol_GridComp.F90.o
[100%] Linking Fortran static library ../../../../../../../../../lib/libAerosol_GridComp.a
[100%] Built target Aerosol_GridComp
[100%] Linking Fortran static library ../../../../../../lib/libGEOSchem_GridComp.a
[100%] Built target GEOSchem_GridComp
[100%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/CMakeFiles/GEOSphysics_GridComp.dir/GEOS_PhysicsGridComp.F90.o
[100%] Linking Fortran static library ../../../../../lib/libGEOSphysics_GridComp.a
[100%] Built target GEOSphysics_GridComp
[100%] Building Fortran object src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/CMakeFiles/GEOSagcm_GridComp.dir/GEOS_AgcmGridComp.F90.o
...

Obviously, if no code has changed, CMake shouldn't need to rebuild.

As discovered by @wmputman, if you change the klid (aka pressure_lid_in_hPa) in GOCART, the model locks up.

It works if pressure_lid_in_hPa: 0.01 but pressure_lid_in_hPa: 0.40 causes it to not work.

I have a hotfix for this incoming.

If you run gcm_setup and ask for AMIP GOCART and Climatological aerosols, the model will crash:

 Integer*4 Resource Parameter: USE_SATSIM_MISR:0
 Parsing of satsim.rc
 Real*4 Resource Parameter: SYNCTQ:1.000000
 ERROR: cannot find export AERO
pe=00094 FAIL at line=07054    MAPL_Generic.F90                         <unknown error>
pe=00094 FAIL at line=00638    MAPL_Generic.F90                         <status=-1>
pe=00094 FAIL at line=00556    MAPL_Generic.F90                         <status=-1>
pe=00094 FAIL at line=04553    MAPL_Generic.F90                         <status=-1>
pe=00094 FAIL at line=04744    MAPL_Generic.F90                         <status=-1>
PHYSICS::SetServices                          1494
AGCMSetServices                                690
GCMSetServices                                 232

I might be able to help @sdrabenh script for this if I knew how to actually run a climatological GOCART2G run. I think it's now all data_driven or something inside here, but I have no idea how to set up for it.

@amdasilva @pcolarco @adarmenov @weiyuan-jiang @tclune @mathomp4

There is this issue reported in UFS that appears to involve gocart2g:
ufs-community/ufs-weather-model#1190

now Matt noticed that a 4 hour vs 2 + 2 start/stop regression was failing. We then remembered there is a hard coded 3 hourly alarm in gocart2g! In these spots:
https://github.com/GEOS-ESM/GOCART/blob/v2.0.6/ESMF/GOCART2G_GridComp/NI2G_GridComp/NI2G_GridCompMod.F90#L393
https://github.com/GEOS-ESM/GOCART/blob/v2.0.6/ESMF/GOCART2G_GridComp/SU2G_GridComp/SU2G_GridCompMod.F90#L480

If I change this to 2 hours, the 4 hour vs 2 + 2 regression passes. So it does seem possible this could be the problem for UFS. At the very least it seems like either whatever the point of this alarm is it either needs to be divisible by the job segment or perhaps we are not saving something in the internal state that needs to be checkpointed.

From @amdasilva

The other issue is the variable names in the multiple-instances of a component, CA, in particular.

CAMASS*    | kg kg-1    | xyz | C   |       | Carbonaceous Aerosol Mass Mixing Ratio                
CACONC*    | kg m-3     | xyz | C   |       | Carbonaceous Aerosol Mass Concentration               
CAEXTCOEF* | m-1        | xyz | C   | size(self%wavelengths_profile) | Carbonaceous Aerosol Extinction Coefficient 
CASCACOEF* | m-1        | xyz | C   | size(self%wavelengths_profile) | Carbonaceous Aerosol Scattering Coefficient

In the case that the instance is called CA.oc, the first variable becomes CAMASSOC, while it should have been OCMASS for backward compatibility. Then long name is now the same for OC, BC and BR. Should not have been done this way, but instead

*MASS    | kg kg-1    | xyz | C   |       | * Carbonaceous Aerosol Mass Mixing Ratio                
*CONC    | kg m-3     | xyz | C   |       | * Carbonaceous Aerosol Mass Concentration               
*EXTCOEF | m-1        | xyz | C   | size(self%wavelengths_profile) | * Carbonaceous Aerosol Extinction Coefficient 
*SCACOEF | m-1        | xyz | C   | size(self%wavelengths_profile) | * Carbonaceous Aerosol Scattering Coefficient

Where * gets replaced by OC, BC, BR. Also he has code like this that is not used for anything.

    if (comp_name(1:5) == 'CA.oc') then
       GCsuffix = 'OC'
    else if (comp_name(1:5) == 'CA.bc') then
       GCsuffix = 'BC'
    else if (comp_name(1:5) == 'CA.br') then
       GCsuffix = 'BR'
    end if

As it turns out, the variables are unnecessarily different. Also, no provisions for multiple instances in the other components.

@tclune will make any needed corrections to the MAPL ACG for this. This ticket is solely about correcting the usage in GOCART gridcomps when those fixes are available.

The Problem

Unlike legacy GOCART, GOCART2G allows export diagnostics to be provided on multiple wavelengths. This is accomplished by these entries in GOCAR2G_GridComp.rc:

`# Set optics parameters

---------------------

aerosol_monochromatic_optics_wavelength_in_nm_from_LUT: 470 550 670 870
wavelengths_for_profile_aop_in_nm: 532 550 # must be included in LUT
wavelengths_for_vertically_integrated_aop_in_nm: 550 # must be included in LUT
`

These resources are parsed, stored in Config and passed down to its children (DU, SS, etc.) The problem is that in each children, the 4 wavelengths above are hardwired in this piece of code:

! Get wavelength index for Mie Table ! Channel values are 4.7e-7 5.5e-7 6.7e-7 8.7e-7 [meter]. Their indices are 1,2,3,4 respectively. if ((wavelength .ge. 4.69e-7) .and. (wavelength .le. 4.71e-7)) then mieTable_index = 1. else if ((wavelength .ge. 5.49e-7) .and. (wavelength .le. 5.51e-7)) then mieTable_index = 2. else if ((wavelength .ge. 6.69e-7) .and. (wavelength .le. 6.71e-7)) then mieTable_index = 3. else if ((wavelength .ge. 8.68e-7) .and. (wavelength .le. 8.71e-7)) then mieTable_index = 4. else print*,trim(Iam),' : wavelength of ',wavelength,' is an invalid value.' return end if

Namely, the code unecessarily hardwires 4 wavelengths: 470 550 670 870 and if the user species something different in

aerosol_monochromatic_optics_wavelength_in_nm_from_LUT: 470 550 670 870

it will produce wrong results.

The Fix

Retrieve the wavelengths specified in the Config inherited from its parent, and load the corresponding Mie tables, eliminating the code fragment above that appears in each children of GOCART2G_GridComp.

Loops could be parallelized using OpenMP to potentially increase GOCART runtime performance. This is crucial for operational applications.

OpenMP directive could be guarded in order to be more finely controlled and to prevent issues with existing applications, if necessary.

@lltakacs and @sdrabenh confirmed 0-increment replays fail to be 0-diff if REPLAY_FILE_FREQUENCY is not the default 21600. This is important since OPS has moved to a 2-hourly 4D IAU. Tests using gcm v10.22.0 and after have the bug while v10.21.1 does not. The implication is that something is amiss in how GOCART-2G interacts with the IAU machinery.

In our C48 out-of-the-box case, a 6-hour AMIP was run as a control. Then, a 0-increment replay was run using the following:

    REPLAY_ANA_EXPID:    x0046a
    REPLAY_ANA_LOCATION: /discover/nobackup/projects/gmao/dadev/dao_it/archive/x0046a
    REPLAY_MODE:         Regular
    REPLAY_FILE:         ana/Y%y4/M%m2/x0046a.ana.eta.%y4%m2%d2_%h200z.nc4
    REPLAY_FILE_FREQUENCY:      21600
    REPLAY_FILE_REFERENCE_TIME: 000000
    REPLAY_P:  NO
    REPLAY_U:  NO
    REPLAY_V:  NO
    REPLAY_T:  NO
    REPLAY_QV: NO
    REPLAY_O3: NO
    REPLAY_TS: NO

The results from using the above configuration have 0-diff restarts compared to the AMIP. This is expected. However, if the following is changed ...

REPLAY_FILE_FREQUENCY: 7200

... 3 restarts become non-0-diff after the first time step. Specifically, there are the following differences:

cdo diffn scratch.amip/achem_internal_checkpoint scratch.0inc-x46a/achem_internal_checkpoint
               Date     Time   Level Gridsize    Miss    Diff : S Z  Max_Absdiff Max_Reldiff : Parameter name
    72 : 2015-05-09 21:07:30      72    13824       0    6484 : F T   3.2613e-16  3.6506e-05 : VOC        
   144 : 2015-05-09 21:07:30      72    13824       0     477 : F T   4.5103e-17  1.6128e-05 : VOCbiob    
  2 of 144 records differ
  0 of 144 records differ more than 0.001
cdo    diffn: Processed 3981312 values from 4 variables over 2 timesteps [0.12s 18MB].

cdo diffn scratch.amip/cabr_internal_checkpoint scratch.0inc-x46a/cabr_internal_checkpoint
               Date     Time   Level Gridsize    Miss    Diff : S Z  Max_Absdiff Max_Reldiff : Parameter name
    72 : 2015-05-09 21:07:30      72    13824       0     418 : F T   2.2204e-16  1.2083e-05 : CAphilicCA.br
  1 of 144 records differ
  0 of 144 records differ more than 0.001
cdo    diffn: Processed 3981312 values from 4 variables over 2 timesteps [0.10s 18MB].

cdo diffn scratch.amip/caoc_internal_checkpoint scratch.0inc-x46a/caoc_internal_checkpoint
               Date     Time   Level Gridsize    Miss    Diff : S Z  Max_Absdiff Max_Reldiff : Parameter name
    72 : 2015-05-09 21:07:30      72    13824       0    6240 : F T   1.7850e-15  0.00031656 : CAphilicCA.oc
  1 of 144 records differ
  0 of 144 records differ more than 0.001
cdo    diffn: Processed 3981312 values from 4 variables over 2 timesteps [0.12s 18MB].

At the end of a 6-hour window, all restarts become non-0-diff. This should not be the case.

I would like to bring this line to your attention. There is bext as an output option. If tau is wanted indeed, we should add a note before the call.. @tclune

The ufs-weather-model coupled test with GOCART failed with gnu compiler gnu/10.1.0 (on Cheyenne) and gnu/9.2.0 (on Hera). The compile jobs actually ran successfully. The test failed at run time with the same error message:

Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
...
117: #1 0x12f5195 in modeldatainitialize
117: at /scratch1/NCEPDEV/stmp4/Minsuk.Ji/ufs-aerosol-gnu-issue/GOCART/ESMF/UFS/Aerosol_Cap.F90:321

An issue was created on ufs-weather-model at ufs-community/ufs-weather-model#1115.

May I ask if there is any suggestion on fixing the issue. Thanks

There are many optional return codes (rc) in the Process_Library that are directly assigned to without checking if the rc is present. If one calls a routine without specifying the optional rc this will cause a runtime error (Segmentation fault).

It would crash here

wavelengths_diagmie is still declared as integer:

I would still recommend including changes from PR #65 into this branch to address this issue and streamline the code. For example, conversion from nm to m should be done only once (see https://github.com/GEOS-ESM/GOCART/pull/65/files#diff-682724d291af0ad213c4e1b2ffa97156de4989543290b60bf362adfa98e3d8da):

Originally posted by @rmontuoro in #68 (comment)

Background

@bena-nasa has refactored ExtData to use a more versatile YAML resource file. This is a mod that goes beyond GOCART (see below for a list of *ExtFiles.rc that need conversion).

Here are the general steps:

• branch: feature/#132/ExtData/yaml
• Start with latest GEOSgcm release 10.21.1
• Update to the latest version of MAPL supporting this feature
• Use Ben's utility to convert current *ExtData.rc to *ExtData.yaml
• Change default in CAP to use the new YAML-style ExtData
• Make sure GEOSgcm runs
• Issue tag: feature/#132/ExtData/yaml/v1.0.0
• Ravi will perform re-forecasts experiments because this is non-zero diff
• Only after Ravi demonstrates it has no impact on reforecasts we will issue a PR.

List of ExtData.rc needing conversion to YAML

./@GEOSchem_GridComp/MAMchem_GridComp/MAM7_ExtData.rc
./@GEOSchem_GridComp/TR_GridComp/TR_ExtData.rc
./@GEOSchem_GridComp/GMIchem_GridComp/GMI_GridComp/CCMI_REF-C2/GMI_ExtData.rc
./@GEOSchem_GridComp/GMIchem_GridComp/GMI_GridComp/CCMI_REF-C1/GMI_ExtData.rc
./@GEOSchem_GridComp/GMIchem_GridComp/GMI_GridComp/GMI_ExtData.rc
./@GEOSchem_GridComp/ChemEnv_ExtData.rc
./@GEOSchem_GridComp/HEMCO_GridComp/HEMCOgocart2g_ExtData.rc
./@GEOSchem_GridComp/HEMCO_GridComp/HEMCOgmi_ExtData.rc
./@GEOSchem_GridComp/GAAS_GridComp/GAAS_GridComp_ExtData.rc
./@GEOSchem_GridComp/Shared/HEMCO/@HEMCO/src/Interfaces/GEOS/HEMCOgocart_ExtData.rc
./@GEOSchem_GridComp/Shared/HEMCO/@HEMCO/src/Interfaces/GEOS/HEMCOgmi_ExtData.rc
./@GEOSchem_GridComp/CARMAchem_GridComp/CARMAchem_GridComp_ExtData.rc
./@GEOSchem_GridComp/DNA_GridComp/DNA_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp/AMIP.20C/CA2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp/AMIP/CA2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp/CA2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SU2G_GridComp/AMIP.20C/SU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SU2G_GridComp/SU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SU2G_GridComp/AMIP/SU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SS2G_GridComp/AMIP.20C/SS2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SS2G_GridComp/SS2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/SS2G_GridComp/AMIP/SS2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp/AMIP.20C/DU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp/AMIP/DU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp/DU2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/NI2G_GridComp/AMIP.20C/NI2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/NI2G_GridComp/AMIP/NI2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART2G_GridComp/NI2G_GridComp/NI2G_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO_GridComp/CO_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO_GridComp/AMIP.20C/CO_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO_GridComp/AMIP/CO_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/Rn_GridComp/Rn_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/Rn_GridComp/AMIP.20C/Rn_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/Rn_GridComp/AMIP/Rn_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CFC_GridComp/AMIP.20C/CFC_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CFC_GridComp/AMIP/CFC_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CFC_GridComp/CFC_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CH4_GridComp/CH4_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CH4_GridComp/AMIP.20C/CH4_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CH4_GridComp/AMIP/CH4_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO2_GridComp/AMIP.20C/CO2_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO2_GridComp/AMIP/CO2_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/CO2_GridComp/CO2_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/O3_GridComp/AMIP.20C/O3_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/O3_GridComp/O3_GridComp_ExtData.rc
./@GEOSchem_GridComp/@GOCART/ESMF/GOCART_GridComp/O3_GridComp/AMIP/O3_GridComp_ExtData.rc
./@GEOSchem_GridComp/StratChem_GridComp/SC_GridComp/StratChem_ExtData.rc
./@GEOSchem_GridComp/GEOSachem_GridComp/AMIP.20C/GEOSachem_ExtData.rc
./@GEOSchem_GridComp/GEOSachem_GridComp/AMIP/GEOSachem_ExtData.rc
./@GEOSchem_GridComp/GEOSachem_GridComp/GEOSachem_ExtData.rc
./@GEOSchem_GridComp/GEOSCHEMchem_GridComp/@geos-chem/run/GEOS/GEOSCHEMchem_ExtData.rc
./GEOSmoist_GridComp/WSUB_ExtData.rc 

It would benefit future shared development to establish naming guidelines for procedures included in the GOCART process library, as well as for their arguments.

I'm not sure if this is an @adarmenov or @pcolarco question...so I've tagged you both. 😄

Namely, I tried running with MERRA2 Emissions today in the year of 2019 (trying to trigger a different issue) and, of course, things go boom:

pe=00084 FAIL at line=02163    MAPL_ExtDataGridCompMod.F90              <From ExtData/PIESA/sfc/QFED/v2.4r6/Y%y4/M%m2/qfed2.emis_bc.005.%y4%m2%d2.nc4 could not find time no extrapolation>

This is because QFED v2.4r6 doesn't exist in 2019. I used Ops emissions and got past this.

So I thought well, maybe this is fixed in GOCART2G, but:

Still v2.4r6.

My question is: should we take this opportunity of moving to GOCART2G (which is non-zero-diff by fiat) to also move to use, say, QFED v2.5r1? That actually does seem to exist in 2019 (and in 2021!). Or is v2.4r6 the only 'validated' version? (And I guess if so...how is MERRA2 running right now?)

An OH_GridComp exists in Icarus-vintage code. It is a key part of the ECCOH functionality (coupled CO/CH4/OH). As such, it should be integrated into the legacy GOCART routines.

It would be helpful to include wet removal efficiency (fwet) as a tuning parameter in each of the GOCART2G subcomponents for increased flexibility.

The FENGSHA dust scheme needs to be added as an optional scheme for the GOCART dust component.
This scheme, developed at NOAA/ARL, will be used in the UFS weather model.

What is it?

The new GEOS-IT that is starting in late 2021 is based on GOCART tag v1.0.1 which only contains the old Legacy GOCART implementation. To improve the simulation of anthopogenic aerosols we will use the CEDS emissions.

New Feature

This new resource file instance should use the CEDS anthropogenic emissions and the MODAPS based (non-NRT, LANCE) version of QFED. Dust and Seasalt will not be modified.

In the GOCART2G resource file, monochromatic wavelengths need to be provided in meters while other wavelengths are in nm:

# monochromatic wavelengths are in meters
aerosol_monochromatic_optics_wavelength: 4.7e-7 5.5e-7 6.7e-7 8.7e-7
wavelengths_for_profile_aop_in_nm: 470 550 870
wavelengths_for_vertically_integrated_aop_in_nm: 470 550 670 870

Using nm also for monochromatic wavelengths may improve consistency and prevent possible mistakes.

Submodules will provide for less contention as individual procedures are modified while preserving the namespace protections of a module.

If there are natural groupings among the routines multiple submodules would be appropriate. Otherwise one large one.

Naming of submodules and associated file is in a bit of flux. With existing GOCART2G conventions, I recommend module has SMod at the end and the file also has ...Smod.F90. (Once we have something working, we can let @amdasilva suggest tweaks to the style.)

Mentioning @rmontuoro for the suggestion to split into multiple files.

When looking at this file to try to find a regression problem I noticed multiple uses of the GOTO statement which should not be used in modern Fortran unless absolutely necessary. From my look at the code, these are not needed as this can be accomplished with the proper modern Fortran way of doing this, which is the exit statement. The exit breaks you out of the do loop and takes you to the next executable statement after the enddo.
i.e. instead of

   do i=1,n
    ! do some work
    if  (logical_condition) then
       goto 15
     endif 
     enddo
15 continue

! The following accomplishes the same thing and makes the code more readable
do i=1,n
  ! do some work
  if (logical_condition) exit
enddo

@tclune - is this valid Fortran?

if (present(a) .and. associated(a)) then

Should this be:

if (present(a)) then
   if (associated(a)) then

Originally posted by @rmontuoro in #68 (comment)

Volcanic SO2 emissions are read incorrectly in UFS-GOCART. This might be related to inconsistent grid coordinate units when running MAPL with an external grid, and it is connected to MAPL closed issue #680 affecting point source emissions.

The picture below shows the SO2EMVN diagnostic variable as written by History:

Until ExtData has been refactored to allow the specification of time intervals for selecting datasets in the ExtData component, this device will allow EMIP runs to work during tge transition from HFED to QFED.

This repo at present does not have a CODEOWNERS file. I can add one, I just need to know who to put in it. I mean, I guess @gmao-esherman should be there in some way.

I suppose it should reflect the GEOSchem_GridComp CODEOWNERS file. But should I make a new "GOCART2G Team"?

ETA: Actually, there was a CODEOWNERS just not in main at the time. There is one in develop at least.

This might be something @amdasilva knows about, but just in case...

When I clone GEOSgcm main and then checkout the develop branch of GOCART, an AMIP run fails:

 Character Resource Parameter: DU_INTERNAL_RESTART_FILE:du_internal_rst
 Bootstrapping du_internal_rst
 Character Resource Parameter: SS_INTERNAL_RESTART_FILE:ss_internal_rst
 Bootstrapping ss_internal_rst
 Character Resource Parameter: CA.oc_INTERNAL_RESTART_FILE:caoc_internal_rst
 Bootstrapping caoc_internal_rst
pe=00091 FAIL at line=00461    CA2G_GridCompMod.F90                     <status=41>
pe=00091 FAIL at line=01799    MAPL_Generic.F90                         <status=41>

It seems to die on this line:

Perhaps HISTORY.rc needs a change for GOCART develop? Not sure. Nope. Does not involve history.

So this is a weird one for @tclune and @aoloso to ponder. Maybe even @atrayano.

To wit, for some work @aoloso is doing looking at the bad performance on the Skylake OPA nodes at NCCS (cc @wmputman), I built Open MPI 4.0.6 for both Skylake/OPA and Cascade Lake/Infiniband at NCCS and then build Baselibs.

I then built the model twice, one on Cascade Lake and one on Skylake (since Open MPI can only handle one interconnect at a time). I did this and the model ran just fine. The history outputs after a day are identical to Intel MPI. The state checkpoints are also identical to Intel MPI EXCEPT for DU and SS!

Comparing NC4 du_internal_checkpoint using nccmp...
Failure!
Checking for data differences
Variable Group   Count        Sum     AbsSum          Min         Max       Range        Mean      StdDev
DU       /     4914428 0.00111215 0.00118107 -7.42602e-07 1.28244e-06 2.02504e-06 2.26302e-10 5.29118e-09
...
Comparing NC4 ss_internal_checkpoint using nccmp...
Failure!
Checking for data differences
Variable Group   Count        Sum     AbsSum          Min         Max       Range        Mean      StdDev
SS       /     4912017 0.00117208 0.00126001 -4.20413e-08 1.49067e-07 1.91109e-07 2.38615e-10 1.89506e-09

And if we look closer, they are not just slightly different, but crazy different:

DIFFER : VARIABLE : DU : POSITION : [1,1,1,1] : VALUES : 0 <> 3.40241e-13
DIFFER : VARIABLE : DU : POSITION : [2,1,1,1] : VALUES : 0 <> 1.23087e-13
DIFFER : VARIABLE : DU : POSITION : [3,1,1,1] : VALUES : 0 <> 3.18833e-14
DIFFER : VARIABLE : DU : POSITION : [4,1,1,1] : VALUES : 0 <> 5.12426e-15
...
DIFFER : VARIABLE : DU : POSITION : [45,288,72,5] : VALUES : 1.06061e-23 <> 0
DIFFER : VARIABLE : DU : POSITION : [46,288,72,5] : VALUES : 3.76284e-25 <> 0
DIFFER : VARIABLE : DU : POSITION : [47,288,72,5] : VALUES : 1.55894e-25 <> 0
DIFFER : VARIABLE : DU : POSITION : [48,288,72,5] : VALUES : 8.44485e-26 <> 0

Weirdly, the differences are mostly "0 in Open MPI" as the "0 <> " on the left ends at [48,288,1,1]:

DIFFER : VARIABLE : DU : POSITION : [47,288,1,1] : VALUES : 0 <> 1.55894e-25
DIFFER : VARIABLE : DU : POSITION : [48,288,1,1] : VALUES : 0 <> 8.44485e-26
DIFFER : VARIABLE : DU : POSITION : [1,1,2,1] : VALUES : 1.17549e-38 <> 0
DIFFER : VARIABLE : DU : POSITION : [2,1,2,1] : VALUES : 1.17549e-38 <> 0

I think the indexing is [lat,lon,lev,unknown_dim1], so it's only level one of the first ungridded dimension??

This is baffling.

Again, all other checkpoints and all of history is zero-diff beyond this!

In reviewing the GOCART2G code, I have notice that there are many blocks of prints which appear to have been commented out. For example in the NI2G_GridComp there is the following block (see 808 - 813):

!if(mapl_am_i_root()) print*,'NI2G Run2 BEGIN sum(NH3) = ',sum(NH3)
!if(mapl_am_i_root()) print*,'NI2G Run2 BEGIN sum(NH4a) = ',sum(NH4a)
!if(mapl_am_i_root()) print*,'NI2G Run2 BEGIN sum(hno3) = ',sum(nitrate_hno3)
!if(mapl_am_i_root()) print*,'NI2G Run2 BEGIN before sum(self%xhno3) = ',sum(self%xhno3)
!if(mapl_am_i_root()) print*,'NI2G sum(DU) = ',sum(DU)
!if(mapl_am_i_root()) print*,'NI2G sum(SS) = ',sum(SS)

There are many more examples scattered throughout all the other GOCART2G GridComps.

It looks like these blocks are being toggled on and off via comments for debugging and diagnostic purposes. If this is the case then we should either remove them in the future or convert them into pFlogger calls (and provide a logging config that turns them on or off).

@tclue what do you think of this?

No longer needed as we do not aggregate across tracers.

There appears to be a bug in getAerosolSum but I'm not sure of the right fix. As this routine is (eventually) used by GAAS, I'll assign to @amdasilva for now and see what he thinks the right fix as I can sort of think of two (and I'll cc @weiyuan-jiang).

The issue is that in this code:

aeroOut is defined as intent(out) but it's never initialized to anything. So the first time it's used is:

          aeroOut = aeroOut + ptr3d

and my guess is if this code was run with the right debugging flags, it would fail since aeroOut appears on the right side.

I can think of two fixes:

1. Initialize aeroOut (to zero?)
2. Make aeroOut intent(inout)

I'm not sure of the right fix. (And they might be equivalent if my reading of get_mixRatioSum is correct.)

It seems beneficial to reorganize the GOCART process library source code into separate units, as pointed out by @tclune. This will provide a code base easier to maintain.

One option would be to arrange the library into separate modules, each including only the procedures required by one GOCART component (e.g. DU, SS, etc.) .

I think the indexes here should start from i1 and j1 @tclune

@tclune