Need a release for #55 for WCOSS2

In the installation of hpc-stack-DTC(with ./build_stack.sh -p /app/hpc-stacks -c config/config_comgsi.sh ), only the grib-util software is not installed successfully in hpc-stack-DTC:

the errors are as follows:

/hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1336: undefined reference to ipolates_' /hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1411: undefined reference to ipolatev_'
/hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1442: undefined reference to ipolatev_' /hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1401: undefined reference to ipolatev_'
/hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1342: undefined reference to ipolates_' /hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1379: undefined reference to ipolatev_'
/hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1364: undefined reference to ipolates_' /hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:1323: undefined reference to ipolates_'
CMakeFiles/copygb.dir/copygb.F.o: In function MAIN__': /hpc-stack-DTC/pkg/grib_util-v1.2.2/sorc/copygb.fd/copygb.F:269: undefined reference to makgds_'
collect2: error: ld returned 1 exit status
make[2]: *** [sorc/copygb.fd/copygb] Error 1
make[2]: Leaving directory /hpc-stack-DTC/pkg/grib_util-v1.2.2/build' make[1]: *** [sorc/copygb.fd/CMakeFiles/copygb.dir/all] Error 2 make[1]: Leaving directory /hpc-stack-DTC/pkg/grib_util-v1.2.2/build'
make: *** [all] Error 2
BUILD FAIL! Lib: grib_util Error:2

How should i resolve the problem?

See NOAA-EMC/NCEPLIBS-g2c#126

Instead, let's fetch and build just what we need.

We have the following directories under sorc:

cnvgrib.fd copygb2.fd copygb.fd degrib2.fd grb2index.fd grbindex.fd grib2grib.fd tocgrib2.fd tocgrib2super.fd tocgrib.fd wgrib.cd

What's with the ".fd" and ".cd" for wgrib? Normally we don't have file extensions on our directory names...

When building like this:

cmake -DCMAKE_PREFIX_PATH=/usr/local/NCEPLIBS-ip-3.3.3/;/usr/local/NCEPLIBS-bacio-2.4.1;/usr/local/NCEPLIBS-sp-2.3.3;/usr/local/jasper-2.0.22;/usr/local/NCEPLIBS-w3nco-2.4.0;/usr/local/NCEPLIBS-g2-3.4.0 ..

I get these CMake warnings.

CMake Warning at sorc/cnvgrib.fd/CMakeLists.txt:24 (add_executable):
  Cannot generate a safe runtime search path for target cnvgrib because there
  is a cycle in the constraint graph:

    dir 0 is [/usr/local/jasper-2.0.22/lib]
      dir 1 must precede it due to runtime library [libjasper.so.4]
    dir 1 is [/usr/local/jasper-2.0.16/lib]
      dir 0 must precede it due to runtime library [libjasper.so.4]

  Some of these libraries may not be found correctly.


CMake Warning at sorc/copygb2.fd/CMakeLists.txt:23 (add_executable):
  Cannot generate a safe runtime search path for target copygb2 because there
  is a cycle in the constraint graph:

    dir 0 is [/usr/local/jasper-2.0.22/lib]
      dir 1 must precede it due to runtime library [libjasper.so.4]
    dir 1 is [/usr/local/jasper-2.0.16/lib]
      dir 0 must precede it due to runtime library [libjasper.so.4]

  Some of these libraries may not be found correctly.


CMake Warning at sorc/degrib2.fd/CMakeLists.txt:12 (add_executable):
  Cannot generate a safe runtime search path for target degrib2 because there
  is a cycle in the constraint graph:

    dir 0 is [/usr/local/jasper-2.0.22/lib]
      dir 1 must precede it due to runtime library [libjasper.so.4]
    dir 1 is [/usr/local/jasper-2.0.16/lib]
      dir 0 must precede it due to runtime library [libjasper.so.4]

  Some of these libraries may not be found correctly.


CMake Warning at sorc/tocgrib2.fd/CMakeLists.txt:12 (add_executable):
  Cannot generate a safe runtime search path for target tocgrib2 because
  there is a cycle in the constraint graph:

    dir 0 is [/usr/local/jasper-2.0.22/lib]
      dir 1 must precede it due to runtime library [libjasper.so.4]
    dir 1 is [/usr/local/jasper-2.0.16/lib]
      dir 0 must precede it due to runtime library [libjasper.so.4]

  Some of these libraries may not be found correctly.


CMake Warning at sorc/tocgrib2super.fd/CMakeLists.txt:12 (add_executable):
  Cannot generate a safe runtime search path for target tocgrib2super because
  there is a cycle in the constraint graph:

    dir 0 is [/usr/local/jasper-2.0.22/lib]
      dir 1 must precede it due to runtime library [libjasper.so.4]
    dir 1 is [/usr/local/jasper-2.0.16/lib]
      dir 0 must precede it due to runtime library [libjasper.so.4]

  Some of these libraries may not be found correctly.

See #92 comments.

@GuangPingLou-NOAA could you describe the issue you're having with copygb2 on WCOSS2?

We are using "develop" branch only.

@BoiVuong-NOAA mentioned this task in an email.

Can we get some more detail please? When is it needed by?

Also, please implement unit tests with this change. No more code changes are permitted without tests to verify their correctness.

Change dependency to w3emc.

See NOAA-EMC/NCEPLIBS#130

grib_util depends on:

• sp
• ip
• w3nco
• g2
• bacio

We need to test against the develop branch of each of these projects. THat way, if we break something by changing those projects, we will find it sooner...

The directory wgrib.cd contains apparently some documentation from wgrib, and a C program. Is this wgrib2? Some old version?

Why is there a copy of it here?

See NOAA-EMC/NCEPLIBS#120

We have the bare minimum of doxygenation in this code base. Every utility needs a complete documentation overhaul to clean up the doxygen to our usual standards.

If anyone wants to jump in and do one of the utilities, feel free. Eventually I will work my way around to each of these, so move quickly if you want to do one. ;-)

@climbfuji reported that. grib_util cannot be built with gfortran-10.
It fails with the usual gfortran-10 error (rank mismatch between actual argument bla bla).

handle bacio library name change with bacio-2.5.0 release

See NOAA-EMC/NCEPLIBS#154

Part of #43

There's a lot of code from the ip project in copygb2. Instead link to the ip project and use the code there.

This must be done after #80 is in place.

We need to start adding some tests for these utilities. Copygb2 seems as good a place to start as any.

A starting point would be to get a small GRIB2 file, and check it into the tests directory. The use copygb2 in various ways and check the output for correctness.

I see we have some tests. We need to get them running with cmake not with scripts, and have them run by the CI.

See NOAA-EMC/NCEPLIBS#130

We have:

find_package(Jasper REQUIRED)
find_package(PNG REQUIRED)
find_package(ZLIB REQUIRED)

find_package(w3nco REQUIRED)
find_package(g2 REQUIRED)
find_package(bacio REQUIRED)
find_package(ip REQUIRED)
find_package(sp REQUIRED)

We should specify some minimum version numbers here...

See NOAA-EMC/NCEPLIBS-g2#151

What has to be done:

• Convert line continuation characters.
• Change comment delimeters.
• Rename file to have .F90 extension.

Update README with NCEP GRIB2 website

Starting with bacio-2.5.0 no more "_4" at the end of bacio library name.

A user is reporting an error with tocgrib2super on WCOSS2.

The error message is GETGB2P: ERROR REQUEST NOT FOUND which comes from the g2 library here;

https://github.com/NOAA-EMC/NCEPLIBS-g2/blob/4e98ae8e10f8994d77365a4fa424faa684e83f84/src/getgb2p.f#L160

After calling getgb2s to search the index here:

https://github.com/NOAA-EMC/NCEPLIBS-g2/blob/develop/src/getgb2s.f

There's a loop in there that searches for the requested index, but it's not found and returns an error:

https://github.com/NOAA-EMC/NCEPLIBS-g2/blob/4e98ae8e10f8994d77365a4fa424faa684e83f84/src/getgb2s.f#L310

@BoiVuong-NOAA @Hang-Lei-NOAA any ideas?

I went back and checked the library kinds and any changes in the program, but everything seems fine. I'm guessing the problem comes from g2.

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM tocgrib2super HAS BEGUN. COMPILED 2010902.50     ORG: NP11
     STARTING DATE-TIME  SEP 28,2021  20:33:21.953  271  TUE   2459486


  filesize            0
 DESC=SUPER WMO HEADER
 WMOHEAD=LAPZ99 KWBP

*******************************************************************************
 Start new record no. =  1
 DESC=1 HOUR SFC DUST
 WMOHEAD=LAPB07 KWBP
 GRIB2 DISCIPLINE= -1
 Section 1= -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 PDT 4. 8 = 13 195 2 0 6 0 0 1 0 103 0 100 103 0 0 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 LUGI =            0
 LUGB =           11
 jrew (number of fields to skip) =            0
  SAGT            0           0           2
 GETGB2P: ERROR REQUEST NOT FOUND

*******************************************************************************
 Start new record no. =  2
 DESC=1 HOUR SFC DUST
 WMOHEAD=LAPB08 KWBP
 GRIB2 DISCIPLINE= -1
 Section 1= -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 PDT 4. 8 = 13 195 2 0 6 0 0 1 1 103 0 100 103 0 0 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 LUGI =            0
 LUGB =           11
 jrew (number of fields to skip) =            0
 GETGB2P: ERROR REQUEST NOT FOUND

*******************************************************************************
 Start new record no. =  3
 DESC=1 HOUR SFC DUST
 WMOHEAD=LAPB09 KWBP
 GRIB2 DISCIPLINE= -1
 Section 1= -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 PDT 4. 8 = 13 195 2 0 6 0 0 1 2 103 0 100 103 0 0 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 LUGI =            0
 LUGB =           11
 jrew (number of fields to skip) =            0
 GETGB2P: ERROR REQUEST NOT FOUND

*******************************************************************************
 Start new record no. =  4
 DESC=1 HOUR SFC DUST
 WMOHEAD=LAPB10 KWBP
 GRIB2 DISCIPLINE= -1
 Section 1= -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 PDT 4. 8 = 13 195 2 0 6 0 0 1 3 103 0 100 103 0 0 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
 LUGI =            0
 LUGB =           11
 jrew (number of fields to skip) =            0
 GETGB2P: ERROR REQUEST NOT FOUND
...
...
...

Flags like -axCORE-AVX2 are sprinkled throughout the code. This can cause problems on heterogenous platforms like Jet.

Does grib_util need these flags? They've been removed from several other libraries.

Seems like README.install refers to an old build system, so can be removed.

In our CMake file we have:

option(OPENMP "use OpenMP threading" OFF)

We need to test this option in CI.

One of the ufs_utils consistency tests use copygb as follows:

• /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/intel-18.0.5.274/grib_util/1.2.2/bin/copygb -M '#1.57' -x seaice.5min.blend.bitmap seaice.5min.blend

copygb should be replacing the bitmap (at land points) in file seaice.5min.blend.bitmap with the flag value of '1.57'. But the flag value is not geolocated correctly. North America is in the wrong location.

This test also uses copygb2. But that step is working. My test does not use any other components of grib_util.

I recompiled grib_util and the copygb step is working correctly. See:

• /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/grib_util/NCEPLIBS-grib_util

There is no documentation block for these programs. Please add one in accordance with doxygen rules.

What do prlevel.f and prvtime.f do in src/degrib2?

We have these two codes, but they have no documentation.

What do they do? Who wrote them? Any ideas?

By using the same subdirectory names we make it easy for others to share the work of maintaining and developing the code, which is one of the big motivations for releasing it to the community. We also make it easier for other NOAA programmers to work on the code without making mistakes.

Just some things I noticed:

wgrib2 is included with grib_util

copygb2 has all the polate routines found in ip lib

cnvgrib contains gds2gdt which is also in the g2 library

THe command
/apps/ops/prod/libs/intel/19.1.3.304/grib_util/1.2.2/bin/copygb -x '-k4*-1 11' -B-1 '-K4*-1 81' '-A<0.5' f1 f2

fails to produce a valid f2 file on WCOSS2. It works on WCOSS and an older, privately built v1.1.1 also works on WCOSS2.

THe full command on Cactus is
/apps/ops/prod/libs/intel/19.1.3.304/grib_util/1.2.2/bin/copygb -x '-k4*-1 11' -B-1 '-K4*-1 81' '-A<0.5' /lfs/h2/emc/eib/noscrub/gwv/copygb/f1 f2

Loaded modules are

1. craype-x86-rome (H) 3) craype-network-ofi (H) 5) PrgEnv-intel/8.1.0 7) intel/19.1.3.304 9) cmake/3.20.2
2. libfabric/1.11.0.4.75 (H) 4) envvar/1.0 6) craype/2.7.8 8) cray-mpich/8.1.7 10) grib_util/1.2.2

Want to tag the fix for GCC 10 in this commit

#44

Bump version by 0.0.1 to 1.2.2

update README to include authors of each utility

See NOAA-EMC/NCEPLIBS#121

For release/public-v1: The following linker flags in NCEPLIBS-grib_util/sorc/copygb2.fd/CMakeLists.txt are requiring a static libgfortran:

if(IntelComp)
  set(fortran_flags "-g" "-O3" "-r8" "-convert" "big_endian" "-auto" "-fpp" "${OpenMP_Fortran_FLAGS}")
  set(link_flags "-static-intel")
elseif(GNUComp)
  set(fortran_flags "-g" "-O3" "-fdefault-real-8" "-fconvert=big-endian" "-cpp" 
    "${OpenMP_Fortran_FLAGS}")
  set(link_flags "-static-libgfortran")
else()
  message("unknown compiler!")
  exit()
endif()

This is not something modern Linux distributions are good at. I commented it out and could build the library on Redhat 8 (which has a broken libgfortran.a, i.e. the packages that "provide" it just have symbolic links to a place where no version exists).

Is it really necessary to use static linking for this? Does anyone know? Thanks.

See NOAA-EMC/NCEPLIBS#154

In our CI we have:

        sudo apt-get update &> /dev/null
        sudo apt-get install libmpich-dev
        sudo apt-get install libnetcdf-dev libnetcdff-dev netcdf-bin pkg-config
        sudo apt-get install libpng-dev
        sudo apt-get install libjpeg-dev doxygen
        python3 -m pip install gcovr

We don't need netCDF or MPICH so these can be removed.

I believe this is documentation that is already in the code so this file can just be removed. Need to check that.

Add memory check, documentation build, warning check to CI runs

See #294.

Before changes are made to cnvgrib or the libraries, some tests for cnvgrib are needed. These can be run by a shell script.

Add macOS build to CI

We faced a similar situation in UFS_UTILS: a bunch of programs that had nothing to do with each other. Making this look good in the documentation is impossible, since it lists the functions etc. from each utility, all in one big jumbled list.

What we ended up doing is a separate doxygen build for each utility. So each utility gets a docs directory, and in it a Doxygen.in file and user_guide.md.

Then the top-level documentation build just has a one-page description of the utilities, with a link to their docs.

This is a pain to set up but, once done, every utility gets nice documentation, and it's easy to maintain because we rarely change the doxyfile.

Right now the CI is not able to run make test. There is no test target in the build.

The documentation landing page is pretty empty. We need to give some high-level overview of grib_util.