goddard-fortran-ecosystem / gftl Goto Github PK

There is a .travis.yml on develop, but no checks are triggered.

Hmm...

This is mostly for debugging purposes - would be useful to examine the state of the container. Should be two cases. One that just prints metadata and the other that prints contained values as well. (If the container is of derived type, another switch will be needed to indicate whether he type includes PDT methods.

The library looks very interesting (and sorely needed in the Fortran world!)

Is V2 ready to play with? The examples all seem to be for V1.

Is there a need to install to consume? As a template library, I was expecting to be able to just include the appropriate files from the include directory into my source.

The challenges I have are cloud build machines (not directly under my control), distributed team (would prefer not to require installs on everyone's machines) and builds for both Linux and Windows. We are using Intel OneAPI Fortan 2022.1.0 on both Windows and Linux. We are not using cmake as our build orchestrator.

Thanks!

I suggest adding the topics generic-programming, fortran, modern-fortran in the About section.

Just need to add a simple gftl.mk file that can be included by make. The file will specify just a few paths and such.

BSD m4 behaves differently from GNU m4. If m4 is hardcoded, the results of build are wrong (e.g. __type_MOVE is generated instead of __TYPE_MOVE). The proposed fix is in #102

There are lots of these warnings, so much that it's unfortunately significantly slowing down single-threaded compilation. Here's an extract:

[200/1384] Building Fortran preprocessed _deps/pfunit-build/extern/fArgParse/extern/gFTL-shared/src/v2/CMakeFiles/gftl-shared-v2.dir/map/IntegerReal128Map.F90-pp.f90
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/set/procedures.inc:1033:0:

 1033 | #define __algorithm_iterator __set_iterator
      | 
Warning: "__algorithm_iterator" redefined
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/map/specification.inc:144:0:

  144 | #define __algorithm_iterator __map_iterator
      | 
note: this is the location of the previous definition
[222/1384] Building Fortran preprocessed _deps/pfunit-build/extern/fArgParse/extern/gFTL-shared/src/v2/CMakeFiles/gftl-shared-v2.dir/ordered_map/IntegerDoubleOrderedMap.F90-pp.f90
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/vector/specification.inc:212:0:

  212 | #define __algorithm_iterator __vector_iterator
      | 
Warning: "__algorithm_iterator" redefined
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/map/specification.inc:144:0:

  144 | #define __algorithm_iterator __map_iterator
      | 
note: this is the location of the previous definition
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/vector/procedures.inc:1013:0:

 1013 | #define __algorithm_iterator __vector_iterator
      | 
Warning: "__algorithm_iterator" redefined
/home/peter/Codes/neasy-f/build/_deps/pfunit-src/extern/fArgParse/extern/gFTL-shared/extern/gFTL/include/v2/map/procedures.inc:428:0:

  428 | #define __algorithm_iterator __map_iterator
      | 
note: this is the location of the previous definition

It looks like there's only three unique sources of the warnings:

• include/v2/set/procedures.inc:1033
• include/v2/vector/procedures.inc:1013
• include/v2/vector/specification.inc:212

I think the above snippet demonstrates all of them.

This is GFTL commit dc85913, ultimately from pfunit 47b3b0ab1.

see discussion here

set_property(
    TARGET myLibrary
    APPEND
    PROPERTY STATIC_LIBRARY_FLAGS "-no_warning_for_no_symbols"
)

PURE would allow more contexts where containers could be used.

The attached reproducer from @bena-nasa shows that the recent workaround for a gfortran memory leak unfortunately introduced a bug in the vector set() method. Previously the scenario in this reproducer relied on intrinsic assignment which was implicitly deallocating in this scenario. The memory workaround used explicit allocation and it was not obvious that a check needed to be done for deallocation.

I think the fix will be easy, but a bit ugly.

use pfio
   implicit NONE
 
   type(Variable) :: v, v1, v2
   type(StringVariableMap) :: vars
 
   v = variable(pfio_real32,dimensions='time')
   call v%add_attribute('long_name','time')
 
   v1 = variable(pfio_real32,dimensions='time')
   call v%add_attribute('long_name','time1')
 
   v2 = variable(pfio_real32,dimensions='lon')
   call v%add_attribute('long_name','longitude')
 
   call vars%insert('time',v1)
   call vars%insert('lon',v2)
 
   call vars%set('time',v)
 
End Program test_modify

(Note the reproducer is using code from GEOS-ESM/MAPL.

One of our builds using pFUnit is failing at runtime with the following back trace:

At line 19 of file /builds/gyrokinetics/utils/build/_deps/pfunit-src/extern/fArgParse/src/StringActionMap.F90
Fortran runtime error: Recursive call to nonrecursive procedure '__copy_fp_stringactionmap_Map_s_node'
Error termination. Backtrace:
#0  0x7f0fa5f95d51 in ???
#1  0x7f0fa5f968a9 in ???
#2  0x7f0fa5f96d8d in ???
#3  0x51747d in __fp_stringactionmap_MOD___copy_fp_stringactionmap_Map_s_node
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/extern/fArgParse/src/StringActionMap.F90:19
#4  0x517553 in __fp_stringactionmap_MOD___copy_fp_stringactionmap_Map_s_node
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/extern/fArgParse/src/StringActionMap.F90:19
#5  0x49bf22 in set_command_line_options
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/src/funit/FUnit.F90:125
#6  0x49eda2 in __funit_MOD_generic_run
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/src/funit/FUnit.F90:60
#7  0x406914 in __pfunit_MOD_run
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/src/pfunit/pFUnit.F90:108
#8  0x4065df in funit_main_
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/src/pfunit/pfunit_main.F90:16
#9  0x40618a in MAIN__
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/include/driver.F90:81
#10  0x4061e0 in main
	at /builds/gyrokinetics/utils/build/_deps/pfunit-src/include/driver.F90:52

It says it's from fArgParse, but I guess it's really from gFTL, though I couldn't work out where exactly.

This build is using gfortran 9.3.1, and the following compiler flags:

-ffpe-trap=invalid,zero -fcheck=all -finit-real=snan -finit-integer=-12345 -finit-logical=true -finit-character=95 -finit-derived -g

-fcheck=all does include recursion, so as a workaround we could probably turn that off.

The GFTL repository itself can be copied to an airgapped system with
.. on connected system

git clone https://github.com/Goddard-Fortran-Ecosystem/gFTL-shared

then copy the gFTL-shared directory to the airgapped system and attempt further build.
Upon doing so ON THE AIRGAPPED SYSTEM though the build generates

cd gFTL-shared/

• rm -rf build
• mkdir build
• cd build
• cmake -DCMAKE_INSTALL_PREFIX=/mnt/lfs4/HFIP/hfv3gfs/gwv/simt/simstack/netcdf140.492.460.mapl235.fms2301.crtm240.z ..
  -- The Fortran compiler identification is Intel 18.0.5.20180823
  -- Detecting Fortran compiler ABI info
  -- Detecting Fortran compiler ABI info - done
  -- Check for working Fortran compiler: /apps/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin/mpiifort - skipped
  -- Performing Test _LOGICAL_DEFAULT_KIND: SUCCESS (value=4)
  -- Performing Test _INT_DEFAULT_KIND: SUCCESS (value=4)
  -- Performing Test _ISO_INT8: SUCCESS (value=1)
  -- Performing Test _ISO_INT16: SUCCESS (value=2)
  -- Performing Test _ISO_INT32: SUCCESS (value=4)
  -- Performing Test _ISO_INT64: SUCCESS (value=8)
  -- Performing Test _REAL_DEFAULT_KIND: SUCCESS (value=4)
  -- Performing Test _DOUBLE_DEFAULT_KIND: SUCCESS (value=8)
  -- Performing Test _ISO_REAL32: SUCCESS (value=4)
  -- Performing Test _ISO_REAL64: SUCCESS (value=8)
  -- Performing Test _ISO_REAL128: SUCCESS (value=16)
  Cloning into '/mnt/lfs4/HFIP/hfv3gfs/gwv/simt/simstack/build/gFTL-shared/extern/gFTL'...
  fatal: unable to access 'https://github.com/Goddard-Fortran-Ecosystem/gFTL.git/': Failed connect to github.com:443; Connection timed out
  fatal: clone of 'https://github.com/Goddard-Fortran-Ecosystem/gFTL.git' into submodule path '/mnt/lfs4/HFIP/hfv3gfs/gwv/simt/simstack/build/gFTL-shared/extern/gFTL' failed
  Failed to clone 'extern/gFTL'. Retry scheduled
  Cloning into '/mnt/lfs4/HFIP/hfv3gfs/gwv/simt/simstack/build/gFTL-shared/extern/gFTL'...

How do we do this build on an airgapped system (NOAA jet tjet node in this example)?

A larger use case in which 10^6 integers are inserted in a set and then deleted one at a time fails at run time. First failure is not finding element 11.

Strongly suspect this is a corner case not properly covered by recent changes to use ALLOCATABLE rather than POINTER for the child nodes.

Reproducer:

program main
   use, intrinsic :: iso_fortran_env, only: INT64, REAL64
   use gFTL2_Integer32Set
   implicit none

   type(Integer32Set) :: s
   type(Integer32SetIterator) :: iter
   integer :: i, j, n
   real(kind=REAL64) :: x

   integer(kind=INT64) :: c0, c1, cr
   
   n = 10**5
   s = Integer32Set()

   call system_clock(c0, cr)
   do i = 1, N
!!$      call random_number(x)
!!$      j = 1 + floor(N*x)/N + i
!!$      call s%insert(j)
      call s%insert(i)
   end do
   call system_clock(c1)

   print*,'construction time: ', real(c1-c0)/cr, s%size()

   call system_clock(c0)
!!$   call s%clear()
   do i = 11, N
!!$      iter = s%find(i)
!!$      print*, i, iter == s%end()
      j = s%erase(i)

      if (j /= 1) then
         print*, i, j
         associate (b => s%begin(), e => s%end())
           iter = b
           do while (iter /= e)
              print*,'iter: ', iter%of()
              call iter%next()
           end do
         end associate
      end if
   end do
   call system_clock(c1)
   print*,'deconstruction time: ', real(c1-c0)/cr
   
end program main

This might break some existing applications, but if the key does not exist, this interface should "throw" an exception.

The existing iterator approach in gFTL mimics C++ behavior, but with a significant wrinkle. Namely, in C++ the loop itself can do the increment, but Fortran requires an explicit call to the next() method at the end of the loop. For simple loops this is not a problem, but users can easily go astray if they attempt to use CYCLE which will not on its own trigger the increment to the iterator.

So rather than

associate (e => container%end())
   iter = container%begin()
   do while (iter /= e)
      if (cond) cycle ! infinite loop
      call iter%next()
   end do
end associate

I would like to have:

associate (e => container%f_end())
   iter = container%f_begin()
   do while (iter /= e)
      call iter%next() ! Top of loop - always triggered
      if (cond) cycle ! works just fine
   end do
end associate

Corner cases:

1. Empty loop. f_begin() must be equal to f_end()
2. 1-item: f_begin() must not equal f_end(). But next(f_begin()) should.

I copied cmake/Intel.cmake to cmake/IntelLLVM.cmake in order to get gftl-shared-v1.5.0 to compile with the LLVM based compilers from Intel.

Note that the LLVM based compilers are very different than the Classic compilers in their support for compiler options. See https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html for further details.

This is a simple one. I found out today as I was trying to build gFTL with PGI 18.5 that it doesn't support PGI. I see two issues:

1. There is no cmake_utils/PGI.cmake file
2. If I add one, it doesn't seem to actually pass the tests.

The use case is to have a container that is part of a family of subclasses. The type definition would start:

TYPE, EXTENDS(AbstractX) :: XVector

The current need is for both Vector and Map, but should treat Set and altSet to be consistent.

I have a use case where I need pointers into a container to remain valid as the container grows. For polymorphic cases, the wrapper type with ALLOCATABLE enables this as gFTL uses MOVE_ALLOC() under the hood.

gFTL v1 also had the ability to use a wrapper type for non-polymorphic types, and it appears that v2 almost has the capability baked in. Just a minor tweak to an FPP conditional is required and then one can use #define T_deferred.

There is a test (see [1]) for vector and deque that calls erase_range(iterator, iterator) (empty range). This currently ends-up calling MOVE_ALLOC(X, X) (at [2] and [3]) which is not legal Fortran (see [4]).
While most compilers do not emit runtime errors, they silently deallocate X, which in the case of erase_range ends-up destroying the vector/deque elements after the iterator. I expect the intention for erase_range(iterator, iterator) is to be a no-op instead, the vector/deque code should probably do nothing in this case.

You can observe the issue if you add an ASSERT to test the element values (like ASSERT(v%at(2), two)) after the erase_range call in the test_erase_empty_range test.

gfortran builds will fail with:

[Test_deferred_stringVector.pf:851]
String assertion failed:
    expected: <"">
   but found: <"two">
  first diff: 

[1] :

I found this issue because flang generate runtime errors for such MOVE_ALLOC.

currently if the key is not found, a null pointer is returned, but the underlying set has grown.

Polymorphism gets interesting in this case.

Testing with the FORD documenter found there were some v1 macros that were being redefined. As this might have bad effects (see Fortran-FOSS-Programmers/ford#421 (comment)), we aim to clean those up. PR incoming.

Sorry for the intrusion, but my employer (NASA) is requesting that all external contributors to my open source projects sign a contributors license agreement. There are two flavors - individual or corporate, depending on whether your contributions are a hobby or part of your day job.

Please download the appropriate CLA at https://github.com/Goddard-Fortran-Ecosystem/cla/tree/main/gFTL and follow the instructions there.

Thank you in advance.

@ZedThree
@MehdiChinoune
@cferenba
@LiamBindle

Consider a case when a user wants to install headers into ${prefix}/include/GFTL (as opposed to dumping them into a common dir and promoting a chaos there). And then, inside GFTL there are V1 and V2 (without an unnecessary another include in between). There seems to be no trivial way to do that, and even patching CMake files requires multiple iterations, because settings in one CMakeLists file are ignored due to the same being redefined elsewhere.
If one does not notice what the build system does, everything gets coerced into ${prefix}/GFTL-${version}.

It looks like Travis is removing oclint from osx images:

https://changelog.travis-ci.com/oclint-is-removed-from-mac-builds-79270

As we currently ask to uninstall it, this triggers faults in Travis tests. Remove it.

When T1_EQ is not defined by T2_EQ is defined, the template generates compile errors because local variables lt and gt are used but not declared.

If multiple (v2) containers are used in a single program unit, some of the procedure names conflict. E.g., before begin() was only a type-bound procedure on containers that support iterators. But it is now also a PUBLIC function that takes an iterator as an argument.

I took the easy path and did not use a generic interface block to expose the method which results in unnecessary name conflicts. The correction is to use interface blocks in each case. Tedious, but straightforward.

The attached reproducer demonstrates a memory leak under Intel 18 compilers. (18.0.3 and 18.0.5 were tested.) This appears to be fixed under Intel 19, but a workaround is desired by some users.

If intrinsic assignment of PAIR objects is replaced with component-wise assignment, the memory leak goes away. (And hints at the necessary workaround.)

module foo_mod
   implicit none
   private

   public :: pair
   public :: container

   

   type :: pair
      private
      character(:), allocatable :: key
      integer :: value
   end type pair

   type :: container
      type(pair), allocatable :: elements(:)
   contains
      procedure :: run
   end type container

   interface pair
      module procedure new_pair
   end interface pair

contains

   function new_pair(key, value) result(p)
      type (Pair) :: p
      character(*), intent(in) :: key
      integer, intent(in) :: value

      p%key = key
      p%value = value
   end function new_pair

   subroutine run(this, obj, n)
      class(container), intent(inout) :: this
      type(Pair), intent(in) :: obj
      integer, intent(in) :: n

      integer :: j
      logical :: flag
      flag = .true.
      
      allocate(this%elements(n))

      do j = 1, n
         if (flag) then
            this%elements(j) = obj
!!$         this%elements(j)%key = obj%key
!!$         this%elements(j)%value = obj%value
         end if
      end do

   end subroutine run
   
end module foo_mod

program main
   use foo_mod
   type(Pair) :: obj
   type(container) :: c

   
   obj = pair('dog', 3)
!!$   obj%key = 'dog'
!!$   obj%value = 3
      
   do i = 1, 1000
      call c%run(obj, 100)
      deallocate(c%elements)
   end do
   
end program main
gs6101-midas:tmp tclune$ 

rc should be set to 0 for in bounds references if present.

Just discovered that _value_allocatable is not used in the map template. This is technically not a bug, as without it the Fortran assignment "=" operator is used instead which is legal.

In the early days move_alloc() was more robust. But even now it is preferable both for efficiency and to allow pointer references to persist when the container is modified. (Without documentation, I can hedge on whether that is guaranteed. But it would be nice ...)

In trying to build MAPL with GCC9 recently, I saw quite a few of these:

[ 54%] Building Fortran object MAPL_Base/CMakeFiles/MAPL_Base.dir/MAPL_ESMFTimeVectorMod.F90.o
/discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-6.0.4-SLES11/x86_64-unknown-linux-gnu/gfortran_9.1.0-openmpi_4.0.1/Linux/GFTL-1.2/include/templates/unused.inc:10:0:

   10 | #define _UNUSED_DUMMY(dummy) if (.false.) print*,shape(dummy)
      | 
Warning: "_UNUSED_DUMMY" redefined
/discover/swdev/mathomp4/Models/MAPL-MAPL20-Cleanup-GCC9/MAPL/MAPL_Base/unused_dummy.H:13:0:

   13 | #define _UNUSED_DUMMY(x) if (.false.) print*,shape(x)
      | 
note: this is the location of the previous definition

I believe this is due to unused.inc being just a #define without the usual quartet:

#ifdef FOO
#undef FOO
#endif
#define FOO

Thus in some files in MAPL that used templates/vector.inc, say, they pick up this new (nigh-identical) macro.

Note: something perhaps should/could be added to all_macros_undef.inc as well? I see it exists but gFTL is more complex than I'm used to.

In include/types, gFTL installs the files Foo.inc, FooPoly.inc, FooPolyPtr.inc, and FooPtr.inc.
These require modules which are not available:

use Foo_mod, only: Foo

This looks like example files; shouldn't they be installed as examples?

The issue is not detectable with most compilers but with aggressive debugging flags NAG will give weird runtime issues if the TO argument of move_alloc() does not have the TARGET attribute. (Standard then allows pointers associated with FROM to become undefined.)

Probably would have been easier to diagnose, but my application causes a NAG internal panic if I use the -C=dangling flag that ought to have detected the issue.

This is a compiler defect, but the workaround is trivial. Replaces ASSOCIATE block with wordier version of one statement.

On Macports buildbots we got a failure: https://trac.macports.org/ticket/68505

CMake Error at CMakeLists.txt:38 (find_package):
  By not providing "FindgFTL.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "gFTL", but
  CMake did not find one.

  Could not find a package configuration file provided by "gFTL" with any of
  the following names:

    gFTLConfig.cmake
    gftl-config.cmake

  Add the installation prefix of "gFTL" to CMAKE_PREFIX_PATH or set
  "gFTL_DIR" to a directory containing one of the above files.  If "gFTL"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!

Presumably, it is due to case-sensitivity, since both locally and on GH IC (which are set to be non-case-sensitive) the build is all good.

Recent development work is breaking with Intel 2021.10.0 in some obscure map scenario. Troubleshooting shows that it is related to the generic ASSIGNMENT(=) rather than relying on the compiler to do a deep copy.

A fix would be to add a CPP switch that disables the redefinition of assignment. For backward compatibility, "enabled" must remain the default. (Even though it is itself a workaround for possibly obsolete use cases/compiler versions.)

2 different (but hopefully related) issues have shown up when migrating GEOS to ifort 17. I suspect these are Intel compiler issues, possibly ones for which bug reports have already been submitted.

In the near term, a workaround is desired to allow GEOS to work.

There is still an issue here

As seen here:

the version number in gFTL is wrong for the release version (should be 1.1.1)

Thanks for the project.

Hi,

for the Debian package, I would need to put the files to places similar to the existing ones, which would be like this:

• Include files to /usr/include/gftl/ (with subdirs templates and types)
• cmake files to /usr/share/cmake/gftl/
• Makefile include file to /usr/share/gftl/gftl.mk

I tried to change the file paths via a patch, but when then building gftl-shared, the gftl include path is not set, and therefore the gftl include files are not found. That has been working with 1.2.7, but now stopped.
But maybe there is a simple way to put the install paths as parameters during the cmake run? Unfortunately, I am not good in CMAKE; could you give me some advise?

Thank you

Best regards, Ole