Code Monkey home page Code Monkey logo

gdal-warp-bindings's Introduction

CircleCI Maven Central Join the chat at https://gitter.im/geotrellis/geotrellis

Intro

Having multi-threaded access to raster data is an important prerequisite to constructing a high-performance GIS tile server. If one wishes to use GDAL's VRT functionality in this way, then it is necessary to ensure that no VRT dataset is simultaneously used by more than one thread, even for read-only operations. The code in this repository attempts to address that issue by wrapping GDAL datasets in objects which abstract one or more identical datasets; the wrapped datasets can be safely used from multiple threads with less contention than would be the case with a simple mutex around one GDAL dataset. APIs are provided for C and Java.

Installation

Beginning with gdal-warp-bindings version 3.7 we've adopted a new versioning scheme intended to simplify linking to native dependencies. Given a {major}.{minor}.{patch} version:

  • {major}.{minor} - matches the GDAL version it is published for
    • e.g. gdal-warp-bindings of version 3.7.x are suitable for GDAL versions 3.7.x major = 3, minor = 7
  • {patch} - this portion of the version corresponds to updates within gdal-warp-bindings and should remain compatible with GDAL library {major}.{minor} versions
    • This implies that there may be multiple gdal-warp-bindings releases for the same GDAL library version. All releases are compatible with the matching GDAL library {major}.{minor} version. Thus, higher patch versions are to be preferred.

These bindings require a GDAL installation on your machine with the appropriate matching version:

GDAL Warp Bindings OS GDAL Shared Library {so,dylib,dll}
3.9.0 Linux, MacOS, Windows 3.9.x libgdal.so.35.3.9.0
3.8.0 Linux, MacOS, Windows 3.8.x libgdal.so.34.3.8.x
3.7.0 Linux, MacOS, Windows 3.7.x libgdal.so.33.3.7.x
3.6.4 Linux, MacOS, Windows 3.6.x libgdal.so.33.3.6.x
1.1.x Linux (AMD64) 3.1.2 libgdal.so.27
1.1.x Linux (ARM64) 2.4.0 libgdal.so.20
1.1.x MacOS (AMD64) 3.1.2 libgdal.27.dylib
1.1.x Windows 3.0.4 --

MacOS

For MacOS users, you must also ensure that the appropriate shared libary file (dylib) above is symlinked to /usr/local/lib from your GDAL installation with matching version.

For best results, we recommend explicitly installing the appropriate GDAL version via Conda and symlinking the appropriate lib. Here's an example for GDAL Warp Bindings 1.1.0 using GDAL 3.1.2:

conda create -n gdal-3.1.2
conda activate gdal-3.1.2
conda install -c conda-forge gdal==3.1.2
sudo ln -s $(conda info --base)/envs/gdal-3.1.2/lib/libgdal.27.dylib /usr/local/lib/libgdal.27.dylib
conda deactivate

APIs

The C and Java APIs are very similar to each other.

GDAL must be initialized prior to use. In these bindings, that is accomplished by calling the init function in C or the GDALWarp.init static method in Java. It is not necessary to call the latter if the default parameters used in the static initializer are statisfactory, but it is okay to do so.

In both APIs, one requests a token (a uint64_t in C and a long in Java) corresponding to a URI, Warp Options pair by calling get_token in C or GDALWarp.get_token in Java.

That token is then passed as an argument to the various wrapper functions, much as one would pass the GDAL Dataset handle into the unwrapped GDAL functions. For example, get_overview_widths_heights (respectively GDALWarp.get_overview_widths_heights) takes a C uint64_t (respectively a Java long) as its first argument.

Both APIs are simple, C-style interfaces. If one is interested in seeing how these bindings can be elaborated into a fancier presentation, please the see geotrellis/geotrellis-contrib repository. There, you can see these bindings used within a Scala API.

C

Please see src/bindings.h for the full C/C++ interface.

Java

Please see src/main/java/com/azavea/gdal/GDALWarp.java for the full Java interface

SonaType Artifacts

The binary artifacts are present on SonaType. Snaphost artifacts are present on the SonaType Snapshots repo.

This jar file contains Linux, Macintosh, and Windows shared libraries.

All native binaries are built for AMD64; the Linux ones are linked against GDAL 2.4.3, The Macintosh ones are linked against GDAL 2.4.2 from Homebrew, and the Windows ones are linked against the GDAL 2.4.3 MSVC 2015 build from GISinternals.com.

The class files in the jar were built with OpenJDK 8.

The jar file is reachable via Maven:

<dependency>
  <groupId>com.azavea.geotrellis</groupId>
  <artifactId>gdal-warp-bindings</artifactId>
  <version>x.x.x</version>
  <type>pom</type>
</dependency>

Repository Structure

The Docker directory contains files used to generate images used for continuous integration testing and deployment. The src directory contains all of the source code for the library; the C/C++ files are in that directory. src/main contains the code for the Java API. src/unit_tests contains the C++ unit tests.

Ports

The repository contains everything needed to compile Linux (AMD64 and ARM64), Macintosh, and Windows versions of the library in a Docker container.

How to Build

All four can easily be compiled in the normal manner outside of a container if all dependencies are present.

Linux

If a recent GDAL and recent JDK are installed, the Linux version can be built by typing make -C src from the root directory of the cloned repository. If one only wishes to use the C/C++ library, then type make -C src libgdalwarp_bindings_amd64.so or ARCH=arm64 make -C src libgdalwarp_bindings_arm64.so.

Macintosh

If a recent GDAL is installed (perhaps from Homebrew) and a recent JDK is installed, the Macintosh version can be built on a Macintosh by typing OS=darwin SO=dylib make -C src from the root directory of the cloned repository. If one only wishes to use the C/C++ library, then type OS=darwin SO=dylib make -C src libgdalwarp_bindings_amd64.dylib.

Windows

The Windows version has only been cross-built with MinGW from within a Linux Docker container. Please see the test script for more.

gdal-warp-bindings's People

Contributors

cloudniner avatar jamesmcclain avatar jisantuc avatar pomadchin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

gdal-warp-bindings's Issues

Core dump on on linux in `GDALOverviewBand::FlushCache`

In the RasterFrames project, when running sbt core/test in CircleCI, the forked JVM process core dumps after the tests are (successfully) finished.

Error

First the process reports

pure virtual method called
terminate called without an active exception

then the JVM terminates with

sbt.ForkMain 38015 failed with exit code 134

Backtrace

gdb reports the following about the generated core file:

Core was generated by `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2G -Djava.library.path=/usr/'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fbebfbfb700 (LWP 11179))]
Installing openjdk unwinder
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 52, in <module>
    class Types(object):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 66, in Types
    nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fbf158da42a in __GI_abort () at abort.c:89
#2  0x00007fbf1466a0ad in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007fbf14668066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007fbf146680b1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fbf14668b8f in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fbf152287f9 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#7  0x00007fbf153cd5d8 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#8  0x00007fbf153cf45f in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#9  0x00007fbf15224ec5 in JVM_handle_linux_signal ()
   from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#10 0x00007fbf15218548 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#11 <signal handler called>
#12 0x00007fbe8764e6af in GDALOverviewBand::FlushCache (this=<optimized out>) at gdaloverviewdataset.cpp:564
#13 0x00007fbe876224e3 in GDALDataset::FlushCache (this=this@entry=0x7fbab99e5340) at gdaldataset.cpp:439
#14 0x00007fbe8764ece3 in GDALOverviewDataset::~GDALOverviewDataset (this=0x7fbab99e5340, __in_chrg=<optimized out>)
    at gdaloverviewdataset.cpp:226
#15 0x00007fbe8764ed49 in GDALOverviewDataset::~GDALOverviewDataset (this=0x7fbab99e5340, __in_chrg=<optimized out>)
    at gdaloverviewdataset.cpp:240
#16 0x00007fbe87623138 in GDALDataset::ReleaseRef (this=0x7fbab99e5340) at gdaldataset.cpp:1199
#17 0x00007fbe875b4261 in VRTWarpedDataset::CloseDependentDatasets (this=this@entry=0x7fbe147b9840)
    at vrtwarped.cpp:401
#18 0x00007fbe875b432a in VRTWarpedDataset::~VRTWarpedDataset (this=0x7fbe147b9840, __in_chrg=<optimized out>)
    at vrtwarped.cpp:354
#19 0x00007fbe875b4399 in VRTWarpedDataset::~VRTWarpedDataset (this=0x7fbe147b9840, __in_chrg=<optimized out>)
    at vrtwarped.cpp:355
#20 0x00007fbe87623138 in GDALDataset::ReleaseRef (this=0x7fbe147b9840) at gdaldataset.cpp:1199
#21 0x00007fbe875b41f8 in VRTWarpedDataset::CloseDependentDatasets (this=0x7fbab9928410) at vrtwarped.cpp:374
#22 0x00007fbe87632593 in GDALDriverManager::~GDALDriverManager (this=0x7fbe157971f0, __in_chrg=<optimized out>)
    at gdaldrivermanager.cpp:191
#23 0x00007fbe87632749 in GDALDriverManager::~GDALDriverManager (this=0x7fbe157971f0, __in_chrg=<optimized out>)
    at gdaldrivermanager.cpp:330
---Type <return> to continue, or q <return> to quit---
#24 0x00007fbe87632216 in GDALDestroyDriverManager () at gdaldrivermanager.cpp:892
#25 0x00007fbe8762e9bc in GDALDestroy () at gdaldllmain.cpp:77
#26 0x00007fbf1649ed6a in _dl_fini () at dl-fini.c:235
#27 0x00007fbf158db940 in __run_exit_handlers (status=0, listp=0x7fbf15c3f5d8 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83
#28 0x00007fbf158db99a in __GI_exit (status=<optimized out>) at exit.c:105
#29 0x00007fbf14fbe52c in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#30 0x00007fbf153d5fc7 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#31 0x00007fbf153d3677 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#32 0x00007fbf153d4af6 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#33 0x00007fbf153d4ff1 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#34 0x00007fbf1521a292 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#35 0x00007fbf162794a4 in start_thread (arg=0x7fbebfbfb700) at pthread_create.c:456
#36 0x00007fbf1598ed0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Machine

Linux 0678881d813c 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 GNU/Linux

JVM

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Libraries

  • "com.azavea.geotrellis" %% "geotrellis-contrib-vlm" % "2.11.0"
  • "com.azavea.geotrellis" %% "geotrellis-contrib-gdal" % "2.11.0"
  • rfSparkVersion := "2.3.2"
  • rfGeoTrellisVersion := "2.2.0"
  • rfGeoMesaVersion := "2.2.1"

Document release process

As a checklist or list of instructions somewhere in this repo.

The current process is approximately:

  • Ensure master is up to date
  • Update changelog like in 9bb7866 and commit
  • Tag CHANGELOG commit with git tag -a vx.y.z -m "vx.y.z" && git push --tags
  • Wait for CI to publish release to OSS SonaType
  • Login to OSS Sonatype https://oss.sonatype.org with credentials in password manager
  • Go to https://oss.sonatype.org/#stagingRepositories and filter by package gdal-warp-bindings
  • Verify that the JAR package contains the bundled shared libs
    image
  • Select staging repo and press Close button on the top of the table with repos
  • Wait for close process to complete
  • Select staging repo again and press Release button. It will be immediately published into sonatype releases repo and synced with maven central ~ in 10 minutes

GDAL 3.8: te parameter not respected if targetResolution is set

Describe the bug

After moving to GDAL 3.8, I see special behaviour when trying to set a target extent.
Normally, if 'te' is set in GDAL Warp options, this extent is respected, and can also be outside the original raster extent.
If however this parameter is set, in combination with a target resolution, even if that resolution matches the native resolution, the extent does not seem to be respected. (It's quite complex, easier to see in running code example below.)

Not yet clear if this is a GDAL issue/regression, or specific to this library.

To Reproduce

    val path = "T31UFS_20230405T105031_B04_10m.jp2"
    val opts = GDALWarpOptions(alignTargetPixels = true,cellSize=Some(CellSize(10.0,10.0)), targetCRS=Some(CRS.fromEpsgCode(32631)),
      te = Some(Extent(600000.0-200, 5590200.0, 709800.0, 5700000.0)), teCRS = Some(CRS.fromEpsgCode(32631))
    )
    val rs = GDALRasterSource(path, options = opts)
    println(rs.extent)

Also reproducible via GDALWarp:

gdalwarp -tap -tr 10.0 10.0 -te_srs EPSG:32631 -te 599800.0  5590200.0 709800.0 5700000.0 -t_srs EPSG:32631  T31UFS_20230405T105031_B04_10m.jp2 /tmp/out.tif

Expected behavior

Normally, the extent that gets printed should be the one provided in 'te'.

Environment

  • OS: Arch Linux
  • GDAL version: 3.8.1

Update to GDAL 3.7.x

Once #99 is merged, so that brew install gdal on MacOS works out of the box again for these bindings and geotrellis.

Feature Request: Option to disable log message colorization

The use of ANSI color in GDAL log and debug messages is helpful during quick
monitoring of running jobs, but adds a lot of visual noise to filter through when doing after-the-fact analysis and forensics. The image below shows a running job on the left, and review of a log file from the previous run.

Screen Shot 2020-02-06 at 10 12 26 AM

Having a GDAL configuration flag to disable colorization would be a nice addition to speed up debugging and tracking down errors.

Add Mac OS dylib build publishment.

Right now if you'll try to run this PR on Mac OS the following error would occur:

[info] GDALConvertedRasterSourceSpec:
[info] geotrellis.contrib.vlm.gdal.GDALReprojectRasterSourceSpec *** ABORTED *** (1 millisecond)
[info]   java.lang.UnsatisfiedLinkError: /private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: dlopen(/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so, 1): no suitable image found.  Did find:
[info] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[info] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[info]   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
[info]   at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
[info]   at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
[info]   at java.lang.Runtime.load0(Runtime.java:809)
[info]   at java.lang.System.load(System.java:1086)
[info]   at cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:109)
[info]   at com.azavea.gdal.GDALWarp.init(GDALWarp.java:51)
[info]   at geotrellis.contrib.vlm.gdal.GDALReprojectRasterSourceSpec$$anonfun$1.apply$mcV$sp(GDALReprojectRasterSourceSpec.scala:48)
[info]   at org.scalatest.SuperEngine.registerNestedBranch(Engine.scala:595)
[info]   at org.scalatest.FunSpecLike$class.describe(FunSpecLike.scala:382)
[info]   at org.scalatest.FunSpec.describe(FunSpec.scala:1630)
[info]   at geotrellis.contrib.vlm.gdal.GDALReprojectRasterSourceSpec.<init>(GDALReprojectRasterSourceSpec.scala:47)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[info]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[info]   at java.lang.Class.newInstance(Class.java:442)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:304)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)
[info] geotrellis.contrib.vlm.gdal.GDALConvertedRasterSourceSpec *** ABORTED *** (1 millisecond)
[info]   java.lang.UnsatisfiedLinkError: /private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: dlopen(/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so, 1): no suitable image found.  Did find:
[info] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[info] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[info]   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
[info]   at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
[info]   at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
[info]   at java.lang.Runtime.load0(Runtime.java:809)
[info]   at java.lang.System.load(System.java:1086)
[info]   at cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:109)
[info]   at com.azavea.gdal.GDALWarp.init(GDALWarp.java:51)
[info]   at geotrellis.contrib.vlm.gdal.GDALConvertedRasterSourceSpec.<init>(GDALConvertedRasterSourceSpec.scala:37)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[info]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[info]   at java.lang.Class.newInstance(Class.java:442)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:304)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)
[error] Uncaught exception when running geotrellis.contrib.vlm.gdal.GDALConvertedRasterSourceSpec: java.lang.UnsatisfiedLinkError: /private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: dlopen(/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so, 1): no suitable image found.  Did find:
[error] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[error] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[error] sbt.ForkMain$ForkError: java.lang.UnsatisfiedLinkError: /private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: dlopen(/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so, 1): no suitable image found.  Did find:
[error] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[error] 	/private/var/folders/r3/gkv3lq7n5jq4cfbsx0h8jbcm0000gn/T/nativeutils128213686854015/libgdalwarp_bindings.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03
[error] 	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
[error] 	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
[error] 	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
[error] 	at java.lang.Runtime.load0(Runtime.java:809)
[error] 	at java.lang.System.load(System.java:1086)
[error] 	at cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:109)
[error] 	at com.azavea.gdal.GDALWarp.init(GDALWarp.java:51)
[error] 	at geotrellis.contrib.vlm.gdal.GDALConvertedRasterSourceSpec.<init>(GDALConvertedRasterSourceSpec.scala:37)
[error] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error] 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] 	at java.lang.Class.newInstance(Class.java:442)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
[error] 	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:304)
[error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] 	at java.lang.Thread.run(Thread.java:748)

Mac OS dylib should be included into the resources directory of the gdal-warp-bindings artifact as well.

screenshot 2019-03-07 at 15 41 16

Crash when CPL_DEBUG=ON

Describe the bug

When reading multiple jpeg2000 files and CPL_DEBUG=ON, the jvm crashes with this error.
Turning off CPL_DEBUG seems to solve it.


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f3cefdd3844, pid=15, tid=71
#
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.14+9) (build 11.0.14+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.14+9-LTS, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# C  [libgdalwarp_bindings.so+0x4a844]  std::_Rb_tree<int, std::pair<int const, std::tuple<int, std::chrono::duration<long, std::ratio<1l, 1000l> > > >, std::_Select1st<std::pair<int const, std::tuple<int, std::chrono::duration<long, std::ratio<1l, 1000l> > > > >, std::less<int>, std::allocator<std::pair<int const, std::tuple<int, std::chrono::duration<long, std::ratio<1l, 1000l> > > > > >::_M_begin()+0xc
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /opt/core.15)
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid15.log
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-11-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Versioning tags release and latest are not updated on publish

maven-metadata.xml versioning.release and versioning.latest tags are not updated:

<metadata>
  <groupId>com.azavea.geotrellis</groupId>
  <artifactId>gdal-warp-bindings</artifactId>
  <versioning>
    <latest>33.567d940</latest>
    <release>33.567d940</release>
...

The task is to investigate what is wrong with it, mb nexus-staging-maven-plugin is outdated and we need to update it up to smth more recent.

SIGSEGV On MacOS

Reproduce

Clone s22s/rasterframes, branch feature/gdal-warp-upgrade.

Or https://github.com/s22s/rasterframes/tree/feature/gdal-warp-upgrade

Then run:

sbt core/it:testOnly org.locationtech.rasterframes.ref.RasterSourceIT

Result

Stop of stack

Stack: [0x0000700004127000,0x0000700004227000],  sp=0x00007000042231a0,  free space=1008k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libgdal.20.dylib+0x8f4aad]  GDAL_LercNS::BitMask::CountValidBits() const+0x29
C  [libgdal.20.dylib+0x16a2ca]  GDAL_MRF::LERC_Band::Decompress(GDAL_MRF::buf_mgr&, GDAL_MRF::buf_mgr&)+0x39a
C  [libgdal.20.dylib+0x33c394]  GDAL_MRF::GDALMRFRasterBand::IReadBlock(int, int, void*)+0x47c
C  [libgdal.20.dylib+0x4df5d1]  GDALRasterBand::GetLockedBlockRef(int, int, int)+0x175
C  [libgdal.20.dylib+0x4f3254]  GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*)+0x8ac
C  [libgdal.20.dylib+0x4de979]  GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*)+0x2c1
C  [libgdal.20.dylib+0x10d53b]  GDALWarpOperation::WarpRegionToBuffer(int, int, int, int, void*, GDALDataType, int, int, int, int, double, double, double, double)+0x2bb
C  [libgdal.20.dylib+0x10e1f8]  GDALWarpOperation::WarpRegionToBuffer(int, int, int, int, void*, GDALDataType, int, int, int, int, double, double)+0x3a
C  [libgdal.20.dylib+0x457fe2]  VRTWarpedDataset::ProcessBlock(int, int)+0xe2
C  [libgdal.20.dylib+0x4583bb]  VRTWarpedRasterBand::IReadBlock(int, int, void*)+0x39
C  [libgdal.20.dylib+0x4df5d1]  GDALRasterBand::GetLockedBlockRef(int, int, int)+0x175
C  [libgdal.20.dylib+0x4e4aab]  GDALRasterBand::ComputeRasterMinMax(int, double*)+0x4dd
C  [libgdalwarp_bindings.dylib+0x96ea]  locked_dataset::get_band_max_min(int, int, int, double*, int*)+0x7a
C  [libgdalwarp_bindings.dylib+0x949f]  get_band_min_max+0x54f
C  [libgdalwarp_bindings.dylib+0x1ec8]  Java_com_azavea_gdal_GDALWarp_get_1band_1min_1max+0xb8
J 3811  com.azavea.gdal.GDALWarp.get_band_min_max(JIIIZ[D[I)I (0 bytes) @ 0x000000010a0afc77 [0x000000010a0afb80+0xf7]
J 3807 C1 geotrellis.contrib.vlm.gdal.GDALDataset$.cellType$extension1(JI)Lgeotrellis/raster/DataType; (192 bytes) @ 0x000000010a0b31b4 [0x000000010a0b2500+0xcb4]
j  geotrellis.contrib.vlm.gdal.GDALDataset$.cellType$extension0(J)Lgeotrellis/raster/DataType;+3
j  geotrellis.contrib.vlm.gdal.GDALBaseRasterSource$$anonfun$cellType$1.apply()Lgeotrellis/raster/DataType;+12
j  geotrellis.contrib.vlm.gdal.GDALBaseRasterSource$$anonfun$cellType$1.apply()Ljava/lang/Object;+1
j  scala.Option.getOrElse(Lscala/Function0;)Ljava/lang/Object;+8
j  geotrellis.contrib.vlm.gdal.GDALBaseRasterSource$class.cellType(Lgeotrellis/contrib/vlm/gdal/GDALBaseRasterSource;)Lgeotrellis/raster/DataType;+14
j  geotrellis.contrib.vlm.gdal.GDALRasterSource.cellType$lzycompute()Lgeotrellis/raster/DataType;+17
j  geotrellis.contrib.vlm.gdal.GDALRasterSource.cellType()Lgeotrellis/raster/DataType;+12

Full report:
hs_err_pid38339.log.zip

Environment

GDAL

$ /usr/local/bin/gdal-config --version
2.4.1
$ /usr/local/bin/gdalinfo --formats
Supported Formats:
  VRT -raster- (rw+v): Virtual Raster
  DERIVED -raster- (ro): Derived datasets using VRT pixel functions
  GTiff -raster- (rw+vs): GeoTIFF
  NITF -raster- (rw+vs): National Imagery Transmission Format
  RPFTOC -raster- (rovs): Raster Product Format TOC format
  ECRGTOC -raster- (rovs): ECRG TOC format
  HFA -raster- (rw+v): Erdas Imagine Images (.img)
  SAR_CEOS -raster- (rov): CEOS SAR Image
  CEOS -raster- (rov): CEOS Image
  JAXAPALSAR -raster- (rov): JAXA PALSAR Product Reader (Level 1.1/1.5)
  GFF -raster- (rov): Ground-based SAR Applications Testbed File Format (.gff)
  ELAS -raster- (rw+v): ELAS
  AIG -raster- (rov): Arc/Info Binary Grid
  AAIGrid -raster- (rwv): Arc/Info ASCII Grid
  GRASSASCIIGrid -raster- (rov): GRASS ASCII Grid
  SDTS -raster- (rov): SDTS Raster
  DTED -raster- (rwv): DTED Elevation Raster
  PNG -raster- (rwv): Portable Network Graphics
  JPEG -raster- (rwv): JPEG JFIF
  MEM -raster- (rw+): In Memory Raster
  JDEM -raster- (rov): Japanese DEM (.mem)
  GIF -raster- (rwv): Graphics Interchange Format (.gif)
  BIGGIF -raster- (rov): Graphics Interchange Format (.gif)
  ESAT -raster- (rov): Envisat Image Format
  FITS -raster- (rw+): Flexible Image Transport System
  BSB -raster- (rov): Maptech BSB Nautical Charts
  XPM -raster- (rwv): X11 PixMap Format
  BMP -raster- (rw+v): MS Windows Device Independent Bitmap
  DIMAP -raster- (rov): SPOT DIMAP
  AirSAR -raster- (rov): AirSAR Polarimetric Image
  RS2 -raster- (rovs): RadarSat 2 XML Product
  SAFE -raster- (rov): Sentinel-1 SAR SAFE Product
  PCIDSK -raster,vector- (rw+v): PCIDSK Database File
  PCRaster -raster- (rw+): PCRaster Raster File
  ILWIS -raster- (rw+v): ILWIS Raster Map
  SGI -raster- (rw+v): SGI Image File Format 1.0
  SRTMHGT -raster- (rwv): SRTMHGT File Format
  Leveller -raster- (rw+v): Leveller heightfield
  Terragen -raster- (rw+v): Terragen heightfield
  GMT -raster- (rw): GMT NetCDF Grid Format
  netCDF -raster,vector- (rw+s): Network Common Data Format
  HDF4 -raster- (ros): Hierarchical Data Format Release 4
  HDF4Image -raster- (rw+): HDF4 Dataset
  ISIS3 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 3)
  ISIS2 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 2)
  PDS -raster- (rov): NASA Planetary Data System
  PDS4 -raster- (rw+vs): NASA Planetary Data System 4
  VICAR -raster- (rov): MIPL VICAR file
  TIL -raster- (rov): EarthWatch .TIL
  ERS -raster- (rw+v): ERMapper .ers Labelled
  JP2OpenJPEG -raster,vector- (rwv): JPEG-2000 driver based on OpenJPEG library
  L1B -raster- (rovs): NOAA Polar Orbiter Level 1b Data Set
  FIT -raster- (rwv): FIT Image
  GRIB -raster- (rwv): GRIdded Binary (.grb, .grb2)
  JPEG2000 -raster,vector- (rwv): JPEG-2000 part 1 (ISO/IEC 15444-1), based on Jasper library
  RMF -raster- (rw+v): Raster Matrix Format
  WCS -raster- (rovs): OGC Web Coverage Service
  WMS -raster- (rwvs): OGC Web Map Service
  MSGN -raster- (rov): EUMETSAT Archive native (.nat)
  RST -raster- (rw+v): Idrisi Raster A.1
  INGR -raster- (rw+v): Intergraph Raster
  GSAG -raster- (rwv): Golden Software ASCII Grid (.grd)
  GSBG -raster- (rw+v): Golden Software Binary Grid (.grd)
  GS7BG -raster- (rw+v): Golden Software 7 Binary Grid (.grd)
  COSAR -raster- (rov): COSAR Annotated Binary Matrix (TerraSAR-X)
  TSX -raster- (rov): TerraSAR-X Product
  COASP -raster- (ro): DRDC COASP SAR Processor Raster
  R -raster- (rwv): R Object Data Store
  MAP -raster- (rov): OziExplorer .MAP
  KMLSUPEROVERLAY -raster- (rwv): Kml Super Overlay
  WEBP -raster- (rwv): WEBP
  Rasterlite -raster- (rwvs): Rasterlite
  MBTiles -raster,vector- (rw+v): MBTiles
  PLMOSAIC -raster- (ro): Planet Labs Mosaics API
  CALS -raster- (rwv): CALS (Type 1)
  WMTS -raster- (rwv): OGC Web Map Tile Service
  SENTINEL2 -raster- (rovs): Sentinel 2
  MRF -raster- (rw+v): Meta Raster Format
  PNM -raster- (rw+v): Portable Pixmap Format (netpbm)
  DOQ1 -raster- (rov): USGS DOQ (Old Style)
  DOQ2 -raster- (rov): USGS DOQ (New Style)
  PAux -raster- (rw+v): PCI .aux Labelled
  MFF -raster- (rw+v): Vexcel MFF Raster
  MFF2 -raster- (rw+): Vexcel MFF2 (HKV) Raster
  FujiBAS -raster- (rov): Fuji BAS Scanner Image
  GSC -raster- (rov): GSC Geogrid
  FAST -raster- (rov): EOSAT FAST Format
  BT -raster- (rw+v): VTP .bt (Binary Terrain) 1.3 Format
  LAN -raster- (rw+v): Erdas .LAN/.GIS
  CPG -raster- (rov): Convair PolGASP
  IDA -raster- (rw+v): Image Data and Analysis
  NDF -raster- (rov): NLAPS Data Format
  EIR -raster- (rov): Erdas Imagine Raw
  DIPEx -raster- (rov): DIPEx
  LCP -raster- (rwv): FARSITE v.4 Landscape File (.lcp)
  GTX -raster- (rw+v): NOAA Vertical Datum .GTX
  LOSLAS -raster- (rov): NADCON .los/.las Datum Grid Shift
  NTv1 -raster- (rov): NTv1 Datum Grid Shift
  NTv2 -raster- (rw+vs): NTv2 Datum Grid Shift
  CTable2 -raster- (rw+v): CTable2 Datum Grid Shift
  ACE2 -raster- (rov): ACE2
  SNODAS -raster- (rov): Snow Data Assimilation System
  KRO -raster- (rw+v): KOLOR Raw
  ROI_PAC -raster- (rw+v): ROI_PAC raster
  RRASTER -raster- (rw+v): R Raster
  BYN -raster- (rw+v): Natural Resources Canada's Geoid
  ARG -raster- (rwv): Azavea Raster Grid format
  RIK -raster- (rov): Swedish Grid RIK (.rik)
  USGSDEM -raster- (rwv): USGS Optional ASCII DEM (and CDED)
  GXF -raster- (rov): GeoSoft Grid Exchange Format
  DODS -raster- (ro): DAP 3.x servers
  BAG -raster- (rwv): Bathymetry Attributed Grid
  HDF5 -raster- (rovs): Hierarchical Data Format Release 5
  HDF5Image -raster- (rov): HDF5 Dataset
  NWT_GRD -raster- (rw+v): Northwood Numeric Grid Format .grd/.tab
  NWT_GRC -raster- (rov): Northwood Classified Grid Format .grc/.tab
  ADRG -raster- (rw+vs): ARC Digitized Raster Graphics
  SRP -raster- (rovs): Standard Raster Product (ASRP/USRP)
  BLX -raster- (rwv): Magellan topo (.blx)
  EPSILON -raster- (rwv): Epsilon wavelets
  PostGISRaster -raster- (rws): PostGIS Raster driver
  SAGA -raster- (rw+v): SAGA GIS Binary Grid (.sdat, .sg-grd-z)
  XYZ -raster- (rwv): ASCII Gridded XYZ
  HF2 -raster- (rwv): HF2/HFZ heightfield raster
  OZI -raster- (rov): OziExplorer Image File
  CTG -raster- (rov): USGS LULC Composite Theme Grid
  E00GRID -raster- (rov): Arc/Info Export E00 GRID
  ZMap -raster- (rwv): ZMap Plus Grid
  NGSGEOID -raster- (rov): NOAA NGS Geoid Height Grids
  IRIS -raster- (rov): IRIS data (.PPI, .CAPPi etc)
  PRF -raster- (rov): Racurs PHOTOMOD PRF
  RDA -raster- (ro): DigitalGlobe Raster Data Access driver
  EEDAI -raster- (ros): Earth Engine Data API Image
  SIGDEM -raster- (rwv): Scaled Integer Gridded DEM .sigdem
  IGNFHeightASCIIGrid -raster- (rov): IGN France height correction ASCII Grid
  GPKG -raster,vector- (rw+vs): GeoPackage
  CAD -raster,vector- (rovs): AutoCAD Driver
  PLSCENES -raster,vector- (ro): Planet Labs Scenes API
  NGW -raster,vector- (rw+s): NextGIS Web
  GenBin -raster- (rov): Generic Binary (.hdr Labelled)
  ENVI -raster- (rw+v): ENVI .hdr Labelled
  EHdr -raster- (rw+v): ESRI .hdr Labelled
  ISCE -raster- (rw+v): ISCE raster
  HTTP -raster,vector- (ro): HTTP Fetching Wrapper

Machine

MacOS 10.13.6 (17G6030)
Darwin DongleBookPro.s22s.net 17.7.0 Darwin Kernel Version 17.7.0: Wed Feb 27 00:43:23 PST 2019; root:xnu-4570.71.35~1/RELEASE_X86_64 x86_64

JVM

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Libraries

  • "com.azavea.gdal" % "gdal-warp-bindings" % "33.58d4965"
  • "com.azavea.geotrellis" %% "geotrellis-contrib-vlm" % "2.11.0"
  • "com.azavea.geotrellis" %% "geotrellis-contrib-gdal" % "2.11.0"
  • rfSparkVersion := "2.3.2"
  • rfGeoTrellisVersion := "2.2.0"
  • rfGeoMesaVersion := "2.2.1"

GitHub Actions release

We need to migrate from CircleCI to GitHub Actions, and overall resurrect builds, I think they are dead.

Early Detection of File Not Found

scala> val token = GDALWarp.get_token("/tmp/aspect.jp2", Array())
token: Long = -6799779891867457380
scala> GDALWarp.get_band_count(token, GDALWarpOptions.EMPTY.datasetType, numberOfAttempts, Array.ofDim[Int](1))
/tmp/aspect.jp2: No such file or directory
res5: Int = -1

3.7.0 Release

Hey gang, lets cut a release? Any objections? If we don't do it for too long we're at a risk of getting behind GDAL 3.8.0 soon.

About versioning: it feels like we want to adopt a new versioning scheme; 3.7.0 (thread: #100 (comment)). Just making sure we're on the same page.

Any πŸ‘ πŸ‘Ž ?

gdalwarp_bindings.dll: Can't find dependent libraries

Describe the bug

when I run this code, throw an Exception:

import com.azavea.gdal.GDALWarp;

public class GdalTest {
  public static void main(String args[]) throws Exception {
    GDALWarp.init(256);
  }
}

To Reproduce

Steps to reproduce the behavior:

  1. add gdal-warp-bindings to pom.xml
  2. run a GdalTest class
  3. throw Exception like this: gdalwarp_bindings.dll: Can't find dependent libraries

Expected behavior

Exception like this:

Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Users\LENOVO\AppData\Local\Temp\nativeutils794907385528900\gdalwarp_bindings.dll: Can't find dependent libraries
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:109)
	at com.azavea.gdal.GDALWarp.init(GDALWarp.java:132)
	at com.azavea.gdal.GDALWarp.<clinit>(GDALWarp.java:142)
	at GdalTest.main(GdalTest.java:5)

Environment

  • OS: Win 10
  • JVM version: 1.8
  • GDAL version: not install

Return GDAL Error Codes on Failure

Right now, we just return a negative number or a Boolean instead of an error code when an operation fails (see ex 1 and ex 2). It would be helpful instead to return this code, as it would provide more helpful information to the user.

Publish project to SonaType

This project is currently dependency of geotrellis-gdal which is published to SonaType for public and easy consumption. However since goal-warp-bindings are not similarly published the geotrellis project can't be fully resolved.

Specifically this will enable pulling in geotrellis into spark shell and notebooks with --packages flag.

Build for GDAL 3.8.1

GDAL<3.8 performs too many reads

An important issue was solved in GDAL 3.8.0:
/vsicurl/: avoid the same region to be downloaded at the same time from concurrent threads (#8041)

We use this library to perform multithreaded reads from S3, which uses vsicurl.
A simple upgrade did not work, because GDALWarp.get_crs_proj4 returned an empty string, so breaking the projection.
A local rebuild against 3.8.1 seems to help.

Would it be possible to get an official 3.8.1 build?

NETCDF / HDF5 Metadata retrieval issue

Describe the bug

Getting the metadata of the NETCDF file can not be performed.
So performing a GDALWarp.get_metadata_domain_list(token, datasetType.value, numberOfAttempts, band, arr) of a file that consists of multiple SUBDATASETS seems to be not possible, even with reading the metadata for the entire file (specifying the band = 0))

To Reproduce

  1. download a file (rarchv.2008_028_00_2d.nc4) (contact me if you need a copy)
  2. retrieve the metadata:
def getMetadataDomainList(token: Long)(datasetType: DatasetType, band: Int): List[GDALMetadataDomain] = {
    val arr = Array.ofDim[Byte](100, 1 << 10)
    val returnValue = GDALWarp.get_metadata_domain_list(token, datasetType.value, numberOfAttempts, band, arr)

    if (returnValue <= 0) {
      val positiveValue = math.abs(returnValue)
      throw new MalformedProjectionException(
        s"Unable to get the metadata domain list. GDAL Error Code: $positiveValue",
        positiveValue
      )
    }

    (arr.map(new String(_, "UTF-8").trim).filter(_.nonEmpty).toList.map(UserDefinedDomain)).distinct
  }

getMetadataDomainList(token)(SOURCE, 0)

Expected behavior

Should behave like usual GDAL JNI bindings:

>>> ds.GetMetadata('SUBDATASETS')
{'SUBDATASET_1_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":qtot', 'SUBDATASET_1_DESC': '[1x2170x4500] surface_downward_heat_flux_in_air (32-bit floating-point)', 'SUBDATASET_2_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":emp', 'SUBDATASET_2_DESC': '[1x2170x4500] water_flux_into_ocean (32-bit floating-point)', 'SUBDATASET_3_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":surface_temperature_trend', 'SUBDATASET_3_DESC': '[1x2170x4500] surface_temperature_trend (32-bit floating-point)', 'SUBDATASET_4_NAME': 'NETCDF:"archv.2008_028_00_2d.nc4":surface_salinity_trend', 'SUBDATASET_4_DESC': '[1x2170x4500] surface_salinity_trend (32-bit floating-point)', 'SUBDATASET_5_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":ssh', 'SUBDATASET_5_DESC': '[1x2170x4500] sea_surface_elevation (32-bit floating-point)', 'SUBDATASET_6_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":surface_boundary_layer_thickness', 'SUBDATASET_6_DESC': '[1x2170x4500] surface_boundary_layer_thickness (32-bit floating-point)', 'SUBDATASET_7_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":mixed_layer_thickness', 'SUBDATASET_7_DESC': '[1x2170x4500] ocean_mixed_layer_thickness (32-bit floating-point)', 'SUBDATASET_8_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":mixed_layer_temperature', 'SUBDATASET_8_DESC': '[1x2170x4500] mixed_layer_temperature (32-bit floating-point)'}

Environment

  • OS: Mac OS 10.14.6
  • JVM version: 1.8.0_162
  • GDAL version: GDAL 2.4.2, released 2019/06/28

Provide mechanism for passing along GDAL settings

Not sure if this is the right place for this ticket

Being able to support the following type of thing was really helpful in the old Swig variant.

gdal {
  settings {
    options {
      // See https://trac.osgeo.org/gdal/wiki/ConfigOptions for options
      CPL_DEBUG = "ON"
    }
  }
}

Q: How to determine buffer sizes to use for metadata query?

I'm looking at pulling metadata using these APIs. Each of the three calls requires a pre-allocated byte-buffer for storing the results. I'm not super familiar with the concept of "metadata domains", or if there are limitations on the size of values in the metadata set. Is there a way of determining what the actual or maximum sizes might be, or some guidelines one might use.

MacOS Catalina fails to load libgdal.27.dylib

Describe the bug

MacOS is unable to link the libgdal.27.dylib when loading GDALRasterSource from geotrellis-gdal package.

Complete error:

[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/97/nzqwjmr53kzbnxbl6xmj1gx40000gp/T/nativeutils159976180077139/libgdalwarp_bindings.dylib: dlopen(/private/var/folders/97/nzqwjmr53kzbnxbl6xmj1gx40000gp/T/nativeutils159976180077139/libgdalwarp_bindings.dylib, 1): Library not loaded: @@HOMEBREW_PREFIX@@/opt/gdal/lib/libgdal.27.dylib
[error]   Referenced from: /private/var/folders/97/nzqwjmr53kzbnxbl6xmj1gx40000gp/T/nativeutils159976180077139/libgdalwarp_bindings.dylib
[error]   Reason: image not found
[error] 	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
[error] 	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383)
[error] 	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227)
[error] 	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169)
[error] 	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2407)
[error] 	at java.base/java.lang.Runtime.load0(Runtime.java:747)
[error] 	at java.base/java.lang.System.load(System.java:1857)
[error] 	at cz.adamh.utils.NativeUtils.loadLibraryFromJar(NativeUtils.java:109)
[error] 	at com.azavea.gdal.GDALWarp.init(GDALWarp.java:128)
[error] 	at com.azavea.gdal.GDALWarp.<clinit>(GDALWarp.java:142)
[error] 	at geotrellis.raster.gdal.GDALDataset$.apply(GDALDataset.scala:338)
[error] 	at geotrellis.raster.gdal.GDALRasterSource.dataset$lzycompute(GDALRasterSource.scala:53)
[error] 	at geotrellis.raster.gdal.GDALRasterSource.dataset(GDALRasterSource.scala:52)
[error] 	at geotrellis.raster.gdal.GDALRasterSource.gridExtent$lzycompute(GDALRasterSource.scala:93)
[error] 	at geotrellis.raster.gdal.GDALRasterSource.gridExtent(GDALRasterSource.scala:93)
[error] 	at geotrellis.raster.RasterMetadata.extent(RasterMetadata.scala:52)
[error] 	at geotrellis.raster.RasterMetadata.extent$(RasterMetadata.scala:52)
[error] 	at geotrellis.raster.RasterSource.extent(RasterSource.scala:43)
[error] 	at Main$.main(Main.scala:6)
[error] 	at Main.main(Main.scala)

To Reproduce

This sbt project reproduces the error when the environment conditions below are satisfied.

macos-gdal-link-failure.zip

Unzip then:

./sbt
run

Expected behavior

The project should successfully print the extent of whatever raster you provide and not throw a link error.

Environment

Additional context

@pomadchin is able to load the dylib and view an extent when he used an old version of Java installed by the old Mac installer that places an entry into the MacOS System Preferences window, [email protected]. This is no longer the supported way to install Java on MacOS, instead you download the JRE/JDK you need directly from Oracle. Does Catalina do more fancy gatekeeper stuff that magically prevents some user-installed dylibs from loading? The error trace has refs to libgdalwarp_bindings.dylib moved or linked to from /private/var.

@pomadchin and I were unable to resolve this error by adding the following to build.sbt in the example project:

javaOptions ++= Seq("-Djava.library.path=/usr/local/lib")
fork := true

Here's my gdal dylibs:

➜  ~/src/gdal-rs-test ls -l /usr/local/lib | grep gdal
lrwxr-xr-x   1 andrew  admin    43 Oct 14 11:30 libgdal.27.dylib -> ../Cellar/gdal/3.1.3_3/lib/libgdal.27.dylib
lrwxr-xr-x   1 andrew  admin    36 Oct 14 11:30 libgdal.a -> ../Cellar/gdal/3.1.3_3/lib/libgdal.a
lrwxr-xr-x   1 andrew  admin    40 Oct 14 11:30 libgdal.dylib -> ../Cellar/gdal/3.1.3_3/lib/libgdal.dylib

This is likely some specific instance of dylib loading failure that would be resolved by #73, but I'm recording it here as a separate issue because it provides a complete example with specific versions of JVM, GDAL and SBT that were tested by @pomadchin and I.

Release 1.2.0 with the initial ARM support

I'm pretty interested in the fresh release (even as is) that includes this PR #107.
It's still not Mac M1 support which could be ideal, but better than nothing: simplifies (performance-wise) binaries usage in ARM docker.

`get_version_info` is padded with 0x00

A minor issue, but it would be nice if the results from get_version_info were trimmed.

import com.azavea.gdal.GDALWarp
GDALWarp.get_version_info("--version")

> res0: String = GDAL 2.4.1, released 2019/03/15?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

The ? values are actually 0x00.

GDALRasterSource unable to read PAM metadata files

GDALRasterSource, using these bindings, throws an error when attempting to read metadata for files that use the PAM xml metadata specification.

Open a GeoTrellis 3.5.1+ SBT console:

cd path/to/geotrellis
./sbt "project gdal"
> console

Then try to read metadata of a tiff with external PAM metadata using GDALRasterSource

scala> import geotrellis.raster.gdal.GDALRasterSource
scala> val rs = GDALRasterSource("s3://aviris-data/test/f130329t01p00r06_corr_v1.tiff")
scala> rs.metadata
geotrellis.raster.gdal.MalformedProjectionException: Unable to get the metadata. GDAL Error Code: 1
  at geotrellis.raster.gdal.GDALDataset$.getMetadata$extension3(GDALDataset.scala:73)
  at geotrellis.raster.gdal.GDALDataset$.$anonfun$getMetadata$1(GDALDataset.scala:61)
  at scala.collection.immutable.List.map(List.scala:293)
  at geotrellis.raster.gdal.GDALDataset$.getMetadata$extension1(GDALDataset.scala:61)
  at geotrellis.raster.gdal.GDALMetadata$.apply(GDALMetadata.scala:49)
  at geotrellis.raster.gdal.GDALRasterSource.metadata$lzycompute(GDALRasterSource.scala:59)
  at geotrellis.raster.gdal.GDALRasterSource.metadata(GDALRasterSource.scala:59)
  ... 36 elided

See:

  • s3://aviris-data/test/f130329t01p00r06_corr_v1.tiff
  • s3://aviris-data/test/f130329t01p00r06_corr_v1.tiff.aux.xml

I'm able to read all metadata for this COG with gdalinfo, the Python GDAL bindings, and Rasterio. For example, gdalinfo /vsis3/aviris-data/test/f130329t01p00r06_corr_v1.tiff prints all metadata contained in associated xml file.

Static linking of MinGW libs

Testing the bindings on windows I found dynamic linking against MinGW:

Dependencies (x64) 3_11_2020 9_26_08 PM

Getting installation of MinGW seems like a pretty high bar for using the bindings. At least not one I could clear with an hour of time. It looks like even locating the GDAL dll requires the use of -Djava.library.path=\Progra~1\GDAL parameter for the JVM.

I propose we statically link the MinGW dependencies, it inflates the dll size only slightly from 1.6MB to 2.1MB. Testing this approach appears to work as far as the dependency checker is concerned at least:

docker run -it --rm \
      -v $(pwd):/workdir \
      -e CROSS_TRIPLE=x86_64-w64-mingw32 \
      -e OS=win32 \
      -e CFLAGS="-Wall -Werror -Os -g" \
      -e JAVA_HOME="/windows/jdk8u202-b08" \
      -e GDALCFLAGS="-I/usr/local/include" \
      -e BOOST_ROOT="/usr/local/include/boost_1_69_0" \
      -e LDFLAGS="-L/windows/gdal/lib -lgdal_i -static-libgcc -static-libstdc++ -Wl,-Bstatic -lstdc++ -lpthread -Wl,-Bdynamic -lws2_32" \
      jamesmcclain/gdal-build-environment:4 make -j4 -C src gdalwarp_bindings.dll || exit -1

Dependencies (x64) 3_11_2020 10_05_08 PM

Add Documentation and Comments

This can be done at any time, but it might be advisable to wait until after testing when it will be known how much has to change.

Consider switching to semver

The library becomes more mature and users start to realy on it. We probably need to consider switching to semver and adding a CHANGELOG that will help users to track down any changes / bugfixes that can be critical.

MacOS build assumes libgdal via Homebrew

Evidenced by looking at the dylib loading rules:

$ curl -o gdal-warp.zip https://repo1.maven.org/maven2/com/azavea/geotrellis/gdal-warp-bindings/33.567d940/gdal-warp-bindings-33.567d940.jar
$ unzip gdal-warp.zip
$ otool -L resources/libgdalwarp_bindings.dylib

resources/libgdalwarp_bindings.dylib:
	libgdalwarp_bindings.dylib (compatibility version 0.0.0, current version 0.0.0)
	@@HOMEBREW_PREFIX@@/opt/gdal/lib/libgdal.20.dylib (compatibility version 26.0.0, current version 26.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Build build should be updated to rely on the standard system lookup rules. Not sure how or what is injecting the @@HOMEBREW_PREFIX@@ bit.

The common use case is that users will have gdal via a conda or pip install.

Add'l context: https://gitter.im/geotrellis/geotrellis?at=5dcdbf95ea7d147cb37d438c

Assertion failed on parallel reads

Describe the bug

I am developing a job in RasterFrames that uses this library and I (sometimes) receive several illegal argument messages, then finally a failed assertion error, process crash and core dump. The job is reading many JP2 (sentinel 2 PDS) files in parallel. The error seems to happen on non-deteriministic files. I have tried reading the files mentioned in the errors and the errors do not occur outside the context of the larger job, and I believe, running on many cores.

A sample of the illegal argument messages:

[116 of 1000] FAILURE(3) CPLE_IllegalArg(5) "Illegal argument." /vsis3/sentinel-s2-l2a/tiles/22/L/FP/2019/8/14/0/R60m/B12.jp2, band 1: Illegal nBlockXOff value (171800590) in GDALRasterBand::GetLockedBlockRef()

[117 of 1000] FAILURE(3) CPLE_IllegalArg(5) "Illegal argument." /vsis3/sentinel-s2-l2a/tiles/22/L/FP/2019/8/14/0/R60m/B12.jp2, band 1: Illegal nBlockXOff value (-494424126) in GDALRasterBand::TryGetLockedBlockRef()

[118 of 1000] FAILURE(3) CPLE_IllegalArg(5) "Illegal argument." /vsis3/sentinel-s2-l2a/tiles/22/L/FP/2019/8/14/0/R60m/B12.jp2, band 1: Illegal nBlockXOff value (-494424126) in GDALRasterBand::GetLockedBlockRef()

[119 of 1000] FAILURE(3) CPLE_IllegalArg(5) "Illegal argument." /vsis3/sentinel-s2-l2a/tiles/22/L/FP/2019/8/14/0/R60m/B12.jp2, band 1: Illegal nBlockXOff value (22130851) in GDALRasterBand::TryGetLockedBlockRef()

[120 of 1000] FAILURE(3) CPLE_IllegalArg(5) "Illegal argument." /vsis3/sentinel-s2-l2a/tiles/22/L/FP/2019/8/14/0/R60m/B12.jp2, band 1: Illegal nBlockXOff value (22130851) in GDALRasterBand::GetLockedBlockRef()

The assertion error is:

java: locked_dataset.hpp:119: locked_dataset& locked_dataset::operator=(locked_dataset&&): Assertion `m_use_count == 0' failed.

I suspect the error and process crash have been foretold in this code comment.

To Reproduce

Steps to reproduce the behavior:

  1. Launch ec2 instance
    1. amazon linux 2 AMI,
    2. perhaps c5.9xlarge, Reproduced on ~16 or more cores
    3. in eu-central-1 region (same as PDS location).
    4. Probably need 50-100 GB storage to collect the core dump
    5. Set up with IAM Role (or other permissions) to read S3 PDS bucket (requester pays)
  2. Install docker
  3. docker pull s22s/rasterframes-notebook:0.9.0-RC1
  4. docker run -p 8888:8888 -p 4040:4040 -v /home/ec2-user/:/home/jovyan/work s22s/rasterframes-notebook:0.9.0-RC1
  5. Upload / scp file minimal.py.zip and unzip
  6. From within the jupyter notebook's terminal: $ python minimal.py
  7. Error may take several minutes to occur.

Expected behavior

I expect the job to complete with all JP2 files read without error in a job that reads one file, or many files. As mentioned above, have tried small single-file jobs on some of the files mentioned in error messages. The job completes fine for these.

Environment

  • Docker container: s22s/rasterframes-notebook:0.9.0-RC1
  • Running on EC2 instance c5.9xlarge, 36 cores on the spark driver.
  • JVM version: 1.8.0_222 (Private Build)
  • GDAL version: 2.4.2

Core Files

I have captured a core dump that is 18GB. We'll have to figure out how to share it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.