Code Monkey home page Code Monkey logo

esri / lerc Goto Github PK

View Code? Open in Web Editor NEW
191.0 29.0 57.0 10.21 MB

Limited Error Raster Compression

Home Page: https://github.com/esri/lerc

License: Apache License 2.0

C++ 70.20% JavaScript 1.14% HTML 2.71% Python 8.21% C 5.42% C# 8.54% CMake 0.37% TypeScript 3.40%
data-management native-development compression lerc-byte-blob raster encoding raster-compression decoding band imagery image image-processing image-compression image-compressor c-plus-plus c-sharp javascript python pixel decoding-images

lerc's Introduction

LERC - Limited Error Raster Compression

What is LERC?

LERC is an open-source image or raster format which supports rapid encoding and decoding for any pixel type (not just RGB or Byte). Users set the maximum compression error per pixel while encoding, so the precision of the original input image is preserved (within user defined error bounds).

This repository contains a C++ library for both encoding and decoding images. You can also do this directly from Python. And we have decoders for JavaScript and C#.

The LERC C API

Function Description
uint lerc_computeCompressedSize(...) Computes the buffer size that needs to be allocated so the image can be Lerc compressed into that buffer. The size is accurate to the byte. This function is optional. It is faster than lerc_encode(...). It can also be called to decide whether an image or image tile should be encoded by Lerc or another method.
uint lerc_encode(...) Compresses a given image into a pre-allocated buffer. If that buffer is too small, the function fails with the corresponding error code. The function also returns the number of bytes written.
uint lerc_getBlobInfo(...) Looks into a given Lerc byte blob and returns an array with all the header info. From this, the image to be decoded can be allocated and constructed. This function is optional. You don't need to call it if you already know the image properties such as tile size and data type.
uint lerc_getDataRanges(...) Looks into a given Lerc byte blob and returns 2 double arrays with the minimum and maximum values per band and depth. This function is optional. It allows fast access to the data ranges without having to decode the pixels.
uint lerc_decode(...) Uncompresses a given Lerc byte blob into a pre-allocated image. If the data found in the Lerc byte blob does not fit the specified image properties, the function fails with the corresponding error code.
uint lerc_decodeToDouble(...) Uncompresses a given Lerc byte blob into a pre-allocated image of type double independent of the compressed data type. This function was added mainly to be called from other languages such as Python and C#.

To support the case that not all image pixels are valid, a mask image can be passed. It has one byte per pixel, 1 for valid, 0 for invalid.

See the sample program src/LercTest/main.cpp which demonstrates how the above functions are called and used. Also see the two header files in the src/LercLib/include/ folder and the comments in there.

About multiple bands, or multiple values per pixel. This has changed with Lerc version 2.4. Before, you could either store each band into its own Lerc byte blob which allowed you to access / decode each band individually. Lerc also allowed to stack bands together into one single Lerc byte blob. This could be useful if the bands are always used together anyway. Now, since Lerc version 2.4, you can additionally store multiple values per pixel interleaved, meaning an array of values for pixel 1, next array of values for pixel 2, and so forth. We have added a new parameter "nDepth" for this number of values per pixel.

While the above can be used as an "interleave flag" to store multiple raster bands as a 3D array as either [nBands, nRows, nCols] for band interleaved or as [nRows, nCols, nDepth] for pixel interleaved, it also allows to do both at the same time and store a 4D array as [nBands, nRows, nCols, nDepth].

Note that the valid / invalid pixel byte mask is not 4D but limited to [nBands, nRows, nCols]. This mask is per pixel per band. For nDepth > 1 or an array of values per pixel, up to Lerc version 3.0, Lerc assumed all values in that array at that pixel are either valid or invalid. If the values in the innermost array per pixel can be partially valid and invalid, use a predefined noData value or NaN.

To better support this special "mixed" case, we have added new Lerc API functions *_4D() in Lerc version 4.0, see Lerc_c_api.h. These functions allow to pass one noData value per band to the encode_4D() function and can receive it back in the decode_4D() function. This way such data can be compressed with maxZError > 0 or lossy, despite the presence of noData values in the data. Note that Lerc will convert noData values to 0 bytes in the valid / invalid byte mask whenever possible. This also allows now to pass raster data with noData values to the encoder without first creating the valid / invalid byte mask. NoData values can be passed both ways, as noData or as byte mask. Note that on decode Lerc only returns a noData value for the mixed case of valid and invalid values at the same pixel (which can only happen for nDepth > 1). The valid / invalid byte mask remains the preferred way to represent void or noData values.

Remark about NaN. As Lerc supports both integer and floating point data types, and there is no NaN for integer types, Lerc filters out NaN values and replaces them. Preferred it pushes NaN's into the valid / invalid byte mask. For the mixed case, it replaces NaN by the passed noData value. If there is no noData value, encode will fail. Lerc decode won't return any NaN's.

When to use

In image or raster compression, there are two different options:

  • compress an image as much as possible but so it still looks ok (jpeg and relatives). The max coding error per pixel can be large.

  • prioritize control over the max coding error per pixel (elevation, scientific data, medical image data, ...).

In the second case, data is often compressed using lossless methods, such as LZW, gzip, and the like. The compression ratios achieved are often low. On top the encoding is often slow and time consuming.

Lerc allows you to set the max coding error per pixel allowed, called "MaxZError". You can specify any number from 0 (lossless) to a number so large that the decoded image may come out flat.

In a nutshell, if jpeg is good enough for your images, use jpeg. If not, if you would use png instead, or gzip, then you may want to try out Lerc.

How to use

Lerc can be run anywhere without external dependencies. This project includes test samples of how to use LERC directly, currently for C++, Python, JavaScript, and C#. We have added a few small data samples under testData/. There is also a precompiled Windows dll and a Linux .so file under bin/.

How to use without compiling LERC

Check out the Lerc decoders and encoders in OtherLanguages/. You may need to adjust the paths to input or output data and the dll or .so file. Other than that they should just work.

Other download sites

How to compile LERC and the C++ test program

For building the Lerc library on any platform using CMake, use CMakeLists.txt. For the most common platforms you can find alternative project files under build/.

Windows

  • Open build/Windows/MS_VS2022/Lerc.sln with Microsoft Visual Studio.
  • Build and run.

Linux

  • Open build/Linux/CodeBlocks/Lerc/Lerc_so.cbp using the free Code::Blocks IDE for Linux.
  • Build it. Should create libLerc_so.so.
  • Open build/Linux/CodeBlocks/Test/Test.cbp.
  • Build and run.

MacOS

  • Open build/MacOS/Lerc64/Lerc64.xcodeproj with Xcode.
  • Build to create dynamic library.

LERC can also be used as a compression mode for the GDAL image formats GeoTIFF (since GDAL 2.4) and MRF (since GDAL 2.1) via GDAL.

LERC Properties

  • works on any common data type, not just 8 bit: char, byte, short, ushort, int, uint, float, double.

  • works with any given MaxZError or max coding error per pixel.

  • can work with a byte mask that specifies which pixels are valid and which ones are not.

  • is very fast: encoding time is about 20-30 ms per MegaPixel per band, decoding time is about 5 ms per MegaPixel per band.

  • compression is better than most other compression methods for larger bitdepth data (int types larger than 8 bit, float, double).

  • for 8 bit data lossless compression, PNG can be better, but is much slower.

  • in general for lossy compression with MaxZError > 0, the larger the error allowed, the stronger the compression. Compression factors larger than 100x have been reported.

  • this Lerc package can read all (legacy) codec versions of Lerc, such as Lerc1, Lerc2 v1 to v5, and the current Lerc2 v6. It always writes the latest stable version.

The main principle of Lerc and history can be found in doc/MORE.md

Benchmarks

Some benchmarks are in doc/LercBenchmarks_Feb_2016.pdf

Bugs?

The codecs Lerc2 and Lerc1 have been in use for years, bugs in those low level modules are very unlikely. All software updates are tested in Esri software for months before they are uploaded to this repo.

Licensing

Copyright 2015-2022 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's LICENSE file.

lerc's People

Contributors

avalentino avatar biswa96 avatar davidjvitale avatar dependabot[bot] avatar dotlambda avatar halmaia avatar hmaarrfk avatar jgravois avatar joalvarez avatar john4744 avatar kmilos avatar kryden avatar lucianpls avatar miurahr avatar nitaym avatar parkske avatar patrickarlt avatar ppoply avatar rouault avatar scw avatar tmaurer3 avatar wju avatar zakjan avatar ziqianming avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lerc's Issues

std::greater

On VS2015 there is this error:

src\LercLib\Lerc.cpp(1541): error C2065: 'greater': undeclared identifier
src\LercLib\Lerc.cpp(1541): error C2062: type 'double' unexpected
src\LercLib\Lerc.cpp(1568): error C2039: 'greater': is not a member of 'std'
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\string(17): note: see declaration of 'std'
src\LercLib\Lerc.cpp(1568): error C2065: 'greater': undeclared identifier
src\LercLib\Lerc.cpp(1568): error C2062: type 'double' unexpected

I fixed it by adding

#include <functional>

std-greater.txt

Improper holes and bumps in the generated surface using Lerc.dll

Latest Windows decoder "Lerc.dll" has some artifacts when generating heights data for the surface. It generates unwanted holes and bumps while decoding some tiles as shown in the following pictures.

Holes

Holes2

Did not test on Mac & Linux yet but it could be the same for these libraries too.

lerc_encode() may return a smaller size than lerc_computeCompressedSize(...)

@tmaurer3

The documentation says that lerc_computeCompressedSize is optional. But the size returned but lerc_encode may be less that what is needed for successful decoding, no way to know what that value is without calling lerc_computeCompressedSize.
I found out the hard way.
Can we have lerc_encode() return the actual number of bytes needed by decoder, aka the return from lerc_computeCompressedSize?

Alignment errors in GetLercInfo on 32-bit ARM

We're using lerc on Android devices. We periodically see unaligned access errors on 32-bit devices (ABI armeabi-v7a) arising from the assignments

    int height = *((const int*)ptr);  ptr += sizeof(int);
    int width  = *((const int*)ptr);  ptr += sizeof(int);
    double maxZErrorInFile = *((const double*)ptr);

in Lerc::GetLercInfo lines 154-156. At the moment we are avoiding the errors by using memcpy instead of pointer casting:

    int height;
    memcpy(&height, ptr, sizeof(int));
    ptr += sizeof(int);
    int width;
    memcpy(&width, ptr, sizeof(int));
    ptr += sizeof(int);
    double maxZErrorInFile;
    memcpy(&maxZErrorInFile, ptr, sizeof(double));

Can you incorporate this as a fix? I can provide a pull request if you'd like.

Is there a reason for the daily release tag for runtimecore?

Trying to find a given Lerc github release is very difficult with daily releases published of the same runetimecore version. Is there a reason it is rebuilt daily? Can it be removed from the release tags except for actual version changes?

how to write lerc byte lerc data to file

the doc not Introduction how write the compress result to file and the lerc data structure.

I have write the pLercBlob to file๏ผŒbut the lerc file need the header,
the code
` for(int i=0;i<numBytesBlob;i++)

outfile.write((char*)&pLercBlob[i],sizeof(pLercBlob[i]));

outfile.close( ); `

Typos

./src/LercLib/fpl_Lerc2Ext.cpp:469: doble ==> double
./src/LercLib/include/Lerc_c_api.h:264: occurence ==> occurrence
./OtherLanguages/js/dist/LercDecode.js:434: compatiblity ==> compatibility
./OtherLanguages/js/src/Lerc.ts:554: compatiblity ==> compatibility

Would be helpful to be able to make static libs

It would be very helpful if the CMakeLists.txt, instead of hard-coding SHARED on the add_library command, instead honored the standard convention of using an option BUILD_SHARED_LIBRARIES (the default can be ON). This would allow a user/site to easily override the default in order to build static libraries if they need it.

The idiom for doing this is simply:

# Make an option, defaulting to shared libs, but allow -DBUILD_SHARED_LIBS=OFF
option (BUILD_SHARED_LIBS "Build shared libraries (set to OFF to build static libs)" ON)

# If no SHARED or STATIC is specified explicitly, add_library will honor BUILD_SHARED_LIBS
add_library (mylibrary ${SOURCES})

I'm sorry I have to just leave these breadcrumbs at your door instead of submitting a proper PR that fully implements it. There's too much bureaucracy in my organization to get legal approval for a one-time, two-line patch.

Thanks for your work with this library!

unpkg.com redirect doesn't handle wasm

If you use <script type="text/javascript" src="https://unpkg.com/lerc"></script>, unpkg.com won't redirect wasm file.

To workaround this issue, use the following url
<script type="text/javascript" src="https://unpkg.com/lerc@latest/LercDecode.min.js"></script>

DLL export API used even on import

The lines in the public header

#if defined(_MSC_VER)
#define LERCDLL_API __declspec(dllexport)
#elif __GNUC__ >= 4
#define LERCDLL_API __attribute__((visibility("default")))
#else
#define LERCDLL_API
#endif

are ok for building a DLL with Visual Studio only, but will not work for projects using the DLL via this header without patching, nor for other compilers on Windows.

See e.g. https://stackoverflow.com/a/538179 and https://gcc.gnu.org/wiki/Visibility#Step-by-step_guide

Another macro is usually needed to distinguish between building the DLL and simply using it (it looks like MSVC and CMake on Windows will readily define LERC_EXPORTS so that could be used for convenience), and one should be testing for _WIN32, not just _MSC_VER.

Doc question: How does compression work for lossless MaxZError = 0?

Sorry, another documentation question. I will try to turn these into a FAQ or more detailed examples or something like that later and make a PR.

LERC can be used with MaxZError = 0 which makes it a lossless process. Still it does compress the data, regardless of the type. How well depends on the data of course. I tested this with GDAL so I might look at some implementation specifics but I would be surprised if it did something it is not meant to do. I tested various data types from Byte to Float32 and they all shrank.

What I read in the patent (on the page break between column 8 and 9) and the byte stream docs says that the values would not be compressed if MaxZError is 0 though so I wonder which parts of the whole LERC algorithm are still applied and when. I probably missed something but I am completely lost somehow.

Thank you!

Doc on how to compile Lerc on different platforms

I think it would be great if we can have some documentation on how to compile Lerc on Linux or OSX. Such as whether there are any special libraries that I need to install.

Also, I assume we can't run LercTest on OSX because of PerfTimer?

showing elevation can survive map zoom but not pan

thanks to @lucianpls, we now display elevation in meters appropriately when using the +/- zoom buttons and dragging a box to zoom in the leaflet sample, but once the map is panned by click dragging, either the elevation shows as undefined, or a value is displayed from an offset location.

Doc question: How are void pixels handled?

I am currently reading through the patent and have a question (probably stupid and there are probably more to come :D ). I hope it is ok if I use an issue here for this.

A LERC block knows about the number of valid/non-negative pixels and then there is the bitstuffed data (which can be unpacked since we know the bits per pixel and the number of pixels). But there could be "void" pixels (~=NODATA pixels).

Looking at https://raw.githubusercontent.com/Esri/lerc/master/doc/LercEncodingSample.png there are four "void" pixels.

What happens with them when encoding? How would I know where to re-insert them in the block when unpacking if I just find 12 pixel values for the 4x4 (=16 pixels) block?

turn current api into pure C api

_ enable call from other languages such as python
_ no change to file format, no version change, just high level interfaces
_ clean up the folder structure for windows and Linux build support

std::min

Using VS2015 the files src\LercLib\fpl_EsriHuffman.cpp and src\LercLib\fpl_Lerc2Ext.cpp dot not compile, the error is:

error C2039: 'min': is not a member of 'std'

I fixed it by adding

#include <algorithm>

std-min.txt

A new sample in LercTest

It would be helpful if we can have a sample in LercTest to demonstrate encoding/decoding of an image on disk. This way, it would help users get a better grasp of MaxZError, and its direct effect on file size and decoded image quality.

The existing 2 samples are great in that it gives users an idea of how fast Lerc is, but leaves me wanting for more of a real life example.

Is it possible to encode a multi-band image service in one LERC blob?

I'm using the Composite Bands python raster function to combine three mosaic datasets into one image service. I'm publishing this service using LERC tiles that are generated on the fly.

I'm reading the tiles using this example from @jgravois, When I log the pixel values, however, I'm only getting data from the first raster, not all three.

Is it possible to encode a multi-band image service in LERC using out of the box ArcGIS Server 10.3? For more information, see my Stack Exchange post.

Apologies if this is the wrong location to post this. Thanks!

Suggestion: gh-pages example

It would be useful when sharing the LERC project and demonstrating what it does to have a gh-pages at http://esri.github.io/lerc/

This could demonstrate a comparison of LERC versus non-LERC imagery as well as enumerate LERC sources that we provide and simple steps for people to publish their own and use with our tools.

upgrade Lerc codec version to 2.4

The biggest change is to allow for multiple (nDim) values per pixel. The API will change accordingly. The function calls will get an extra parameter "nDim" in addition to "nCols", "nRows", and "nBands". So an RGB image can now be represented as [RGB, RGB, ...] by setting nDim = 3 and nBands =1, or it can be represented as before as [RRRR ..., GGGG ..., BBBB ...] by setting nDim =1 and nBands = 3.
The Lerc API release version will change from 1.0 to 2.0.
There will be other improvements and updates as well.

Update on PyPI

Hi there, just wondering why Esri updates the arcgis package on PyPI, but not lerc. The PyPI lerc package is severely outdated. I understand that conda is the preferred install method, but if one pacakge is kept up to date on PyPI, then why not another?

Thanks

Official release?

I've noticed a number of buffer overflow fixes/ UB fixes going in in the past few months. Is there any chance of an official release containing these going out?

Typescript export interfaces

Hi,

is I just installed 4.0.0., but I cannot import interfaces from your library. Can you please export those interfaces?

Thanks you

BitStuffer2::BitStuff undefined bitshift

InBitStuffer2::BitStuff, if bitPos <= 0 and (32 - bitPos >= numBits) is False, an undefined bitshift will occur when (*srcPtr++) >> (32 - bitPos).

LERC functions which use BitStuff make this scenario impossible by returning false if numBits >= 32 before calling BitStuff. Users of the library may not do this. If that scenario is likely, could the BitStuff function be patched?

LERC adoption FYI

Hi all,
just wanted to draw your attention to this issue.
As it's relevant to the future adoption of LERC, I thought it might be interesting for you.

Regards, Marco

Question: what should be the official name of libs files?

I'm creating a recipe of lerc for conan: conan-io/conan-center-index#1159

It's not really clear what names should be given to generated lib files: LercLib (in CMakeLists in master)? Lerc (in bin)?

By the way, why do you force using shared lib in build scripts? Seems to also properly work as static lib.

For example, here is how I've patched v2.1 for conan in order to support static or shared, but also to quickly fix strange things in macros related to LERCDLL_API:

--- /dev/null
+++ b/CMakeLists.txt
@@ -0,0 +1,24 @@
+cmake_minimum_required(VERSION 3.0.2)
+project(Lerc)
+
+file(GLOB SOURCES 
+  "src/LercLib/*"
+  "src/LercLib/Lerc1Decode/*"
+)
+
+add_library(LercLib ${SOURCES})
+set_property(TARGET LercLib PROPERTY CXX_STANDARD 11)
+if(BUILD_SHARED_LIBS)
+  set_property(TARGET LercLib PROPERTY CXX_VISIBILITY_PRESET hidden)
+  if(WIN32)
+    target_compile_definitions(LercLib PRIVATE LERC_EXPORTS)
+  endif()
+endif()
+
+set(HEADER_FILES include/Lerc_c_api.h include/Lerc_types.h)
+install(FILES ${HEADER_FILES} DESTINATION include)
+install(TARGETS LercLib
+  RUNTIME DESTINATION bin
+  LIBRARY DESTINATION lib
+  ARCHIVE DESTINATION lib
+)
--- a/include/Lerc_c_api.h
+++ b/include/Lerc_c_api.h
@@ -28,13 +28,7 @@ Contributors:  Thomas Maurer
 extern "C" {
 #endif
 
-#if defined(_MSC_VER)
-  #define LERCDLL_API __declspec(dllexport)
-#elif __GNUC__ >= 4
-  #define LERCDLL_API __attribute__((visibility("default")))
-#else
-  #define LERCDLL_API
-#endif
+#define LERCDLL_API
 
 
   //! C-API for LERC library
--- a/src/LercLib/Lerc_c_api.h
+++ b/src/LercLib/Lerc_c_api.h
@@ -28,8 +28,12 @@ Contributors:  Thomas Maurer
 extern "C" {
 #endif
 
-#if defined(_MSC_VER)
-  #define LERCDLL_API __declspec(dllexport)
+#ifdef _WIN32
+  #ifdef LERC_EXPORTS
+    #define LERCDLL_API __declspec(dllexport)
+  #else
+    #define LERCDLL_API
+  #endif
 #elif __GNUC__ >= 4
   #define LERCDLL_API __attribute__((visibility("default")))
 #else

LERC support in ArcGIS Image Server 10.8.1 (Linux)

For reference I've posted this issue here as well as I am using the OptimizeRasters toolkit to create rasters with LERC.

I am creating MRFs, uploaded to S3, then adding them to a mosaic dataset and publishing the mosaic dataset to Image Server on Linux.

When I add the image service to ArcGIS Pro 2.6, the rasters do not draw. When I try to use the Image Server Rest API export image command, I can't get an image to be returned.

Through troubleshooting, I've found that if I add the same mosaic dataset to Pro locally, the rasters are able to be drawn, so I don't think it's an issue with access to S3 or the rasters. If I publish the image service with rasters using the JPEG compression, the service draws fine in ArcGIS Pro from image server.

Does anyone know if Image Server should be able to serve out rasters with LERC and what I need to configure to get them to display?

Different result if raster is 2D or 3D

I am writing a python wrapper that round-trips a numpy array through encode and decode. This wrapper works on both 2D and 3D numpy arrays, however, I get a surprising result when I compare the post-LERC results of individual bands.

I expected that if I encode and then decode a 256x256x2 array and do the same for each of the bands and stack them together that I should get identical results. However, this is not the case.

Is this incorrect?

image

Python test

This test fails on the last assert.

class TestLercCompressor:

    @classmethod
    def _zerror(cls, dtype: np.dtype):
        zerror = 5.0  # in unit8 space
        if np.issubdtype(dtype, np.integer):
            return dtype(zerror * (np.iinfo(dtype).max / np.iinfo(np.uint8).max))
        else:
            return dtype((zerror + 0.5) / (np.iinfo(np.uint8).max + 1))

    @pytest.mark.parametrize("dtype", [np.uint8, np.uint16, np.float32, np.double])
    def test_compress_3d(self, request, test_data_dir: Path, dtype: np.dtype):
        testname = f"{self.__class__.__name__}.{request.node.name}"
        rs = np.random.RandomState(sum(map(ord, testname)))
        if np.issubdtype(dtype, np.integer):
            raster = rs.randint(np.iinfo(dtype).min, np.iinfo(dtype).max, size=(256, 256, 2), dtype=dtype)
        else:
            raster = rs.uniform(0, 1, size=(256, 256, 2)).astype(dtype)

        z_error = self._zerror(dtype)

        raster2d = np.stack(
            [
                lerc.apply(raster=raster[..., 0], max_z_error=z_error),
                lerc.apply(raster=raster[..., 1], max_z_error=z_error),
            ],
            axis=2,
        )
        raster3d = lerc.apply(raster=raster, max_z_error=z_error)

        assert raster2d.dtype == raster.dtype
        np.testing.assert_raises(AssertionError, np.testing.assert_equal, raster, raster2d)
        np.testing.assert_allclose(raster, raster2d, rtol=0.0, atol=z_error)  # check raster2d changed
        if np.issubdtype(dtype, np.integer):
            np.testing.assert_array_less(raster2d, np.iinfo(dtype).max)
        else:
            np.testing.assert_array_less(raster2d, 1.0)

        assert raster3d.dtype == raster.dtype
        np.testing.assert_raises(AssertionError, np.testing.assert_equal, raster, raster3d)
        np.testing.assert_allclose(raster, raster3d, rtol=0.0, atol=z_error)  # check raster3d changed
        if np.issubdtype(dtype, np.integer):
            np.testing.assert_array_less(raster3d, np.iinfo(dtype).max)
        else:
            np.testing.assert_array_less(raster3d, 1.0)

        # there should be no difference between 2d and 3d
        np.testing.assert_equal(raster2d, raster3d)

C++ wrapping code using pybind11

#include "wrapLerc.hpp"

#include <pybind11/numpy.h>
#include <pybind11/stl_bind.h>

#include <LercLib/Defines.h>
#include <LercLib/Lerc.h>
#include <LercLib/Lerc_types.h>

namespace py = pybind11;

namespace {
static constexpr int LERC_VERSION = -1;  // use the latest

template <typename DataT> struct NumpyInfo {
    using NumpyT = typename py::array_t<DataT, py::array::c_style>;

    NumpyInfo() : array(), info(), data(nullptr), nDim(1), nCols(0), nRows(0), nBands(0) {}

    NumpyInfo(NumpyT inArray)
        : array(inArray), info(array.request()), data(nullptr), nDim(1), nCols(0), nRows(0), nBands(0) {
        if (info.ndim != 2 && info.ndim != 3) {
            throw std::runtime_error("LERC compression requires a 2D or 3D array");
        }
        if (info.ptr == nullptr) {
            throw std::runtime_error("The array has no data!");
        }
        data = static_cast<DataT *>(info.ptr);
        nCols = info.shape[0];
        nRows = info.shape[1];
        nBands = (info.ndim == 2) ? 1 : info.shape[2];
        if (array.size() != nCols * nRows * nBands) {
            throw std::runtime_error("Failed to correctly extract array size");
        }
    }

    NumpyInfo(const NumpyInfo &other) : NumpyInfo(std::move(NumpyT(other.info.shape, other.info.strides))) {
        // copy data into the array
        std::copy(other.data, other.data + other.array.size(), data);
    }

    NumpyInfo copy() const { return NumpyInfo(*this); }

    /// numpy array
    NumpyT array;
    /// numpy buffer protocol
    py::buffer_info info;
    /// Raw image data, row by row, band by band. This is a raw pointer, assume someone else is looking after memory
    DataT *data;
    /// number of values per pixel (for us, this is always 1)
    const int nDim;
    /// data shape
    int nCols, nRows, nBands;
};

/// Calculate the size of the required buffer size
template <typename DataT> struct LercBufferSize {
    using RasterInfo = NumpyInfo<DataT>;
    using RasterT = typename RasterInfo::NumpyT;

    static uint32_t calculate(const RasterT &raw, double maxZError) {
        RasterInfo raster(raw);
        return calculateImpl(raster, maxZError);
    }

    static uint32_t calculateImpl(const RasterInfo &raster, double maxZError) {
        LercNS::BitMask *bitMask = nullptr;  // We don't care about the mask
        uint32_t bs = 0;                     // size of outgoing Lerc blob
        auto status = LercNS::Lerc::ComputeCompressedSizeTempl<DataT>(
            raster.data, LERC_VERSION, raster.nDim, raster.nCols, raster.nRows, raster.nBands, bitMask, maxZError, bs);
        if (status != LercNS::ErrCode::Ok) {
            throw std::runtime_error("Failed to fetch the number of bytes required for the buffer");
        }
        return bs;
    }
};

template <typename DataT> struct LercCompressor {
    using RasterInfo = NumpyInfo<DataT>;
    using RasterT = typename RasterInfo::NumpyT;

    static void applyInPlace(RasterT &raw, double maxZError) {
        RasterInfo raster(raw);
        impl(raster, maxZError);
    }

    static RasterT apply(const RasterT &raw, double maxZError) {
        auto raster = RasterInfo(raw).copy();
        impl(raster, maxZError);
        return raster.array;
    }

  private:
    static void impl(RasterInfo &raster, double maxZError) {
        LercNS::BitMask *bitMask = nullptr;  // We don't care about the mask

        // --- get the buffer size
        uint32_t numBytesNeeded = LercBufferSize<DataT>::calculateImpl(raster, maxZError);

        // --- encode
        // buffer to write to, function will fail if buffer too small
        std::vector<LercNS::Byte> lercBlob(numBytesNeeded, 0);
        auto pLercBlob = lercBlob.data();

        uint32_t numBytesUsed = 0;  // num bytes written to buffer
        auto status = LercNS::Lerc::EncodeTempl<DataT>(raster.data, LERC_VERSION, raster.nDim, raster.nCols,
                                                       raster.nRows, raster.nBands, bitMask, maxZError, pLercBlob,
                                                       numBytesNeeded, numBytesUsed);
        if (status != LercNS::ErrCode::Ok) {
            throw std::runtime_error("Failed to encode data");
        }

        // --- decode
        status = LercNS::Lerc::DecodeTempl<DataT>(raster.data, pLercBlob, numBytesUsed, raster.nDim, raster.nCols,
                                                  raster.nRows, raster.nBands, bitMask);
        if (status != LercNS::ErrCode::Ok) {
            throw std::runtime_error("Failed to decode data");
        }
    }
};

template <typename DataT> void wrapLercImpl(py::module &m) {
    using LBS = LercBufferSize<DataT>;
    m.def("buffer_size", &LBS::calculate, py::arg("raster"), py::arg("max_z_error"));

    using LC = LercCompressor<DataT>;
    m.def("apply_in_place", &LC::applyInPlace, py::arg("raster"), py::arg("max_z_error"));
    m.def("apply", &LC::apply, py::arg("raster"), py::arg("max_z_error"), py::return_value_policy::move);
}
}  // namespace

namespace cppaiutils {
namespace python {

void wrapLerc(py::module &m) {
    auto mLerc = m.def_submodule("lerc");
    wrapLercImpl<uint8_t>(mLerc);
    wrapLercImpl<uint16_t>(mLerc);
    wrapLercImpl<float>(mLerc);
    wrapLercImpl<double>(mLerc);
}
}  // namespace python
}  // namespace cppaiutils

Homebrew GDAL

Currently GDAL installed from Homebrew doesn't support LERC.

Is it possible to publish LERC to Homebrew, and link it with libtiff there?

See Homebrew/homebrew-core#106450 for how ZSTD support was enabled.

Problem with PyPi package pushed out 2023-03-15 10:36:12 UTC

Currently when trying to pip install lerc, the published package isn't packaged correctly:

$ pip install lerc
Collecting lerc
  Using cached lerc-4.0.0.tar.gz (721 kB)
ERROR: lerc from https://files.pythonhosted.org/packages/1d/49/84f7e28247c82809417a9a2e898693da8f57b36d6a9358b146bdc0b182f0/lerc-4.0.0.tar.gz does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

Checking the package, it looks like there's an one directory too many (dist/ shouldn't be the top level):

$ wget https://files.pythonhosted.org/packages/1d/49/84f7e28247c82809417a9a2e898693da8f57b36d6a9358b146bdc0b182f0/lerc-4.0.0.tar.gz && tar tvf lerc-4.0.0.tar.gz
--2023-03-15 09:45:11--  https://files.pythonhosted.org/packages/1d/49/84f7e28247c82809417a9a2e898693da8f57b36d6a9358b146bdc0b182f0/lerc-4.0.0.tar.gz && tar tvf lerc-4.0.0.tar.gz
drwxrwxrwx  0 0      0           0 Mar 14 19:31 dist/
drwxrwxrwx  0 0      0           0 Mar 14 19:31 dist/lerc-4.0.0/
drwxrwxrwx  0 0      0           0 Mar 14 19:31 dist/lerc-4.0.0/lerc/
-rwxrwxrwx  0 0      0      526336 Jun 14  2022 dist/lerc-4.0.0/lerc/Lerc.dll
-rwxrwxrwx  0 0      0        4158 Jun 14  2022 dist/lerc-4.0.0/lerc/Lerc.lib
-rwxrwxrwx  0 0      0      951344 Jun 22  2022 dist/lerc-4.0.0/lerc/libLerc.dylib
-rwxrwxrwx  0 0      0      816528 Jun 22  2022 dist/lerc-4.0.0/lerc/libLerc.so.4
-rwxrwxrwx  0 0      0       41442 Aug 18  2022 dist/lerc-4.0.0/lerc/_lerc.py
-rwxrwxrwx  0 0      0          20 Feb 24  2020 dist/lerc-4.0.0/lerc/__init__.py
drwxrwxrwx  0 0      0           0 Mar 14 19:31 dist/lerc-4.0.0/lerc.egg-info/
-rwxrwxrwx  0 0      0           1 Mar  5  2020 dist/lerc-4.0.0/lerc.egg-info/dependency_links.txt
-rwxrwxrwx  0 0      0         410 Mar 14 18:59 dist/lerc-4.0.0/lerc.egg-info/PKG-INFO
-rwxrwxrwx  0 0      0         216 Mar 14 19:14 dist/lerc-4.0.0/lerc.egg-info/SOURCES.txt
-rwxrwxrwx  0 0      0           5 Mar  5  2020 dist/lerc-4.0.0/lerc.egg-info/top_level.txt
-rwxrwxrwx  0 0      0         410 Mar 14 18:58 dist/lerc-4.0.0/PKG-INFO
-rwxrwxrwx  0 0      0        4008 Jun 24  2022 dist/lerc-4.0.0/README.md
-rwxrwxrwx  0 0      0          38 Mar  5  2020 dist/lerc-4.0.0/setup.cfg
-rwxrwxrwx  0 0      0        1703 Mar 14 19:04 dist/lerc-4.0.0/setup.py

Looking at https://pypi.org/project/lerc/#history it looks like this was pushed out at 2023-03-15T10:36:12+0000

Upgrade Lerc codec version to Lerc2v5

Need to sync with upcoming Esri releases Pro 2.4 and ArcMap 10.7.1.
Minor release version update, no API changes.
Codec update from Lerc2v4 to Lerc2v5.

Unable to static link lerc on Windows OOTB

Hello,

One of my project is using lerc as a sub-sub dependency and I am currently unable to link it on Windows (using msys2 and MINGW) because apparently lerc doesn't provide an out-of-the-box solution to build a static library for Windows.

I've opened an issue on msys2's MinGW package repository where people are already talking about potential solutions: most of them consist of providing a way to define LERCDLL_API to nothing.

Thank you for your time and your help,
Arignir

Lerc:EncodeTempl missing from final .so

I think this PR by @tmaurer3 introduced a small API issue: #147

By introducing the EncodeInternal template function, the compiler now fails to include exported versions of either Lerc::EncodeTempl or Lerc:: ComputeCompressedSizeTempl.

$ nm libLerc.so  | c++filt | grep "Lerc::Encode"
0000000000016260 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<signed char>(signed char const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
00000000000190b0 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<double>(double const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
00000000000185a0 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<float>(float const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
0000000000016700 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<unsigned char>(unsigned char const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
0000000000017a80 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<int>(int const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
0000000000017f20 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<unsigned int>(unsigned int const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
0000000000016d90 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<short>(short const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
0000000000017400 W LercNS::ErrCode LercNS::Lerc::EncodeInternal<unsigned short>(unsigned short const*, int, int, int, int, int, int, unsigned char const*, double, unsigned int&, unsigned char*, unsigned int, unsigned int&)
000000000001a0f0 T LercNS::Lerc::Encode(void const*, int, LercNS::Lerc::DataType, int, int, int, int, int, unsigned char const*, double, unsigned char*, unsigned int, unsigned int&)
$ nm libLerc.so  | c++filt | grep "Lerc::.*Templ"
00000000000112a0 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<signed char>(signed char*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
0000000000015310 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<double>(double*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
0000000000014b00 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<float>(float*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
0000000000011c40 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<unsigned char>(unsigned char*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
0000000000013940 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<int>(int*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
00000000000142f0 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<unsigned int>(unsigned int*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
00000000000125e0 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<short>(short*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)
0000000000012f90 W LercNS::ErrCode LercNS::Lerc::DecodeTempl<unsigned short>(unsigned short*, unsigned char const*, unsigned int, int, int, int, int, int, unsigned char*)

This means that a user of the library cannot include these template functions in their own code as it leads to:

undefined symbol: _ZN6LercNS4Lerc11EncodeTemplIdEENS_7ErrCodeEPKT_iiiiiiPKhdPhjRj

and

$ c++filt _ZN6LercNS4Lerc11EncodeTemplIdEENS_7ErrCodeEPKT_iiiiiiPKhdPhjRj
LercNS::ErrCode LercNS::Lerc::EncodeTempl<double>(double const*, int, int, int, int, int, int, unsigned char const*, double, unsigned char*, unsigned int, unsigned int&)

lerc v2 pure-python

Is there going to be an implementation of a pure-python solution for Lerc v2?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.