Code Monkey home page Code Monkey logo

geoengine's Introduction

Geo Engine

CI Coverage Status Documentation

This workspace contains the Geo Engine crates.

Documentation

Development

  • While Geo Engine should build on Linux and Windows environments, we currently only support Ubuntu Linux 22.04 LTS.
  • You need a recent Rust environment with a Rust nightly compiler. We recommend rustup to manage Rust https://rustup.rs/.

Dependencies

# Build essentials
apt install build-essential
# lld linker
apt install clang lld
# GDAL (>=  3.4.1)
apt install libgdal-dev gdal-bin
# Proj build dependencies (if libproj >= 7.2 not installed)
apt install cmake sqlite3
# Protocl Buffers
apt install protobuf-compiler

Lints

Rust Code

Please run Clippy with cargo clippy --all-targets --all-features before creating a pull request.

SQL Files

We use SQLFluff to lint our SQL files.

You can install it with pip install sqlfluff.

To lint all SQL files, run sqlfluff lint. This will also be checked in our CI.

There is a VSCode extension available here. This helps to format the files.

Testing

Please provide tests with all new features and run cargo test before creating a pull request.

Edit Settings-test.toml for environment-specific test parameters.

PostgreSQL

For running the PostgreSQL tests, you need to have it installed. Furthermore, you need to create a default user geoengine with the password geoengine.

sudo -u postgres psql << EOF
\set AUTOCOMMIT on
CREATE USER geoengine WITH PASSWORD 'geoengine' CREATEDB;
CREATE DATABASE geoengine OWNER geoengine;
\c geoengine
CREATE EXTENSION postgis;
EOF

During development, you can use the following setting in your Settings.toml to clean the database on server startup:

[postgres]
clear_database_on_start = true
NFDI / GFBio

For running the NFDI/GFBio ABCD data providers (GfbioAbcdDataProviderDefinition and GfbioCollectionsDataProviderDefinition) during development, you need to integrate the test data like this:

# delete existing data
sudo -u postgres psql --dbname=geoengine -c \
  "DROP SCHEMA IF EXISTS abcd CASCADE;"

# insert data
cat test_data/gfbio/init_test_data.sql test_data/gfbio/test_data.sql | \
  sudo -u postgres psql --dbname=geoengine --single-transaction --file -
GBIF

For running the GBIF data provider (GbifDataProviderDefinition) during development, you need to integrate the test data like this:

# delete existing data
sudo -u postgres psql --dbname=geoengine -c \
  "DROP SCHEMA IF EXISTS gbif CASCADE;"

# insert data
cat test_data/gbif/init_test_data.sql test_data/gbif/test_data.sql | \
  sudo -u postgres psql --dbname=geoengine --single-transaction --file -

Benchmarks

For performance-critical features, we aim to provide benchmarks in the benches directory. If you plan on optimizing a feature of Geo Engine, please confirm it this way.

Deployment

Deploy an instance using cargo run --package geoengine-services --bin main --release.

Features

The PostgreSQL storage backend can optionally be enabled using --features postgres in the cargo command.

Configuration

Copy Settings-default.toml to Settings.toml and edit per your requirements.

Troubleshooting

Running out of memory map areas

Problem: When running Geo Engine or its tests you encounter one of the following (or similar) errors:

  • Cannot allocate Memory (OS Error 12)
  • Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

Solution: Adjust the system's max_map_count parameter:

  • Temporary configuration:
    • sudo sysctl -w vm.max_map_count=262144
  • Permanent configuration:
    • Add the following line to /etc/sysctl.d/local.conf
      • vm.max_map_count=262144
    • Reload the system config:
      • sudo sysctl --system

Contributing

We are grateful for any contributions. Please make sure to read our contributing guide.

geoengine's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

geoengine's Issues

Building the container as described in the readme fails

docker build --file=dev.Dockerfile --tag=geoengine:0.0.1 .
   Compiling actix-multipart v0.4.0-beta.9
   Compiling actix-web-httpauth v0.6.0-beta.4
error: failed to run custom build command for `geoengine-services v0.1.0 (/app/services)`

Caused by:
  process didn't exit successfully: `/app/target/release/build/geoengine-services-91e999261e8ed61b/build-script-build` (exit status: 1)
  --- stderr
  Error: could not find repository from '/app/services'; class=Repository (6); code=NotFound (-3)
warning: build failed, waiting for other jobs to finish...
error: build failed
The command 

Inconsistent loading of CSV files with empty fields

Consider the following csv file: lonlat_empty_field.csv

Longitude,Latitude,Name
1.1,2.2,foo
1.1,2.2,

This file will be loaded using a standard OgrSource.
The FeatureDataRef returned by the data-method identifies the string in the scond row as NULL. This can be seen in the following test:

let column: FeatureDataRef = result.data("Name").unwrap();

assert_eq!(column.nulls(), vec![false, true]);
assert_eq!(column.get_unchecked(0), FeatureDataValue::NullableText(Some("foo".to_string())));
assert_eq!(column.get_unchecked(1), FeatureDataValue::NullableText(None));

But at the same time the MultiPointCollection itself stores an empty string:

let pc = MultiPointCollection::from_data(
    MultiPoint::many(vec![vec![(1.1, 2.2)], vec![(1.1, 2.2)]]).unwrap(),
    vec![Default::default(); 2],
    {
        let mut map = HashMap::new();
        map.insert("Name".into(), FeatureData::NullableText(vec![Some("foo".to_owned()), Some("".to_owned())]));
        map
    },
)
    .unwrap();

assert_eq!(result, pc);

This discrepancy could lead to the problem that the Point in Polygon-Operator throws an error.

Possible solution

The GDAL OpenOption "EMPTY_STRING_AS_NULL=YES" can be set (which is done in #486). In that case the MultiPointCollection will store the second string cell as NULL and the Operator functions properly:

let pc = MultiPointCollection::from_data(
    MultiPoint::many(vec![vec![(1.1, 2.2)], vec![(1.1, 2.2)]]).unwrap(),
    vec![Default::default(); 2],
    {
        let mut map = HashMap::new();
        map.insert("Name".into(), FeatureData::NullableText(vec![Some("foo".to_owned()), None]));
        map
    },
)
    .unwrap();

assert_eq!(result, pc);

I do not know the exact reason why the data inside the FeatureDataRef and the internal data in MultiPointCollection happens to be different yet.

Frontend cannot contact backend

After biulding the backend I started it with cargo run:

cargo run  
    Finished dev [unoptimized + debuginfo] target(s) in 0.66s
     Running `target/debug/main`
[flexi_logger][ERRCODE::Time] flexi_logger has to work with UTC rather than with local time, caused by IndeterminateOffset
    See https://docs.rs/flexi_logger/latest/flexi_logger/error_info/index.html#time
[2021-12-15 07:53:20.927504 +00:00] INFO [geoengine_services::server] Starting server… http://localhost:3030/
[2021-12-15 07:53:20.929092 +00:00] INFO [geoengine_services::server] Using in memory backend
[2021-12-15 07:53:20.974746 +00:00] WARN [geoengine_services::datasets::add_from_directory] Skipped adding provider from directory entry: DirEntry("./test_data/provider_defs/gfbio.json") error: SerdeJson: unknown variant `GfbioDataProviderDefinition`, expected `MockExternalDataProviderDefinition` at line 2 column 39
[2021-12-15 07:53:20.979882 +00:00] INFO [actix_server::builder] Starting 8 workers
[2021-12-15 07:53:20.980225 +00:00] INFO [actix_server::server] Tokio runtime found; starting in existing Tokio runtime

However the frontend cannot attach to it:

npm run serve:geoengine:local


> [email protected] serve:geoengine:local
> ng serve --configuration development geoengine-app --proxy-config proxy-local.conf.json

Node.js version v17.2.0 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/releases/.
⠋ Generating browser application bundles (phase: setup)...[HPM] Proxy created: /api  ->  http://localhost:3030/
[HPM] Proxy rewrite rule created: "^/api" ~> ""
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
[HPM] Proxy created: /mapcache  ->  http://sng107.sng.uni-frankfurt.de/mapcache/
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
✔ Browser application bundle generation complete.

Initial Chunk Files | Names         |      Size
vendor.js           | vendor        |  15.59 MB
main.js             | main          |   1.28 MB
polyfills.js        | polyfills     | 128.64 kB
styles.css          | styles        | 104.68 kB
runtime.js          | runtime       |   6.82 kB

                    | Initial Total |  17.11 MB

Build at: 2021-12-15T07:57:53.105Z - Hash: 15809ea91ba4b70668bb - Time: 15892ms

** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **


✔ Compiled successfully.
[HPM] Rewriting path from "/api/session" to "/session"
[HPM] GET /api/session ~> http://localhost:3030/
[HPM] Error occurred while trying to proxy request /session from localhost:4200 to http://localhost:3030/ (ECONNREFUSED) (https://nodejs.org/api/errors.html#errors_common_system_errors)
[HPM] Rewriting path from "/api/anonymous" to "/anonymous"
[HPM] POST /api/anonymous ~> http://localhost:3030/

On the root URL it responds with

{"error":"NotFound","message":"Not Found"}%

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.