koopjs / koop-provider-elasticsearch Goto Github PK

I love this project, I am curious if you have started to think about leveraging aggregations in es much the way BDS does? We have done some work on this and might be able to help along this journey.

I just upgraded my koop instance and it wont work with this provider, any thoughts?

(node:5648) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 15): SyntaxError: Unexpected token o in JSON at position 1

I have just updated koop with it's latest version 4.0.3 and found that Elastic search provider is throwing error - {"error":"features.forEach is not a function"}
Anything needs to be updated in elastic search model?

Opening attribute table creates 500 internal server error. Postman get for the query
FeatureServer/0/query?f=json&returnIdsOnly=true&where=(DT_FIELD > timestamp '2023-01-02 05:59:59') AND (1=1)&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=OBJECTID&outSR=102100
returns
{    "error": "Cannot read properties of undefined (reading 'OBJECTID')"}

If using this
FeatureServer/0/query?f=json&returnIdsOnly=true&where=(DT_FIELD > timestamp '2023-01-02 05:59:59') AND (1=1)&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=OBJECTID

the query works.

why is outSR=102100 causing OBJECTID error

ohh through some quotes around outSR='102100' works too, or drop it entirely also works

f=json&where=navstat%20%3D%20%27under%20way%20using%20engine%27&returnGeometry=true&spatialRel=esriSpatialRelIntersects&geometry=%7B"xmin"%3A-15028131.257092927%2C"ymin"%3A5009377.085700966%2C"xmax"%3A-10018754.171396947%2C"ymax"%3A10018754.171396947%2C"spatialReference"%3A%7B"wkid"%3A102100%2C"latestWkid"%3A3857%7D%7D&geometryType=esriGeometryEnvelope&inSR=102100&outFields=*&outSR=102100&callback=dojo_request_script_callbacks.dojo_request_script2413

It appears the filter is not working with string values that multiple words or spaces

In hitconverter.js, line 174 const flatMapping = flatten(mapping) can create issues, if mapping is undefined. can it be moved post condition defined in line 175?

I am unable to get a feature set to display on an AGOL map viewer.

I have the service up and running and it is populating the console.log ("Features "" Found) with the correct number of records. However the features never make it back to a feature set and into AGOL

I want to hightlight a couple key points / issues we have run into.

1. You do not need to add the geometry field into the returnfields array
2. In order to get the URL route to correctly resolve we currently have to name the esConnection, id and index the same name.

"esConnections": {
"master": {
"id": "master",
"protocol": "http://",
"port": 9200,
"hosts": [
"localhost"
],
"indices": [{
"index": "master",
"maxResults": 10000,
"geometryField": "geometry",
"geometryType": "geo_point",
"returnFields": [

3. You do need to add the date field in both returnfields and dateFields array.

Hello all,

Does anyone know where I can access the metadata.fields?

Thanks

I have added the following to the metadata section of the provider.

            "timeInfo": {
                "startTimeField": "eventtime",
                "endTimeField": "eventtime",
                "trackIdField": null,
                "hasLiveData": true,
                "timeExtent": [new Date(Date.now() - 86400 * 1000).getTime(), Date.now()],
                "timeInterval": 1,
                "timeIntervalUnits": "esriTimeUnitsDays"
            }

I can get the time slider to popup, but the features that come back are not time aware, the query still brings back everything.

https://github.com/koopjs/koop-provider-elasticsearch/blob/master/provider/models/esmodel.js#L22-L24

I was thinking the :host variable was the name of the cluster and the :id was the name of my index (http://myurl.com/es/mycluster/myindex/FeatureService/0), but when I plug those in I get back "cannot get ..."

Any thoughts?

Found it

http://myurl.com/koop/es/mycluster/myindex/FeatureService/0

Hello,

Does anyone know how to change the Coordinate Reference System for geo_point fields.

I know that we can do it for geo_shape something like:
geometry: feature.geometry ? {
type: 'Polygon',
coordinates: [(feature.geometry.rings || [[]])[0].map(coordinates => proj4('EPSG:3857', 'EPSG:4326', coordinates))]
} : null
}));

But if i have my geo_point like this:
geometry: !!feature.attributes?.LATITUDE && !!feature.attributes?.LONGITUDE
? { lat: feature.attributes?.LATITUDE, lon: feature.attributes?.LONGITUDE }
: null

How can i specify the projection for the geo_point because when reading the geohash i am getting the following:
WARNING: [object Object] is not a valid spatial reference; defaulting to none
WARNING: spatial reference "[object Object]" could not be normalized. Defaulting to EPSG:4326.

Hello,

I have an elasticsearch index which does not require a username or password(http://10.4.2.41:9500/hurricane_dorian_2019_test) and I have koop installed where i have set the relevant config and proxy routes. When i run the koop server it is working fine, and when i access the /es/res/info it is showing fine. However, when i access /es/rest/services/10.4.2.41/local/FeatureServer it gives a proxy error (even though i have the route specified). Can anyone let me know what the problem is here and how I can resolve this?

Koop running

Config File

Sample JSON (trying to retrieve lat and lon from location_info --> resolved_location)

Info showing fine

Proxy Error on Webpage

Koop Error

Mod Proxy File

I have a multilinestring geometry type which has nested query. What should I provide in the config to make query something like below:
esquery {
"index": "test_index",
"body": {
"size": 6000,
"query": {
"nested": {
"path": "test_abc",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "test_abc.pqr"
}
}
]
}
}
}
}
}
}

If I give "geometryField": "test_abc.pqr" in the config, It does not work.

Hello developers,

I have a complex elasticsearch mapping where one of my geo_point fields are within other fields e.g. user_location.resolved_location.geometry and when i specified this in the mapping of koop, it would not run successfully. Instead, i had to move the necessary fields outside on the top level so that i can use geometry only in the mapping and get koop to work.

Do you think this feature is something that can be incorporated?

Reason: failed to parse [geo_bounding_box] query. [numeric value expected] - ResponseError: parse_exception: [parse_exception] Reason: failed to parse [geo_bounding_box] query. [numeric value expected]
at onBody (D:\mykoop\node_modules@koopjs\provider-elasticsearch\node_modules@elastic\elasticsearch\lib\Transport.js:367:23)
at IncomingMessage.onEnd (D:\mykoop\node_modules@koopjs\provider-elasticsearch\node_modules@elastic\elasticsearch\lib\Transport.js:291:11)
at IncomingMessage.emit (node:events:525:35)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

"dependencies": {
"@koopjs/koop-core": "^8.0.1",
"@koopjs/provider-elasticsearch": "^3.4.1"
}

Not sure why I cant get polygons to render. In es how should i set the mapping, geo_shape or shape. For value to insert should it be string wkt?

"geometry": "POLYGON ((-95.62829707113862 29.99388366204915, -95.62853292235582 29.994088598628586, -95.62853598596803 29.994126824084642, -95.62836662022042 29.994282742975376, -95.62828537578142 29.994223282183523, -95.62810862399301 29.994053385238534, -95.62811578108486 29.9940470115642, -95.62829707113862 29.99388366204915))"

}
}

    "geometry": {
      "type": "shape"
    },


  "indices": {
    "polyService": {
      "index": "hcad2",
      "geometryField": "geometry",
      "geometryType": "Polygon",
      "reversePolygons": false,
      "returnFields": [
        "hcad_num",
        "owner",
        "appr_val"
      ],
      "dateFields": [],
      "subLayers": [],
      "maxResults": 1000
    }
  }

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update dependency @koopjs/cli to v1.1.9
• Update dependency @elastic/elasticsearch to v8
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency moment to v2.29.4 [SECURITY]
• Update dependency @koopjs/koop-core to v8.0.11
• Update dependency @mapbox/geojson-rewind to v0.5.2
• Update dependency gh-release to v6.0.4
• Update dependency polygon-splitter to v0.0.11
• Update dependency @elastic/elasticsearch to v7.17.12
• Update dependency proj4 to v2.9.0
• Update dependency @koopjs/koop-core to v9
• Update dependency gh-release to v7
• Update dependency h3-js to v4
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Hello developers,

I have several tweet texts in elasticsearch stored as a text, but since they contain emojis which are utf-8 encoded and when i try to display the text on the map, the emojis don't show up properly. Therefore, one suggestion is to incorporate this functionality if its possible.

Thanks

I am not able to see the results in the table.

I have narrowed it down to this error

{
error: "Cannot read property 'timestamp' of undefined"
}

If returnGeometry=true then the features come across and draw on the map.
if returnGeometry=false the you get the error message uptop.

Hi, thanks, we have fixed the issue with data loading in the table, but I have a new question.

If I set up the appConfig file to return 10,000 records it tries to load all of those into the table, and there is no pagination set up.

Hello developers,

After extensive testing I have found that Koop does not work with geohash type as it cannot decode the lat and lon properly, instead, i had to change the mapping to a string type for the geo_point field.

It would be great, if Koop could support geohashes.

I have done some work in the old koop es provider to handle the query string statistics request within es, so that you can a true representation of unique values (for example) in order to do this we intercept the request coming from agol and modify the request going to es.

Example below, I have not looked yet into how to do this in the new es provider

`if (query.groupByFieldsForStatistics) {
            esQuery.body.aggs = {
                "uniqueValueCount": {
                    "cardinality": {
                        "field": query.groupByFieldsForStatistics + ".keyword"
                    }
                },
                [indexConfig.index]: {
                    "terms": {
                        "field": query.groupByFieldsForStatistics + ".keyword",
                        "size": 10000
                    }
                }
            };
        }
        if (query.objectIds) {
            esQuery.body.query.bool.filter = {
                ids: {}
            };


            esQuery.body.query.bool.filter.ids.values = [query.objectIds];

   `

`var whereParser = new WhereParser();
        if (where) {
            if (where.includes("UPPER")) {
                where = where.replace(/UPPER/g, '');
            }
            if (where.includes("objectid")) {
                where = where.replace("objectid", '_id');
            }
            if (where.includes("OBJECTID")) {
                where = where.replace("OBJECTID", '_id');
            }
            var boolClause = whereParser.parseWhereClause(where, indexConfig.dateFields, indexConfig.returnFields);
            if (boolClause) {
                if (boolClause.bool) {
                    if (boolClause.bool.must) {
                        boolClause.bool.must.push({ exists: { field: indexConfig.geometryField } });
                    }
                    else {
                        boolClause.bool.must = [
                            { exists: { field: indexConfig.geometryField } }
                        ];
                    }
                    esQuery.body.query = boolClause;
                }
                else {
                    esQuery.body.query.bool.must.push(boolClause);
                }
            }

        }`

All, thanks for posting this updated es provider, we are just now digging in. I have a couple clarifications.

What is the param in the config file called "shapeIndices" it looks like it has something to do with the join index?

With regard to aggregations I notice that there is two options, one is geohash, one is geohex, when I look in the code I notice a reference to an aggServerConfig

const options = {
hostname: aggServerConfig.hostname,
port: aggServerConfig.port,
path: '/algorithm/execute',
method: 'POST',
headers: {
'Content-Type': 'application/json'
}
};

I am wondering in what is that in reference to?

A property in elasticsearch can be a single value or an array. When the value is an array this provider currently serializes it as

[object Object]

but could instead flatten the array to e.g.

property.0: "foo", 
property.1: "bar"

or for arrays of objects with nested properties:

property.0.a: "foo", 
property.0.b: "bar", 
property.1.a: "fizz", 
property.1.b: "buzz"

Its a super-simple fix to make this work: in src/utils/hitConverter.js: change the opts given to flatten on line 34 to {safe: false}.

The fix is so simple that it made me wonder if this was on purpose (that there's a good reason not to try to support arrays at all) and being new to koop and esri featureserver i'm missing something?

Has anyone been able to get this to work?

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Ok, I will post some of my thoughts here on this, starting with the config file. I think a config file would need to following changes in order to trigger results as an aggregation. basically the aggregation feature would return a hash polygon leveraging the geojson that is returned from ES.

Getting the spatial results back should be very straight forward. The hard part will be in ensure that all the attribute fields still come back across because of the need for filtering and time series interactions.

{
  "appInfo": {
    "protocol": "http",
    "listenPort": 80
  },
  "esConnections": {
    "firstESCluster": {
      "id": "clusterID",
      "protocol": "http://",
      "port": 9200,
      "hosts": [
        "escluster.mynetwork.com"
      ],
      "indices": [
        {
          "index": "indexOrAliasName",
          "maxResults": 6000,
          "geometryField": "geometry",
          "geometryType": "geo_point/Point/Polyline/Polygon",
          "returnFields": [
            "fieldFromIndex", "SeenInFeatureService", "SomeDateField"
          ],
          "dateFields": [
            "pickup_date", "SomeDateField"
          ],
          **"aggregation": {
               "type": "count",  (sum/average/max/min/count)
               "output": "geo_point", (hash centroid geo_point, hash polygon, turf hexagon)
               "field": "GEOMETRY", (GEOMETRY for count, field for all other types)
               "resolution": 9  //this might not be needed because it can be calulated on the fly
            },**
          "timeInfo": {
            "startTimeField": "start_date",
            "endTimeField": "end_date",
            "timeExtent": [1438401615000, 1439833466000],
            "timeInterval": 1,
            "timeIntervalUnits": "esriTimeUnitsDays"
          }
        }
      ]
    }
  }
}

Hello developers.

I am returning an image url in the return fields but would like to visualize these images on the map. Is this feature something that can be implemented?

Thanks

Have you tested the native app support for android and iphone?

I am getting this error when I try and bring in to an android phone

the legend comes in fine, just no data

index > TypeError: Cannot read property 'wkid' of undefined │
│[12] index Mem: 71 MB CPU: 0 % onli ││ index > at buildESQuery (es/provider/models/esmodel.js:113:37) │
│[13] index Mem: 84 MB CPU: 0 % onli ││ index > at indexInfo.getMapping.then.mapping (es/provider/model │
│[14] index Mem: 88 MB CPU: 0 % onli ││ s/esmodel.js:36:31) │
│[15] index Mem: 70 MB CPU: 0 % onli ││ index > at process._tickDomainCallback (internal/process/next_tick.js:135:7) │
│[16] index Mem: 82 MB CPU: 0 % onli ││ index > You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection: │
│[17] index Mem: 82 MB CPU: 0 % onli ││ index > TypeError: Cannot read property 'wkid' of undefined │
│[18] index Mem: 71 MB CPU: 0 % onli ││ index > at buildESQuery (/home/ec2-/es/provider/models/esmodel.js:113:37) │
│[19] index Mem: 70 MB CPU: 0 % onli ││ index > at indexInfo.getMapping.then.mapping (/es/provider/model │
│ ││ s/esmodel.js:36:31) │
│ ││ index > at process._tickDomainCallback (internal/process/next_tick.js:135:7)

I am currently working with geo-point data in ES, and my geo_point data have a format like this [1].

PUT my-index-000001/_doc/1
{
  "text": "Geo-point as an object",
  "location": { 
    "lat": 41.12,
    "lon": -71.34
  }
}

When I gave it a try, accessing the FeatureServer, it gave me this error.

...
{"message":"Failed to parse {\"_index\":\"INDEX_NAME\",\"_type\":\"_doc\",\"_id\":\"DOC_ID\",\"_score\":1,\"_source\":{\"@timestamp\":TIMESTAMP,\"ident\":\"IDENT\",\"location.lon\":LONGITUDE,\"location.lat\":LATITUDE}} with following error:","level":"warn"}
{"level":"error"}
...

I found out that the location object is flattened and split into two different fields (location.lon and location.lat).

I assume that the error comes from these section of code at hitConverter (link)

               if(Array.isArray(feature.geometry)){

                    // If the index config is set to allow multiple points. If it isn't, then only the first point is used.
                    if (indexConfig.allowMultiPoint === true) {
                        pointType = "MultiPoint";
                        coords = [];
                        if (!isNaN(feature.geometry[0])){ // case where we allow multipoint but this is a single point
                            coords.push(feature.geometry);
                        } else {
                            feature.geometry.forEach((geom) => {
                                if(geom.hasOwnProperty('lon')){
                                    coords.push([geom.lon, geom.lat]);
                                } else {
                                    coords.push(geom);
                                }
                            });
                        }
                    } else {
                        // The index was not configured for multipoint, so only the first point is used.
                        pointType = "Point";
                        if(feature.geometry[0].hasOwnProperty('lon')){
                            coords = [feature.geometry[0].lon, feature.geometry[0].lat];
                        } else {
                            coords = feature.geometry;
                        }
                    }
                } else {
                    pointType = "Point";
                    coords = feature.geometry.split(",").map( coord => {
                        return parseFloat(coord);
                    }).reverse();
                }

This block assumes that the incoming hit with single point has a format of string only (something like "41.12,-71.34").

Is this truly an unhandled format? Or is there any extra configuration in my format case? Or should I change the format to string?
Thank you 😊

Reference

[1] Geo-point field type

It appears that other filters are working in agol / portal for arcgis. however when I try to use the time filter it no longer works, thowing

(node:7314) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 41): SyntaxError: Syntax error found near Identifier (WHERE Clause)

This is the qs param that I am passing

query?f=json&where=timestamp%20BETWEEN%20timestamp%20%272018-04-08%2005%3A00%3A00%27%20AND%20timestamp%20%272018-04-09%2004%3A59%3A59%27&returnGeometry=true&spatialRel=esriSpatialRelIntersects&geometry=%7B"xmin"%3A-10801469.341038998%2C"ymin"%3A3443946.7464246675%2C"xmax"%3A-10644926.307111034%2C"ymax"%3A3600489.7803526316%2C"spatialReference"%3A%7B"wkid"%3A102100%2C"latestWkid"%3A3857%7D%7D&geometryType=esriGeometryEnvelope&inSR=102100&outFields=*&outSR=102100&callback=dojo_request_script_callbacks.dojo_request_script38

This is a subset of our config file, for some reason the join is not working, it comes in as a table, but even the table doesnt populate with data, not sure what we are doing wrong

`{

`"esConnections":` {

	"y": {

		"id": "y",

		"protocol": "https://",

		"hosts": [

			"yyy"

		],

		"shapeIndices": {

			"zzz": {

				"geometryField": "GEOMETRY",

				"geometryType": "geo_point",

				"joinField": "MMSI"

			}

		},

		"indices": {

			"ddd": {

				"index": "hhh",

				"allowMultiPoint": false,

				"returnFields": [

					"CAPABILITIES",

					"DECK_SIZE",

					"FLAG",

					"LINK_TO_SPECS",

					"OWNER",

					"TOP_SPEED",

					"SHIPNAME",

					"YEAR_BUILT",

					"VESSEL CLASS"

				],

				"aggregations": [],

				"shapeIndex": {

					"name": "zzz",

					"joinField": "MMSI"
				},

				"maxResults": 1000

			}
		}
	}
}

}`

First round of proposed improvements

• Restructure provider to follow standards (https://github.com/koopjs/koop-provider-example#koop-provider-file-structure)
  • provider/models/esmodel.js -> src/model.js and refactor to be an ES6 class
  • move initialization logic here and here to functions called from the Model class constructor and nix app.js
  • provider/controllers/index.js -> src/index.js
  • provider/utils.js -> src/utils.js and refactor utils to be pure, classless JS functions exported from a utils module using a "bucket" file.
  • Drop Express and Koop instantiation logic
• Drop these pointless things: https://github.com/koopjs/koop-provider-elasticsearch/blob/master/app.js#L118-L128
• Clean this up: https://github.com/koopjs/koop-provider-elasticsearch/blob/master/provider/models/esmodel.js#L28-L63
• Add tests

Outstanding Questions

• What do we do to adapt appConfig.json and this bespoke cert-loading system into the standardized spec?

Hello everyone !

I'm Remi from France, I'm working for the city of Versailles and I'm highly interested by this provider.

My main problematic is :
We have several smart cities components connected to an elastic search database, we wanted to connect to index in this database without making replication in order to visualize datas with a map server in webmap, dashboard etc.... I understood that it could be possible with koop and this provider, am I right ?

I have actually some difficulties with using this provider. My first question is : Is it possible to connect to es with a user/password authentification ? I couldn't find an example with this issue and I tried naively to add a password variable in the esConnection class inside the default.json file.....

I'm really sorry in advance that I can't contribute with the coding part but I can maybe use some time in order to write some more "user friendly", detailed documentation if you're interested.

Best regard,

Rémi

I have issues getting things to work since there is missing lots of documentation.
Or is it a requirement that i should read and understand all code to make it work?

Whats the ES support? Versions? All should work?

Parameters like:
timeInfo parameters - (start,end, intervall..) important for timeEnabled layers
timefields have to be in unix_ms to get the time extent correct.

Aggregations parameters - ["geohash"] doesn't work at all, cant even see layer/1 anymore since 2.5. Works fine in Kibana.
How should custom aggregations be specified?

Custom symbolizer...

the list goes on...