Code Monkey home page Code Monkey logo

winnow's Introduction

Winnow

DEPRECATED - now migrated to a package in the Koop monorepo. Update any npm usage to @koopjs/winnow.

No Maintenance Intended npm version js-standard-style Greenkeeper badge

Winnow is made for applying sql to geojson in memory. It is useful for working against geojson objects but also has built-in primitives for piping streams.

API

winnow.query

Build and apply a query to a feature collection object or an array of features

const features = Object
const options = {
  where: String // A sql where statement
  geometry: Object // GeoJSON or Esri geometry Object
  spatialPredicate: String // ST_Within || ST_Contains || ST_Intersects
  fields: Array // Set of fields to select from feature properties
  aggregates: Object // Describes the set of aggregations to perform on fields
  groupBy: Array // Set of fields for grouping statistics
  limit: Number // number of results to return
  offset: Number // number of return features to offset
  order: Array // Set of fields or aggregates by which to order results
  outputCrs: Number || String // An EPSG code, an OGC WKT or an ESRI WKT used to convert geometry
  inputCrs: Number || String // An EPSG code, an OGC WKT or an ESRI WKT defining the coordinate system of the input data. Defaults to 4326 (WGS84)
  toEsri: Boolean // return Esri feature collection
  geometryPrecision: Number // number of digits to appear after decimal point for geometry
  classification: Object // GeoJSON or geoservices classification Object
}
winnow.query(features, options)
// Returns the set of features that match the query

where

A sql where statement.

  • 'Trunk_Diameter > 10'
  • 'Trunk_Diameter > 10 AND Genus like 'Quercus%'
  • (Genus like '%Quercus%' OR Common_Name like '%Live Oak%') AND Street_Type like '%AVE%'

geometry

A GeoJSON or Esri Geometry Object

// GeoJSON Polygon
{
  type: 'Polygon',
  coordinates: [[[-118.163, 34.162], [-118.108, 34.162], [-118.108, 34.173], [-118.163, 34.173], [-118.163, 34.162]]],
}

// Esri Envelope (aka Bounding box)
{
 xmin: -13155799.066536672,
 ymin: 4047806.77771083,
 xmax: -13143569.142011061,
 ymax: 4050673.16627152,
 spatialReference: {
   wkid: 102100
 }
}

spatialPredicate

Specifies the relationship between the passed-in geometry and the features in the data

  • ST_Within: Features in the data must be completely within the passed-in geometry
  • ST_Contains: Features in the data must completely contain the passed-in geometry
  • ST_Intersects: Features in the data must intersect the passed-in geometry

Can also specify the esri-style predicates esriSpatialRelWithin, esriSpatialRelContains, esriSpatialRelIntersects

fields

An array that specifies fields should be returned from each features properties or attributes. Will also accept a comma-delimited string.

e.g. [Trunk_Diameter, Common_Name, Genus]

aggregates

An array that specifies aggregations to apply across all features or properties. Must specify at least a type and a field. Providing a name for the aggregation is optional

Types: [sum, avg, count, max, min, first, last]

e.g:

[
  {
    type: 'sum',
    field: 'Trunk_Diameter',
    name: 'Total_Trunk_Diameter'
  },
  {
    type: 'avg',
    field: 'Trunk_Diameter'
  }
]

groupBy

An array of fields used to group the results. Must be used with aggregates. Note this will not return any geometry

limit

The total number of results to return

offset

The amount to offset returned features (e.g., 10 will skip the first 10 returned features). limit is required to used offset.

order

An array of fields use to sort the output

outputCrs (formerly projection)

Can be an EPSG code, an OGC wkt or an Esri wkt. This parameter controls how the geometry will be projected to another coordinate system.

inputCrs

Can be an EPSG code, an OGC wkt or an Esri wkt.. This parameter defines the coordinate system of input data. If the incoming data is a GeoJSON feature collection with a "named" crs property defined according to specification, winnow will use it's value if inputCrs is undefined. If neither is defined, Winnow assumes the input data has coordinate system WGS84.

toEsri

If true, the object returned will be an esri feature collection.

Winnow will automatically determine field types from the first feature passed in. If a given attribute is null, Winnow will assume it is a string.

You can also pass in a metadata object that describes the fields in the feature collection. This is recommended if you know your schema ahead of time

e.g.

{
  type: 'FeatureCollection',
  features: [],
  metadata: {
    fields: [
      {
        name: 'SomeDateField',
        type: 'Date'
      },
      {
        name: 'SomeDoubleField',
        type: 'Double'
      },
      {
        name: 'SomeIntegerField',
        type: 'Integer'
      },
      {
        name: 'SomeStringField',
        type: 'String'
      }
    ]
  }
}

geometryPrecision

A number for geometry precision. Geometry values will be truncated to the supplied decimal place.

classification

An object for classification aggregation. Classification ouputs an array of breaks on features. There are two supported classification types: Class Breaks and Unique Value. Classification supports input from FeatureServer's generateRenderer classificationDef.

Class Breaks

Class Breaks is used to classify numeric data based on a number of breaks and a statistical method. Features can also be normalized before being classified.

{
  *type: 'classes',
  *field: '<field1>',
  *method: 'equalInterval' | 'naturalBreaks' | 'quantile' | 'std',
  *breakCount: 7,
   normType: 'field' | 'log' | 'percent',
   normField: '<field2>' // mandatory if normType === 'field'
}
*required

e.g. An example feature collection has a field called field1 ranging in value from 0 - 29.

Input:

{
  type: 'classes',
  field: 'field1',
  method: 'equalInterval',
  breakCount: 5,
}

Output (array of class intervals):

[ [0-5],
  [6-11],
  [12-17],
  [18-23],
  [24-29] ]
Unique Value

Unique Value is used to classify data based on a unique field(s). The output is an array of objects for each unique value combination. Each object contains an instance count, and the classifying unqiue field names and values.

{
  *type: 'unique',
  *fields: ['<field1>', '<field2>', '<field3>'] // up to three fields
}
*required

e.g. An example feature collection has unique fields called employeeID and customerID.

Input:

{
  type: 'unique',
  fields: ['employeeID', 'customerID']
}

Output (array of instance objects):

[
  {count: 3, employeeID: 'A', customerID: 'M'},
  {count: 1, employeeID: 'A', customerID: 'N'},
  {count: 1, employeeID: 'B', customerID: 'M'},
  {count: 2, employeeID: 'B', customerID: 'N'},
  {count: 2, employeeID: 'B', customerID: 'O'},
  {count: 1, employeeID: 'C', customerID: 'O'},
]

winnow.prepareQuery

Returns a function that can be applied directly to a feature collection object, an array of features, or a single feature. Useful when you want to pass a stream of features through a filter.

const options = {
  where: String,
  geometry: Object,
  spatialPredicate: String,
  fields: Array,
  aggregates: Array
}
const filter = winnow.prepareQuery(options)
filter(geojson)
// returns the set of feature that match the query

winnow.querySql

Execute sql directly against the query engine.

  • Replace any variables with ?
  • Table name should always be replaced by ?
  • Non-string values always be replaced by ?
const statement = 'Select * from ? where Genus in ?'
const data = geojson
const genus = ['Quercus']
winnow.querySql(statement, [geojson, genus])
// returns all features that match the query

winnow.prepareSql

Pass in a statement and return a filter than can be applied to a feature collection object, an array of features or a single feature. Variables work in the same way as winnow.sql

const statement = 'Select Trunk_Diameter from ? where Trunk_Diameter > 100'
const filter = winnow.prepareSql(statement)
filter(geojson)
// returns all the features that match the query

OBJECTID generation

When option toEsri is true, winnow will check that features have a unique-identifier field. This is field is flagged with the idField option. If not present, winnow will look for a field named OBJECTID and use it as the unique-identifier by default. If no OBJECTID is present, winnow creates one by generating a numeric hash of the feature. By default, winnow uses FarmHash for hashing. Some cloud deployments currently have trouble executing Farmhash (Heroku, 2020-09-30), so you can use native Javascript hashing as an alternative. Simply set the environment variable OBJECTID_FEATURE_HASH=javascript.

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing and the RELEASE.md for a description of our release process.

License

Apache 2.0

winnow's People

Contributors

ahamilt avatar corinchappy avatar dependabot[bot] avatar efbenson avatar greenkeeper[bot] avatar gwright avatar haoliangyu avatar jahndis avatar jgravois avatar mathiasrw avatar mishugana avatar nheminger avatar renovate[bot] avatar rgwozdz avatar slibby avatar thomas-hervey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

winnow's Issues

Geometry not removed when option returnGeometry = false

Queries that include the option returnGeometry: false should not return features with a geometry property. However, winnow version <= v1.12.6 returns geometry property and data even when returnGeometry: false.

This was found to prevent ArcGIS Pro 2.1.2 from loading an attribute table for a koop service (Pro reported "Unknown Error"), initially noted in koopjs/koop#304.

Adding @jgravois @jptheripper @WalterPayne and @rminderhoud as participants from that thread.

OBJECTID queries

In cases where providers do not define an idField and koop generates the OBJECTID property by default, requests with where parameters that include the OBJECTID will return zero features. This happens because the alasql library applies the WHERE clause to the data before the esriFy function that creates the OBJECTID. Since OBJECTIDs don't exist at that point, zero features are selected.

When no idField is set, and the where includes OBJECTID we need to strip that part of the where before it heads off to alasql.

override groupByFieldsForStatistics in winnow

we are dealing with very large datasets. These large data sets need the field stats calculated at the server level, not at a winnow level. We are interested in doing groupby stats against possible tens of thousands of different unique field values.

I can easily do this and mirror the necessary response the client is expecting in arcgis, however the stats are getting calculated again in winnow (i think?) and thus the wrong counts are provided (usually a count of 1 or 0) is there a way to easily request winnow to except the values provided by the geojson as the correct field stats?

Version 10 of node.js has been released

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

  • Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

More information on this issue

Greenkeeper has checked the engines key in any package.json file, the .nvmrc file, and the .travis.yml file, if present.

  • engines was only updated if it defined a single version, not a range.
  • .nvmrc was updated to Node.js 10
  • .travis.yml was only changed if there was a root-level node_js that didn’t already include Node.js 10, such as node or lts/*. In this case, the new version was appended to the list. We didn’t touch job or matrix configurations because these tend to be quite specific and complex, and it’s difficult to infer what the intentions were.

For many simpler .travis.yml configurations, this PR should suffice as-is, but depending on what you’re doing it may require additional work or may not be applicable at all. We’re also aware that you may have good reasons to not update to Node.js 10, which is why this was sent as an issue and not a pull request. Feel free to delete it without comment, I’m a humble robot and won’t feel rejected 🤖


FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

OBJECTID queries

In cases where providers do not define an idField and koop generates the OBJECTID property by default, requests with where parameters that include the OBJECTID will return zero features. This happens because the alasql library applies the WHERE clause to the data before the esriFy function that creates the OBJECTID. Since OBJECTIDs don't exist at that point, zero features are selected.

When no idField is set, and the where includes OBJECTID we need to strip that part of the where before it heads off to alasql.

WHERE clause with multiple conditions returns nothing

When using a WHERE clause that has multiple conditions, nothing will be returned. Example:

const feature = {
            "type": "Feature",
            "geometry": {
                "type": "Point",
                "coordinates": [-73.5, 45.5]
            },
            "properties": {
                "city": "montreal",
                "province": "quebec"
            }
 }

const options = {
        where: "city='montreal' AND province='quebec'",
};

const result = winnow.query([feature], options); // Returns nothing

Stepping through the query creation shows that the where clause is being created as:

"properties->city = 'montreal' AND province='quebec'".

I think the problem might be in the pad() method in where.js. It doesn't seem to be able to handle more than one condition.

Winnow does not support point coordinates for geometry param

While using koop, I ran into an issue where a url like:
https://featureserver.com/provider/id/FeatureServer/0/query?f=json&geometryType=esriGeometryPoint&geometry=7652714.335571289%2C-1985856.184782909&geometryType=esriGeometryPoint&inSR=102100&outSR=102100
would cause koop to respond with a 500 internal server error.

It appears that winnow currently doesn't support the geometry=x,y query param format for the esriGeometryPoint geometryType as specified here: https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer-.htm
It currently tries to construct a bbox from the two x,y coordinates, resulting in an invalid bbox with NaN values, which causes koop to respond with 500's (at least as of koop version 4.1.1).

We are able to work around it in our provider by overwriting the geometry with a "bounding box" consisting of x/y min/max that are the same values (i.e. "-10,10" => "-10,10,-10,10") doing the following:

  if (!!query.geometry && query.geometry.split(',').length === 2) {
    query.geometry = `${query.geometry},${query.geometry}`;
  }

But obviously, this is less than ideal.

I have a PR that fixes this issue for our use case, although I'd appreciate someone else taking a look to see if there are any other implications of introducing this support. #202

When returnGeometry is false and outSR is 102100 winnow throws an error

It seems contradictory to have them both sent, but AGOL does it when opening the grid of a layer in the map viewer and it throws a modal error.

image

I console dir'ed the params for esrify and properties are null.

This is a regression at some point, it was working a before at 1.12.5

{ properties: undefined,
  geometry: undefined,
  dateFields: '',
  requiresObjectId: 'true',
  idField: 'OBJECTID' }
Trace: TypeError: Cannot read property 'OBJECTID' of undefined
    at Object.esriFy (F:\proj\htrack\src\mapping-api\node_modules\winnow\dist\sql.js:155:38)
    at alasql.Query.eval [as selectfn] (eval at yy.Select.compileSelect2 (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:9734:9), <anonymous>:3:40)
    at doJoin (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:7461:27)
    at doJoin (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:7545:8)
    at queryfn3 (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:6963:2)
    at queryfn2 (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:6937:9)
    at Object.eval [as datafn] (eval at <anonymous> (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:8415:20), <anonymous>:3:57)
    at F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:6885:19
    at Array.forEach (<anonymous>)
    at queryfn (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:6881:16)
    at statement (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:8026:14)
    at Function.alasql.dexec (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:4619:11)
    at Function.alasql.exec (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:4601:17)
    at alasql (F:\proj\htrack\src\mapping-api\node_modules\alasql\dist\alasql.fs.js:138:16)
    at standardQuery (F:\proj\htrack\src\mapping-api\node_modules\winnow\dist\executeQuery.js:34:18)
    at Object.Winnow.query (F:\proj\htrack\src\mapping-api\node_modules\winnow\dist\index.js:33:17)
    at query (F:\proj\htrack\src\mapping-api\node_modules\featureserver\dist\query.js:27:28)
    at execQuery (F:\proj\htrack\src\mapping-api\node_modules\featureserver\dist\route.js:36:16)
    at Object.route (F:\proj\htrack\src\mapping-api\node_modules\featureserver\dist\route.js:27:14)
    at F:\proj\htrack\src\mapping-api\node_modules\koop-output-geoservices\index.js:9:24
    at F:\proj\htrack\src\mapping-api\node_modules\koop\dist\models\index.js:15:9
    at route.then (F:\proj\htrack\src\mapping-api\lib\provider\model.js:40:13)
    at execQuery (F:\proj\htrack\src\mapping-api\node_modules\featureserver\dist\route.js:38:52)
    at Object.route (F:\proj\htrack\src\mapping-api\node_modules\featureserver\dist\route.js:27:14)
    at F:\proj\htrack\src\mapping-api\node_modules\koop-output-geoservices\index.js:9:24
    at F:\proj\htrack\src\mapping-api\node_modules\koop\dist\models\index.js:15:9
    at route.then (F:\proj\htrack\src\mapping-api\lib\provider\model.js:40:13)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)

An in-range update of alasql is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! 💜 🚚💨 💚

Find out how to migrate to Snyk at greenkeeper.io


The dependency alasql was updated from 0.6.0 to 0.6.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

alasql is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 5 commits.

  • 9747404 Let ADD COLUMN use the given dbtypeid to fix #1196
  • 3865012 Mergefix
  • 4779e00 Adds BETWEEN op scenario to the toString prototype for yy.Ops class (#1194)
  • c610d72 Update README.md
  • b123866 Updated version in files to 0.6.0

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Ordering is broken

const params = Query.params([feature], options)

Because the way that this is implemented and querying per feature vs against all of them order does not work. I am happy to do a PR to fix this, however I need some direction. I think the cleaner way would be to run the query against all the features so you get proper order and limiting/skipping but I don't know if there was a reason for one offing the queries. The other option is to build out a custom (or use some lib) to do ordering on the data outside of alasql.

An in-range update of farmhash is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! 💜 🚚💨 💚

Find out how to migrate to Snyk at greenkeeper.io


The optionalDependency farmhash was updated from 3.0.0 to 3.1.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

farmhash is a optionalDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 4 commits.

  • 43e540a Release v3.1.0
  • 946779f CI: add Node.js 14
  • fa3bbb7 Bump all deps (Node.js 10+ already required)
  • f26a6f4 Add index.d.ts type declaration (#33)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Normalize options not working when inSR is JSON string

Insights 2.3 is passing a query using URLs that look like this

https://<featurelayer-url>/query?f=json&geometry=%7B%22xmin%22%3A-20037508.342788905%2C%22ymin%22%3A-20037508.342788905%2C%22xmax%22%3A0%2C%22ymax%22%3A0%7D&inSR=%7B%22wkid%22%3A102100%7D&geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&outFields=url%2COBJECTID&returnGeometry=true&outSR=%7B%22wkid%22%3A102100%7D&resultOffset=0&where=1%3D1&callback=dojo.io.script.jsonp_dojoIoScript12._jsonpCallback

The inSR is a JSON string and is not being parsed into a JSON object so is not being normalized correctly.

Is this an issue with the way Insights is passing in the Spatial reference JSON object or should the string be parsed into a JSON object by winnow?

An in-range update of ngeohash is breaking the build 🚨

The dependency ngeohash was updated from 0.6.0 to 0.6.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ngeohash is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of flora-sql-parser is breaking the build 🚨

The dependency flora-sql-parser was updated from 0.9.1 to 0.9.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

flora-sql-parser is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 3 commits.

  • abe092c 0.9.2
  • 0e070d9 implemented "NOT BETWEEN" operator in createBinaryExpr()
  • 790b859 Update dependencies and Travis CI for Node.js 12

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of @esri/proj-codes is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! 💜 🚚💨 💚

Find out how to migrate to Snyk at greenkeeper.io


The dependency @esri/proj-codes was updated from 2.3.0 to 2.5.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@esri/proj-codes is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of proj4 is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! 💜 🚚💨 💚

Find out how to migrate to Snyk at greenkeeper.io


The dependency proj4 was updated from 2.6.1 to 2.6.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

proj4 is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

High volumes of requests cause date queries to break

I am thinking this is where the issue would go. When we are in prod, ~4 million requests a day across 3 instances, there are cases where date queries return 0 results. You can keep querying over and over though koop (using querystrings generated by AGOL) and once and a while (frequency tied to loading) you get 0 results, even with no changes in data (running out of koop's memory cache). This does not happen with string or number data based queries. I am doing my best to try and run down the issue myself, but any ideas would be helpful.

Sample where: eta>timestamp '2019-01-01 04:59:59'
queryString: query?f=json&where=eta%3Etimestamp%20%272019-01-01%2004%3A59%3A59%27&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=*&outSR=102100&resultOffset=0&resultRecordCount=25

An in-range update of simple-statistics is breaking the build 🚨

Version 4.2.0 of simple-statistics just got published.

Branch Build failing 🚨
Dependency simple-statistics
Current Version 4.1.1
Type dependency

This version is covered by your current version range and after updating it in your project the build failed.

simple-statistics is a direct dependency of this project this is very likely breaking your project right now. If other packages depend on you it’s very likely also breaking them.
I recommend you give this issue a very high priority. I’m sure you can resolve this 💪

Status Details
  • continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

License issue with flora-sql-parser

The flora-sql-parser package used for SQL parsing is a fork from @florajs/sql-parser, a package with GPL-2 license. It requires all downstream software to be open-source, which creates a problem for those who are developing proprietary solutions with winnow. The package should be replaced by an MIT-licensed package such as pgsql-ast-parser.

Aggregate queries should not include LIMIT

Aggregate queries are currently being constructed with a LIMIT fragment, i.e SELECT SUM(properties->price) as sum_price FROM ? LIMIT 2000.

In cases where a limit or resultRecordCount parameter was not part of the request, a LIMIT is still added to the SQL using the maxRecordCount or a default value of 2000. This doesn't give an accurate sum of the record set, because its only calculated over the subset of 2000 records.

I checked the behavior of feature services; requests with both outStatistics and resultRecordCount parameters return a 400 error. Requests with both outStatistics and limit calculate the statistics over the full record set. Winnow should follow this pattern.

Any spatial indexing like rtree, btree, rbush was used?

I am new to koop.js

Any spatial indexing like rtree, btree, rbush was used in winnow?

for example: 

    terraformer.js use rtree
    sqlite  use rtree
   sql server spatial use btree
   oracle use ?
   postgis use ?

I have not test yet,
How the spatial query performance is? Would rtree make it faster?

An in-range update of flora-sql-parser is breaking the build 🚨

Version 0.7.10 of flora-sql-parser was just published.

Branch Build failing 🚨
Dependency flora-sql-parser
Current Version 0.7.9
Type dependency

This version is covered by your current version range and after updating it in your project the build failed.

flora-sql-parser is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

Commits

The new version differs by 1 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

The request from ArcGIS Pro with projection seems to be failing.

I am testing koop.js to use some custom data (basically geojson source) in ArcGIS Pro through koop.js.
So far it is working pretty great. Thanks.

I ran into one problem which is
"When the spatial reference is other than 4326, it seems reprojection in koop.js seems to be failing."

The symptoms

It looks like ArcGIS Pro send the request to koop.js with the following info.
(This is California State Plane III but the SR does not matter it seems to happen with all the SRs)

{"wkid":102643,"latestWkid":2227,"xyTolerance":0.003280833333333333,"zTolerance":0.001,"mTolerance":0.001,
"falseX":-115860600,"falseY":-93269500,"xyUnits":3048.0060960121928,"falseZ":-100000,"zUnits":10000,
"falseM":-100000,"mUnits":10000}

The with WARNINGS as follows.

WARNING: [object Object] is not a valid spatial reference; defaulting to none
WARNING: spatial reference "[object Object]" could not be normalized. Defaulting to EPSG:4326.

Possible cause and solution

Quick debug seems to indicate that only the validation seems to be failing.
https://github.com/koopjs/winnow/blob/master/lib/normalize-query-options/spatial-reference.js#L19

It seems only the validation is failing.
The projection works perfectly, if simply the validation is bypassed or the input is stringified.

const { error } = schema.validate(JSON.stringify(input))

Joi validation may have to updated or the input has to be serialized?

Thanks.

An in-range update of flora-sql-parser is breaking the build 🚨

Version 0.7.8 of flora-sql-parser was just published.

Branch Build failing 🚨
Dependency flora-sql-parser
Current Version 0.7.7
Type dependency

This version is covered by your current version range and after updating it in your project the build failed.

flora-sql-parser is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

Commits

The new version differs by 8 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of alasql is breaking the build 🚨

Version 0.4.7 of alasql was just published.

Branch Build failing 🚨
Dependency alasql
Current Version 0.4.6
Type dependency

This version is covered by your current version range and after updating it in your project the build failed.

alasql is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

unable to import v1.16.10

{ Error: Cannot find module 'winnow'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/home/linux/Documents/Repositories/output-ogcapi-features/src/route-handlers/get-collection-item.js:3:16)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3) code: 'MODULE_NOT_FOUND' }

v1.16.9 works fine.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • Replace dependency @hapi/joi with joi ^17.1.1
  • Update dependency fs-extra to v11.1.1
  • Update dependency pgsql-ast-parser to v11.1.0
  • Update dependency tape to v5.6.6
  • Update dependency wkt-parser to v1.3.3
  • Update dependency alasql to v4
  • Update dependency husky to v8
  • Update dependency lint-staged to v14
  • 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/ci.yml
  • actions/checkout v3
  • actions/setup-node v3
  • actions/checkout v3
  • actions/setup-node v3
  • actions/cache v3
  • rhysd/github-action-benchmark v1
npm
package.json
  • @esri/proj-codes ^3.0.0
  • @hapi/joi ^17.1.1
  • @terraformer/arcgis ^2.1.1
  • @terraformer/spatial ^2.1.2
  • @turf/bbox-polygon ^6.0.1
  • @turf/centroid ^6.0.0
  • @types/geojson ^7946.0.7
  • alasql ^3.0.0
  • classybrew 0.0.3
  • lodash ^4.17.4
  • moment ^2.18.1
  • ngeohash ^0.6.3
  • pgsql-ast-parser ^11.0.0
  • proj4 ^2.6.2
  • simple-statistics ^7.0.0
  • string-hash ^1.1.3
  • wkt-parser ^1.2.2
  • benchmark ^2.1.4
  • fs-extra ^11.0.0
  • husky ^4.3.0
  • lint-staged ^13.0.0
  • proxyquire ^2.1.3
  • sinon ^15.0.0
  • standard ^14.3.0
  • tap-spec ^5.0.0
  • tape ^5.0.0
  • farmhash ^3.1.0

  • Check this box to trigger a request for Renovate to run again on this repository

Esri envelope geometry option converted to invalid GeoJSON polygon

An Esri envelope option specified to Winnow is normalized to a GeoJSON Polygon incorrectly.

Input of envelope:

{"xmin":-180,"ymin":-180,"xmax":0,"ymax":0,"spatialReference":{"wkid":4326,"latestWkid":4326}}

Results in a geometry inside sql.js of:

{"type":"Polygon","coordinates":[[[-180,-180],[-180,0],[0,0],[0,-180],[-180,-180]]]}

This geometry does not validate according to http://geojsonlint.com/ because it is clockwise.

GeoJSON IETF RFC 7946 section 3.1.6. Polygon specifies a counter-clockwise order of coordinates for exterior polygon rings:

A linear ring MUST follow the right-hand rule with respect to the
area it bounds, i.e., exterior rings are counterclockwise, and
holes are clockwise.

The order of coordinates in transform-envelope.js could use adjusting to be compliant with the RFC. This will enable better validation of the GeoJSON and any tests on them.

How to do full text search?

I need full text search, how to do it in winnow?

Sqlite support full text search by create a virtual table in memory( not physical table on disk like other tables)

Cannot import winnow in a React.js project

Hi,

I would like to use winnow package in a React.js app.
I face some problems while importing the module.

TS module

Informative message (triggered by ESLint)

Could not find a declaration file for module 'winnow'. '/home/lealp/Documents/Projects/odyssee/front/node_modules/winnow/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/winnow` if it exists or add a new declaration (.d.ts) file containing `declare module 'winnow';`ts(7016)

Alright, let's try this : yarn add --dev @types/winnow fails, so I followed the next option.
A winnow.d.ts file was created at root directory.

module-winnow

It seems to work, but i'm not sure if I import this module correctly.
I tried different import declarations

import winnow from 'winnow';
import * as winnow from 'winnow';
import { querySQL } from 'winnow';
const winnow = require('winnow');
const winnow = require('<absolutePath>/winnow/index.js');
const querySQL = require('<absolutePath>/winnow/index.js').querySQL;

Which one is adapted ?

Node loader

Main problem : .node file loader. Compiling fails.

Failed to compile.

./node_modules/farmhash/build/Release/farmhash.node 1:0
Module parse failed: Unexpected character ' ' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)

I configured webpack.config.js as recommended (with this tutorial), nothing changed.
Then I realized this kind of configuration is only available for a specific type of applications : Electron.

So... I would like to know if there's an easy way to import winnow package in a React.js app ?

Thanks !

can you provide browser distribution version?

I try to use browserify.js to generate the browser distribution version of it, so I can use script tag include it in browser use.

But, I get build error:

can you try to build it see if you can success?

  `Last login: Mon Apr 27 16:28:49 on ttys003
   nicole@nicoles-iMac ~ % cd /Users/nicole/node/winnow 
   nicole@nicoles-iMac winnow % npm install
       npm WARN deprecated [email protected]: request has been deprecated, see       https://github.com/request/request/issues/3142

 > [email protected] install /Users/nicole/node/winnow/node_modules/farmhash
  > prebuild-install --runtime napi || prebuild-install || node-gyp rebuild

npm notice created a lockfile as package-lock.json. You should commit this file.
added 439 packages from 310 contributors and audited 1074 packages in 15.164s

33 packages are looking for funding
run npm fund for details

found 0 vulnerabilities

nicole@nicoles-iMac winnow % browserify src/index.js --s winnow -o dist/winnow.js

/Users/nicole/node/winnow/node_modules/farmhash/build/Release/farmhash.node:1
����
0���__TEXT�__text__TEXT�
�m�
�__stubs__TEXT�y ��__stub_helper__TEXT�z��z�__const__TEXT�|%�|__cstring__TEXT�}8�}__gcc_except_tab__TEXT��__unwind_info__TEXT�8�__eh_frame__TEXTX��
X��__DATA��__nl_symbol_ptr__DATA��0__got__DATA��2__la_symbol_ptr__DATA(��(�5__mod_init_func__DATA��� __data__DATA�����H__LINKEDIT�t�t"�0� �Pp��h�8�94�@ P6��e�꤫�1=��X ��$ ^ ParseError: Unexpected character '�'

Offset seems broken

https://github.com/FeatureServer/winnow/blob/master/src/executeQuery.js#L38

when you use the query as a filter per feature an offset in the sql query like:
SELECT type, properties as attributes, esriGeom(geometry) as geometry FROM ? LIMIT 100 OFFSET 1
will skip each feature

So if the feature query is returning 50 features and you add 1 resultOffset you will get no features.

I would think 1 of two things would need to happen
A) load all features in and apply the query
or
B) remove the offset from the query and do it in post processing of the results in winnow

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.