Code Monkey home page Code Monkey logo

koop's Introduction

koop

Turn data into FeatureServices

Koop provides a flexible server for exposing 3rd party data sources (APIs) as both Feature Services and other data formats (GeoJSON). This project is meant to provide a simple / plugable platform for experimenting with various data within the ArcGIS platform. Koop aims to provide a platform for accessing any API and making it easy to consume within the realm of Esri's geospatial web products.

Architecture

lots of geojson into featureservices

Dependencies

The following dependencies are needed in order to run Koop on your local machine / server:

  • Node.js (version > 0.10.0)
  • PostgreSQL / PostGIS - In PostgreSQL 9.3 you can create a PostGIS enabled database by executing CREATE EXTENSION postgis; inside an existing database.

Installation

  1. clone the repo git clone [email protected]:Esri/koop.git
  2. enter the koop project directory cd koop
  3. install the node.js dependencies npm install

If npm install fails trying to install node-canvas (on a mac)

  1. install xquartz - https://xquartz.macosforge.org/landing/
  2. install cairo brew install cairo
  3. Update the pkgconfig path export PKG_CONFIG_PATH=/usr/X11/lib/pkgconfig
  4. try again to install node.js dependencies npm install

Configuration

Koop uses a config called /config/local.js to setup its database connection and data directories for caching data.

NOTE: it is crucial that you make the appropriate edits to your config/local.js file so that the values match your environment.

  1. copy the example config file cp config/local.js.example config/local.js
  2. edit config/local.js for your environment.

In particular you will need to:

  • Ensure that the specified logfile directory exists and you have write permissions on it.
  • Ensure that the PostGIS database configuration matches the host, port, user, password, and name of your database.
  • In order to use the Github and Gist providers you'll need to correctly set the "github_token" property to match your personal Github API token.

Running Koop

Koop runs its own http server and will use port 1337 by default. You can start koop in several ways (see below) and then visit http://localhost:1337

start the server with auto-reload via grunt:

grunt server

start the server with node:

node app.js

to run on a non default port

PORT=9999 node app.js

Tests

Currently the tests depend on a locally running instance of koop. Once you have koop running:

  1. install grunt if you dont have it sudo npm install -g grunt-cli
  2. run the tests grunt test

Demo + More Docs

We've deployed a sample server as a testbed / proof of concept to http://koop.dc.esri.com

Data Providers

Koop is now designed to expose 3rd party services as FeatureServices that are consumable within Esri products and services. Currently Koop has the following providers shipped by default:

  • gists
  • github
  • agol
  • socrata

Each provider resides in api/providers/

Defining a new provider

Each provider defines custom routes, a controller, and a model. Each of these uses module.exports to export an object (common js modules). Each is then fused into koop at start up time and becomes available within the server.

"Note": The name of the provider dir is used to define the name of the provider and its controller within koop.

Routes

  • Define custom routes in the "routes" index.js file:
        // defined in api/providers/sample/routes/index.js
 
        module.exports = {
          'get /sample': {
            controller: 'sample',
            action: 'index'
          }
        }
  • The above creates a /sample route that calls the index method on the sample controller ( defined in /api/providers/sample/controller/index.js ).

Controller

  • Defines the handlers to used to respond to routes
      module.exports = {
        // this tells koop to treat this provider like AGS service and show up at the root data provider endpoint 
        provider: false,

        // our index method to simple print text 
        index: function(req, res){
          res.send('Sample Providers, to make this a real one set provider true');
        }
  
      }; 
  • each method takes in a request and response property and needs to send something to the reponse.

Models

  • Should be used to interact directly with 3rd party services and databases
  • Models make the http requests to API and should hand back raw data to the controllers

Example URL Structure 1: Gists as a FeatureService

Example URL Structure 2: Github Repo as a FeatureService

Feature Service Support

We'd like to have full feature service query support from any geojson data source. Currently the coverage is limited to the following params:

To see a full list of what koop will support in time visit http://resources.arcgis.com/en/help/arcgis-rest-api/

Caching

Koop uses either a local cache or a MongoDB instance to store data so that it doesnt flood external API with requests. This is helpful for avoiding API rate limits and general makes Koop faster (quick access to data). By default Koop will use a local hash to store data, but this is inefficient and doesnt persist across processes. To get around this Koop can use MongoDB which works well for non-relational data and has the added bonus of having geospatial query support.

Resources

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing.

Credit

Licensing

Copyright 2013 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

[](Esri Tags: ArcGIS Web Mapping GeoJson FeatureServices) [](Esri Language: JavaScript)

koop's People

Contributors

chelm avatar calvinmetcalf avatar plaxdan avatar benheb avatar ajturner avatar dbouwman avatar patrickarlt avatar mjuniper avatar

Stargazers

Raj Singh avatar

Watchers

Russell Branca avatar Mike Miller avatar Jay Doane avatar Eric Avdey avatar Sam Bisbee avatar Will Holley avatar Alan Hoffman avatar Benjamin Anderson avatar Garren avatar Pen Gale avatar Henning Diedrich avatar Dan DeMichele avatar Michael Wallace avatar Sue Lockwood avatar Mike Elsmore avatar Cleto Martín avatar James Cloos avatar DFB avatar David Tyler avatar Sean Hudgston avatar Andy Ellicott avatar Joshua McBride avatar Kyle Snavely avatar Scott Raven avatar Rhys Short avatar Derek Schoettle avatar Paul Szczypka avatar Anthony Abdulla avatar Chris Glew avatar Stefan Kruger avatar Yang Lei avatar Kevin Wolf avatar John Harrison avatar  avatar Jerry Jeong avatar Dan Cheeseman avatar chinajuanbob avatar David Taieb avatar Eric Appelt avatar Jake Epstein avatar Srdjan Jendruh avatar Michael Breslin avatar  avatar Emil Czajka avatar Ryan Millay avatar Aaron Nelson avatar Asim Singh avatar Tony Sun avatar Marion Behnen avatar Rob Silvagni avatar Riccardo Brognara avatar Christopher J Quinones avatar Justin Campbell avatar  avatar Dwight Ford avatar Sean Conway avatar Kent Kjaer avatar Robert Mckenzie avatar Wayne Deleo avatar Joe Cavanaugh avatar Aruna Kolluru avatar Vincent Ong avatar Vincent Kelly avatar Adrian Warman avatar Marc Suesser avatar Becky Wright avatar  avatar  avatar nierui avatar Esteban Laver avatar Rohit Agarwal avatar Andrew-Rombach avatar Paul Tinagbedia avatar  avatar Todd Shulman avatar Ben Perlmutter avatar TS avatar Darek Czapinski avatar Brian Wilkins avatar  avatar Yali Wang avatar Scott Davidson avatar Lakshmi Palaniappan avatar Jozef de Vries avatar David J Schlegel avatar Dr Doug Cowie avatar HUI ZHAO avatar  avatar  avatar Yusuf Khan avatar liujjcdl avatar Keerthi Sukumaran avatar Paul Nightingale avatar  avatar SEF MUTARI avatar Graham Thackrah avatar Mercy Ilade avatar coughlan@uk.ibm.com avatar Ram Maganti avatar Kristin Keating avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.