Code Monkey home page Code Monkey logo

ckan.js's Introduction

CDNJS

A Javascript client library for CKAN designed for both the browser and NodeJS.

The library provides full support for accessing both the CKAN Catalog and CKAN DataStore API.

It also provides a Recline compatible backend.

Installing

Browser

The JQuery library is required for use in the browser. After that, just add the ckan.js script to your page.

<script src="jquery-2.1.4.min.js"></script>
<script src="ckan.js"></script>

You can also use our hosted one:

<script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
<script src="http://okfnlabs.org/ckan.js/ckan.js"></script>

Using the JsDelivr CDN you can get everything you need with this webpacked bundle:

<script src="https://cdn.jsdelivr.net/gh/okfn/ckan.js@master/dist/ckan.bundle.js"></script>

Node

npm install ckan

Then in your code:

var CKAN = require('ckan')

Usage

Usage is generally similar across Node and Browser versions.

Callback structure follows Node conventions, that is:

function(err, data)

Catalog

Set it up:

var client = new CKAN.Client('http://my-ckan-site.com');

// You can also provide an API key (for operations that require one)
var client = new CKAN.Client('http://my-ckan-site.com', 'my-api-key');

// If your portal disallows POST requests (note: limited support in browser module)
client.requestType = 'GET';

You can now use any part of the action API:

client.action('action_name', data, callback)

For example, to create a dataset using dataset_create action you would do:

client.action('dataset_create', { name: 'my-dataset' }, function(err, result) {
  console.log(err);
  console.log(result);
})

Here's a more complex example showing several commands to do a Dataset upsert (create if not exists, otherwise update):

var datasetInfo = {
  name: 'ckan.js-example',
  title: 'CKAN.JS Example',
  tags: ['amazing']
};
client.action('dataset_show', { id: datasetInfo.name }, function(err, out) {
  // dataset exists
  if (!err) {
    // TODO: you'd really want to extend the existing dataset object returned
    // in out with the datasetInfo we have but we are being simple here
    client.action('dataset_update', datasetInfo, cb);
  } else {
    client.action('dataset_create', datasetInfo, cb);
  }
});

DataStore

The DataStore feature of CKAN allows you to store structured data in CKAN and to create a rich API for it. It is also accessible via the action API - details in the docs here - and and you can therefore access using the CKAN client. Here are a few examples:

Store Data

Store data into the DataStore for an existing resource

// 2 rows or data (with columns/fields named 'A' and 'B'
var data = [
  { A: 1, B: 2 },
  { A: 10, B: 16}
];
// the id of a CKAN DataSet resource (the data that we store will be associated with that resource)
// this resource will need to already exist
resourceId = 'abc-efg';
client.action('datastore_create', {
    resource_id: resourceId,
    records: data
  },
  function(err) {
    if (err) console.log(err);
    console.log('All done');
  })

Store data into a new DataStore resource:

// the id of a CKAN dataset that already exists
packageId = 'the-best-dataset-ever';
client.action('datastore_create', {
    resource: {package_id: packageId},
    records: data
  },
  function(err) {
    if (err) console.log(err);
    console.log('All done');
  })

Here's an example of loading data from a CSV file into the DataStore:

// npm's csv file
var csv = require('csv');
csv()
  .from('path/to/csv-file.csv', {columns: true})
  .to.array(function(data, count) {
    client.action('datastore_create', {
        resource_id: resourceId,
        records: data
      },
      function(err) {
        if (err) console.log(err);
        console.log('All done');
      })
    })
    ;

Search Data

Search data using the Data API - see datastore_search for details of options:

client.action('datastore_search', {
    resource_id: '...',
    q: '...'
  },
  function(err, out) {
    if (err) console.log(err);
    console.log(out);
  })
});

Or using SQL support:

client.action('datastore_search_sql', {
    sql: '...'
  },
  function(err, out) {
    if (err) console.log(err);
    console.log(out);
  })
});

There are also a couple of nice wrapper methods:

// queryObj should be like the Recline Query structure
// http://okfnlabs.org/recline/docs/models.html#query
client.datastoreQuery(queryObj, function(err, out) {
  // out will follow recline structure, viz
  {
    total: ..
    fields: ... (fields will have Recline / JSON Table Schema types)
    hits: array of results ...
  }
});

And for SQL

client.datastoreSqlQuery(sql, function(err, out) {
  // out will follow recline structure, viz
  {
    total: ..
    fields: ... (fields will have Recline / JSON Table Schema types)
    hits: array of results ...
  }
});

Recline JS Backend

This module also provides a Recline compatible backend available as:

recline.backend.Ckan

The backend supports fetch and query but does not provide write support at the present time.

Developer notes

We are using QUnit for unit testing the webpack'ed version.

npm install -D

To install developer libraries, then:

npm run dev

To update the dist version of the library. Start a local web server, e.g.:

python -m http.server

And in your web browser, open http://0.0.0.0:8000/test/

ckan.js's People

Contributors

rufuspollock avatar loleg avatar davidmiller avatar vitorbaptista avatar krzysztofmadejski avatar davidmegginson avatar marks avatar kaocy avatar

Stargazers

chakri avatar Godfrey Takavarasha avatar Joel Berman avatar Bryden Wayne avatar Luis Castillo avatar Mizar avatar Yuji Azama avatar Sara Tasche avatar Talliny Dalla Nora avatar MaX avatar Anuar Ustayev (aka Anu) avatar  avatar Michelle Tang avatar Ilya Ilyankou avatar  avatar Terje Christensen avatar David Morton avatar Robley Gori avatar João Rocha da Silva avatar Fei (Faye) Fang avatar Jiang Dongke avatar Ricardo Borges avatar roll avatar  avatar  avatar Alexander Murphy avatar Matthew avatar Nick Weber avatar Lorenzo Bragaglia avatar Alex Rollin avatar Pedro avatar Angus H. avatar Osahon Okungbowa avatar  avatar Gustavo RPS avatar Luis Capelo avatar Denis Zgonjanin avatar Anibal avatar Matt Moyles avatar German Martinez avatar Dhi Aurrahman avatar Mita avatar Damon Oehlman avatar Aleksandr Blekh avatar Alfredo Serafini avatar Andhika Nugraha avatar Moritz Klack avatar Seth Vincent avatar Kanit Wongsuphasawat avatar JT5D avatar Alex Sadleir avatar Fumihiro Kato avatar Dominik Moritz avatar  avatar

Watchers

Alex Sadleir avatar  avatar Darwin Peltan avatar Adrià Mercader avatar Daniel Dietrich avatar Edgar Z. Alvarenga avatar Anja Jentzsch avatar  avatar  avatar James Cloos avatar Anders avatar Michael Bauer avatar Sam Leon avatar Jonathan Gray avatar Stephen Abbott Pugh avatar  avatar  avatar Andres Vazquez avatar Nikesh Balami avatar Yared Getachew avatar Shashi Gharti avatar Oscar Montiel avatar Patricio Del Boca avatar  avatar  avatar Lucas Pretti avatar Georgiana Bere avatar  avatar  avatar

ckan.js's Issues

Errors could be returned as object

Errors are returned as strings:

CKANJS API Error. HTTP code 409. Message: {
  "help": "http://localhost:5000/api/3/action/help_show?name=package_create",
  "success": false,
  "error": {
    "__type": "Validation Error",
    "id": [
      "Dataset id already exists"
    ]
  }
}

It would be helpful it they were returned as json object:

{ "status": 409,
  "message": {
  "help": "http://localhost:5000/api/3/action/help_show?name=package_create",
  "success": false,
  "error": {
    "__type": "Validation Error",
    "id": [
      "Dataset id already exists"
    ]
  }
}
}

So one can easily check err.message.error.id

Return the fields on query()

Right now, to get the fields, I need to call fetch().

As I'm calling query() anyway, it would save one request to make it return the fields as well. We already have that available, we just don't add it into the returned object.

400 Bad Request

Hi,

I've tried to use this module but it's not working for me. CKAN is always returning 400 ValidationError.

I've check the code and I think that I've found the error (at least the solution is working for me). In ckan.js:144 I've replaced request(conf,... by request.post(conf,.... By including .post, the request library is making a POST request so the parameters can be sent. Otherwise, the request library will make a GET request without parameters.

BR

A widget based on ckan.js

Recently I worked on ckan-embed, supported by opendata.swiss. It is a small jQuery-based widget which uses CORS/JSONP to fetch data from CKAN using this library (see related issue #27).

I would like to encourage some discussion/adoption of this approach, at least in the interest of promoting data reuse. Would this make sense as a feature request in CKAN itself (akin to data viewer), or something for OKFN labs, or just write about it in a wiki somewhere?

It's tightly coupled to ckan.js, which is why I'm asking here. I also figure other users of this library have built similar things.

Webpack dependencies update?

I am getting an error when trying to add this package via NPM to my angular app

in doing some research, it looks like some of Ckan's dependencies are built in to webpack now (crypto, net, tls, etc). is there hope for an update?

Error:

[ng] ERROR in ./node_modules/ckan/node_modules/aws-sign2/index.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\aws-sign2'
[ng] ERROR in ./node_modules/ckan/node_modules/http-signature/lib/verify.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\http-signature\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/http-signature/lib/util.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\http-signature\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/http-signature/lib/signer.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\http-signature\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/oauth-sign/index.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\oauth-sign'
[ng] ERROR in ./node_modules/ckan/node_modules/request/lib/helpers.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\ckan\node_modules\request\lib'
[ng] ERROR in ./node_modules/cryptiles/lib/index.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\cryptiles\lib'
[ng] ERROR in ./node_modules/hawk/lib/crypto.js
[ng] Module not found: Error: Can't resolve 'crypto' in 'Z:\ionic\node_modules\hawk\lib'
[ng] ERROR in ./node_modules/sntp/lib/index.js
[ng] Module not found: Error: Can't resolve 'dgram' in 'Z:\ionic\node_modules\sntp\lib'
[ng] ERROR in ./node_modules/sntp/lib/index.js
[ng] Module not found: Error: Can't resolve 'dns' in 'Z:\ionic\node_modules\sntp\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/form-data/lib/form_data.js
[ng] Module not found: Error: Can't resolve 'fs' in 'Z:\ionic\node_modules\ckan\node_modules\form-data\lib'
[ng] ERROR in ./node_modules/hoek/lib/index.js
[ng] Module not found: Error: Can't resolve 'fs' in 'Z:\ionic\node_modules\hoek\lib'
[ng] ERROR in ./node_modules/boom/lib/index.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\boom\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\ckan\node_modules\forever-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/form-data/lib/form_data.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\ckan\node_modules\form-data\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/http-signature/lib/signer.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\ckan\node_modules\http-signature\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/request/request.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\ckan\node_modules\request'
[ng] ERROR in ./node_modules/ckan/node_modules/tunnel-agent/index.js
[ng] Module not found: Error: Can't resolve 'http' in 'Z:\ionic\node_modules\ckan\node_modules\tunnel-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'https' in 'Z:\ionic\node_modules\ckan\node_modules\forever-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/form-data/lib/form_data.js
[ng] Module not found: Error: Can't resolve 'https' in 'Z:\ionic\node_modules\ckan\node_modules\form-data\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/request/request.js
[ng] Module not found: Error: Can't resolve 'https' in 'Z:\ionic\node_modules\ckan\node_modules\request'
[ng] ERROR in ./node_modules/ckan/node_modules/tunnel-agent/index.js
[ng] Module not found: Error: Can't resolve 'https' in 'Z:\ionic\node_modules\ckan\node_modules\tunnel-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'net' in 'Z:\ionic\node_modules\ckan\node_modules\forever-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/request/request.js
[ng] Module not found: Error: Can't resolve 'net' in 'Z:\ionic\node_modules\ckan\node_modules\request'
[ng] ERROR in ./node_modules/ckan/node_modules/tunnel-agent/index.js
[ng] Module not found: Error: Can't resolve 'net' in 'Z:\ionic\node_modules\ckan\node_modules\tunnel-agent'
[ng] ERROR in ./node_modules/tough-cookie/lib/cookie.js
[ng] Module not found: Error: Can't resolve 'net' in 'Z:\ionic\node_modules\tough-cookie\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/form-data/lib/form_data.js
[ng] Module not found: Error: Can't resolve 'path' in 'Z:\ionic\node_modules\ckan\node_modules\form-data\lib'
[ng] ERROR in ./node_modules/bl/node_modules/readable-stream/lib/_stream_writable.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\bl\node_modules\readable-stream\lib'
[ng] ERROR in ./node_modules/bl/node_modules/readable-stream/lib/_stream_readable.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\bl\node_modules\readable-stream\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/assert-plus/assert.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\ckan\node_modules\assert-plus'
[ng] ERROR in ./node_modules/ckan/node_modules/combined-stream/lib/combined_stream.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\ckan\node_modules\combined-stream\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/delayed-stream/lib/delayed_stream.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\ckan\node_modules\delayed-stream\lib'
[ng] ERROR in ./node_modules/ckan/node_modules/request/request.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\ckan\node_modules\request'
[ng] ERROR in ./node_modules/stringstream/stringstream.js
[ng] Module not found: Error: Can't resolve 'stream' in 'Z:\ionic\node_modules\stringstream'
[ng] ERROR in ./node_modules/ckan/node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'tls' in 'Z:\ionic\node_modules\ckan\node_modules\forever-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/tunnel-agent/index.js
[ng] Module not found: Error: Can't resolve 'tls' in 'Z:\ionic\node_modules\ckan\node_modules\tunnel-agent'
[ng] ERROR in ./node_modules/ckan/node_modules/request/request.js
[ng] Module not found: Error: Can't resolve 'zlib' in 'Z:\ionic\node_modules\ckan\node_modules\request'

package.json

"dependencies": {
"@angular/common": "~7.0.2",
"@angular/core": "~7.0.2",
"@angular/forms": "~7.0.2",
"@angular/http": "~7.0.2",
"@angular/platform-browser": "~7.0.2",
"@angular/platform-browser-dynamic": "~7.0.2",
"@angular/router": "~7.0.2",
"@ionic-native/core": "5.0.0-beta.21",
"@ionic-native/splash-screen": "5.0.0-beta.21",
"@ionic-native/status-bar": "5.0.0-beta.21",
"@ionic/angular": "4.0.0-beta.15",
"core-js": "^2.5.7",
"rxjs": "6.3.3",
"spotcrime": "^1.0.1",
"zone.js": "^0.8.26"
},
"devDependencies": {
"@angular-devkit/architect": "~0.10.4",
"@angular-devkit/build-angular": "~0.10.4",
"@angular-devkit/core": "~7.0.4",
"@angular-devkit/schematics": "~7.0.4",
"@angular/cli": "~7.0.4",
"@angular/compiler": "~7.0.2",
"@angular/compiler-cli": "~7.0.2",
"@angular/language-service": "~7.0.2",
"@ionic/angular-toolkit": "^1.0.0",
"@types/jasmine": "~2.8.6",
"@types/jasminewd2": "~2.0.3",
"@types/node": "~10.12.0",
"codelyzer": "~4.5.0",
"jasmine-core": "~2.99.1",
"jasmine-spec-reporter": "~4.2.1",
"karma": "~3.0.0",
"karma-chrome-launcher": "~2.2.0",
"karma-coverage-istanbul-reporter": "~2.0.0",
"karma-jasmine": "~1.1.1",
"karma-jasmine-html-reporter": "^0.2.2",
"protractor": "~5.4.0",
"ts-node": "~7.0.0",
"tslint": "~5.11.0",
"typescript": "~3.1.6"
},

Increment version on npm

Just spent a long while debugging only to realize npm install ckan installs version 0.2.2 instead of the latest, 0.2.3 :P

Getting ckan.js working in a browser requires CORS setting

Hi
I just want to share how to get ckan.js to work in a browser so that you don't need to waste your time figuring it out.

Below you will see to javaScript programs that look similar. They do the same thing. But one works and the other dont.

In order to get the version that is running in the browser working you need to enable CORS on your CKAN server.
You do this by adding this line in the configuration file and restarting the server.

ckan.cors.origin_allow_all = true

After you do this the browser version will also work.

Node version

Run it by typing

node ckan_dataset_list.js

The node version ckan_dataset_list.js

var CKAN = require('ckan')
var myAPIkey = "";
var resource_id = "3ae312b4-0e9b-4542-92b7-cf017b586f0c";
var myCKANhost = "http://urbalurba.no";

var client = new CKAN.Client(myCKANhost, myAPIkey);

client.action('datastore_search', { resource_id: resource_id }, function(err, result) {
    console.log(err);
    //console.log(result);
    console.log("We can read all records in a dataset:", JSON.stringify(result));
  })

Browser version

The browser version ckan_dataset_list_browser.js

var myAPIkey = "";
var resource_id = "3ae312b4-0e9b-4542-92b7-cf017b586f0c";
var myCKANhost = "http://urbalurba.no";

var client = new CKAN.Client(myCKANhost, myAPIkey);
  
client.action('datastore_search', { resource_id: resource_id }, function(err, result) {
    console.log(err);
    //console.log(result);
    console.log("We can read all records in a dataset:", JSON.stringify(result));
    document.getElementById("app").innerText = "Result:" + JSON.stringify(result);
  })

The browser version html file

<html>
<head>
    <script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
    <script src="http://okfnlabs.org/ckan.js/ckan.js"></script>
</head>
<body>
    <h1>Listing a dataset resource</h1>
        <div id="app"></div>
    <script src="ckan_dataset_list_browser.js"></script>
</body>
</html>

Pivot support for datastore ...

Postgresql has reasonable pivot support. We could wrap this in a nice API ...

Research on Postgresql pivoting

Pivoting using crosstab

Hack Pivots

CKAN v3 support

This is more of a question than an issue at this point, but as CKAN goes through a major revision, the utility and form of this library should be put into question. A well documented JSON API and the Frictionless Data libraries may be preferable to maintaining this module for anything more than backwards compatibility. Could someone from the core team weigh in, please?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.