opendatatrentino / ckan-api-client Goto Github PK

then, compare behavior/bugs

Create a web application exposing client functionality through RESTful API.

This might be worth putting in its own separate project.

Apparently there is only logical deletion, plus there are some known issues..

Datasets have the following keys:

• id
• name

Resources have the following keys:

• id (enforced database-level)
• name (enforced database-level)
• url (not enforced)
• dataset_id + ordering position (apparently not enforced, need checks)

Todo:

• write tests to clearly pin-point the behavior when key conflicts happen
• figure out a way to properly handle collisions (ignore all keys but one, randomize other keys should they conflict)

Problems:

• we cannot pre-load list of keys used in the database, as logically deleted items are missing from lists but still there to cause issues
• we cannot rely on 409 errors as:
  • they are misused quite often (in place of 404s) and I'm pretty sure in certain cases 500 errors get returned instead
  • there still is no way to figure out which key caused a conflict..

• new CkanDataset
• new CkanResource
• new CkanGroup
• new CkanOrganization

Otherwise, things will fail in weird ways (eg. in most cases we don't care if a string field is actually None or ""..)

Eg. it should have a merge_datasets(old, new) method to allow merging datasets in custom ways

When I update the name of a CKAN Group, the group members are not updated accordingly.

we want to expose all the client functionality through the CLI:

• objects CRUD
• synchronization facilities

Then remove XFAIL marks from:

XFAIL ckan_api_client/tests/functional/client_lowlev/test_data_loss_on_update.py::test_data_loss_on_update
  reason: [NOTRUN] Requires rewrite of prepare_dataset()
XFAIL ckan_api_client/tests/functional/client_lowlev/test_dataset_crud.py::test_delete_dataset
  reason: [NOTRUN] Requires rewrite of prepare_dataset()
XFAIL ckan_api_client/tests/functional/client_lowlev/test_dataset_crud.py::test_updating_extras
  reason: [NOTRUN] Requires rewrite of prepare_dataset()

• create
• read
• list
• update
• delete

Need to check:

• update of core fields
• update of extras
• update of groups
• update of resources
• update of relationships (?)

After installing it using pip install ckan-api-client I've the following exception when executing from ckan_api_client.high_level import CkanHighlevelClient:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-4-c95dae115e31> in <module>
----> 1 from ckan_api_client.high_level import CkanHighlevelClient

~/software/pyenv/versions/3.9.2/envs/covid19-br/lib/python3.9/site-packages/ckan_api_client/high_level.py in <module>
      3 
      4 from .objects import CkanDataset, CkanOrganization, CkanGroup
----> 5 from .low_level import CkanLowlevelClient
      6 from .exceptions import OperationFailure, HTTPError
      7 

~/software/pyenv/versions/3.9.2/envs/covid19-br/lib/python3.9/site-packages/ckan_api_client/low_level.py in <module>
      1 import json
----> 2 import urlparse
      3 
      4 import requests
      5 

ModuleNotFoundError: No module named 'urlparse'

The correct import inside low_level.py is probably:

from urllib.parse import urlparse

Installed version from PyPI:

$ pip freeze | grep ckan
ckan-api-client==0.1b5

• create
• read
• list
• update
• delete

• create
• read
• list
• update
• delete

The synchronization client is used to import harvested data into ckan.

The import strategy is, roughly:

• upsert all the needed groups, keep a (source_id, ckan_id) map
• upsert all the needed organizations, keep a (source_id, ckan_id) map
• prepare datasets for insertion
  • replace group/org names with ckan ids
• figure out which datasets require creation/update/deletion
• perform the above activity

We'd like to support:

• plain json files
• sqlite databases
• mongodb databases
• postgresql databases w/ text column
• postgresql databases w/ hstore
• postgresql databases w/ json fields

It seems like something really useful, to automatically sync datasets to a CKAN instance, but I can't figure out how it works. The documentation is very vague. It says:

data – Data to be synchronized. Should be a dict (or dict-like) with top level keys coresponding to the object type, mapping to dictionaries of {'id': object}.

But so what is <object> supposed to be like ? What dictionary should I call that method with ? An example usage would really be welcome.