ckan / ckanext-scheming Goto Github PK

And maybe, making it a bit meta - using CKAN to catalog scheming schemas created by the community :)

Since you can specify URLs when defining schemas, maybe even enforce a convention (using scheming?) to point to different versions of a schema.

Hi,

Below is a .json schema I would like to implement. It is about 60% of the final schema we have planned. I have not added all the relevent fluent parts as all labels will be available in swedish and english. I was hoping you could run your eye over it for me as it is the first I have written. My development.ini file additions looks like this:

scheming.dataset_schemas = ckanext.scheming:ckan.json
                                                 ckanext.scheming:camel_photos.json
                                                 ckanext.scheming:sluschemav014.json

scheming.presets = ckanext.scheming:presets.json
                                ckanext.fluent:presets.json
                                ckanext.repeating:presets.json

Also: scheming-datasets has been added to my plugins list

When I serve the development.ini file, ckan seems to run AOK, I have checked and the plugins are all loading but I do not know how to get any of the schemas to actually appear when a new dataset is created. Should this be automatic? If you have multiple schemas, does the author chose which to use?

sluschemav014.json

{
  "scheming_version": 1,
  "dataset_type": "SLU Base Metadata schema v014",
  "about_url": "http://tilda.slu.se/base-schema-v014",
  "dataset_fields": [
    {
      "field_name": "metadatalanguage",
      "label": "MetaDataLanguage",
      "help_text": "Which language will you use?",
      "help_inline": true,
      "preset": "English",
      "choices": [
        {
          "value": "english",
          "label": "English"
        },
        {
          "value": "swedish",
          "label": "Swedish"
        }
      ]
    },
    {
      "field_name": "mdcOrganisation",
      "label": "Organisation",
      "help_text": "Which organisation?",
      "help_inline": true,
      "preset": "Sveriges lantbruksuniversitet (SLU)",
      "form_placeholder": "eg. SLU",
      "choices": [
        {
          "value": "sveriges lantbruksuniversitet (slu)",
          "label": "Sveriges lantbruksuniversitet (SLU)"
        },
        {
          "value": "swedish university of agricultural sciences",
          "label": "Swedish University of Agricultural Sciences"
        }
      ]
    },
    {
      "field_name": "mdcPerson",
      "label": "Person",
      "preset": "person",
      "form_placeholder": "eg. John Smith",
      "display_property": "dc:creator"
    },
    {
      "field_name": "mdcEMail",
      "label": "EMail",
      "preset": "email",
      "form_placeholder": "[email protected]",
      "display_property": "dc:creator",
      "display_snippet": "email.html",
      "display_email_name_field": "mdcPerson"
    },
    {
      "field_name": "mdcRole",
      "label": "Role",
      "help_text": "What role?",
      "help_inline": true,
      "preset": "pointOfContact - kontakt",
      "form_placeholder": "eg. point of contact",
      "choices": [
        {
          "value": "pointOfContact - kontakt",
          "label": {"en": "pointOfContact","sv": "kontakt"}
        },
        {
          "value": "principalInvestigator - undersökningsansvarig",
          "label": {"en": "principalInvestigator","sv": "undersökningsansvarig"}
        },
        {
          "value": "author - upphovsman",
          "label": {"en": "author","sv": "upphovsman"}
        },
        {
          "value": "originator - producent",
          "label": {"en": "originator","sv": "producent"}
        },
        {
          "value": "custodian - förvaltare",
          "label": {"en": "custodian","sv": "förvaltare"}
        },
        {
          "value": "owner - ägare",
          "label": {"en": "owner","sv": "ägare"}
        },
        {
          "value": "processor - vidareförädlare",
          "label": {"en": "processor","sv": "vidareförädlare"}
        },
        {
          "value": "publisher - utgivare",
          "label": {"en": "publisher","sv": "utgivare"}
        },
        {
          "value": "distributor - distributör",
          "label": {"en": "distributor","sv": "distributör"}
        },
        {
          "value": "resourceProvider - leverantör",
          "label": {"en": "resourceProvider","sv": "leverantör"}
        }
      ]
    },
    {
      "field_name": "metadata_date",
      "label": "Dataset date",
      "preset": "date"
    },
    {
      "field_name": "md_standard",
      "label": "Metadata Standard"
    },
    {
      "field_name": "md_standardversion",
      "label": "Metadata Standard version"
    },
    {
      "field_name": "resourcetype",
      "label": "Resource Type",
      "help_text": "What type is the resource?",
      "help_inline": true,
      "preset": "dataset - datamängder",
      "form_placeholder": "eg. dataset",
      "choices": [
        {
          "value": "dataset - datamängder",
          "label": {"en": "dataset","sv": "datamängder"}
        },
        {
          "value": "series - serier av datamängder",
          "label": {"en": "series","sv": "serier av datamängder"}
        },
        {
          "value": "service - tjänster",
          "label": {"en": "service","sv": "tjänster"}
        },
        {
          "value": "software - applikationer",
          "label": {"en": "software","sv": "applikationer"}
        }
      ]
    },
    {
      "field_name": "guidurl",
      "label": "Unique URL",
      "form_placeholder": "http://tilda.slu.se/dataset",
      "display_property": "foaf:homepage",
      "display_snippet": "link.html"
    },
    {
      "field_name": "title",
      "preset": "fluent_text",
      "label": {
         "en": "Title",
         "sv": "Titel"
    },
      "form_languages": ["en", "sv"]
    },
    {
      "field_name": "version",
      "label": "Version of the dataset",
      "validators": "ignore_missing unicode package_version_validator",
      "form_placeholder": "1.0"
    },
    {
      "field_name": "ds_citations",
      "label": {"en": "Quoting data sets", "se": "Citering av dataset"}
    },    
    {
      "field_name": "reference_date",
      "label": "Reference date",
      "preset": "date"
    },
    {
      "field_name": "resource_event",
      "label": "Resource Event",
      "help_text": "What event?",
      "help_inline": true,
      "preset": "creation - skapades",
      "form_placeholder": "eg. creation",
      "choices": [
        {
          "value": "creation - skapades",
          "label": {"en": "creation","sv": "skapades"}
        },
        {
          "value": "publication - publicerades",
          "label": {"en": "publication","sv": "publicerades"}
        },
        {
          "value": "revision - reviderades",
          "label": {"en": "revision","sv": "reviderades"}
        }
      ]
    },
    {
      "field_name": "resource_status",
      "label": "Resource Status",
      "help_text": "Which status is the resource at?",
      "help_inline": true,
      "preset": "planned - planerad",
      "form_placeholder": "eg. planned",
      "choices": [
        {
          "value": "planned - planerad",
          "label": {"en": "planned","sv": "planerad"}
        },
        {
          "value": "underDevelopment - under utveckling",
          "label": {"en": "under development","sv": "under utveckling"}
        },
        {
          "value": "onGoing - pågående/driftsatt",
          "label": {"en": "on going","sv": "pågående/driftsatt"}
        },
        {
          "value": "completed - färdigställd",
          "label": {"en": "completed","sv": "färdigställd"}
        },
        {
          "value": "historicalArchive - arkiverad",
          "label": {"en": "historical archive","sv": "arkiverad"}
        },
        {
          "value": "obsolete - inte längre i bruk",
          "label": {"en": "obsolete","sv": "inte längre i bruk"}
        },
        {
          "value": "required - åtgärd krävs",
          "label": {"en": "required","sv": "åtgärd krävs"}
        }
      ]
    },
    {
      "field_name": "rcOrganisation",
      "label": "Organisation",
      "help_text": "Which organisation?",
      "help_inline": true,
      "preset": "Sveriges lantbruksuniversitet (SLU)",
      "form_placeholder": "eg. SLU",
      "choices": [
        {
          "value": "sveriges lantbruksuniversitet (slu)",
          "label": "Sveriges lantbruksuniversitet (SLU)"
        },
        {
          "value": "swedish university of agricultural sciences",
          "label": "Swedish University of Agricultural Sciences"
        }
      ]
    },
    {
      "field_name": "rcPerson",
      "label": "Person",
      "preset": "person",
      "form_placeholder": "eg. John Smith",
      "display_property": "dc:creator"
    },
    {
      "field_name": "rcEMail",
      "label": "EMail",
      "preset": "email",
      "form_placeholder": "[email protected]",
      "display_property": "dc:creator",
      "display_snippet": "email.html",
      "display_email_name_field": "mdcPerson"
    },
    {
      "field_name": "rcRole",
      "label": "Role",
      "help_text": "What role?",
      "help_inline": true,
      "preset": "pointOfContact - kontakt",
      "form_placeholder": "eg. point of contact",
      "choices": [
        {
          "value": "pointOfContact - kontakt",
          "label": {"en": "pointOfContact","sv": "kontakt"}
        },
        {
          "value": "principalInvestigator - undersökningsansvarig",
          "label": {"en": "principalInvestigator","sv": "undersökningsansvarig"}
        },
        {
          "value": "author - upphovsman",
          "label": {"en": "author","sv": "upphovsman"}
        },
        {
          "value": "originator - producent",
          "label": {"en": "originator","sv": "producent"}
        },
        {
          "value": "custodian - förvaltare",
          "label": {"en": "custodian","sv": "förvaltare"}
        },
        {
          "value": "owner - ägare",
          "label": {"en": "owner","sv": "ägare"}
        },
        {
          "value": "processor - vidareförädlare",
          "label": {"en": "processor","sv": "vidareförädlare"}
        },
        {
          "value": "publisher - utgivare",
          "label": {"en": "publisher","sv": "utgivare"}
        },
        {
          "value": "distributor - distributör",
          "label": {"en": "distributor","sv": "distributör"}
        },
        {
          "value": "resourceProvider - leverantör",
          "label": {"en": "resourceProvider","sv": "leverantör"}
        }
      ]
    },
    {
      "field_name": "resource_id",
      "label": "Resource ID"
    },
    {
      "field_name": "resource_id_code",
      "label": "Resource ID Code"
    },
    {
      "field_name": "notes",
      "label": "Description",
      "form_snippet": "markdown.html",
      "form_placeholder": "eg. Some useful notes about the data"
    },
    {
      "field_name": "geographic_extent",
      "label": "Geographic extent"
    },
    {
      "field_name": "geographic_boudingbox",
      "label": "Geographic bounding box"
    },
    {
      "field_name": "minmax_height",
      "label": " Minimum and maximum height"
    },
      "field_name": "vertcal_date",
      "label": "Vertical Date"
      "preset": "date"
    },
      "field_name": "temporal_date_from",
      "label": "Temporal Date From"
      "preset": "date"
    },
      "field_name": "temporal_date_to",
      "label": "Temporal Date To"
      "preset": "date"
    },
    {
      "field_name": "topic",
      "label": "Topic",
      "help_text": "Which topic",
      "help_inline": true,
      "preset": "inlandWaters - sjöar och vattendrag",
      "form_placeholder": "eg. inlandWaters",
      "choices": [
        {
          "value": "farming - areella näringar",
          "label": {"en": "farming","sv": "areella näringar"}
        },
        {
          "value": "biota - biologi och ekologi",
          "label": {"en": "biota","sv": "biologi och ekologi"}
        },
        {
          "value": "boundaries - administrativa gränser",
          "label": {"en": ""boundaries","sv": "administrativa gränser"}
        },
        {
          "value": "climatologyMeteorologyAtmosphere - ?",
          "label": {"en": "climatologyMeteorologyAtmosphere","sv": "?"}
        },
        {
          "value": "economy - ekonomiska förhållanden",
          "label": {"en": "economy","sv": "ekonomiska förhållanden"}
        },
        {
          "value": "elevation - höjddata",
          "label": {"en": "elevation","sv": "höjddata"}
        },
        {
          "value": "environment - miljörisker, skydd, landskap",
          "label": {"en": "environment","sv": "miljörisker, skydd, landskap"}
        },
        {
          "value": "geoscientificInformation - geovetenskap",
          "label": {"en": "geoscientificInformation","sv": "geovetenskap"}
        },
        {
          "value": "health - hälsovård, human ekologi",
          "label": {"en": "health","sv": "hälsovård, human ekologi"}
        },
        {
          "value": "imageryBaseMapsEarthCover - baskartor",
          "label": {"en": "imageryBaseMapsEarthCover","sv": "baskartor"}
        },
        {
          "value": "intelligenceMilitary - försvar",
          "label": {"en": "intelligenceMilitary","sv": "försvar"}
        },
        {
          "value": "inlandWaters - sjöar och vattendrag",
          "label": {"en": "inlandWaters","sv": "sjöar och vattendrag"}
        },
        {
          "value": "oceans - kust och hav",
          "label": {"en": "oceans","sv": "kust och hav"}
        },
        {
          "value": "location - adresser, stomnät, ortnamn",
          "label": {"en": "location","sv": "adresser, stomnät, ortnamn"}
        },
        {
          "value": "planningCadastre - fysisk planering",
          "label": {"en": "planningCadastre","sv": "fysisk planering"}
        },
        {
          "value": "society - samhälle och kultur, demografi",
          "label": {"en": "society","sv": "samhälle och kultur, demografi"}
        },
        {
          "value": "structure - byggnader och konstruktioner",
          "label": {"en": "structure","sv": "byggnader och konstruktioner"}
        },
        {
          "value": "transportation - transportinfrastruktur",
          "label": {"en": "transportation","sv": "transportinfrastruktur"}
        },
        {
          "value": "utilitiesCommunication - tekniska nät",
          "label": {"en": "utilitiesCommunication","sv": "tekniska nät"}
        }
      ]
    },
    {
      "field_name": "summary",
      "label": "Summary / Abstract",
      "form_snippet": "markdown.html",
      "form_placeholder": "eg. Abstract or summary"
    },
    {
      "field_name": "sampleimage_url",
      "label": "Sample Image URL",
      "form_placeholder": "http://example.com/image.jpg",
      "display_snippet": "link.html"
    },
    {
      "field_name": "sampleimage_caption",
      "label": "Image description"
    },
    {
      "field_name": "sampleimage_format",
      "label": "Image format"
    },
    {
      "field_name": "tag_string",
      "label": "Keywords",
      "preset": "tag_string_autocomplete",
      "form_placeholder": "eg. "
    },
    {
      "field_name": "spatialrepresentation",
      "label": { "en": " Method for Spatial Representation", "se":" Metod för rumslig 

representation" },
      "help_text": "Which spatial method?",
      "help_inline": true,
      "preset": "vector",
      "form_placeholder": "eg. point of contact",
      "choices": [
        {
          "value": "vector",
          "label": "vector"
        },
        {
          "value": "grid - raster",
          "label": {"en": "grid","sv": "raster"}
        },
        {
          "value": "textTable - text eller tabell",
          "label": {"en": "text table","sv": "text eller tabell"}
        },
        {
          "value": "tin - TIN",
          "label": {"en": "tin","sv": "TIN"}
        },
        {
          "value": "stereoModel - stereomodell",
          "label": {"en": "stereo Model","sv": "stereomodell"}
        },
        {
          "value": "video",
          "label": "video"
        }
      ]
    },
    {
      "field_name": "scalefactor",
      "label": " Scale Factor"
    },
    {
      "field_name": "distance_btwn_obs",
      "label": "Distance between observations"
    },
    {
      "field_name": "type_of_service",
      "label": "Type of Service"
    },
    {
      "field_name": "Coupled_resource",
      "label": "Distance between observations"
    },
    {
      "field_name": "character_in_resource",
      "label": " Character in the resource"
    }
  ],
  "resource_fields": [
    {
      "field_name": "url",
      "label": "URL",
      "preset": "resource_url_upload"
    },
    {
      "field_name": "name",
      "label": "Name",
      "form_placeholder": "eg. January 2011 Gold Prices"
    },
    {
      "field_name": "guid",
      "label": "UUID",
    },
    {
      "field_name": "created_date",
      "label": "Created Date"
      "preset": "date"
    },
    {
      "field_name": "lastupdated_date",
      "label": "Last Updated Date"
      "preset": "date"
    },
    {
      "field_name": "description",
      "label": "Description",
      "form_snippet": "markdown.html",
      "form_placeholder": "Some useful notes about the data"
    },
    {
      "field_name": "format",
      "label": "MIMEType",
      "preset": "resource_format_autocomplete"
    },
    {
      "field_name": "format_version",
      "label": "Format Version"
    },
    {
      "field_name": "function",
      "label": "Function of File",
    },
    {
      "field_name": "size",
      "label": "File Size",
    },
    {
      "field_name": "checksum",
      "label": "File Checksum",
    },
    {
      "field_name": "checksum_type",
      "label": "File Checksum Type",
    }
  ]
}

It would be nice to add additional fields to the user profile, e.g. twitter, github, API keys to 3rd-party services, etc.

I imagine these additional fields can be used by CKAN extensions to implement additional functionality - e.g. tweeting when a dataset is updated using the user's twitter account, rendering a CartoDB viz using their CartoDB account, etc.

I ran into a problem when trying to create a dataset with resource fields defined in the schema. I have a validator that builds the name field, but the validator doesn't get run in time. When the browser tries to go to the form which would allow me to enter a resource, it fails because it can't complete the url since it's missing the name of the dataset I want to create.

https://github.com/ckan/ckan/blob/release-v2.2.3/ckan/controllers/package.py#L618

I have an error when opening a dataset generated by a harvester: ckan complains that the author_email is missing.

Seems that it's ckanext-scheming that triggers the error.

See error trail: http://pastebin.com/Ssneu6sP

The harvester used is the ckanext-dcat used to harvest a DCAT-RDF file coming build using the following example: https://github.com/ckan/ckanext-dcat/blob/master/examples/catalog_datasets_list.rdf

When the harvester creates the datasets, it does not provide/create a author_email field. If I edit a dataset and resave it (without changing anything), the field is created and the error does not appear anymore.

Here is an API output for 2 dataset generated by the harvester: the first one has author_email (it has been edited/saved) while the second is directly how produced by the harvester

http://pastebin.com/Vt78AgmX

Since display snippet link.html seems to link back to the dataset page, it would be nice to have a display snippet that simply renders the field content as a link.

So instead of

{{ h.link_to(data[field.field_name], rel=field.display_property, target='_blank') }}

this:

{{ h.link_to(data[field.field_name], rel=data[field.field_name], target='_blank') }}

does that make sense?

I know this is not supported, but what would be the best way to define on your schema that you don't want a field to be displayed on the Additional Info table?

This check led me to try something like:

    {     
      "field_name": "private",    
      "label": "Visibility",    
      "required": true,    
      "validators": "boolean_validator datasets_with_no_organization_cannot_be_private opendatani_private_datasets",
      "form_snippet": "visibility_form_field.html",
      "display_snippet": null
    },

But it had no effect.

Arguably the private dataset should be one of the ones excluded, but I'm thinking more about a generic option.

What do you think about "display_snippet": null or a new explicit "display": false key?

Hi,

Does anybody know how do I should to get a button of "Add new dataset" in the main window for each schema that I have defined? Is that something implemented in this extension?

I have been following the documentation and the examples but I am a bit stuck right now.

Thanks,

Jordi.

Not sure whether this is a bug or a feature, but it seems that enabling ckanext-scheming with a custom schema will, on first saving a dataset with the new schema, delete any existing "extra" fields. Even to only retain the existing extra fields to safely migrate data would be helpful - this could be achieved if config settings would toggle the display of existing extra fields and/or form fields.

Maybe I'm doing something wrong on my end, apologies if that's the case.
It appears that the hidden organization input is missing from the package form.

Symptoms: admins and editors of organizations cannot update datasets of their own organization if they did not create them themselves, resulting in

The form contains invalid entries:

Owner org: A organization must be supplied

The organization snippet follows
ckan's snippet, and when this bug happens, the template's state is:

• dataset_has_organization: fe234287-6219-4335-b6d9-4cad894eef44 (so we have an org)
• group id: none! therefore, the hidden input that should preserve the group id is not shown, so there's no organization/group_id in the form, hence the error.
• owner org: fe234287-6219-4335-b6d9-4cad894eef44

Modifying the organization snippet gives me two options:

• "if not show_organizations_selector", including a hidden input with name "owner_org" and the owner_org id (== what the organizations selector provides after all the js magic, or did I go wrong there?) does nothing, still getting "missing owner_org" error.
• disabling the "if show_organizations_selector" if -clause, and therefore always showing the org input (actually good to give the editors control over moving datasets between their respective orgs, different story though) - works. of course this goes against ckan's existing permissions handling.

I'd assume any production instance running ckanext-scheming, where organization admins/editors update datasets they didn't create themselves, will soon run into the same problem - if it's not something on my end. Also there's a typo in the error message, and it could contain more info, e.g. show_organizations_selector et al. would be useful.

Running Ckan 2.3.1

Attempting to update to scheming release-1.0.0, however I am seeing my description field being duplicated below tags.

The issue (for me) appears after this commit 8e83c0e

If I revert to the previous commit e4941f5 the duplication goes away.

e4941f5

8e83c0e

What are steps are needed to switch links from the default schema to a new schema?
Is it just a matter of changing a few templates to link "/catalog", instead of "/dataset"?

Hello,
I am creating this issue, for there is a lack of translation for the package state.

Coverage reports from coveralls seem to be broken.

As above, when trying to clone : sudo pip install -e "git+https://github.com/open-data/ckanext-scheming.git#egg=ckanext-scheming"

Output:
Obtaining ckanext-scheming from git+
https://github.com/open-data/ckanext-scheming.git#egg=ckanext-scheming
Cloning
https://github.com/open-data/ckanext-scheming.git to ./src/ckanext-scheming
fatal: I don't handle protocol '
https'
Complete output from command /usr/bin/git clone -q "
https://github.com/open-data/ckanext-scheming.git" /home/ckan/ckan/lib/default/src/ckan/src/ckanext-scheming:

Command /usr/bin/git clone -q "
https://github.com/open-data/ckanext-scheming.git" /home/ckan/ckan/lib/default/src/ckan/src/ckanext-scheming failed with error code 128

I have installed the most recent MASTER of ckan/ckan and have it running with a development.ini configuration. The project I am a part of at Swedens Agricultural University would really like to get going with CKAN and the scheming extension. Any help gratefully received.

As the title says, I can't get to h. e.g.,

{% set map_config = h.get_common_map_config() %}

will throw server errors. Would anyone know a simple fix?

Edit: not available in macros. Templates are fine.

Is there a specific need to add a schemed field in the search feature of CKAN?

I've added a field that I need searchable:

        {
            "field_name": "featured_dataset",
            "label": "Jeu de données vedette",
            "preset": "multiple_checkbox",
            "choices": [
                {
                    "label": "Oui",
                    "value": "is_featured"
                }
            ],
            "help_text": "Affiche le jeu de données sur la page d'accueil"
        }

I've update the solr schema.xml:

    <field name="tags" type="string" indexed="true" stored="true" multiValued="true"/>
    <field name="groups" type="string" indexed="true" stored="true" multiValued="true"/>
    <field name="featured_dataset" type="string" indexed="true" stored="true" multiValued="true"/>

    <field name="capacity" type="string" indexed="true" stored="true" multiValued="false"/>

Then reloaded everything.

I'm no SOLR expert, but I would expect a search query like q=featured_dataset:is_featured to work, but it does not. q=is_featured works, but clearly it's not specific enough.

Schema is still loaded and validated when creating datasets through the API, but the forms no longer show.

CKAN saves datetimes, but the jquery-ui datepicker shows only dates and will be blank if set to a datetime. This causes a dataset with date fields to lose their value when updating.

The new date preset should format CKAN's datetime to dates only, so the datepicker won't lose the value, and the date field is, as the name says, just a date field.

Filling date fields for resources results in StatementError: datetime.datetime(2017, 1, 1, 0, 0) is not JSON serializable when trying to save a non-null value. Date fields work fine for datasets.

Below is a schema for verifying this behavior. Create a dataset and a resource without filling the date fields first, then edit the resource and fill the date fields to reproduce the error.

{
  "scheming_version": 1,
  "dataset_type": "dataset",
  "about": "A reimplementation of the default CKAN dataset schema",
  "about_url": "http://github.com/open-data/ckanext-scheming",
  "dataset_fields": [
    {
      "field_name": "title",
      "label": "Title",
      "preset": "title",
      "form_placeholder": "eg. A descriptive title"
    },
    {
      "field_name": "name",
      "label": "URL",
      "preset": "dataset_slug",
      "form_placeholder": "eg. my-dataset"
    },
    {
      "field_name": "notes",
      "label": "Description",
      "form_snippet": "markdown.html",
      "form_placeholder": "eg. A detailed description"
    },
    {
      "field_name": "tag_string",
      "label": "Tags",
      "preset": "tag_string_autocomplete",
      "form_placeholder": "eg. economy, mental health, government"
    },
    {
      "field_name": "owner_org",
      "label": "Organization",
      "preset": "dataset_organization"
    }
  ],
  "resource_fields": [
    {
      "field_name": "url",
      "label": "URL",
      "preset": "resource_url_upload"
    },
    {
      "field_name": "name",
      "label": "Name"
    },
    {
      "field_name": "description",
      "label": "Description",
      "form_snippet": "markdown.html",
      "form_placeholder": "eg. A detailed description"
    },
    {
      "field_name": "validSince",
      "label": "Valid since",
      "preset": "date"
    },
    {
      "field_name": "validUntil",
      "label": "Valid until",
      "preset": "date"
    },
    {
      "field_name": "format",
      "label": "Format",
      "preset": "resource_format_autocomplete"
    }
  ]
}

I have some trouble trying to edit a resource which has custom fields. For instance, I have a custom field using : "preset": "multiple_choice" with 2 choices. During the creation of the resource I selected both checkbox so I got the two values in the additional information box.
Then I tried to edit it by deselecting them, but these two values didn't change in the additional information box, because in the dictionnary (api/3/action/package_show?id=....) values was unchanged.
And this is true for all additional fields.

So I was wondering if you had already had this kind of behavior?

When using the default errors template, it seems that type and classes are always empty which leaves the resulting output as:

<div class="error-explanation alert alert- ">

The macros/form.html errors macro allows both to be specified, but then you lose the custom labels etc. I don't need the styling to change so I just created a custom template and set it to:

<div class="error-explanation alert alert-error">

The schema should be able to accept non-ascii utf-8 encoded character since we can add the description/label of the field. JSON is supposed to be always utf-8.

However, adding non-ascii chars (for example 'é') results in an error when loading the schema file.

The follow configuration item

        {
            "field_name": "update_frequency",
            "label": "Frequence de mise à jour",
            "form_snippet": "update_frequency_qc.html"
        },

Gives following error

Module ckanext.scheming.helpers:29 in scheming_language_text          view
>>  return t.decode('utf-8')
Module encodings.utf_8:16 in decode          view
>>  return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 18: ordinal not in range(128)

Full error logs http://pastebin.com/piGmWhqF

Follow package/snippets/additional_info and roll author/maintainer name/email into one field here, ideally avoiding the use of c in the snippet as per ckan/ckan#2132

Hi again,

I have been customizing a schema and I'm doing some tests. It seems that is working quite very well but I am having problems for a Selectable field: If there is no choice selected from a selectable field then the following value appears in the field (once I've created a new dataset).

Project-Id-Version: CKAN Report-Msgid-Bugs-To: EMAIL@ADDRESS POT-Creation-Date: 2015-01-26 11:55+0000 PO-Revision-Date: 2015-02-25 10:36+0000 Last-Translator: Adrià Mercader [email protected] Language-Team: Catalan (http://www.transifex.com/projects/p/ckan/language/ca/) Plural-Forms: nplurals=2; plural=(n != 1) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Generated-By: Babel 0.9.6

My dataset configuration is below:

{
  "field_name": "city_integration",
  "label": "Integration",
  "help_text": "Is integrated in City DB?",
  "help_inline": true,
  "preset": "select",
  "choices": [
    {
      "value": "yes",
      "label": "Yes"
    },
    {
      "value": "no",
      "label": "No"
    }
  ]
}

And preset configuration is below:

{
  "preset_name": "select",
  "values": {
    "form_snippet": "select.html",
    "display_snippet": "select.html",
    "validators": "scheming_required scheming_choices"
  }
}

What I've made it wrong? May do I should to change the "validators" specified on the preset?

I would appreciate any advice.

Thanks.

We are currently encountering a situation where when we enable validation on
either a multiple_select or multiple_checkbox type we will lose data if we enter
more than one option at a time.

If we remove validation from these fields they appear to behave normally.
Any tips you can provide, were only interested at present in ensuring the fields
are required using multiple choices

{
"field_name": "langcheck",
"label": "Language Checkbox",
"preset": "multiple_checkbox",
"required": true ,
"validators": "scheming_required",
"choices": [
{"value": "en-CA [default]", "label": "English (Canadian)"},
{"value": "fr-CA", "label": "French (Canadian)"}
]
},
{
"field_name": "langselect",
"label": "Language Select",
"preset": "multiple_select",
"required": true ,
"validators": "scheming_required",
"choices": [
{"value": "en-CA [default]", "label": "English (Canadian)"},
{"value": "fr-CA", "label": "French (Canadian)"}
]
},

Hi,

I'm hitting a 500 Server error when trying to create an instance of my custom dataset.

Configuration

• CKAN v 2.4.1
• ckanext-scheming @969995b
• My schema

Steps to reproduce

• Create a fresh CKAN instance with ckanext-scheming
• Visit the form to create the new dataset (i.e., http://192.168.99.100/beataml_patient/new)
• Fill out the form, get to the next page to add resources, click next...
• 💥
• Visiting http://192.168.99.100/beataml_patient/test-patient shows that it's a "DRAFT".
• Clicking "Modify" takes me back to the same "new" form.
• Rinse, repeat.

Solr logs

I see that there are calls being made to Solr which could be from process_solr.

solr_1       | 1021391 [qtp1894627749-17] INFO  org.apache.solr.core.SolrCore  – [ckan] webapp=/solr path=/select params={q=name:"test-patient"+OR+id:"test-patient"&wt=json&fq=site_id:"default"&rows=1} hits=1 status=0 QTime=0
solr_1       | 1021401 [qtp1894627749-15] INFO  org.apache.solr.core.SolrCore  – [ckan] webapp=/solr path=/select params={q=name:"test-patient"+OR+id:"test-patient"&wt=json&fq=site_id:"default"&rows=1} hits=1 status=0 QTime=1
solr_1       | 1021409 [qtp1894627749-14] INFO  org.apache.solr.core.SolrCore  – [ckan] webapp=/solr path=/select params={q=name:"test-patient"+OR+id:"test-patient"&wt=json&fq=site_id:"default"&rows=1} hits=1 status=0 QTime=0
etc...

There's nothing being generated in CKAN logs.

Do you have any thoughts as to what could be causing this?

Thanks!

The pagination for organizations relies on an undocumented method on IGroupForm (see ckan/ckan#2771). Because this method (group_controller()) does not exist on SchemingOrganizationsPlugin, it falls back to the group controller.

This leads to the generation of links in the pagination like this /group/org-name?page=2.

PR follows.

using iFacets plugin interface cc @jguo144

When I set "required" : "true" and as my first validator : "scheming_required", nothing happened because the validator not_missing is present here instead of not_empty :

@scheming_validator
def scheming_required(field):
"""
not_empty if field['required'] else ignore_missing
"""
if field.get('required'):
return not_missing
return ignore_missing

ckanext-hierarchy instantiates the view singleton for organisations, and so does scheming, so there can only be one. Is there a way to have both, apart from implementing hierarchy as part of scheming?

I want to accept arbitrary dicts for a field that i defined in a scheme. It seems that CKAN tries to apply some kind of sub-schema if it encounters a list of dicts. Is it possible to define such sub-schemas with this extension?
And if the fields are not known, all the data is put into the key "__junk" as a flattened dict.

What I ended up doing is writing my own validator that gets the values out of the __junk, but I'm pretty sure this is a bad solution (see here for details: https://github.com/ogdch/ckanext-switzerland/blob/421de6c546462f0ac5764f93064d68d285addda2/ckanext/switzerland/validators.py#L39-L62).

Any ideas how to handle this?

After installing and enabling the extension, CKAN stops loading and throws a 500 Internal Server Error.

Steps to produce:
Install CKAN 2.3 from package
Clone this repo and then python setup.py install
Install requirements pip install -r requirements.txt
Add "scheming_datasets" to ckan.plugins
Add "scheming.dataset_schemas = ckanext.scheming:ckan_dataset.json"

Below is the error from the webserver:

mod_wsgi (pid=29654): Target WSGI script '/etc/ckan/default/apache.wsgi' cannot be loaded as Python module.
mod_wsgi (pid=29654): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
Traceback (most recent call last):
File "/etc/ckan/default/apache.wsgi", line 10, in
application = loadapp('config:%s' % config_filepath)
File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, *_kw)
File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, *_context.local_conf)
File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
val = callable(_args, *_kw)
File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware.py", line 57, in make_app
load_environment(conf, app_conf)
File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 232, in load_environment
p.load_all(config)
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 134, in load_all
load(*plugins)
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 149, in load
service = _get_service(plugin)
File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 255, in _get_service
return plugin.load()(name=plugin_name)
File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1989, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/lib/ckan/default/lib/python2.7/site-packages/ckanext_scheming-0.0.1-py2.7.egg/ckanext/scheming/plugins.py", line 6, in
from ckan.plugins.toolkit import (get_validator, get_converter,
ImportError: cannot import name get_validator

Any help/ direction would be much appreciated,

Julian

If the CKAN search should "know" about all the defined fields of a schema, this extensions should make use of the before_index method and customize the dict that gets sent to Solr.

This issue is probably related to #54.

Sometimes, the data provided for scheming is a list with the form (id, value), mainly in the case of a value list.

For example, to declare an update frequency, one has to provide a standardized value (eg. freq:irregular), but we want to display something more fitted for human (e.g "Ponctuel").

Given how things are working it is far from straighforward. Since the place where it is the main concerns is when not using english, at least the display should use the translation feature.

Hi I'm using ckan 2.4 just trying to display the ckan_dataset schema provided in the package.

When I go to select an organisation from the drop down (or there an organisation already selected as I have started making a dataset within the organisation) the selected organisation doesn't show up in the box. It is definitely selected, as can be seen in the developer tools (where the selected option updates) and the dataset can be create and shows up under the correct organisation. But the text just isn't showing up, it just remains blank.

Thanks,
Renee

Using the select widget with its choices defined in the custom schema becomes a performance issue for large lists of choices, as they are fully loaded when the dataset edit form is rendered. E.g., 2000 lines of geojson for the spatial snippet will significantly slow down the read view.

@wardi could you point me in the right direction: Is there an elegant way to provide an autocompleting box (like the tag snippet) for custom lists of label:value pairs?

Hi Ian.

I have developed some opendata portals in Spain (e.g. https://datos.upo.gob.es) which use CKAN as a backend with Wordpress + Wijmo as a frontend. I have a portal that needs multilingual support and I have solved this problem by adding extra fields in the dataset form following the CKAN extension docs, but I would like to try out your ckanext-fluent extension.

I have installed ckanext-scheming, but when I activate the plugin and I access the server CKAN shows the following error:

File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/plugins.py", line 6, in 
   from ckan.plugins.toolkit import (get_validator, get_converter,
ImportError: cannot import name get_validator

What am I doing wrong?

Please, what must be included in ckanext.spatialx:spatialx_schema.json, ckanext.spatialx:spatialxy_schema.json?

In many cases organization and license should be mandatory.

However, adding "required": true does not seem to be effective. Although it is possible to add a validator not_empty, it does not add the "required" red star next to the field.

The scheming for datasets works like a charm, but for groups and organization only the about page seems to be changed. I can save additional fields via the API, but the form to add or edit a group/organization do not show the fields defined in the schema.

How does that work?

Scheming currently does not handle using tag vocabularies in schemas. I am aware that choices targets selection from a given list. However, using tag vocabs for that purpose would allow creating, deleting and modifying the possible values programmatically via the remote API. This is very useful for uses-cases where scheming is used to create and maintain a number of different interfaces for different user-groups and where the list of potential choices for different fields might change frequently (e.g. due to user feedback).

Here is an implementation: #116

Ok this is stretching it a bit, but since scheming tackles custom form widgets, and ckanext-spatial provides this really good to have but (for end users) quite challenging spatial field, it would be very useful to integrate both.

A form_snippet spatial.html should allow users to

• paste a valid GeoJSON geometry as per ckanext-spatial, or
• draw a polygon on a leaflet map, writing back the geojson geometry of the convex hull (if several polygons are drawn) or
• upload a geojson file with the bounding polygon (or a bunch of features), then save the convex hull's geometry

A display_snippet spatial.html could render the dataset extent just like ckanext-spatial does as a map, hiding the unhelpful WKT blurb. Edit: done

Also posted to ckanext-spatial.

If the JSON definition includes a key help_text, display next to or below the form input.

For the organization field, the first organization is selected by default which mean that except if the user specifically select "no organization", there is always a default.

For licence, it should work the same, at least if the fields is set to "required" as it work for other select fields. (Maybe this will be resolved if #48 is resolved?)

This is a very useful plugin, thanks for creating it in the first place! How much hassle would it be to create a date preset with some shiny js date picker?

The ini files are located on the tests folder, which is fine but not where generally extensions have them, so worth mentioning on the README.
The use = config:../../../links/ckan/test-core.ini has a links folder not present on a default set up, I guess this is set for Travis?
There are two ini files, what's test_subclass.ini for?

Because this check is wrong. AFAICT what you are getting from the helper already is schema.dataset_fields, which is a list of [{'field_name': 'notes',... }].

An input string of "2014-03-27" being loaded into a plain-text solr field (type="string" or type="text" in the schema) will result in a plain-text string containing the unix timestamp of "2014-3-27" instead of a plain-text string containing "2014-03-27".

This only occurs when using the built-in date preset. Removing the preset results in "2014-03-27" being stored.