npolar / api.npolar.no Goto Github PK
View Code? Open in Web Editor NEWSearchable data storage
Home Page: https://api.npolar.no
Searchable data storage
Home Page: https://api.npolar.no
A lot of tests are failing. They shouldn't.
Users will often tag an entry with only the most relevant tag. This tag is sometimes a "child" tag (e.g. ICE Fluxes), and this will result in a missing "parent" tag (e.g. ICE).
The result is somewhat counter-intuitive filtering options; a filter for a parent tag may produce far less hits than its child filter. Real-world example:
ICE Fluxes -> 50 hits
ICE -> 5 hits
Proposed solution: Automagically apply the parent tag (before saving) whenever a user has applied a child tag only.
The issue has been identified for ICE and its sub-programmes (e.g. ICE Fluxes), and some area tags, like areas contained within the Arctic. (E.g. entries may be tagged only "Svalbard", not "The Arctic".)
Example 1: http://www.npolar.no/en/publications/ (see "Programme" filters)
Example 2: http://www.npolar.no/en/projects/ (see "Area" filters)
And indexing rivers
Because the mime type "application/vnd.geo+json" does not pass the simplistic #is_json? test in
https://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/jsonp.rb
With external data dumped to some sink, like rsync to a disk folder, we need a way to prove after imports that a system actually contains the data it should
We can inject source metadata in /source and compare source metadata like number of documents that day (or hour or timeframe) to the actual content in the destination API.
For PBSG's polar bears/human interaction database
For configureless CORS
Part of the problem is the DELETEs here https://github.com/npolar/api.npolar.no/blob/master/lib/npolar/api/bootstrap.rb#L100
The text areas "summary" and "abstract" in projects would benefit from allowing simple markup (specifically: italic, bold, headings, subscript, superscript, link).
$ curl -IXOPTIONS http://address
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 0
Allow: DELETE, GET, HEAD, OPTIONS, POST, PUT
To avoid inconsistency issues, we switch /schema to a file system backed API.
There are 4 simple rules to follow
0. Naming convention is semantic like /schema/dataset-1.x.y[.dev].json (where x is major and y minor edit number)
Employee and project entries suffer from a potential conflict between timestamps and their related "labels":
NPI Employee
and On Leave
State
These labels should ideally be "auto-updated" by the system, deduced from the timestamps. (As opposed to the current solution, which requires manual updating. Alternatively, they could be dropped – they are, after all, redundant. However, they do make very useful filters.)
Example situation:
An employee can be set as "currently employed" by adding a "hired" timestamp, but the NPI Employee
field could indicate the opposite, until manually updated. Same vice versa, and the same type of conflict exists for On Leave
, as well as projects and their state (planned
, active
, etc.).
The current system effectively forces editors to continuously monitor all these timestamps/statuses manually, and manually update the entries. Furthermore, such updates must be done at very specific times (the exact times specified in the timestamps).
Description: When requesting a feed from couch it will return information on all documents in the database including the couch _design documents holding couch views.
Reproduction:
Bugged Code: Code in couch.rb
Datasets are validated against XML and JSON schemas before saving, but vocabularies are not cheked.
Could improve XML schema validation by checking vocabularies against http://api.npolar.no/gcmd/concept/?q= or equivalent ie. https://github.com/npolar/gcmd/blob/master/lib/gcmd/concepts.rb
Some services will not survive republishing if ES mapping is out of sync with the data.
Also, need to kill old, manually setup indexing rivers
Must support DIF XML (metadataPrefix=dif) and must provide set support.
Probably use
https://github.com/code4lib/ruby-oai
Should support all 6 verbs from v2 spec http://www.openarchives.org/OAI/openarchivesprotocol.html#ProtocolMessages
Need both XML schema validation and JSON schema validation
Improve XML schema validation by checking vocabularies against http://api.npolar.no/gcmd/concept/?q= or equivalent ie. https://github.com/npolar/gcmd/blob/master/lib/gcmd/concepts.rb
Planned changes from .pre:
But rely on the formats in the service definition
Until now we don't have official URIs for getting storage-layer data out of the API.
We have a couple of hacks, like _feed, but this isn't a scalable solution.
Auto-updating "updated" is sometimes unfortunate, when you have external data coming in you loose the last "updated" date.
In AtomPub "edited" is used for system-generated edit-time, while updated can be user provided. Suggest we move to that.
Currently the only thing that works is using a folder/file naming convention
This is perhaps OK as default, but should be overridable with either inline or linked mapping
Reimplement core functionality in Go
We need a robust mechanism to GET all documents in a collection.
The API co-evolved from ideas similar to those expressed in http://tools.ietf.org/html/draft-kelly-json-hal-06 - with widespread use of hyperlinks between documents. However, at the collection level, we lack links from the Collection URI to bundles of documents (similar to a database view). With such links, RESTful clients could follow a simple set of rules to download entire collections
For environmental monitoring timeseries.
Endpoint
/monitoring
workspace: monitoring
3 document types
collection:timeseries
collection:parameter
collection:indicator
Allowing for re-arranging the order of affiliated institutions (Participants -> Organisations) for projects with drag-and-drop, like one can for participants (Participants -> People), would be a handy enhancement.
One of the major features we are missing in version 1.0.0 support for async work flows like streaming.
Every client should declare explicit schema versions, and the server should reject (422) documents that are lower than the minimal schema version accepted.
Fra datapolitikken:
Beskyttelsesperioden skal datofestes, og bør normalt ikke settes til mer enn 2 år etter at datainnsamlingen er avsluttet. Innenfor rammene av de kontraktsvilkår og eksterne betingelser som gjelder for prosjektet, kan beskyttelsesperioden gjøres lengre hvis det er nødvendig for å ivareta førsteretten til publisering. Seksjonsleder er ansvarlig for å sette frigivelsesdato og kan utvide beskyttelsesperioden inntil 4 år etter at datainnsamlingen er avsluttet. Utvidelse av beskyttelsesperioden ut over 4 år skal kun gis unntaksvis og må godkjennes av avdelingsdirektør.
Når beskyttelsesperioden er utløpt, eller når ingen beskyttelsesperiode er datofestet, vil data som er ferdig prosessert, dokumentert og lagret ved instituttet være åpent tilgjengelige fra data.npolar.no.
When reducing the response fields there are situations that one or more documents don't have these fields. when this happens the search engine will return a null entry.
Add a null check to see if there is anything in the entry => https://github.com/npolar/api.npolar.no/blob/master/views/api/index.rb#L162
Original bug reference npolar/icelastic#4
Currently it is possible to define the same path multiple times in different service documents.
When creating two service documents with different names but using the same path the API will boot and show you the same /endpoint twice. Both seem to be pointing to the same API
http://api.npolar.no/dataset/oai
Tools
http://oval.base-search.net/
http://validator.oaipmh.com/
delete-persistence?
resumptionToken?
granularity?
OK with oai:npolar.no:dataset - gives errors but should work according to the examples given at http://www.openarchives.org/OAI/2.0/guidelines-oai-identifier.htm - linked from
http://www.openarchives.org/OAI/2.0/oai-identifier.xsd#repositoryIdentifierType
From
GCMD (18) IPY (18) DOKIPY (15) NMDC (12)
To
ipy.org
cryoclim.net
gcmd.nasa.gov
marine (for NMDC - also set topics to include "marine")
/expedition
/expedition/track
We should allow endpoints to filter documents before indexing
https://github.com/elasticsearch/elasticsearch-river-couchdb#script-filters
For /oceanography and other large and numeric-heavy indexes, we should reduce precision to e.g. 1e-2 and benchmark
Uses "string" even for object like locations...
"locations":{"type":"string","index":"not_analyzed"},
https://tools.ietf.org/html/rfc2616#section-10.4.11
Just trap 404 and the following body...
{
"error": "not_found",
"reason": "deleted"
}
Create an authorizer middleware that can validate with a JWT
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.