Comments (3)
So the particular problem is that source documents (and transformation definition) are specified as URLs and implementers are instructed to retrieve e.g. the data.csv from that URL, or to find the metadata alongside the csv file.
We operate in a different context, we want to feed csv/ json files into a pipeline - e.g. by passing filenames at the cli or contents in an HTTP body. In other words the csvw authors want to put csv on the web, we want to put csv into pipelines!
I believe we could follow the spirit of the spec by hosting the prepared csv and json metadata at a given URL, then instructing the csv2rdf step to retrieve it (note that you're supposed to be able to find one from the other by convention and (?) because the json metadata contains a url
reference to the csv).
In practice we might want to ignore this part of the spec, such that the url
parameter is just an identifier for the source (that mightn't resolve) and make our csv2rdf implementation expect csv and json as input parameters as we have with other grafter pipelines.
from table2qb.
Thanks for expanding on the specific problem @Robsteranium, that is indeed what I was referring to.
I should also point out that searching the CSVW working groups motivating use cases none of them come from the perspective of transformation.
I think you're right that we're operating in a different context, and there is stuff we should do to reconcile it. Some strategies are:
A: Build the declarative CSVW world
As I think @Robsteranium is suggesting an option is to have a prep phase which creates real world side effects to ensure the world is in a state that agrees with the view of the tabular-metadata file. When run in a remote context this might involve:
- In a real world context this involves upload the input files to a URL (we should probably also use a hash of file contents in the URL string to ensure idempotency e.g.
https://csvinputs.s3.amazonaws.com/0d599f0ec05c3bda8c3b8a68c32a1b47.csv
(or in a test context this just creates the file locally on your filesystem possibly with a hash and setsurl
to the file path). - Execute the transformation and build out all the target URIs, observations etc inside your draftset. Essentially we materialise the implications of the csvw jsonld metadata document into the RDF world.
- For completeness we could store the jsonld somewhere too and link it in with
Link
headers on CSV downloads etc and whatever the spec suggests. So CSVW readers can build an RDF view of our CSV). (Though personally I think this is probably pointless as you can more easily just consume the RDF.)
How would such an approach handle updates / revisions and appends of new data? It's not clear to me how this should be represented. Do you use a csvw:TableGroup
and append new tables for each new quarter?
Additional difficulties with this approach are considerations about how this might work with draftsets. In a perfect world the csv input file would be inside the draftset and downloadable too. Except we probably want to store it on S3, so will need to munge permissions on draftset state changes / publication etc.
B: We identify paramaterisable things in the spec and handle them as custom extensions
There are many ways we could do this, some of which might keep within the spec. One way that strays out of the spec might be to replace "tables"
with a new extension called "arguments"
which is a set of named input arguments providable via environment/command-line/web-form.
from table2qb.
Yeah, there's a 1:1 (lossless?) correspondence between the csvw and the rdf. If we have rdf, you should be able to offer it for download as csvw. And if you have csv we get get the rdf triples that represent it.
So a client-pipeline would go from our agreed interchange format (for that client), to the csvw input format, and then run our csvw.clj implementation
from table2qb.
Related Issues (20)
- Trim column headings when looking for matches in columns.csv HOT 1
- Trim column headings in codelist-pipeline HOT 1
- cube-pipeline ignores + character when generating code list URIs HOT 6
- cube-pipeline does not pick up datatype config from columns input HOT 1
- Update README - multi-measure cubes
- codelist pipeline CSVW should not generate ?scheme skos:member ?concept triples HOT 1
- Make use of csvw validation features
- Introduce whitespace trimming under a feature flag
- Allow arbitrary (non-component) columns in cube-pipeline
- Make observation-uri configurable - allow guid hashes
- Label should only be required if it's used in the URI
- Creating multiple URIs from multi-valued cells HOT 4
- Don't generate used-codes skos:Collections
- Put variables in virtual columns so we don't need a meta-templating language HOT 3
- Attempting to render a template with missing variables should raise an error
- Feature request: specify an additional class for the concepts HOT 1
- Make uncoded components datatype properties
- Fix dependencies in library
- regional-trade example csvw outputs don't include @context HOT 1
- Add images from coloured-spreadsheet example to documentation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from table2qb.