Artifacts of the EKGF Data Product Workgroup
The specification is generated by Respec and (temporarily) published here: https://ekgf.github.io/dprod/dprod
We're using the CLA assistant service to collect CLA signatures of all contributors.
Artifacts of the EKGF Data Product Workgroup
Home Page: https://ekgf.github.io/dprod
License: MIT License
Artifacts of the EKGF Data Product Workgroup
The specification is generated by Respec and (temporarily) published here: https://ekgf.github.io/dprod/dprod
We're using the CLA assistant service to collect CLA signatures of all contributors.
In the 16th January version, the lifecycle status of the Data Product is defined to be: Ideation, Design, Build, Deploy, Consume.
The CDMC Designing and Managing Data Products document includes a status for an obsolete Data Product - 'Retire'.
Example of a Data Product with Equity Trades. Contains two datasets, one for trades in LSEG and one in Euronext
"@context": "https://ekgf.github.io/data-product-spec/dprod.jsonld",
"id": "equity trade-xxx",
"@id": "https://y.com/products/equity-trade-xxx",
"@type": "DataProduct",
"title": "Equity Trade XXX",
"description": "Trade data defining the outcome of equity trades between parties in different stock markets, where the terms are primarily reflected in the tradable product. Additionally, Trade includes attributes such as the trade date, transacting parties, and settlement terms. Some attributes, such as the parties, are already defined in the Party Product and are simply referenced in Trade",
"dataProductOwner": "https://www.schema.xxx/person/AnnTaylor",
"lifecycle" : "Consume",
"outputPort": {
"@type": "dcat:DataService",
"id": "equity-trade-euronext-xxx-tabular-adls-prod",
"@id": "https://y.com/service/equity-trade-euronext-xxx-adls-prod-1",
"dcat:endpointURL": "abfss://[email protected]/demo/full/trade-euronext",
"dcat:endpointDescription": "Details for accessing storage account",
"isAccessServiceOf": {
"@id": "https://y.com/service/equity-trade-euronext-xxx-tabular",
"id": "equity-trade-euronext-xxx-tabular",
"@type": "dcat:Distribution",
"dcterms:format": "application/parquet",
"dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
"isDistributionOf": {
"@type": "dcat:Dataset",
"@id": "https://y.com/dataset/equity-trade-euronext-paris,
"datasetOwner": "https://www.schema.xxx/person/JohnBarks",
"title": "Equity Trade Euronext Paris XXX",
"id": "equity-trade-euronext-paris-xxx",
"dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
}
}
}
"outputPort": {
"@type": "dcat:DataService",
"id": "equity-trade-lseg-xxx-tabular-adls-prod",
"@id": "https://y.com/service/equity-trade-lseg-xxx-adls-prod-1",
"dcat:endpointURL": "abfss://[email protected]/demo/full/trade-lseg",
"dcat:endpointDescription": "Details for accessing storage account",
"isAccessServiceOf": {
"@id": "https://y.com/service/equity-trade-lseg-xxx-tabular",
"id": "equity-trade-lseg-xxx-tabular",
"@type": "dcat:Distribution",
"dcterms:format": "application/parquet",
"dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
"isDistributionOf": {
"@type": "dcat:Dataset",
"@id": "https://y.com/dataset/equity-trade-lseg-xxx,
"title": "Equity Trade LSEG XXX",
"id": "equity-trade-lseg-xxx",
"dcat:conformsTo": "https://cdm.finos.org/docs/event-model"
}
}
}
}
Example of a Data Product providing sba pool rates either through an api or a database connection
"@context": "https://ekgf.github.io/data-product-spec/dprod.jsonld",
"id": "sba-pool-rates",
"@id": "https://y.com/products/sba-pool-rates",
"@type": "DataProduct",
"title": "SBA Pool Rates",
"description": "Rates for SBE Pool that are MortgageBackedSecurities ",
"dataProductOwner": "https://www.schema.xxx/person/johnSmith",
"lifecycle" : "Consume",
"outputPort": {
"@type": "dcat:DataService",
"id": "sba-pool-tabular-prod1",
"@id": "https://y.com/service/sba-pool-rates-tabular-prod1",
"dcat:endpointURL": "jdbc:oracle:[email protected]:43534/PGPERG.WORLD",
"isAccessServiceOf": {
"@id": "https://y.com/distribution/sba-pool-tabular",
"id": "sba-pool-tabular",
"@type": "dcat:Distribution",
"dcterms:format": "https://www.iana.org/assignments/media-types/application/sql",
"table": [
"http://server/xxxx/database/yyyy/schema/zzzz/table/sba-pool",
"http://server/xxxx/database/yyyy/schema/zzzz/table/sba-pool-identfier",
]
"isDistributionOf": {
"@type": "dcat:Dataset",
"@id": "https://y.com/dataset/sba-pool,
"id": "sba-pool",
"dcat:conformsTo": "https://spec.edmcouncil.org/fibo/ontology/SEC/Debt/MortgageBackedSecurities/SBA-Pool"
}
}
}
"outputPort": {
"@type": "dcat:DataService",
"id": "sba-pool-rates-json-prod1",
"@id": "https://y.com/service/sba-pool-rates-json-prod1",
"dcat:endpointURL": "http://y.com/api/sbl-rates-us",
"isAccessServiceOf": {
"@id": "https://y.com/distribution/sba-pool-rates-json",
"id": "sba-pool-rates-json",
"@type": "dcat:Distribution",
"dcterms:format": "https://www.iana.org/assignments/media-types/application/json",
"conformsTo": [
"http://confluent-registry-y/rates-json-schema.json,
]
"isDistributionOf": {
"@type": "dcat:Dataset",
"@id": "https://y.com/dataset/sba-pool-rates",
"id": "sba-pool-rates",
"geographicalCoverage" :"https://y.com/country/US"
"dcat:conformsTo": "https://spec.edmcouncil.org/fibo/ontology/SEC/Debt/MortgageBackedSecurities/SBA-Pool"
}
}
}
}
** Note **
Representation of the different physical models in the distributions is not defined yet from the DPROD model. This work will be coordinated according to the odrl market licencing requirements
Maybe I overlooked it but would have expected that the dprod:
itself is defined in the namspaces section. like: https://www.w3.org/TR/vocab-dcat-3/#namespaces
cf. https://github.com/rpietzsch/data-product/pull/1/files
I tried to make sense of the examples and encountered some issues:
ttl
for human use (if for technical purpose json
is needed it is only one rapper / riot / jena call away)
ttl
json
distribution of the examples I suggest to make ttl
the authoritative format and generate the json on every pushdprod:
although they do not seam to exist there yet either:
rdf:computedOn
rdf:isMeasurementOf
rdf:Metric
rdf:title
to rdfs:label
(maybe dct:title
is what you intended, but inconsistent with the other examples)Please don't take offense. My Black Friday search wasn't successful yet and so I need to vent a little bit.
Writing and commenting as I read. Especially, this may mean that I comment on something that you define much further down without a forward reference. That is somewhat intentional.
DCAT is a catalog. DPROD is for products listed in the catalog. Why would one be a profile/extension of the other?
You use the term “Semantic Data…” What is the semantic of “semantic” in this context?
As pointed out elsewhere: Why emphasize or limit this for “Mesh.” It is a cool word and very popular. But what exactly makes products only suitable in a Mesh? Imagine a data-product that lists the geo-coordinates of fire hydrants around the country. Couldn’t I just download the file and move on? What would be meshy or semantic about that?
I disagree with your notion of shift left/right. The data-owner (maker) should also own the meta-data. Granted, all publishers cannot and should not handle data-integration. But they should handle explaining what it is they are offering. Likewise, I really don’t want a central team to manage my ontology. They are just going to make a mess of it. It is my ontology.
“...and ontological classes…” what does this mean?
“... input and output ports…” ports is undefined
If memory serves, in DCAT a data service is a means to access the dataset described in the catalog. Are you suggesting that data services are extended to describe meta-data?
“...semantic meaning…” This basically says: “meaning meaning.” You sure that is what you mean? ;-)
“...This allows for…” Of course it allows. Its absence also would not prohibit it. It just might make it less convenient.
“...semantics ensure that all stakeholders have a common understanding…” You ought to be really careful with such statements. I have seen ontologies and that didn’t mean that they were useful or supported understanding, especially not by humans. Imagine I wrote you an ontology and obfuscated all names and IRIs, replaced them with UUIDs and eliminated all regular comments and explanations.The formal structure and hence implied formal semantics would remain the same. Good luck emailing one this ontology, say as an RDF/Turtle file and expecting anybody to find that it creates a “common understanding.”
“...fundamental idea behind a Data Mesh…” Colloquialism. Intuitive but neither semantic nor appropriate. If you want to explain what a data mesh is supposed to be, put a reference there assuming that something even half normative exists.
“...can be programmatically understood…” will there be LLMs involved? Or did you mean “...can be verified against something and interpreted programmatically…”?
“...ensure that these products can interact…” are products “active” in the sense that they do something? Like “interact?”
“DPROD maps to the Data Mesh notion of a port to the DCAT notion of a DataService, so we can declare a DataProduct by and specify and input and output ports and these ports are Dataservice.” Could not parse.
DCAT states that a data service is defined as: “A collection of operations that provides access to one or more datasets or data processing functions.” You might make it very clear that you are changing that definition. Is it really just that: One inputPort and one OutputPort? Nothing else?
“A data product is a rational, managed, and governed collection of data, with purpose, value and ownership, meeting consumer needs over a planned life-cycle.”
How do you define and measure “rational?” How do you assign and measure “value?” How is value expressed? In US$? I suppose I could write more here. I’ll leave it as: You don’t define what a data product is. You list a bunch of cool, marketable attributes.
I’ll stop here because the rest seems unfinished.Maybe a parting thought: Reading all of this, I still don’t get at all how I would understand (you know, the semantic bit) what a specific dataproduct (like listOfFireHydrants) would be described and how I would use that. I get that somehow a port is involved. Now what?
Footnote: You mention odrl but don’t seem to use/reference it elsewhere.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.