Prefix section disrepencies?

Section on Externally Defined Prefixes need requirements defined. Also an example of how Externally Defined Prefixes should be included, i.e. using the proposed VANN vocabulary. Also, were we going to evaluate the viability of json-ld style prefixes/context files? see https://opengeospatial.github.io/ELFIE/json-ld/

Refer to #39

We discussed adapting requirements 7.3 around aliases to:

• allow plain JSON encoding of aliases as an option
• providing JSON schema for conformance testing of the plain JSON encoding of aliases

@adamml in #59 proposes the following JSON Schema. we can add that into the repo as a resource for conformance tests.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "netcdf-ld__alias",
    "type": "object",
    "patternProperties": {
        "^.*$": {
            "type": "string",
              "pattern": "^(https?)://"
        }
    }
}

This issue is about edits to the spec to include the above and adding the json schema into the repo

Based on #34 and other discussions around schema.org, it would be good to add a worked example of using schema.org in the informative section.

Section 6.5.7 explains how to resolve the value of an attribute to a single variable reference, or multiple variable references as ordered and unordered collections. The description and examples make sense and look implementable.

However, I'm curious as to why this system of string decomposition is chosen rather than allowing the attribute to simply have multiple values, each of which could be separately resolved to a single variable. Is this a more natural way of defining a multitude of variable references on a single attribute? Since the set of values on an attribute is ordered, is it necessary to distinguish between an ordered and unordered set of variable references?

In conformance class A.5 and worked example 6.6.5, the bald__references attribute on list_collection and set_collection is resolved as if it is a resource-valued property (ie. its values are resolved as links to variables), however no alias file containing a description of bald:references is given, and those variables don't have a "shape" from which to construct a bald:Reference. Is bald:references a special case and is there a part of the spec that describes this behaviour?

Some terms have been suggested in the clause 4 of the specification. This issue is about filling out the text of the terms and definitions section.

5 people are listed - Jim is no longer at NOAA, need to check with Aleksander if he would like to continue

Hi, the draft looks excellent. It looks to me as though it would be technically easy to map this into NCML to produce XML files with declared namespaces. Have you considered this option?

The prefixes section of the spec explains how prefix mappings can be defined in the NetCDF file as attributes on a specific group or variable.
The attributes section states that all attributes on groups and variables in the file should be mapped to statements in the RDF graph, and that if the attribute name contains a prefix mapping, it should be applied.

So the description of the prefix group:

group: prefix_list {
    :bald__ = "https://www.opengis.net/def/binary-array-ld/";
    :skos__ = "http://www.w3.org/2004/02/skos/core#";
}

would be:

<file.nc/prefix_list> a bald:Container ;
    bald: "https://www.opengis.net/def/binary-array-ld/" ;
    skos: "http://www.w3.org/2004/02/skos/core#" .

With bald: and skos: acting as properties. Although this is permitted in RDF, it seems like it's not how the attributes are intended to be used, since bald: and skos: are not actually properties.

Should the spec be changed to omit the prefix group / var or its attributes from the resulting graph?

Section 6.5.6 states that "An attribute name shall be mapped to an alias URI if, and only if, there is an exact match for the full attribute name as a dct:identifier for an entity within the alias graph".

Conformance class A.3 implies that the attribute name "title" should be aliased to "https://raw.githubusercontent.com/opengeospatial/netcdf-ld/v0.1/standard/abstract_tests/aliases/NetCDF.ttl/title", however the alias file contents describe the resource "http://def.scitools.org.uk/NetCDF/title" as having the dct:identifier "title".

I think this is the same inconsistency that is discussed here.

A statement is made "A set of RDF vocabularies are used within this standard."

... without giving the reason for using RDF vocabularies.

Propose expanding this a little to cover the reason and use of RDF vocabularies in netCDF-LD

A concrete example would complement the text

In the spec, there are a few references to the bald: namespace and the BALD ontology.

From the user/reader's perspective, it would be good to describe it in a bit more and provide context for its role in the netCDF-Classic-LD spec. We need to be careful that we don't re-define what the BALD ontology is (as that's what the OWL does).

I'm sorry if this has been covered elsewhere, but I was wondering why you use "__" rather than ":" or "@" to indicate that a variable has a namespace?

The use of the colon is supported since NetCDF 3.6.3 (according to NUG), and "@" has been supported before that. The "@" gives clearer CDL, e.g. float cmip@tas rather than float cmip\:tas, where the escape character is intrusive.

Note - some of this is subjective, from my POV as someone who is not very familiar with NetCDF features, so just a suggestion.

Section 6.6.3 describes the requirements for "variable reference between variables defined with respect to netCDF dimensions", however at this point it's not clear what that terminology actually means. It's later clarified in section 6.6.4 that coordinate variables are an example of such a reference, but it's not clear whether they are the only examples that are expected to be supported by NetCDF-LD, or if there are others that are not explicitly mentioned. I think it would be clearer to define the concept of coordinate variables (and any other dimensional variable references) before 6.6.3, and then to give simple examples of the expected representation of those references in 6.6.3.

When describing reference entities, the section contains the following statements:

• The <$referenceEntity> may define a bald:sourceShape, where that source shape is required to be different from the defining shape of the source array.
• The <$referenceEntity> shall define a bald:sourceShape, whether that source shape is required to be different from the defining shape of the source array or whether the shape is the same.

which seem to be either contradictory or redundant.

The worked example 6.6.5 doesn't appear to implement the requirements described in 6.6.3, despite containing dimensional variable references. It includes the shapes of the variables as described at the beginning of 6.6, and the resolution of explicit bald__references attributes through prefix expansion and variable location, but does not include any inference of bald:references relationships or any shape / reshape descriptions.

Propose that we define requirements classes in the doc. Something like as separate classes:
a. Core
b. Prefixing
c. Aliases
d. Attribute names
e. Variable references

Describe Design Goals and Rationale for nc-classic-ld.

There is scope to describe nc-classic-ld in a similar way to JSON-LD - e.g "Linked Data" approach and perhaps aligned principles around Design Goals and Rationale? Also differences.

Feedback from adopters has revealed some unwanted effects from the definition of prefixes within a file.

With a netCDF classic file the prefixes may only be stored as a variable.

However, a variable carries data, which may be missing, but is still an oddity for a metadata only element.

NetCDF software is prone to read this variable and assume that it is a content data variable and present it to users.

With this in mind, it is proposed that the external mechanism used for alias definitions is extended to enable the definition of prefixes external to the netCDF file.

In this way, a file can use prefixed entities, but resolve them using a remote resource to provide the predicate URIs

This proposal reduces the self-describing nature of the file somewhat.

This proposal reduces an obstruction to adoption for users and makes the netCDF files simpler and easier to manage.

The namespace functionality of the prefixed attribute and value remains intact.

Within the standard, the term broadcast is used extensively. This was initially adopted from the Python Numpy implementation, e.g.
https://numpy.org/devdocs/user/theory.broadcasting.html

I think the document assumes too much knowledge, and that this term should be defined, and it's use expanded on in the standard, with pictures and with external references.

The bald vocabulary explicitly defines two properties: childBroadcast and parentBroadcast
These are related to broadcasting, but are actually shape statements, or perhaps reshape statements.

I think that the use of parent and child imply too strong a hierarchy. There isn't a hierarchy here, there is just a directional relationship: A has an array reference to B

I think that the shape aspect of the properties is the key, broadcasting is the downstream use. I think we should consider renaming these two properties

@adamml @jyucsiro

Clause 3: Only list standards by Standards Development Organisations (SDO) in the References clause. Examples of SDO's include OGC, ISO, W3C, and IETF.

All other cited documents should be listed in the Bibliography.

Analyse consistency problems between chapters 6 - informative and 7 - normative

Where is chapter 6 inconsistent with chapter 7?

please create new tickets for targeted issues, using the 6/7Gaps label

In the asciidoc, paths to images/* get rendered in the Github viewer. However, the automated build doesn't pick it up. The ones with the images/ path in the source don't get rendered.

I'm still finding my way around the BALD vocabulary and variable-to-variable referencing (coming from an RDF world rather than a netCDF world). I'm wondering about the role of the class bald:Subject.... whether in needs to be any more constrained than rdfs:Resource... and if so whether bald:Resource would be a better name to avoid possible confusion with subject roles in an RDF triple.

Create tests to address requirements in Chapter 7. This will provide the input test files which can also be used in the other chapters in the specifications, e.g. Chapter 6.

Examples of RDF statements are provided in this section, however, there isn't an example of how to implement explicit aliases as input and the resulting encoding from + =

#39

Section 6.5.2 of the spec defines the media type of a distribution of a netCDF file to be a dct:MediaType, however conformance class A.1.2 requires the media type on the distribution to be a dcat:MediaType instead. I assume this is a typo since DCAT doesn't define the latter and states that the range of dcat:mediaType should be dct:MediaType.

This is the case for all of the conformance classes and the worked examples in section 6.

Rationale

In discussions on the group telecon, it came to light that the Binary Array LD (BALD) specification would describe the contents of the NetCDF file for NetCDF-LD, but not provide a link to the NetCDF file itself. It became apparent that an extra predicate would be needed in the RDF representation of a Binary Array file in order to support this.

The file location should be an optional, user-specified parameter supplied at runtime.

Approach

A number of options have been considered:

• The NSF EarthCube's cyberinfrastructure GeoLink project
• Schema.org
• W3C's DCAT Data Catalog Vocabulary - Version 2

Due to the stability and maturity of the vocabularies, it was decided to focus on the Schema.org or DCAT options.

A further consideration was the grouping of NetCDF files into collections, which may be acheived in either Schema.org or DCAT if the contents of the NetCDF file are considered to be a Dataset and the collection of the NetCDF files a DataCatalog. The ability to nest, or to create heirarchies of catalogues was also considered, such as a collection of NetCDF files being available with other files or collections through a THREDDS server. While we do not provide an implementation pathway for this, the consideration motivated us to focus on DCAT which at the time of writing supports nesting catalogues, whereas Schema.org does not.

Boilerplate code

First an addition to the BALD ontology will be required:

@prefix dcat: <http://www.w3.org/ns/dcat#>.

bald:Container a dcat:Dataset. 

Then the following boilerplate would allow a software agent to traverse the graph to find the file to download the NetCDF data from:

@base <http://foo.bar/my-netcdf-file.nc>.

@prefix bald: <https://www.opengis.net/def/binary-array-ld/>.
@prefix dcat: <http://www.w3.org/ns/dcat#>.
@prefix dct: <http://purl.org/dc/terms/>.

<./> a bald:Container;
	dcat:distribution [
		a dcat:Distribution;
		dcat:downloadURL <>;
		dcat:mediaType [
			a dct:MediaType;
			dct:identifier "application/x-netcdf"
		];
		dct:format [
			a dct:MediaType;
			dct:identifier <http://vocab.nerc.ac.uk/collection/M01/current/NC/>
		]
	].

Further Considerations

• If a supplied file name ends in a '/', then the base URL should not have the '/' appended in as the subject of the graph.

Questions

@jyucsiro, @marqh a couple of questions/topics for discussion:

1. Does this look like the approach we discussed on the call?
2. I think there may be a subtlty I am missing in the way @base is parsed, at least one library I have used ignored the filename beyond the final slash when converting to RDF/XML. We may want to have a discussion about using the full URI if we take this to production.
3. Are we ok with the introduction of blank nodes here?
4. Is there a better URI which defines NetCDF than the oine I have used here?
5. The MIME type I used is not actually registered with IANA, and also there is a suggestion that THREDDS also has a different MIME type for NetCDF 3 and NetCDF 4. Can we handle this?

Apologies if this is already covered, I couldn't find a mention of it in the current document or open issues.

NetCDF Classic allows up to one dimension to be of UNLIMITED length; NetCDF-4 allows multiple such dimensions.

The current draft represents the shape of a variable as a list of integers, the lengths of the corresponding dimensions. So how should a variable with an UNLIMITED dimension be represented?

I imagine possibilities might include:

1. Define a constant bald:unlimited to represent this case.

2. Use an otherwise illegal integer (e.g. -1).

3. Use INF^^xsd:float. There is no INF in the value space for xsd:decimal (and thus xsd:integer) but there is in xsd:float and xsd:double.

Non of these feel particularly clean though arguably (1) most directly signals the intention.

/cc @marqh @skwlilac

The specification includes an alias mechanism to map attributes names to RDF predicates. These alias mappings are supplied to a netCDF-LD processor as RDF declarations. Section 6.3.4 proposes use of dct:identifier for this purpose. So an alias declaration looks like:

myns:meaningfulPredicate a rdf:Property; dct:identifier 'attributeName' .

Would it be better to use a separate property defined within the specification (e.g. bald:alias) rather than repurposing dct:identifier?

Logging this as an issue is intended to simply give a place where the question can be discussed, not to indicate any fundamental problem with the current approach.

Framing discussion

There are a number of considerations here:

This is certainly a reasonable choice. This usage is consistent with the meaning of dct:identifer and dct is a widely accepted vocabulary.

The meaning of dct:identifier is broader than it's usage here so, arguably, a more specific term such as bald:alias would enable a publisher to more clearly signal that such a declaration was intended to introduce a netCDF-LD alias rather than anything else. Indeed bald could formally state:

bald:alias rdfs:subPropertyOf dct:identifier .

In a closed world in which the configuration file supplied to the netCDF processor is self-contained and under full control of the processing agent then the choice of identifier is largely arbitrary.

If a goal is to encourage communities to publish and maintain vocabularies of terms that can be used in netCDF files and support semantic mapping then the choice might make a difference.

On the one hand, if a publisher already includes dct:identifier declarations for other reasons then this may cause issues. A processing agent taking that vocabulary as part of the input to a netCDF-LD run would need to modify it to remove any clashing or inappropriate mappings which would otherwise be unintentionally treated as aliases. Since RDF is purely additive it is easier to add missing aliases (e.g. via a supplemental mapping file) than it is to suppress unintended aliases caused by differing uses of of dct:identifier.

On the other hand, in approaching a publisher to ask them to supplement a vocabulary with aliases for netCDF use then it may be easier to persuade them to use a common and well known term than to persuade them to adopt the currently-unknown bald.

create a tagged draft release

• 0.1

ensure doc release number is updated, consistent with release tag

Distinguish between local identity and explicit identity (url). We make the point that each netCDF file has an identity but then refer to explicit identity (URI/URL )in the informative text, whereas in the normative text, we use identity a bit ambiguously.

Propose we be deliberate and qualify the specific identity being referred to.

See #39

[Apologies if this has already been considered and is out of scope.]

In looking at example NetCDF files I can see cases where attribute values could reasonably be interpreted as something other than string values. Should this be supported by NetCDF-LD?

For example, consider the following (from sresa1b_ncar_ccsm3-example.nc available from NetCDF Examples):

    float tas(time, lat, lon) ;
        tas:missing_value = 1.e+20f ;
        tas:standard_name = "air_temperature" ;

Since the CF term missing_value is defined as having the type of the variable it is applied to then it would be reasonable for that first CDL attribute declaration to be mapped to the statement:

    :tas cf:missing_value '1.e+20f'^^xsd:float .

Should NetCDF-LD support this? Mapping to a fixed type would be relatively easy to specify by e.g. allowing an rdfs:range declaration as part an alias definition. Supporting the CF convention that the type should match that of the variable it is applied to would be harder. That would need some additional vocabulary machinery in the alias declarations such as:

    cf:missing_value  dct:identifier 'missing_value';
                      bald:valueType  bald:typeOfBaseVariable .

Now consider the second attribute in the sample tas:standard_name. In CF a standard_name is just a string so a simple attribute mapping would be perfectly reasonable:

    :tas cf:standard_name "air_temperature" .

However, in the interests of semantic grounding one could imagine users wishing to link to terms to in an agreed vocabulary resource. So if cfsn were the prefix for a vocabulary server containing descriptions of standard names then it might be desirable to be able to map to:

    :tas cf:standard_name cfsn:air_temperature .

Should NetCDF-LD allow users to specify such a mapping?

If so it could also be achieved through additional vocabulary machinery in the alias definition:

    cf:standard_bname  dct:identifier 'standard_name';
                      bald:valueType  rdfs:Resource ;
                      bald:valueBase cfsn:  .

I could imagine that, even if desirable in the abstract, the complexity of specifying and implementing such machinery could be beyond the point of diminishing returns for at least the first version. However, it seemed worth at least asking the question.

/cc @marqh @skwlilac

In example 6.6.5, several variables are described as bald:Arrays rather than as bald:Resources as indicated by the conformance class A examples.
Presumably this has to do with those variables being defined with respect to dimensions, however the spec doesn't directly mention bald:Array or provide any requirement to describe such variables as bald:Arrays.

Section 6.1 needs a little work to frame the design principles and rationale clearer. At the moment it's mushed together and some bits are not complete.

I have few gnit-picky comments and suggestions for the Chapter 4 Terms and Defs wrt to the description of RDF terminology

1. wrt: 4.2 "RDF statements (or triples)" - The second sentence says:
   "It [a triple] is a set of three entities which codify a statement about semantic data".
   I suggest this be re-written as:
   "It [a triple] is used to make a logical statement about the triple's subject (which could be anything)".
   In particular, RDF in general does not restrict statements to be "...about semantic data"
   they can be about anything (or at least anything that could be named with a URI).

2. wrt: 4.2 first sentence: I don't think the "...(or a semantic triple)" in the first sentence helps. We already have statements and triples as near synonyms - I don't think a 3rd variant helps, certainly I snagged on it. As regards triples and statements, I tend toward a syntactic/semantic distinction in that a triple is the constrained syntactic form used to make a statement whereas the statement is its logical interpretation (which has a truth value) - but there is a whole rabbit hole there to disappear into around interpretations (mappings of URI to things in the world) and RDF model and syntax.

3. wrt: 4.3 "RDF Graph" at the level of this narrative, I suggest changing the last part of the last sentence to "...while the predicate is used to label the edges....". Just as in RDF in general there is some triple/statement duality, there is also some property/predicate duality. The latter is routed in the notion of a logical function, p(s,o) -> true|false, a binary predicate, which has an extension that is all the {s,o} pairs for which p holds true. Anyway that's back on the edge of the rabbit hole.

4. wrt 4.4 URI the first sentence says:
   "A Uniform Resource Identifier (URI) is a string of characters that precisely identify a particular information object."
   There is no such constraint - URI in general can be used it identify anything - eg. http://dbpedia.org/resource/Elvis_Presley is used to identify the person and global phenomenon that is/was Elvis - indeed the example later focusses on B.B. King.

Clause 2: The third paragraph needs a complete rewrite. The section makes reference to Annex B, but Annex B is empty. There are instances of TBD. A blank line should be placed above the first bullet point so that asciidoctor recognises the text as a bullet list.

The current example has this:

dcat:distribution [
 	a dcat:Distribution;
 	dcat:downloadURL <{}>;
 	dcat:mediaType [
 		a dct:MediaType;
 		dct:identifier "application/x-netcdf"
 	];

Should we add an example downloadURL? Or is this meant to signal something else? @adamml

conformance tests in Annex A are not up to date w.r.t. to the ch7 requirements

(only reqA* are done, and #78 has broken this numbering)

consistency is essential here

the linked topics of variable referencing, shape, refShape and coordinate variables are under review, and this issue is to capture that review and to collate feedback, which may be implemented in numerous targeted PRs

These are a challenging area for the specification, and further thought and consideration is required on intent and implementation.

http://docs.opengeospatial.org/DRAFTS/19-002.html#_netcdf_dimensions
http://docs.opengeospatial.org/DRAFTS/19-002.html#_requirement_class_e_variable_to_variable_referencing

Propose adding an example to show how attribute names are encoded...

6.5.8. Attribute values map to Requirement E-2 in Section 7.5.2. Should this be a new Requirements class to match the subsections in section 6? Took me a little while to find the matching requirement from the informative to the normative sections.

Similarly for 6.6. Dimensions

Refer to #39

could netcdf be a MIME type within IANA?

https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2015/msg00054.html

https://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg02847.html

http://docs.opengeospatial.org/DRAFTS/19-002.html#_media_types_for_any_data_encodings

http://www.iana.org/assignments/media-types/index.html

check Unidata for mime type

netcdf netcdf4

In #85 it was clarified that the BALD vocabulary would be used to interpret variable to variable references for resource-valued properties, in particular the bald:references relationship.

The vocab also specifies that bald:isPrefixedBy is a resource-valued property, and therefore its values should be interpreted as possible variable references. In conformance class A.2.1 (and in many places throughout the document), the bald__isPrefixedBy attribute is "prefix_list", which is the name of a variable, however the bald:isPrefixedBy property in the output graph is evaluated as a raw string rather than as a variable URI. This is a slightly obscure case since the prefix_list variable is excluded from the output graph.

As I see it there are a few options:

• Do not treat "prefix_list" as a variable reference since the variable is excluded from the output (current spec).
• Do treat "prefix_list" as a variable reference since it satisfies all of the criteria in 6.5.8.
• Exclude the bald:isPrefixedBy relationship from the output graph because it is acting as an annotation for the linked data converter and does not necessarily contain meaningful information about the binary array itself.

@marqh

There are several figures/images in Clause 6 that do not have captions explaining what the images are showing.

All figures should have captions.

The text in the specification should then refer to the figure.

Do we need a matching requirements class for the 6.5.3 Containment section?

Links to #39

As it says above - you can assign this one to me (@adamml)

Section 6.5.8 states that "A parsing process shall map attribute values to URIs using identified prefixes first", however the conformance class A.2.1 implies that the attribute value "rdfs__label" with the prefix "rdfs" being defined should be evaluated as "rdfs__label", ie. without prefix mapping.

Is this an inconsistency in the spec, and if so which version is correct?

Requirements class E-1 requires the range of an RDF property to be bald:Subject (or a sub-class of), however section 6.5.7 requires the range to be bald:Resource (or a sub-class of).
Which one is correct?

On a related note, the spec explicitly permits the use of direct sub-classes of the required class, but doesn't mention the need to support indirect sub-classes, eg. <classA> rdfs:subClassOf <classB> . <classB> rdfs:subClassOf bald:Subject ., which seems like an equally valid case (note - implementation should guard against circular hierarchies).