geopython / pygeometa Goto Github PK

Does it make sense to tag the current pygeometa version on GitHub as 0.1.0? It currently has no version number and this is annoying when trying to reference to a pygeometa version. Furthermore, pygeometa's output currently includes:

<gco:CharacterString>This metadata record was generated by pygeometa-0.1.0 (https://github.com/geopython/pygeometa)</gco:CharacterString>

It specifically states 0.1.0, so it would make sense that this value is in sync with release tags in the code repository. Makes sense? Thanks! -- Alex

Possible bug, at least a feature request for HNAP2.3 compliance.

In current HNAP XML output, for the EPSG we have:

    <gmd:role gco:nilReason="missing"/>

while ECDC expects something along the lines of:

<gmd:role xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:napec="http://www.ec.gc.ca/data_donnees/standards/schemas/napec">
  <gmd:CI_RoleCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_90" codeListValue="RI_413">originator; auteur</gmd:CI_RoleCode>
</gmd:role>

In this example, I chose 'Originator' as role, however, the ECDC proposes 15 possible values, such as Author, Distributor, Editor, Owner, Publisher, Resource provider, Rights holder, etc.

Let me know how I can help, thanks -- Alex

pygeometa currently setting and emitting a single topiccategory/gmd:topicCategory. Support multiple elements so users can identify more than one topic category.

As discussed with @tomkralidis ; pygeometa's MCF parameters should support a generic design pattern for language.

Example for the 'format' parameter:

• given format, format_en, or format_fr
• if !format_en, then single lang output, else multiple lang

Same applies for all parameters (eg description, description_en, description_fr)

In the output XML, for the HNAP profile, the French locale must be #fra instead of #locale-fr

Thanks for the fix @tomkralidis -- Alex

We get an error related to the missing MD_Format from the distribution info. I don't know if this is an HNAP-only issue or not. And I admit I know what is the desired value for the parameter (provided we already provide distribution info elsewhere!).

Here's the gmd:MD_Format tag missing in a code sample that has been provided to me, along with some context before+after:

  <gmd:distributionInfo>
    <gmd:MD_Distribution>
      <gmd:distributionFormat>
        <gmd:MD_Format>
          <gmd:name>
            <gco:CharacterString>dsa</gco:CharacterString>
          </gmd:name>
          <gmd:version>
            <gco:CharacterString>sfda</gco:CharacterString>
          </gmd:version>
        </gmd:MD_Format>
      </gmd:distributionFormat>
      <gmd:distributor>
        <gmd:MD_Distributor>
          <gmd:distributorContact>
            <gmd:CI_ResponsibleParty>

Let me know if this makes sense to you, and if so, if you have questions or comments. Thanks

Long term feature request, no rush to implement this on my end.

It would be useful if pygeometa was to support YAML as input format for metadata in addition to the MCF format.

There should be a mechanism in pygeometa that validates that the MCF version stated in an input MCF file is indeed supported by pygeometa.

Feature request to have pygeometa log messages to be exposed when running from the command line. The user should be able to specify the log level to output (e.g. info, warning, error, debug).

cc @alexandreleroux

By default pygeometa writes to stdout. Add --output parameter which writes to specified file. This allows for catching errors and exiting gracefully without shell redirecting.

So $date$ in pygeometa creates a datetime representation. ECDC would like us to change our date format for publication:

• "Publication date uses gco:DateTime, if the metadata is edited the value is lost as in ECCC metadata editor only is managing gco:Date. For FGP version this was updated, but was requested to not include in ECCC version. Please if you can create the metadata for now using gco:Date "

Should we update pygeometa’s magic $date$ as follows? :

• $date$ is a magic keyword for YYYY-MM-DD
• $datetime$$ is a magic keyword for YYYY-MM-DDTHH:MM:SSZ

HNAP2.3 requires the use of xlink:role to specify language for WMS Map Resources. Current pygeometa does not duplicate WMS Map Resources as required by HNAP2.3.

Here's an example of a desired HNAP2.3 output XML where you can see that gmd:transfer Options is duplicated for the WMS Map Resource to add French via xlink:role="urn:xml:lang:fra-CAN":

                       <gmd:transferOptions>
                                <gmd:MD_DigitalTransferOptions>
                                        <gmd:onLine xlink:role="urn:xml:lang:eng-CAN" xlink:title="tor1d5678012e1497">
                                                <gmd:CI_OnlineResource>
                                                        <gmd:linkage>
                                                                <gmd:URL>http://dd.weather.gc.ca/model_gem_global/25km/grib2/lat_lon/</gmd:URL>
                                                        </gmd:linkage>
                                                        <gmd:protocol>
                                                                <gco:CharacterString>WWW:LINK</gco:CharacterString>
                                                        </gmd:protocol>
                                                        <gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>MSC Datamart</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">Datamart du SMC</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:name>
                                                        <gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>The Meteorological Service of Canada (MSC) HTTP data server is a source of several raw meteorological data types and forecast data. This service is aimed at specialized users with good meteorological and IT knowledge, and is mainly meant to be accessed in an automatic manner via the internet (e.g. with scripts). The server&apos;s URL is: http://dd.weather.gc.ca/</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">Le Service météorologique du Canada (SMC) met à la disposition des usagers spécialisés une multitude de données météorologiques et prévisionnelles sur son serveur de données HTTP, à l&apos;adresse : http://dd.meteo.gc.ca/ Les usagers qui désirent accéder à ces données doivent avoir une bonne connaissance de la météorologie et de l&apos;informatique afin d&apos;en faire bon usage. Le mode d&apos;accès privilégié est via des routines d&apos;accès automatiques (scripts).</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:description>
                                                        <gmd:function>
                                                                <gmd:CI_OnLineFunctionCode codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode" codeListValue="download" codeSpace="ISOTC211/19115">download</gmd:CI_OnLineFunctionCode>
                                                        </gmd:function>
                                                </gmd:CI_OnlineResource>
                                        </gmd:onLine>
                                        <gmd:onLine xlink:role="urn:xml:lang:eng-CAN" xlink:title="tor1d5678012e1551">
                                                <gmd:CI_OnlineResource>
                                                        <gmd:linkage>
                                                                <gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&amp;service=WMS&amp;request=GetCapabilities&amp;layers=GDPS.ETA_TT&amp;legend_format=image/png&amp;feature_info_type=text/plain</gmd:URL>
                                                        </gmd:linkage>
                                                        <gmd:protocol>
                                                                <gco:CharacterString>OGC:WMS</gco:CharacterString>
                                                        </gmd:protocol>
                                                        <gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>MSC GeoMet</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">SMC GeoMet</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:name>
                                                        <gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>Web Service;WMS;eng</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">Service Web;WMS;eng</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:description>
                                                        <gmd:function>
                                                                <gmd:CI_OnLineFunctionCode codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode" codeListValue="download" codeSpace="ISOTC211/19115">download</gmd:CI_OnLineFunctionCode>
                                                        </gmd:function>
                                                </gmd:CI_OnlineResource>
                                        </gmd:onLine>
                                </gmd:MD_DigitalTransferOptions>
                        </gmd:transferOptions>
                        <gmd:transferOptions>
                                <gmd:MD_DigitalTransferOptions>
                                        <gmd:onLine xlink:role="urn:xml:lang:fra-CAN" xlink:title="tor1d5678012e1611">
                                                <gmd:CI_OnlineResource>
                                                        <gmd:linkage>
                                                                <gmd:URL>http://geo.weather.gc.ca/geomet/?lang=F&amp;service=WMS&amp;request=GetCapabilities&amp;layers=GDPS.ETA_TT&amp;legend_format=image/png&amp;feature_info_type=text/plain</gmd:URL>
                                                        </gmd:linkage>
                                                        <gmd:protocol>
                                                                <gco:CharacterString>OGC:WMS</gco:CharacterString>
                                                        </gmd:protocol>
                                                        <gmd:name xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>MSC GeoM</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">SMC GeoMet</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:name>
                                                        <gmd:description xsi:type="gmd:PT_FreeText_PropertyType">
                                                                <gco:CharacterString>Web Service;WMS;fra</gco:CharacterString>
                                                                <gmd:PT_FreeText>
                                                                        <gmd:textGroup>
                                                                                <gmd:LocalisedCharacterString locale="#fra">Service Web;WMS;fra</gmd:LocalisedCharacterString>
                                                                        </gmd:textGroup>
                                                                </gmd:PT_FreeText>
                                                        </gmd:description>
                                                </gmd:CI_OnlineResource>
                                        </gmd:onLine>
                                </gmd:MD_DigitalTransferOptions>
                        </gmd:transferOptions>

If you need additional info for supporting this, let me know! Thanks -- Alex

HNAP2.3 needs the WMS layer also specified in the gmd:linkage gmd:URL element of the output XML.

Current output XML:

<gmd:CI_OnlineResource>
 <gmd:linkage>
   <gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&amp;service=WMS&amp;request=GetCapabilities</gmd:URL>
 </gmd:linkage>

While HNAP2.3 needs:

  <gmd:CI_OnlineResource>
      <gmd:linkage>
            <gmd:URL>http://geo.weather.gc.ca/geomet/?lang=E&amp;service=WMS&amp;request=GetCapabilities&amp;layers=GDPS.DIAG_NW&amp;legend_format=image/png&amp;feature_info_type=text/plain</gmd:URL> 
  </gmd:linkage>

Let me know if you need additional details.

Our local variable for french in the yaml definition are "fr", but only "fra" is supported in ECDC. When generating the xml, some keywords are created with:

• <gmd:LocalisedCharacterString locale="#fr"
  instead of:
• <gmd:LocalisedCharacterString locale="#fra"

The result is that in ECDC, those keywords are not translated in french...

I suggest we modify a little bit https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2#L202-205 .

to:

{% else %}
  {% for kw1, kw2 in zip(keywords[0], keywords[1]) %}
    {% if record['metadata']['language_alternate'] == "fr" %}
      {% set language_alternate = "fra" %}
    {% endif %}
    {{ cs.get_freetext('keyword', language_alternate, [kw1, kw2]) }}
  {% endfor %}
{% endif %}

I think this is the easier solution.

Other solution:

• Modify another part of the script (more tricky)
• Change keywords_fr to keywords_fra in the yaml files
• Ask ECDC to support "fr" tag and "fra"

thoughts @tomkralidis @josegar74 ?

Feature request for the capability to specify a post-processing script to be ran after pygeometa is done generating the output xml. The post-processing script could be invoked with --postprocessing=script-name.py. The pygeometa arguments would be sent to the post-processing script, which could thus identified which input mcf file has been specified and which output xml file has been generated.

Usage example: this would allow pygeometa to automatically launch Python scripts to apply processing which is not within pygeometa's scope. This post-processing capability could be used in conjunction with the batch mode (feature request in #63).

In a future version, the --postprocessing argument could also write in the pygeometa log.

Thoughts on the pertinence of this feature request?

Feature request. (moved from the to-do list to the issue tracker)

Would be useful to have a validation mechanism for MCF files. Validating things such as: MCF syntax, missing non-optional values based on schema, duplicate base_mcf, etc

As discussed offline with @tomkralidis , we could eventually create a website for pygeometa (for docs, news via blog, etc). Could be hosted on http://geopython.github.io/pygeometa and leverage Jekyll.

More discussions required before proceeding.

As a metadata manager
I want to create "initial" or "stub" metadata records in ISO XML format using OGC Web Services (OWS) as the source for metadata
So that I can reference these "stub" metadata records in my OWS (eg WMS Metadata URL) or import them into a catalogue (eg GeoNetwork)

Use case:
An organization already has one or more OWS services. Data managers have already created services with minimum metadata. This organization wants to further its data governance maturity by adding a catalogue. Catalogue systems are able to harvest services and create MD (eg Geonetwork), however any edits made to the MD records are overwritten on each harvest. Creating stub records from the service will act as a starting point to import into such catalogues.

(issue moved from initial repos to GitHub) Feature request. Not a blocker. Would constitute a significantly useful new feature.

The existing mcf file inheritance feature via base_mcf= is useful but has a limitation that mcf files can have only a single 'parent' and a single 'child'. This is a tree structure. This limitation means that whole branches of the mcf file tree would need to be duplicated to accommodate, as an example, the combination of dissemination methods for similar datasets.

Instead of modifying the existing template inheritance capability (i.e. base_mcf=), Tom proposed to use the more flexible 'include' command to allow multiple mcf file inclusions by pygeometa. This ticket is the request for the implementation of such 'multiple mcf file inclusion' capability.

Unlike what's mentioned in the doc, pygeometa doesn't seem to support chain nesting.

It looks like it's only reading one level of MCF nesting.

For example, first I create a MCF for the radar model using a base_mcf for the contact information. Then I'm trying to create a specific radar layer MCF using the radar model MCF and it fails because pygeometa is not able to read the information from the contact file.

contact.yml:

Contact:
    blablabla

radar_model.yml:

base_mcf: ../contact.yml
Identification:
     titre:
     abstract:

radar_layer.yml:

base_mcf: ../radar_model.yml
Identification:
     titre: another title
     abstract: another abstract

Then I get an error because I have no contact key in my radar_layer.yml:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'contact'

If I copy manually the whole contact.yml file into my radar_layer.yml, I don't get an error...

copying @tomkralidis and @alexandreleroux for comments

This is a question, not a change request.

HNAP2.3 use an aggregation of 3 controlled fields for the description of online resources. It does not allow providing an actual description of the online resource in its description parameter. My questions:

1. Can we provide the description at another location in the output XML?
2. Is there a better way of specifying the HNAP-compliant description other than pushing it directly in the MCF file?

Here's an example of what HNAP2.3 wants as a description of an online resource: Web Service;WMS;eng

Thanks for your thoughts and comments -- Alex

Documentation specify that the CRS needs to be an 'EPSG code identifier'. Can pygeometa ingest anything else at the moment? Can it ingest OGC WKT directly? If not, should this be a feature request? Thanks -- Alex

Feature request. It would be useful to have keywords in the MCF files to be substituted when pygeometa is ran.

As an example, if the keyword $date$ would be substituted by the actual datestamp at time of execution that would allow MCF files to require less modifications before generating XML files with pygeometa, thus less chances of errors.

Examples of where $date$ and $year$ keywords could be used in an MCF file:

• datestamp=$date$ would become datestamp=2016-09-13 at running time
• rights_en=Copyright (c) $year$ Her Majesty the Queen in Right of Canada would become rights_en=Copyright (c) 2016 Her Majesty the Queen in Right of Canada

When trying to use a creation or publication date older than 1900, I get an error:

  File "/home/ops/afss/lor/ENV/ven-p2/bin/pygeometa", line 9, in <module>
    load_entry_point('pygeometa==0.4.dev0', 'console_scripts', 'pygeometa')()
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 377, in generate_metadata
    schema_local=schema_local)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 325, in render_template
    software_version=VERSION).encode('utf-8')
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ops/afss/lor/ENV/ven-p2/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/templates/iso19139-hnap/main.j2", line 149, in top-level template code
    {% set datestamp = date|normalize_datestring %}
  File "/home/ops/afss/lor/ENV/ven-p2/local/lib/python2.7/site-packages/pygeometa-0.4.dev0-py2.7.egg/pygeometa/core.py", line 122, in normalize_datestring
    datestring2 = datestring.strftime('%Y-%m-%dT%H:%M:%SZ')
ValueError: year=1871 is before 1900; the datetime strftime() methods require year >= 1900

Looks like the problem is with the datestring library.

HNAP2.3 has mandatory requirements for 'Corporate information'. There are 3 parameters: Branch, Directorate and Project. The first two are mandatory and 'Project' is optional.

The ECDC provides a dropdown list for selecting Branch and Directorate, along with a freetext box for Project (in both language). In the pygeometa spirit, I suggest specifying 'Corporate info' directly in the MCF file. Parameter names could be HNAP_Branch, HNAP_Directorate and HNAP_Project-en / HNAP_Project-fr.

Here's an example of an output XML which has the 'Corporate info' in it. This section comes right after /gmd:supplementalInformation

                        <napec:EC_CorporateInfo>
                                <napec:EC_Branch>
                                        <napec:EC_Branch_TypeCode codeListValue="meteorological_service_canada" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Branch"/>
                                </napec:EC_Branch>
                                <napec:EC_Directorate>
                                        <napec:EC_Directorate_TypeCode codeListValue="weather_environmental_operations" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Directorate"/>
                                </napec:EC_Directorate>
                                <napec:EC_Project xsi:type="gmd:PT_FreeText_PropertyType">
                                        <gco:CharacterString>Test project in English</gco:CharacterString>
                                        <gmd:PT_FreeText>
                                                <gmd:textGroup>
                                                        <gmd:LocalisedCharacterString locale="#fra">Projet test en français</gmd:LocalisedCharacterString>
                                                </gmd:textGroup>
                                        </gmd:PT_FreeText>
                                </napec:EC_Project>
                                <napec:GC_Security_Classification>
                                        <napec:GC_Security_Classification_TypeCode codeListValue="" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#GC_Security_Classification"/>
                                </napec:GC_Security_Classification>
                                <napec:EC_Program>
                                        <napec:EC_Program_TypeCode codeListValue="" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Program"/>
                                </napec:EC_Program>
                        </napec:EC_CorporateInfo>
                </napec:MD_DataIdentification>

Let me know if you need additional info. Thanks! -- Alex

Request for supporting 'inapplicable' for [identification]:language values.

In the case where a dataset is made of numerical values, it makes sense for the dataset's language to be set to 'none' or its ISO19115 equivalent. pygeometa needs to support this value.

cc @alexandreleroux

Currently pygeometa's API allows for only passing an MCF file by filepath:

from pygeometa import render_template
xml_string = render_template('/path/to/file.mcf', schema='iso19139')

Allow for passing of either plain old strings or ConfigParser objects:

from pygeometa import render_template
mcf_string = '...'  # some string
xml_string = render_template_string(mcf_string, schema='iso19139')

from pygeometa import render_template
mcf_cp = '...'  # some ConfigParser object
xml_string = render_template_string(mcf_cp, schema='iso19139')

I looked at https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2#L421-454 and saw that only if the CRS value is not equal to '4326' the BBOX value should be written on the same line. If the CRS is set to 4326, it should use the {else} tag where the max lat, lon and min lat lon should be written separately on different lines.

We have a few models which uses CRS 4326 and they all have the same behaviour.

Ex for GDPS (xml):

<gmd:polygon>
  <gml:Polygon gml:id="P001" srsDimension="2" srsName="4326">
    <gml:exterior>
      <gml:LinearRing>
        <gml:posList srsDimension="2" srsName="4326">-180  -90 -180  90  180  90  180  -90 -180  -90</gml:posList>
     </gml:LinearRing>
   </gml:exterior>
</gml:Polygon>

In the YAML:

spatial:
    datatype: grid
    geomtype: solid
    crs: 4326
    bbox: -180, -90, 180, 90

Copying @alexandreleroux and @tomkralidis for comments.

Feature request as discussed offline.
Ability to specify the output XML filename and location from a parameter when running pygeometa.

Example:
generate_metadata.py --mcf=sample.mcf --schema=iso19139-hnap --outputxml=sample.xml

One advantage of proceeding this way is being able to handle errors before the outputxml file is created.

Documentation will need to be updated accordingly (I can do that :-))

If pygeometa is to support the 'WMO Core Profile', users will be able to use pygeometa to fulfil WMO WIS requirements.

http://www.wmo.int/pages/prog/www/WIS/metadata_en.html

Missing xsd for the HNAP schema.

<gmd:MD_Metadata xsi:schemaLocation="http://www.ec.gc.ca/data_donnees/standards/schemas/n
apec schema.xsd">

Feature request. Currently, an empty XML output file is generated if pygeometa can't parse the whole MCF file because the MCF file contains syntax errors. Would be nice if the output XML file is written to file at the end of pygeometa's process, thus if pygeometa ends prematurely in a ParsingError, at least there won't be an empty file created.

This is different from feature request #5 which requires more code. Thanks -- Alex

Seems like running pygeometa self tests fail after a fresh installation.

Beginning of the error message:

/my-env/pygeometa/tests$ python run_tests.py
__main__.PygeometaTest.test_get_charstring: Test support of unilingual or multilingual value(s)
.__main__.PygeometaTest.test_get_supported_schemas: Test supported schemas
.__main__.PygeometaTest.test_mcf_model: test mcf model and types
.__main__.PygeometaTest.test_nested_mcf: test nested mcf support
.__main__.PygeometaTest.test_pretty_print: Test pretty-printing
.__main__.PygeometaTest.test_read_mcf: Test reading MCFs, strings or dict
.__main__.PygeometaTest.test_render_template: test template rendering
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
    msg = self.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
    return fmt.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 573, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/lib/python3.4/logging/__init__.py", line 523, in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
  File "/usr/lib/python3.4/traceback.py", line 169, in print_exception
    for line in _format_exception_iter(etype, value, tb, limit, chain):
  File "/usr/lib/python3.4/traceback.py", line 146, in _format_exception_iter
    for value, tb in values:
  File "/usr/lib/python3.4/traceback.py", line 125, in _iter_chain
    context = exc.__context__
AttributeError: 'NoneType' object has no attribute '__context__'

Copying @RousseauLambertLP . Thanks!

Request for the adjustment of pygeometa's HNAP XML output for dates. As of HNAP 2.3, both publication and creation dates are mandatory (with their respective 'Date Type').

In a pygeometa MCF file, in the [identification] section, if I provide:
date=1991-03-12T00:00:00Z
datetype=publication
date=1996-01-01T00:00:00Z
datetype=creation
the first date is ignored (probably overwritten by the second one).

A potential solution would be to, in MCF files, specifying dates with parameters such as:
creationdate=value
publicationdate=value
A creationdate parameter in an MCF would create both the date and datetype in the XML.

Better ideas? Thanks -- Alex

In the pygeometa MCFs, there's a value for the MCF version as stated in the documentation:

mcf:
    version: 1.0.0

Given we changed the MCF format from INI to YAML, does it make sense to specify mcf.version: 2.0.0 now? Thanks. Copying @RousseauLambertLP

cc @alexandreleroux

Our current base_mcf support requires the base MCF to be a fully qualified MCF. Update base_mcf support so that base MCF files are relative from where they are defined. This results in, by design, MCF snippets which are not required to be fully qualified MCF files. In addition, inheritence should continue to be supported (i.e. if value in base MCF and not in child MCF, then set; if in child and in base, override).

HNAP2.3 has mandatory requirements for 'Information classification'. There are 5 parameters:

• Information Category: mandatory
• Geography: mandatory
• Content: optional, can have multiple values
• Program: optional, can have multiple values
• GC Security Level: mandatory

ECDC provides dropdown lists of value values for these parameters. In the pygeometa spirit, I suggest specifying 'Information classification' directly in the MCF file. Parameter names could be HNAP_info-cat, HNAP_geography, HNAP_info-content, HNAP_info-program and HNAP_GCSecLevel

Example of output XML which has 'Information classification' specified in it. For Program and GC Security Level, the associated code goes in the same section EC_CorporateInfo mentioned in issue #19 :

                                <napec:GC_Security_Classification>
                                        <napec:GC_Security_Classification_TypeCode codeListValue="unclassified" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#GC_Security_Classification"/>
                                </napec:GC_Security_Classification>
                                <napec:EC_Program>
                                        <napec:EC_Program_TypeCode codeListValue="weather_observations_forecasts_and_warnings" codeList="http://www.ec.gc.ca/data_donnees/standards/schemas/napec#EC_Program"/>
                                </napec:EC_Program>
                        </napec:EC_CorporateInfo>
                </napec:MD_DataIdentification>

For info-cat, geography and content, they go in the gmd:descriptivekeywords section, such as:

                        <gmd:descriptiveKeywords>
                                <gmd:MD_Keywords>
                                        <gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
                                                <gco:CharacterString>Weather and Climate</gco:CharacterString>
                                                <gmd:PT_FreeText>
                                                        <gmd:textGroup>
                                                                <gmd:LocalisedCharacterString locale="#fra">Temps et climat</gmd:LocalisedCharacterString>
                                                        </gmd:textGroup>
                                                </gmd:PT_FreeText>
                                        </gmd:keyword>
                                        <gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
                                                <gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="theme"/>
                                        </gmd:type>
                                        <gmd:thesaurusName>
                                                <gmd:CI_Citation id="local.theme.EC_Information_Category">
                                                        <gmd:title>
                                                                <gco:CharacterString>local.theme.EC_Information_Category</gco:CharacterString>
                                                        </gmd:title>
                                                        <gmd:date>
                                                                <gmd:CI_Date>
                                                                        <gmd:date>
                                                                                <gco:Date>2012-05-25</gco:Date>
                                                                        </gmd:date>
                                                                        <gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
                                                                                <gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
                                                                        </gmd:dateType>
                                                                </gmd:CI_Date>
                                                        </gmd:date>
                                                        <gmd:citedResponsibleParty/>
                                                </gmd:CI_Citation>
                                        </gmd:thesaurusName>
                                </gmd:MD_Keywords>
                        </gmd:descriptiveKeywords>
                        <gmd:descriptiveKeywords>
                                <gmd:MD_Keywords>
                                        <gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
                                                <gco:CharacterString>International</gco:CharacterString>
                                                <gmd:PT_FreeText>
                                                        <gmd:textGroup>
                                                                <gmd:LocalisedCharacterString locale="#fra">Internationale</gmd:LocalisedCharacterString>
                                                        </gmd:textGroup>
                                                </gmd:PT_FreeText>
                                        </gmd:keyword>
                                        <gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
                                                <gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="place"/>
                                        </gmd:type>
                                        <gmd:thesaurusName>
                                                <gmd:CI_Citation id="local.place.EC_Geographic_Scope">
                                                        <gmd:title>
                                                                <gco:CharacterString>local.place.EC_Geographic_Scope</gco:CharacterString>
                                                        </gmd:title>
                                                        <gmd:date>
                                                                <gmd:CI_Date>
                                                                        <gmd:date>
                                                                                <gco:Date>2012-05-25</gco:Date>
                                                                        </gmd:date>
                                                                        <gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
                                                                                <gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
                                                                        </gmd:dateType>
                                                                </gmd:CI_Date>
                                                        </gmd:date>
                                                        <gmd:citedResponsibleParty/>
                                                </gmd:CI_Citation>
                                        </gmd:thesaurusName>
                                </gmd:MD_Keywords>
                        </gmd:descriptiveKeywords>
                        <gmd:descriptiveKeywords>
                                <gmd:MD_Keywords>
                                        <gmd:keyword xmlns:srv="http://www.isotc211.org/2005/srv" xsi:type="gmd:PT_FreeText_PropertyType">
                                                <gco:CharacterString>Environmental Model</gco:CharacterString>
                                                <gmd:PT_FreeText>
                                                        <gmd:textGroup>
                                                                <gmd:LocalisedCharacterString locale="#fra">Modèle environnemental</gmd:LocalisedCharacterString>
                                                        </gmd:textGroup>
                                                </gmd:PT_FreeText>
                                        </gmd:keyword>
                                        <gmd:type xmlns:srv="http://www.isotc211.org/2005/srv">
                                                <gmd:MD_KeywordTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_101" codeListValue="theme"/>
                                        </gmd:type>
                                        <gmd:thesaurusName>
                                                <gmd:CI_Citation id="local.theme.EC_Content_Scope">
                                                        <gmd:title>
                                                                <gco:CharacterString>local.theme.EC_Content_Scope</gco:CharacterString>
                                                        </gmd:title>
                                                        <gmd:date>
                                                                <gmd:CI_Date>
                                                                        <gmd:date>
                                                                                <gco:Date>2012-05-25</gco:Date>
                                                                        </gmd:date>
                                                                        <gmd:dateType xmlns:srv="http://www.isotc211.org/2005/srv">
                                                                                <gmd:CI_DateTypeCode codeList="http://nap.geogratis.gc.ca/metadata/register/napMetadataRegister.xml#IC_87" codeListValue="RI_367">publication; publication</gmd:CI_DateTypeCode>
                                                                        </gmd:dateType>
                                                                </gmd:CI_Date>
                                                        </gmd:date>
                                                        <gmd:citedResponsibleParty/>
                                                </gmd:CI_Citation>
                                        </gmd:thesaurusName>
                                </gmd:MD_Keywords>
                        </gmd:descriptiveKeywords>

Let me know if you need additional info. Thanks! -- Alex

Feature request for the capability to launch pygeometa and process multiple MCF files located in a user-specified directory with a single command.

This 'batch mode' could be invoked directly from pygeometa with the --batch argument. If --batch is specified, then pygeometa is ran for every mcf files in the directory specified with --mfc= and all the outputs files are named the same as input files but end with .xml and are saved in the directory specified by --output. Other pygeometa arguments such as --schema= are applied to all MCF files processed in batch.

An initial version of this batch mode can simply loop on all mcf files and generate the corresponding xml files. The batch mode should skip base_mcf mcf files, which can't be processed on their own.

Future versions could:

• Recursively look in the --mfc= folder to process .mcf files located in subfolders as well and output the same directory structure to --output=
• Provide a log with warnings, errors and outputs. The log location and named could be specified by the --log= argument
• The batch mode could also look in the output directly if a corresponding xml exists and skip running if the input mcf file has not change since last time the batch mode was ran. The last time the batch mode was ran could be specified in the log file found at the --log= location

Thoughts / comments?

When the example 'language_alternate' directive is removed from an MCF (i.e.: creating a unilingual MCF), or if the directive is set to an empty value (e.g.: "language_alternate="), iso19139 Jinja2 template appears to raise UndefinedError

Example:

>>> from pygeometa import render_template
>>> xml = render_template('unilingual.mcf', 'iso19139')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bjv/git/pygeometa/pygeometa/__init__.py", line 202, in render_template
    software_version=__version__).encode('utf-8')
  File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/bjv/git/pygeometa/my-env/lib/python3.4/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/home/bjv/git/pygeometa/pygeometa/templates/iso19139/main.j2", line 163, in top-level template code
    {% for kw1, kw2 in zip(keywords[0].split(','), keywords[1].split(',')) %}
jinja2.exceptions.UndefinedError: 'None' has no attribute 'split'

(sample MCF attached)

When pygeometa generates the xml metadata files, it creates a <gmd:distribution format> for each value in distribution even if the distribution format already exists.

Example in MSC case:
Under distribution in the YAML file we have 3 distribution types:

• Datamart
• GeoMet
• AMQP

For each of these distribution type we have a format_en and format_fr tag. pygeometa creates a <gmd:distribution format> tag in the xml for each of these format_xx even if it already exist in the xml. This creates many <gmd:distribution format> with the same value. Note that each <gmd:distribution format> has an English and French value.

I think we need to modify the code so that pygeometa checks if the format already exists in the xml, and if it does, it doesn't create another <gmd:distribution format> with the same value.

Copying @alexandreleroux and @tomkralidis for comments.

Feature request for the support of the HNAP schema, specifically the implementation of language in HNAP which doesn't use gmd:language but rather uses xlink:role for specifying language.

Possible implementation (not from me) proposes supporting tokens such as [distribution:wms:eng-CAN] and [distribution:wms:fra-CAN], thus if there's three tokens (1:2:3), pygeometa can output the last token as xlink:role

Since latest change 99d6b7b , I get an error parsing options.

Trace:

(my-env)/PATH/my-env/pygeometa$ pygeometa generate-metadata --mcf /local/home/data/afssalx/content/discovery-metadata/mcf/climate/msc_climate-normals.yml --schema=schema=iso19139-hnap --output msc_climate-normals.xml
Usage: pygeometa generate-metadata [OPTIONS]

Error: Invalid value for "--schema": invalid choice: schema=iso19139-hnap. (choose from iso19139, iso19139-hnap, wmo-cmp, wmo-wigos)

With another schema such as iso19139, I get the same error.

Hello,

Not sure if this issue is specifically with pygeometa or if there is another issue somewhere in my workflow. I'm working to create metadata files for all of my companies geospatial data. As a test, I created a test yaml file based on the sample one from this repository, and updated it with information pertinent for my use. Pygeometa works fine and creates an xml file if I have everything hardcoded into that yaml file.

However, I had hoped to be able to update the yaml on the fly as I go through the data files so that I can update it with the extent information, file type (vector, grid, or something else) and other information based on the file. When I do this, I can update the yaml file as expected and save it to be able to be used by pygeometa. If I do this though, I recieve the error as mentioned in the title above. So once again, if I manually update my yaml file and save it and use pygeometa it works but if I try to do it on the fly it does not. Any help would be useful if anyone can see something in my code that would be cause this issue to arise. Thanks in advance.

System: Win7
Env: JupyterLab notebook for testing
Python Version: 3.6.6

import yaml
from osgeo import gdal, ogr, osr
import datetime
import os
from pygeometa.core import render_template

path = "path_to_my_geospatial_data"

# Then I grab a bunch of the information about the geo datafile to input into my *.yml file

# Next open the basic file
with open("test.yml") as f:
     data = yaml.load(f)

# change values in data to account for the information I got from the geo datafile

# Now save to a new temporary .yml file
with open("test_new.yml", "w") as ff:
    yaml.dump(data, ff)
    
ff.close()

# Finish with giving the new .yml file to pygeometa to create the metadata xml
xml_string = render_template("test_new.yml", schema='iso19139')
with open('test_new.xml', 'w') as x:
    x.write(xml_string)

I tried to import some metadata xml files to ECDC but I get some errors. One of them is because we use a variable #locale-fra.

Jose Garcia working on ECDC told me that we should use #fra instead.

@tomkralidis @alexandreleroux should I update https://github.com/geopython/pygeometa/blob/master/pygeometa/templates/iso19139-hnap/main.j2 to output #fra?

In a pygeometa MCF file, we must specify the Identification 'url' parameter. We cannot currently specify different URLs for different languages. At least for HNAP, pygeometa stores this information in the output XML as gmd:supplementalInformation. However, for HNAP, this information needs to be provided in the output XML in both language.

Currently, pygeometa generates:

      <gmd:supplementalInformation>
        <gco:CharacterString>http://weather.gc.ca/grib/grib2_glb_25km_e.html</gco:CharacterString>
      </gmd:supplementalInformation>

and the following is expected:

                        <gmd:supplementalInformation xsi:type="gmd:PT_FreeText_PropertyType">
                                <gco:CharacterString>http://weather.gc.ca/grib/grib2_glb_25km_e.html</gco:CharacterString>
                                <gmd:PT_FreeText>
                                        <gmd:textGroup>
                                                <gmd:LocalisedCharacterString locale="#fra">http://weather.gc.ca/grib/grib2_glb_25km_f.html</gmd:LocalisedCharacterString>
                                        </gmd:textGroup>
                                </gmd:PT_FreeText>
                        </gmd:supplementalInformation>

One possibility would be to support url_en and url_fr in MCF files. If no url_en or url_fr exists, then url is used. This would allow specifying language-specific URLs in pygeometa MCF files.

Let me know if you need any additional info. Thanks -- Alex

Attempting to install pygeometa from source using pip fails, seemingly because setup.py imports 'pygeometa'.. the package which is to be built.

(test1)bjv@host:~$ pip install git+https://github.com/geopython/pygeometa.git#egg=pygeometa
You are using pip version 7.0.3, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pygeometa from git+https://github.com/geopython/pygeometa.git#egg=pygeometa
  Cloning https://github.com/geopython/pygeometa.git to /tmp/pip-build-sj02i_4c/pygeometa
p11-kit: invalid config filename, will be ignored in the future: /etc/pkcs11/modules/gnome-keyring-module
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-build-sj02i_4c/pygeometa/setup.py", line 53, in <module>
        import pygeometa
      File "/tmp/pip-build-sj02i_4c/pygeometa/pygeometa/__init__.py", line 52, in <module>
        from jinja2 import Environment, FileSystemLoader
    ImportError: No module named 'jinja2'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sj02i_4c/pygeometa

Documentation currently indicate:

• generate_metadata.py --mcf=path/to/file.mcf --schema_local=/path/to/my-schema --output some_file.xml
  which works, but for homogeneity, I'd suggest adding the = to --output and thus changing to
• generate_metadata.py --mcf=path/to/file.mcf --schema_local=/path/to/my-schema --output=some_file.xml
  which also works without any code change, only documentation would be slightly updated.

Agreed? Thx

Potential bug, or at least a parameter missing from the output HNAP XML. Under gmd:MD_Distribution, there should be a gmd:distributionFormat. Currently, distributionformat is nowhere in pygeometa's output XML.

Example I've been provided:

  <gmd:distributionInfo>
    <gmd:MD_Distribution>
      <gmd:distributionFormat>
        <gmd:MD_Format>
          <gmd:name>
            <gco:CharacterString>dsa</gco:CharacterString>
          </gmd:name>
          <gmd:version>
            <gco:CharacterString>sfda</gco:CharacterString>
          </gmd:version>
        </gmd:MD_Format>
      </gmd:distributionFormat>
...

Let me know if you need more details, thanks! -- Alex

There are most probably errors in the MCF_Reference.md file regarding which properties are indeed mandatory when it comes localization.

Example, 'organization' is flagged as mandatory, but 'organization_en' and 'organization_fr' as well. Same for 'url', 'url_en' and 'url_fr'. I suggest that properties flagged as mandatory should be the ones indicated as such for the ISO19115:2003 schema.

Makes sense @tomkralidis ?

Feature request for pygeometa to be able to import ISO metadata records from external sources and create corresponding pygeometa MCF records in the yaml format.

Example use cases:

• help users create a first version of their existing metadata records in the pygeometa yaml MCF format, for those wanting to adopt pygeometa in their workflow
• allow users relying on metadata records in pygeometa's MCF format to automatically create MCFs from external sources