turbomam / examples-first-cookiecutter Goto Github PK

based on discussions with Kai B

and also NMDC Studies

Emphasize the properties of a SlotDefinition that effect validation?

PyCharm puts its configuration in .idea/

this is might be separate from adding it to the repo created by the cookiecutter

Also clarify the Collection bias of this repo.

It's totally fine to make LinkML data that aren't wrapped in a Collection

@pkalita-lbl says this would take a lot of effort. Probably not worth it.

I really messed up the formatting of the templated schema file

see

• #17

• add examples in Collections later
• confirm invalidity and minimality
• add exhaustive later
  • add exhaustiveness checker

make test from ./Makefile currently does the following, at a high level:

• make test-schema
  • make gen-project
• make test-python
  • python -m unittest discover
    • applies tests/test_valid_collections.py to selected files in src/data/examples/valid
• make test-examples
  • runs linkml-run-examples
  • input: src/data/examples
  • output: examples/output

make check-jsonschema-example from ./project.Makefile provides a minimal example of checking one hard-coded data file with check-jsonschema

Change that to iterate over all valid and invalid files. Wrap with ! (not -!) for the invalid files.

Clarify that check-jsonschema exhaustively reports schema violations but is not part of the LinkML ecosystem. Its functionality will probably be replaced by a (potentially even more useful) LinkML tool at some point.

and take them form the valid examples dir

see also

• #2

Problem: moving files is likely to loose GH history

linkml-run-examples ... --output-formats ttl

adds more constraints that some users may not want to be bothered by

See https://github.com/turbomam/mixs-subset-examples-first

• #70
• #71
• combine the two scripts into one

probably habitual use of auto-format on a file that isn't really pure YAML

and not multi-valued except for enums

see also

• #37
• #47
• #49

@pkalita-lbl provided a solution for this in the NMDC submissions schema repo

Eventually to be replaced by a LinkML tool

This benefits from the fact that the Collection is the tree root.

And from the fact that we are making Collection examples.

I really mean detecting the tree root class at runtime, not at the time of repo creation

git diff tests/test_valid_collections.py 
diff --git a/tests/test_valid_collections.py b/tests/test_valid_collections.py
index 6f2a618..d929828 100644
--- a/tests/test_valid_collections.py
+++ b/tests/test_valid_collections.py
@@ -4,14 +4,14 @@ import glob
 import unittest
 
 from linkml_runtime.loaders import yaml_loader
-from identifiers_sandbox.datamodel.identifiers_sandbox import ThingCollection
+from identifiers_sandbox.datamodel.identifiers_sandbox import NamedThingCollection
 
 ROOT = os.path.join(os.path.dirname(__file__), '..')
 DATA_DIR = os.path.join(ROOT, "src", "data", "examples", "valid")
 
 EXAMPLE_FILES = glob.glob(os.path.join(DATA_DIR, '*.yaml'))
 
-MAIN_SCHEMA_CLASS_NAME = ThingCollection.class_name
+MAIN_SCHEMA_CLASS_NAME = NamedThingCollection.class_name
 
 ACCEPTABLE_PREFIX = DATA_DIR + "/" + MAIN_SCHEMA_CLASS_NAME
 
@@ -23,7 +23,7 @@ class TestData(unittest.TestCase):
         """Date test."""
         for path in EXAMPLE_FILES:
             if path.startswith(ACCEPTABLE_PREFIX):
-                obj = yaml_loader.load(path, target_class=ThingCollection)
+                obj = yaml_loader.load(path, target_class=NamedThingCollection)
                 assert obj
             else:
                 print()

make setup

cp src/data/examples/* examples
cp: -r not specified; omitting directory 'src/data/examples/invalid'
cp: -r not specified; omitting directory 'src/data/examples/valid'
make: *** [Makefile:100: gen-examples] Error 1

last line is really
make: *** [Makefile:100: gen-examples] Error 1

• add issue to DH repo
• try normalizer part of reference validator in linkml normalizer

• #59
• #48
  • range is already range: uriorcurie!
• #49

• #15
• #13
• add exhaustive later
  • add exhaustiveness checker

closed by #14

based on discussions with Kai B

and also NMDC Studies

After generation of the repo, in src/data/examples/invalid/PersonCollection-dupe-ids.yaml

with

entries:
  - id: example:Person001
    name: foo bar
  - id: example:Person001
    name: foo bar

make test

ValueError: Counter example src/data/examples/invalid/PersonCollection-dupe-ids.yaml succeeded validation
make: *** [Makefile:138: examples/output] Error 1

• src/data/examples/unexpected-fail
• src/data/examples/unexpected-pass

with .gitignore files

see

• cidgoh/DataHarmonizer#387

• #65
• #80

see also

• #59
• #62

actually, not completely as they are:

• #82
• #91

One part of make test:

poetry run python -m unittest discover

Can't instantiate the following:

entries:
  - id: example:Thing001
  - id: example:Thing002

/Users/MAM/Documents/gitrepos/linkml-projects/identifiers-sandbox/tests/../src/data/examples/valid/NamedThingCollection-single-slot.yaml
E
======================================================================
ERROR: test_data (tests.test_valid_collections.TestData)
Date test.

Traceback (most recent call last):
File "/Users/MAM/Documents/gitrepos/linkml-projects/identifiers-sandbox/tests/test_valid_collections.py", line 27, in test_data
obj = yaml_loader.load(path, target_class=NamedThingCollection)
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 85, in load
results = self.load_any(*args, **kwargs)
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/loaders/yaml_loader.py", line 32, in load_any
return self.load_source(source, loader, target_class, accept_header="text/yaml, application/yaml;q=0.9",
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 66, in load_source
return target_class(**data_as_dict)
File "", line 4, in init
File "/Users/MAM/Documents/gitrepos/linkml-projects/identifiers-sandbox/src/identifiers_sandbox/datamodel/identifiers_sandbox.py", line 96, in post_init
self._normalize_inlined_as_list(slot_name="entries", slot_type=NamedThing, key_name="id", keyed=True)
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/utils/yamlutils.py", line 97, in _normalize_inlined_as_list
self._normalize_inlined(slot_name, slot_type, key_name, keyed, True)
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/utils/yamlutils.py", line 173, in _normalize_inlined
order_up(list_entry[lek], slot_type(list_entry))
File "", line 6, in init
File "/Users/MAM/Documents/gitrepos/linkml-projects/identifiers-sandbox/src/identifiers_sandbox/datamodel/identifiers_sandbox.py", line 70, in post_init
self.id = NamedThingId(self.id)
File "/Users/MAM/Library/Caches/pypoetry/virtualenvs/identifiers-sandbox-c7uJR0Fx-py3.10/lib/python3.10/site-packages/linkml_runtime/utils/metamodelcore.py", line 100, in init
raise ValueError(f"{v} is not a valid URI or CURIE")
ValueError: JsonObj(id='example:Thing001') is not a valid URI or CURIE

• install npm
  • https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/#installing-nodejs-and-npm-using-nvm
  • curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
  • nvm --version
• make create-data-harmonizer
  • ? What would you like your new project to be called?
    • dataharmonizer
  • Which [classes] should be used as DataHarmonizer templates?
    • only include Person for now
    • we should add more collections to illustrate DH functionality
      • switching template
      • template parameter on URLs
      • multiview