TG2-VALIDATION_SCIENTIFICNAME_FOUND about bdq HOT 22 OPEN

I have written a blog post about unmatched names GBIF receives from publishers that might be somewhat relevant to this discussion:
https://data-blog.gbif.org/post/2022-03-24-reasons-why-names-don-t-match-to-the-gbif-backbone/

This work relies heavily on the GBIF name parser, which while not perfect, does a fairly good job of telling whether a name is a valid dwc:scientificName.
https://www.gbif.org/tools/name-parser
https://api.gbif.org/v1/parser/name?name=Steroma%20superba%20nr.%20Butler,%201868

from bdq.

Changed Notes from

The purpose of this test is to detect errors in the scientific name but is dependent on the abilities of the parsing of the bdq:sourceAuthority. For research users of biodiversity data doing quality assurance, #70 handles their needs, but for curators of data sets doing quality control, #46 provides a specific subset of targeted data cleaning, making this a valuable test to include for the quality control case.

To

The purpose of this test is to detect errors in the scientific name but is dependent on the abilities of the parsing of the bdq:sourceAuthority. For research users of biodiversity data doing quality assurance, VALIDATION_TAXON_UNAMBIGUOUS (4c09f127-737b-4686-82a0-7c8e30841590) handles their needs, but for curators of data sets doing quality control, this test provides a specific subset of targeted data cleaning, making it a valuable test to include for the quality control case.

from bdq.

Specification leaves case of a mononomial scientific name as undefined. Specification doesn't explain what to do with information elements other than dwc:scientificName. The may be the VALIDATION_SCUENTIFICNAME_NOTFOUND that @Tasilee is looking for.

from bdq.

So, should this test be "VALIDATION_TAXON_NOTFOUND" to match #122, #77, #83, #22, #28, with Information elements dwc:scientificName and dwc:genus?

from bdq.

@Tasilee I'm thinking this test takes just dwc:scientificName and is VALIDATION_SCIENTIFICNAME_NOTFOUND. TAXON_NOTFOUND would imply more terms, perhaps dwc:taxonID, but perhaps dwc:genus and higher. We do need to consider if #101 needs additional companion tests that examine the TAXON, as phrased in the specification, this one currently isn't.

from bdq.

I agree @chicoreus - I would be much happier with VALIDATION_SCIENTIFICNAME_NOTFOUND to match the other VALIDATIONS.

Regards #101 (and #46) - I am now wondering about the set of TAXON type tests we should conform to. For example, #123 and #70

from bdq.

See above where it was once called "TG2-VALIDATION_SCIENTIFICNAME_NOTSTANDARD" but was changed duiring our Gainseville discussions. Same with #45. Do you have some notes, @Tasilee on the Gainesville discussion?

from bdq.

@ArthurChapman - I checked my Gainesville notes and I had nothing specific to this one or #45. Maybe related tests have changed leaving this orphaned?

from bdq.

I have changed the wording of the notes to make a little clearer

FROM: This test is not intended to detect errors of a taxonomic nature. The intent of this test is not to detect errors or inconsistencies in the format of the Authorship. For the purpose of this amendment, if the genus in the dwc:genus field does not match the genus of the polynomial, the genus of the polynomial takes precedence for standardization.

TO: The purpose of this test is to detect errors in spelling and typography only. It is not intended to detect errors of a taxonomic nature or to detect errors or inconsistencies in the format of the Authorship. For the purpose of this amendment, if the genus in the dwc:genus field does not match the genus of the polynomial, the genus of the polynomial takes precedence for standardization.

from bdq.

from bdq.

I have also deleted the last paragraph - as that refers to the AMENDMENT (#45)

from bdq.

With the quorum of agreement (email response July 15, 2020), I have changed this test to match the higher taxonomic equivalents, e.g., #122 with the recognition that the utility of the test is heavily dependent on the abilities of the bdq:sourceAuthority.

from bdq.

This one again overlaps with #82 If there was workflow then if #82 was run before #46, then there would be no point in running #46 (cf. comment under #101 which is similar)

from bdq.

@ArthurChapman no overlap, #82 tests for non-emptyness, that is on an axis of completeness, #46 for value being found in an authority, thus on an axis of completeness. The framework treats the the response status (internal prerequisites not met) with the response result (not compliant) as orthogonal concerns, and test order is not specified (to allow for implementations that run tests in paralell, or implementations that run tests (as in @tucotuco 's sql implementation in the paper) on distinct values.

from bdq.

@chicoreus - I have no problem with running all. Just saying that if you run #82 and it fails - i.e. the Scientific Name is EMPTY, then there would be no use running #42 (or #101) as they would both just return INTERNAL_PREREQUISITES_NOT_MET. But if all tests are standalone - and not part of a workflow, then it doesn't really matter.

from bdq.

Changed "NOT COMPLIANT" to "NOT_COMPLIANT" in Example

from bdq.

Feedback from deployment of the implementation sci_name_qc implementation at the MCZ: Most services, including GBIF's API, support a lookup on the scientific name without authorship, but not on the literal dwc:scientificName including the authorship. This means that implementations which aren't working off a local data store must implement a parser to extract the part of the name to look up in the authority, then compare the returned name and authorship with the provided name and authorship. When parsing fails (as happens readily with historical name orthographic variants where specific epithets based on people's names started with a capital letter, e.g. Ophiocoma Alexandri Lyman, 1860 or Pentagonaster Alexandri Perrier, 1881, both of which fail in MCZbase through having the validation lookup the generic name rather than the binomial), the lookup fails.

I would recommend that we address this by including dwc:scientificNameAuthorship in the information elements, and changing the specification from

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:scientificName is EMPTY; COMPLIANT if there is a match of the contents of dwc:scientificName with the bdq:sourceAuthority; otherwise NOT_COMPLIANT

to:

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if dwc:scientificName is EMPTY; COMPLIANT if there is a match of the contents of dwc:scientificName with the bdq:sourceAuthority (using dwc:scientificNameAuthorship to assist with separating which part of the dwc:scientificName to query on a source authority that does not support lookup on scientific name with authorship); otherwise NOT_COMPLIANT

Or, leaving the specification unchanged, and changing the notes from:

The purpose of this test is to detect errors in the scientific name but is dependent on the abilities of the parsing of the bdq:sourceAuthority.

To:

The purpose of this test is to detect errors in the dwc:scientificName. dwc:scientificNameAuthorship is included as an information element to allow implementors to potentially identify which portion of the dwc:scientificName is the authorship string without having to parse the dwc:scientificName into component parts. Many possible bdq:sourceAuthority services support the lookup of a scientific name by the scientific name without authorship, and return no results when given the full dwc:scientificName with authorship, so implementations are likely to have to (1) extract the portion of the name to look up from the dwc:scientficName, (2) perform the lookup, and (3) determine if a full name with authorship in lookup results is an exact match on the provided dwc:scientificName. Step (1) is simplified if dwc:scientificNameAuthorship contains the scientific name authorship portion of dwc:scientificName, with parsing being a fallback when it is not available. An empty dwc:scientificNameAuthorship or an inconsistency between dwc:scientificName and dwc:scientificNameAuthorship should result in dwc:scientificNameAuthorship being ignored.

from bdq.

@jhnwllr thanks for the post. I think many will find it helpful as it illuminates and provides transparency about the decision making process going on. I see this related post that really helps those submitting data to engage them to supply names -- that is, in the cases where their names are not matched because they fill gaps. In other words, great to see ways to get the system to work bidirectionally. We need to think of ways to spread the knowledge / use of this information you've collated so nicely ...

from bdq.

In our zoom today, we (@tucotuco, @chicoreus and @ArthurChapman) concluded that as #70 is providing a testing of dwc:scientificName (in part 2 of the Expected Response), we can set this test to NON CORE. Yea, one less test!

There is a recognition that the 'hard part' of dwc:scientificName is the likely/potential inclusion of dwc:scientificNameAuthorship as per Darwin Core Standard (https://dwc.tdwg.org/list/#dwc_scientificName) and the way some source authorities handle dwc:scientificName. For example, GBIF will not return a match if dwc:scientificName contains an authorship.

A principle (?) was also suggested by @tucotuco in the Zoom meeting of August 17: We should avoid parsing 'input strings'. Output strings yes.

from bdq.

We should consider adding this back to core tests. (1) For research users of biodiversity data doing quality assurance, #70 handles their needs, but for curators of data sets doing quality control, #46 provides a specific subset of targeted data cleaning, making this a valuable test to include for the quality control case.
(2) This can be handled without parsing the dwc:scientificName even for services that require just the name without authorship by running a two phase process (as in early Filtered Push implementations), querying a scientific name string authority followed by a query on a nomenclator or taxon authority.

from bdq.

Changed Parameter(s) to "bdq:sourceAuthority" as per discussions 12th June 2023 and restructured Source authority entries

from bdq.

Amended Source Authority values to align with @chicoreus syntax

From

bdq:sourceAuthority default = "GBIF Backbone Taxonomy" [https://doi.org/10.15468/39omei] |
| | API endpoint [https://api.gbif.org/v1/species?datasetKey=d7dddbf4-2cf0-4f39-9b2a-bb099caae36c&name=]

to

bdq:sourceAuthority default = "GBIF Backbone Taxonomy" {[https://doi.org/10.15468/39omei]} {API endpoint [https://api.gbif.org/v1/species?datasetKey=d7dddbf4-2cf0-4f39-9b2a-bb099caae36c&name=]}

from bdq.