• use the Wikidata musicians and bands sample;
• implement the following baseline matchers:
• #6;
• #7;
• #8;
• #11;
• compute the following ratio: # unique matches / (sample length - #already linked Wikidata items);
• report the result here: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline#Big_fishes

Currently, only 37 out of 256k albums in Wikidata have a MusicBrainz identifier:

SELECT (COUNT(DISTINCT ?item) AS ?count) WHERE { ?item wdt:P31/wdt:P279* wd:Q482994 ; wdt:P434 ?identifier . }

Investigate how we could leverage the album data in MusicBrainz to both populate album identifiers and performer statements, like:
Animals, performer, Pink Floyd

The whole project should work in a virtual environment like Docker

Currently, it is loaded from a file.
See check_existence checks.py

Setup a logging mechanism.

• create a bot request, after #66;
• implement the bot in soweego/ingestor;
• import accurate links as per #62.

• BIBSYS; Dropped: see #73
• Discogs;
• IMDb;
• MusicBrainz;
• Twitter.

We cannot be sure that the url building from Wikidata is the same available in the target databases.

We could split for '/' and '.' the urls to get the tokens. Excluding grammar words like "https" and so on we can try to understand similarity between urls.

E.g. (trivial)
https://twitter.com/BarackObama -> "https" "twitter" "com" "BarackObama"
After removing grammar words we have "twitter" "BarackObama" which are actually the core informations we need to match against another form of twitter url like http://www.twitter.com/BarackObama -> "twitter" "BarackObama" (after cleaning)

Import them into 2 different tables in MariaDB

- [ ] Port this Java implementation into Python: https://github.com/fbk/utils/blob/master/utils-core/src/main/java/eu/fbk/utils/core/strings/JaroWinklerDistance.java
based on Apache Commons lang: https://commons.apache.org/proper/commons-text/javadocs/api-release/index.html?org/apache/commons/text/similarity/JaroWinklerDistance.html
original source code should be contained here: http://it.apache.contactlab.it//commons/lang/source/commons-lang3-3.7-src.tar.gz;

• use jellyfish;
• for each full name of a target identifier, compute the Jaro-Winkler edit distance against all Wikidata item labels.

Currently musicbrainz calculation are done with an internal identifier and not with the one reachable from the outside

• code pointer in User:Hjfocs
• test edits:
• 35 Discogs
• 35 MusicBrainz
• 35 Twitter

It should perform the following actions:

• download the dump at http://data.bibsys.no/autreg/linked_data_auth_reg.nt;
• slice the subset about people (if applicable), i.e., all the subjects that have rdf:type foaf:Person. For instance:

<http://data.bibsys.no/data/notrbib/authorityentry/x90054225> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .

• import the dataset into the s51434__mixnmatch_large_catalogs_p database on Toolforge.

It should perform the following actions:

• download the dump at http://datos.bne.es/datadumps/autoridades.nt.bz2;
• slice the subset about people;
• import the dataset into the s51434__mixnmatch_large_catalogs_p database on Toolforge.

1. Remove honorific words, e.g., Sir, Mlle, Jr., Sr., PhD, MD M.D. (prefixes), de, de la, of, von (infixes). See mix'n'match regexps below:

/[, ]+(Jr\.{0,1}|Sr\.{0,1}|PhD\.{0,1}|MD|M\.D\.)$
/^(Sir|Baron|Baronesse{0,1}|Graf|Gräfin)\s+/
/\b(Mmle|pseud\.|diverses)\b/
/ Bt$/' , ' Baronet'
'/^(.+)[ ,]+[sj]r\.{0,1}$/i'
/^(.+)[ ,]+I+\.{0,1}$/i

2. remove name initials with or without dot, e.g. M.;
3. remove commas, dashes and quotes;
4. normalization: convert to ASCII via a diacritics map, e.g., { 'à': 'a' };
5. lowercase;
6. split on white spaces.

they are all artist nodes in the dump.
Musicians have a groups tag.
Bands have a members tag.

Import them into 2 different tables in MariaDB.

Soundex, metaphone, and other phonetic algorithms can be useful to normalize ideographic languages like Japanese and Chinese.

The jellyfish Python library we use implements those algorithms, but does not seem to support ideographic languages (exceptions raised or empty strings as output).

This is the most straightforward one.

For each full name of a target identifier:

1. lowercase the string;
2. match against Wikidata item labels (also lowercased).

The final version will be published on the project page: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline

The investigation should answer the question: is a full-text index column in MariaDB better than a Lucene-based solution?

In bibsys_baseline_helper.link_scraper.

Currently, only 12 out of 256k albums in Wikidata have a Discogs identifier.

Discogs has a huge amount of albums: we could either leverage the releases_url tag of each artist or the releases dump directly, e.g.:
https://discogs-data.s3-us-west-2.amazonaws.com/data/2018/discogs_20180801_releases.xml.gz

• build the Wikidata sample of authors and teachers with no links to BIBSYS;
• implement the following baseline matchers:
• #6;
• #7;
• #8;
• #11; No dates available!
• compute the following ratio: #matches / sample length;
• report the result here: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline#August_2018:_big_fishes_selection

For each (birth, death) couple (death is optional) of a target identifier, for each of [#6, #9, #10], match against Wikidata item birth and death dates, and labels.

As per #19, if required criteria 2 or 3 are not met, then deprecate the statement.

• use the Wikidata musicians and bands sample;
• implement the following baseline matchers:
• #6;
• #7;
• #8;
• #11;
• compute the following ratio: # unique matches / (sample length - #already linked Wikidata items);
• report the result here: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline#Big_fishes

For each link to another catalog of a target identifier, match against Wikidata item identifiers.

• use the Wikidata sample;
• implement the following baseline matchers:
• #6;
• #7;
• #8;
• #11;
• compute the following ratio: #matches / (sample length - #already linked Wikidata items);
• report the result here: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline#Target_selection

• standard analyzer (tokenization, bag of words);
  - [ ] fuzzy analyzer;
• optimize #9 #10 #33 runtime from O(nm) to O(n), where n is the length of the source term list, m is the length of the target term list. We can drop a "constant", coming from the result set length of a query against the index.

As per #35, MariaDB on Toolforge doesn't seem to support one.
Therefore, the fuzzy analyzer must be implemented before ingesting datasets into MariaDB.

- [ ] Pick the best implementation that fits our use case here: https://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python;

• use jellyfish;
• for each full name of a target identifier, compute the Levenshtein edit distance against all Wikidata item labels.

The whole project should be structured to serve both as a command line interface and as a Python module.

• Get inspired by the StrepHit structure: https://github.com/Wikidata/StrepHit/tree/master/strephit: the folder strephit is the parent module and all its subfolders are submodules;
• for module usage, you can see the typical __init__.py and __main__.py files;
• for CLI usage, click is the way to go.

Follow steps 1 to 3 of this link: https://tools.wmflabs.org/

For each Wikipedia/DBpedia link of a target identifier, match against Wikidata item site links.

We have 3 criteria, 2 generic and 1 specific to item type.

1. existence: whether the target ID still exists in the target catalog;
2. links: URLs overlap;
3. metadata: statements overlap.

Community discussion initiated at:

• https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2021/07#Item_validation_criteria
• https://lists.wikimedia.org/hyperkitty/list/[email protected]/message/FZD6DPIQAQ3LJMEUV5IA7ATY2RXA2UQ5/
• https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/Soweego_bot_4

So far, we have developed several different things in the target_selection module.
It's time now to move logic to appropriate places:

• discogs, bne, musicbrainz, matching_strategies.py, -> linker module;
• data extraction functions, e.g., discogs/baseline_matcher#extract_data_from_dump -> importer module, with appropriate sub-folders.

Besides labels, aliases may be useful to augment matching.

In particular:

• For paths, no string concatenation but os.path.join
• Loading of resources should be done using resources.get_data
• Paths for Input should be set from click.argument
• Paths for Output should be set from click.option

• Get Wikidata bands;
• implement #54;
• implement #59;
• run name-based matchers over album titles.

It should be used at dump extraction time to feed the link table in MariaDB with "good" URLs.
A starting point would be to query Wikidata for URL domains of existing catalog identifiers.

Python 3.4 is the version deployed on Toolforge.

pkgutil.get_data returns a binary string, i.e., bytes.
json.loads docstring:

• Python 3.4
  Deserialize s (a str instance containing a JSON document) to a Python object.
• Python 3.6.5
  Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object.`

Temporary fix: don't use get_data
See discogs baseline matcher.

• Get Wikidata musical albums;
• implement #55;
• implement #56;
• run name-based matchers.

As per #19, if required criterium 1 is not met, then delete the statement.

It should perform the following actions:

• download the dump;
• slice the subset about artists;
• import the subset into the s51434__mixnmatch_large_catalogs_p database on Toolforge.

• use jellyfish;
• for each full name of a target identifier, compute the Damerau-Levenshtein edit distance against all Wikidata item labels.

INPUT: full set of Wikidata musicians and bands QIDs with MusicBrainz ID; full set of MusicBrainz artists;
OUTPUT: set of Wikidata QIDs with invalid MusicBrainz ID
Please put the implementation under the soweego/validator folder.

While working on #62, we realized that BIBSYS suffers from data inconsistency:

1. the available dump is out of sync with the online resources, with identifiers yielding HTTP 404;
2. an identifier may have multiple cross-catalog links (VIAF, GND);
3. an identifier may have correct sitelinks, but wrong cross-catalog links.

IMDB is the next big fish in line.

• build the Wikidata sample of actors, directors and producers with no links to IMDB;
• implement the following baseline matchers:
• #6;
• #7; No links available
• #8; No Wikilinks available
• #11;
• compute the following ratio: #matches / sample length;
• report the result here: https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego/Timeline#August_2018:_big_fishes_selection

It should perform the following actions:

• download the dump at https://data.discogs.com/;
• slice the subset about artists;
• import the dataset into the s51434__mixnmatch_large_catalogs_p database on Toolforge.

INPUT: full set of Wikidata musicians and bands QIDs with Discogs ID; full set of Discogs artists;
OUTPUT: set of Wikidata QIDs with invalid Discogs ID
Please put the implementation under the soweego/validator folder.