filak / mtw-mesh Goto Github PK

View Code? Open in Web Editor NEW

6.0 5.0 1.0 7.51 MB

Medical Subject Headings (MeSH) Translation Workflow

Home Page: https://github.com/filak/MTW-MeSH/wiki

License: MIT License

Batchfile 1.03% Python 51.92% JavaScript 3.38% CSS 0.74% HTML 42.94%

mesh workflow translation sparql rdf apache-jena apache-jena-fuseki python3 sqlite3 flask

mtw-mesh's Introduction

MTW-MeSH

Medical Subject Headings Translation Workflow

Please see the Wiki for details of this project.

Codacy status

mtw-mesh's People

Contributors

Stargazers

Watchers

Forkers

julienbacquart

mtw-mesh's Issues

Marc Exports - add registryNumber and casn1_label to 450 fields

Target for the MeSH 2025 version.

TBD:

modify getMarcFields, getMarc functions
update the docs for MSH $r and $s fields

Exports - add dictionary spell check output for all terms

Check if /search/do:clear works as expected

Support conversion of the official UMLS TSV translation file to RDF dataset

Use trans_only_YYYY_extended.txt

  DescriptorUI | ConceptUI | Language | Term Type | String | TermUI | ScopeNote | Tree | Created | Relation | ParentCUI

Do not use the TermUI - generate UUID v4 instead

Exports - make lookups & lookups_rest export files versioned by target year

Move Talisman GCSP dict to config.ini file

Unwanted OBSOLETE terms in UMLS export

Fix missing empty cells in UMLS based tsv exports

Exports - missing scopenotes in check-tags

Version 1.3.3

Fix export query: lookups_notes.sparql

?c meshv:scopeNote ?scn =>
OPTIONAL { ?c meshv:scopeNote ?scn }

Fix gen_date value

MTW-MeSH/flask-app/application/utils.py

Line 696 in 211dcff

gen_date = timestamp[0:10].replace('-','')

According to https://loc.gov/marc/bibliographic/bd005.html

16 chars length:
```
yyyymmddhhmmss.f
```

Missing add to clipboard button in version 1.4.14

Non-existant MeSH DUI causes 500 status response

It shall return 404 instead.

Exports - JSON for Elastic - fix query to include missing items

Query template to fix:

lookups_use_instead

Ie.
https://id.nlm.nih.gov/mesh/D001769Q000201.html
https://id.nlm.nih.gov/mesh/D004798Q000097.html

Test query:

PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd:  <http://www.w3.org/2001/XMLSchema#>
PREFIX owl:  <http://www.w3.org/2002/07/owl#>
PREFIX meshv:  <http://id.nlm.nih.gov/mesh/vocab#>
PREFIX mesh:   <http://id.nlm.nih.gov/mesh/>
PREFIX text:   <http://jena.apache.org/text#>
PREFIX meshx:  <http://mesh.medvik.cz/link/>
PREFIX mesht:  <http://www.medvik.cz/schema/mesh/vocab/#>

#SELECT ?dui (GROUP_CONCAT(DISTINCT ?o ; separator='|') AS ?qn)
SELECT *
WHERE {
  BIND(mesh:D001769 as ?d)
  ?dq meshv:hasDescriptor ?d .
  #?dq meshv:active ?active .
  #FILTER(?active)
  ?d meshv:identifier ?dui .
  ?dq meshv:useInstead ?ud .
  #?dq meshv:hasQualifier ?q .
  #?q meshv:identifier ?qui .
  #?ud meshv:identifier ?udui .
  #BIND( CONCAT(STR(?qui), '~', STR(?udui)) AS ?o)
}
#GROUP BY ?dui
LIMIT 100

Use Flask-Session - with configurable params

https://flask-caching.readthedocs.io/en/latest/
Ie.:
SESSION_TYPE
SESSION_COOKIE_NAME
SESSION_COOKIE_HTTPONLY
SESSION_COOKIE_SECURE
SESSION_KEY_PREFIX
...

Config usage in mtw.ini as:

SESSIONS = {"SESSION_TYPE": "filesystem", ... }

Remove xtr field from Elastic export

Fix json exports - missing xtr field

It has been removed as being dedundant to xtr_en and xtr_cs fields

Checking for duplicate terms in translation data when updating concept

Store all the trx terms in a db table

/* terms */
--drop table if exists terms;
create table audit (
  id integer primary key autoincrement,
  dui text not null,   -- DescriptorUI | QualifierUI
  cui text not null,   -- ConceptUI
  term text not null, 
  created text DEFAULT (strftime('%Y-%m-%dT%H:%M:%S','now', 'localtime'))
);

Analyze how to:

populate the table initially
check terms for duplicates before concept update
insert rows for new terms
update when deleting terms

If duplicate is found issue a danger alert - including created + dui + link .../search/dui:

Implement caching using Memcached

pymemcache==3.5.2

Ie.

    if app.config.get('CACHE_TYPE') == 'MemcachedCache':
        memhost = app.config.get('CACHE_MEMCACHED_SERVERS',['127.0.0.1:11211'])[0]
        try:
            memclient = PMClient(memhost, ignore_exc=True, connect_timeout=3, timeout=3)
            memclient.set('test_alive_key', 'ALIVE')
        except:
            err = '\n\nMemcached instance NOT running ! ' + str(memhost) + ' \n'
            app.logger.error(err)
            warn = '\n\nSetting CACHE_TYPE to Flask-Caching : FileSystemCache \n'
            app.logger.warning(warn)
            app.config.update({"CACHE_TYPE": "FileSystemCache"})
            CACHE_DIR = mtu.get_instance_dir(app, app.config.get('CACHE_DIR','cache_fs'))
            app.config.update({'CACHE_DIR': CACHE_DIR})

Version 1.3.5 Cache does not get updated in some cases

Missing params in sparql.updateTriple(... dui=dui, cache=cache)

Some exports not working - MARC21, Lookups, Custom

Broken in version 1.6.0

Investigate if the mesh: prefix shall be defined in mesh.ttl

https://github.com/filak/MTW-MeSH/blob/master/flask-app/instance/conf/mesh.ttl

  @prefix mesh:   <http://id.nlm.nih.gov/mesh/> .

Custom concepts are NOT exported to UMLS TSV dataset

Add dateEstablished property to JSON exports

Upgrade Flask-session to 0.5.0

Add autofocus to searchForm input

https://github.com/filak/MTW-MeSH/blob/master/flask-app/application/templates/snips/search-form.html#L6

Marc exports - mixed up descriptor classes

geo, publ & check tags

Update MESH_RDF endpoint url to https

MESH_RDF = https://id.nlm.nih.gov/mesh/sparql

Check that missing trns are not being exported as fields 072 and 686

Ie. check

MTW-MeSH/flask-app/application/utils.py

Line 827 in 54f3275

for trn in item.get('trn',[]):

Add MeSH Browser link to Concept view

https://meshb.nlm.nih.gov/

https://meshb.nlm.nih.gov/record/ui?ui=...

Better relevancy while searching

Try to use propLists

https://jena.apache.org/documentation/query/text-query.html#lists-of-indexed-properties

text:propLists (
    [ text:propListProp mesht:defQuery ;
      text:props ( 
         rdfs:label
         meshv:identifier 
         meshv:prefLabel
         meshv:altLabel
         meshv:casn1_label
         meshv:registryNumber
         meshv:relatedRegistryNumber
             mesht:identifier 
             mesht:prefLabel
             mesht:altLabel
             mesht:casn1_label
          ) ;
    ]
    [ text:propListProp mesht:includeNotes ;
      text:props ( 
         rdfs:label
         meshv:identifier 
         meshv:prefLabel
         meshv:altLabel
         meshv:casn1_label
         meshv:registryNumber
         meshv:relatedRegistryNumber
             mesht:identifier 
             mesht:prefLabel
             mesht:altLabel
             mesht:casn1_label
                mesht:translatorsNote
                mesht:annotation
                mesht:historyNote
                mesht:scopeNote
          meshv:annotation
          meshv:historyNote
          meshv:note
          meshv:onlineNote
          meshv:publicMeSHNote
          meshv:scopeNote
         ) ;
    ]
) ;

Add filter - checkbox - to the form (searchForm)

includeNotes

{% set prop_list = 'mesht:defQuery' %}
{% if includeNotes == 'yes' %}
    {% set prop_list = 'mesht:includeNotes' %}
{% endif %}

  {% if slang == 'target' %}
    (?s ?score) text:query ({{ prop_list }} "{{ query }}" 500 "lang:{{ lang }}") .
  {% elif slang == 'source' %}
    (?s ?score) text:query ({{ prop_list }} "{{ query }}" 500 "lang:en") .
  {% else %}
    (?s ?score) text:query ({{ prop_list }} "{{ query }}" 500) .
  {% endif %}
...