qurator-spk / mods4pandas Goto Github PK

The mods:name now has a mods:nameIdentifier:

/home/mike/devel/qurator-mono-repo/modstool/qurator/modstool/modstool.py:428: UserWarning: Exception in /srv/digisam_mets/PPN1678618276.xml:
Unknown tag "{http://www.loc.gov/mods/v3}nameIdentifier"
  warnings.warn('Exception in {}:\n{}'.format(mets_file, e))

Also, the mods:name/mods:displayForm (optional according to DFG MODS-Anwendungsprofil) got dropped in favor of the mandatory name part fields.

Following up on #12, we need a better name for altotool.

• Rename altotool to alto4pandas
• Rename modstool to mods4pandas
• Rename qurator.modstool
• Rename project to mods4pandas

From the feedback by Maria Federbusch:

PPN813655765
Zusätzlich fehlen in diesem Beispiel (MONO) noch die Schlagwörter - hier aus dem ARMA-Projekt:

<mods:subject authority="getty">
<mods:topic valueURI="http://vocab.getty.edu/aat/300411614">reading culture</mods:topic>
<mods:topic valueURI="http://vocab.getty.edu/aat/300020756">Medieval (European)</mods:topic>
</mods:subject>

<mods:subject authority="wikidata">
<mods:topic valueURI="https://www.wikidata.org/wiki/Q107274053">Reading culture (medieval)</mods:topic>
</mods:subject>

<mods:genre valueURI="https://www.wikidata.org/wiki/Q1261026" type="class" authority="wikidata">
<mods:genre>printed matter</mods:genre>
</mods:genre>

• Review a. DFG MODS Anwendungsprofil for mods:genre and b. more examples of mods:genre - here the structure is clearly different from the mods:subject examples. Would make more sense if the nested genre had the valueURI?

By now, only the content of <mods:identifier type="PPNanalog"> seems to be included in the output but not the equivalent information from

<mods:relatedItem type="original">
	<mods:recordInfo>
		<mods:recordIdentifier source="gbv-ppn">PPNxyz</mods:recordIdentifier>
	</mods:recordInfo>
</mods:relatedItem>

• Should this be in here? Or in "codename altotool"?

• What info would be relevant? What would be metadata, what would be data (count words?)

• Include metadata from the Description section

• Include descriptive statistics for the Layout section etc.

  • E.g. word count
  • E.g. mean STRING WC (confidence)

• When that's done review the comments below for things we may have missed

• Test using all available versions of ALTO

• NER annotated ALTO should at least be identifiable

• Include ALTO version/namespace

• <LayoutTag ID="layouttag-marginalia" LABEL="marginalia"/>

• Any language infos?

• Update README that we now support ALTO

• Test grouping name columns by role
• Consider using a list in the column (e.g. multiple authors, multiple publishers)
• Review #1

Using "qurator" requires coordination between all projects. → Avoid this.

• Do not hardcode METS/MODS version, e.g. in modstool.py
• Review the rest

• Mention CSV + Excel
• Show example data in the README

Dissect this comment by @joergleh:

Ansonsten: Da nur die oberste dmdSec verwendet wurde, sind Felder wie der Titel des übergeordneten Werks, alternative Titel einzelner Werks, Informationen darüber, dass dieses Werk Teil einer mehrbändigen Publikation ist, strukturelle Informationen (z. B. über den Umfang des Bandes oder darüber, wo das Inhaltsverzeichnis zu finden ist, falls ein solches vorhanden ist), über den Eigentümer des physischen Buchs, das Digitalisierungsprojekt oder einige spezifische Verschlagwortungen möglicherweise nicht in der Tabelle enthalten.

• Da nur die oberste dmdSec verwendet wurde,
• Titel des übergeordneten Werks
• alternative Titel einzelner Werks
• Informationen darüber, dass dieses Werk Teil einer mehrbändigen Publikation ist,
• strukturelle Informationen (z. B.
• über den Umfang des Bandes oder
• darüber, wo das Inhaltsverzeichnis zu finden ist, falls ein solches vorhanden ist),
• über den Eigentümer des physischen Buchs,
• das Digitalisierungsprojekt oder
• einige spezifische Verschlagwortungen möglicherweise nicht in der Tabelle enthalten.

@labusch had questions regarding structure information (from METS metadata) and @joergleh already had suggestions regarding missing information (#23, #24).

While there is certainly information that I find out of scope for this tool (like the location of a title page → should use the original METS for this) there is certainly information we should include (like the count/presence of a title page).

(Edit: Moved the missing field documentation to #27.)

• structMap[@TYPE="LOGICAL"]: should count the divs, grouped by their type. They are nested, so this needs to be accounted for.
• structMap[@TYPE="PHYSICAL"]: Count?
• Should reformat the files in tests/data to read them more easily

Newly downgrade to Pandas 1.0.5 broke Python 3.10

Language attributes are LANG and the deprecated language:

• https://www.loc.gov/standards/alto/v4/alto-4-3.xsd
• In my test data, only the deprecated TextBlock/@language tags are used

mods:names now may have more than one role:

    <mods:mods xmlns:mods="http://www.loc.gov/mods/v3">                                                                  
    <mods:name type="personal" valueURI="http://d-nb.info/gnd/117357669">                                                
      <mods:displayForm>Wurm, Mary</mods:displayForm>                                                                    
      <mods:namePart type="given">Mary</mods:namePart>                                                                   
      <mods:nameIdentifier type="gbv-ppn">078789583</mods:nameIdentifier>                                                
      <mods:namePart type="family">Wurm</mods:namePart>                                                                  
      <mods:role>                                                                                                        
        <mods:roleTerm authority="marcrelator" type="code">cmp</mods:roleTerm>                                           
      </mods:role>                                                                                                       
      <mods:role>                                                                                                        
        <mods:roleTerm authority="marcrelator" type="code">aut</mods:roleTerm>                                           
      </mods:role>                                                                                                       
    </mods:name>                                                                                                                                             
    </mods:mods>           

This should be merged into one column, e.g. d['name0_role_roleTerm'] == {'cmp', 'aut'}

• Review
• Improve documentation
• Add types (mypy)
• Group TagGroup methods (some are fluent API calls on itself, some produce values)

Since the ALTO functionality, imports are mess. This should be reviewed

Currently, the README does not show any results (i.e. an excerpt of the resulting table).

The obvious thing that could be included:

• Include page count per filegroup

In addition to that:

• Investigate tags which were ignored for now

Newer input files have two <mods:languageTerm>s in one <mods:language>:

<mods:language>
  <mods:languageTerm authority="iso639-2b" type="code">ger</mods:languageTerm>
  <mods:languageTerm authority="iso639-2b" type="code">eng</mods:languageTerm>
</mods:language>

modstool should not throw an exception here.

Aim: Reduce number of columns for better manageability of data frame
Proposal: Group names given in the MODS-file according to given roles
Explanation: Each "name" entry in the mods-file consists of at least four parts:
"nameXX_namePart.family"
"nameXX_namePart.given"
"nameXX_displayForm"
"nameXX_role_roleTerm"
The number of columns could significantly be reduced if the names would first be grouped according to the roles and then concatenated into a fewer number of columns.
Examples:
PPN735425078 contains 76 names with the role "asn" (= associated name); this amounts up to 304 columns, but could be reduced to three columns ("nameASN_namePart.family", "nameASN_namePart.given", "nameASN_displayForm"), each containing 76 names in nested form (Mauschwitz; Baudis; Hoberg; ...)
PPN858144891 contains 50 names with the role "oth" (= other); this amounts up to 200 columns, but could be reduced to three columns ("nameOTH_namePart.family", "nameOTH_namePart.given", "nameOTH_displayForm")
PPN1774254956 contains 42 names with the role "ctb" (= contributor); this amounts up to 168 columns, but could be reduced to three columns ("nameCTB_namePart.family", "nameCTB_namePart.given", "nameCTB_displayForm")
The most frequently used roles are asn (associated name), oth (other), ctb (contributor), dte (dedicatee), fnd (funder), auth (author), isb (issuing body), egr (engraver), hnr (honoree), ill (illustrator), prt (printer).

Our pandas dependency is broken on Python 3.12.

How could I get the total number of pages/images/canvases from the METS files of all objects returned in a query like https://digital.staatsbibliothek-berlin.de/suche?queryString=type%3Aannotation%20date_issued%3A%3E1455%20date_issued%3A%3C1800&category=Naturwissenschaften%20%2F%20Mathematik

ERROR:mods4pandas:Exception in /srv/digisam_mets/PPN1830497871.xml: One or more element has unexpected attributes: <mods:recordIdentifier xmlns:mods="http://www.loc.gov/mods/v3" xmlns:mets="http://www.loc.gov/METS/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" source="dnb-ppn">1236513355</mods:recordIdentifier>

Edit after feedback from a co-worker:

• For top-level mods:recordIdentifier we need to check that's it's a GBV PPN ("our" PPN)
• For mods:relatedItem//mods:recordIdentifier we need to distinguish GBV PPNs from DNB PPN (or others). This happens when we have digitized works from other libraries (in this case DNB)

(Moved from #26)

There's also the issue that I usually forget what we already included in our current mods4pandas/current mods_info export. We should have a complete description of the fields. This should ideally be generated automatically, or at least checked for completeness. This would also enable checking what changed between versions of the export. (This is related #26 because I simply could not tell @labusch what currently is possible with the info we already export.)