mrmiguez / pymods Goto Github PK

View Code? Open in Web Editor NEW

18.0 6.0 0.0 8.77 MB

process MODS records from Python

Home Page: https://pypi.python.org/pypi/pymods

License: MIT License

Python 100.00%

mods xml oai metadata o

pymods's Introduction

pymods

pymods is utility module for working with the Library of Congress's MODS XML standard: Metadata Description Schema (MODS). It is a utility wrapper for the lxml module specific to deserializing data out of MODSXML into python data types.

If you need a module to serialize data into MODSXML, see the other pymods by Matt Cordial.

Installing

Recommended:

pip install pymods

Using

Basics

XML is parsed using the MODSReader class:

mods_records = pymods.MODSReader('some_file.xml')

Individual records are stored as an iterator of the MODSRecord object:

In [5]: for record in mods_records:
  ....:    print(record)
  ....:
<Element {http://www.loc.gov/mods/v3}mods at 0x47a69f8>
<Element {http://www.loc.gov/mods/v3}mods at 0x47fd908>
<Element {http://www.loc.gov/mods/v3}mods at 0x47fda48>

MODSReader will work with mods:modsCollection documents, outputs from OAI-PMH feeds, or individual MODSXML documents with mods:mods as the root element.

pymods.MODSRecord

The MODSReader class parses each mods:mods element into a pymods.MODSRecord object. pymods.MODSRecord is a custom wrapper class for the lxml.ElementBase class. All children of pymods.Record inherit the lxml._Element and lxml.ElementBase methods.

In [6]: record = next(pymods.MODSReader('example.xml'))
In [7]: print(record.nsmap)
{'dcterms': 'http://purl.org/dc/terms/', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance', None: 'http://www.loc.gov/mods/v3', 'flvc': 'info:flvc/manifest/v1', 'xlink': 'http://www.w3.org/1999/xlink', 'mods': 'http://www.loc.gov/mods/v3'}

In [8]: for child in record.iterdescendants():
  ....:    print(child.tag)
    
{http://www.loc.gov/mods/v3}identifier
{http://www.loc.gov/mods/v3}extension
{info:flvc/manifest/v1}flvc
{info:flvc/manifest/v1}owningInstitution
{info:flvc/manifest/v1}submittingInstitution
{http://www.loc.gov/mods/v3}titleInfo
{http://www.loc.gov/mods/v3}title
{http://www.loc.gov/mods/v3}name
{http://www.loc.gov/mods/v3}namePart
{http://www.loc.gov/mods/v3}role
{http://www.loc.gov/mods/v3}roleTerm
{http://www.loc.gov/mods/v3}roleTerm
{http://www.loc.gov/mods/v3}typeOfResource
{http://www.loc.gov/mods/v3}genre
...

Methods

All functions return data either as a string, list, list of named tuples. See the appropriate docstring or the API documentation for details.

>>> record.genre?
Type:        property
String form: <property object at 0x0000000004812C78>
Docstring:
Accesses mods:genre element.
:return: A list containing Genre elements with term, authority,
    authorityURI, and valueURI attributes.

Examples

Importing

from pymods import MODSReader, MODSRecord

Parsing a file

In [10]: mods = MODSReader('example.xml')
In [11]: for record in mods:
   ....:    print(record.dates)
   ....:
[Date(text='1966-12-08', type='{http://www.loc.gov/mods/v3}dateCreated')]
None
[Date(text='1987-02', type='{http://www.loc.gov/mods/v3}dateIssued')]

Simple tasks

Generating a title list

In [14]: for record in mods:
   ....:     print(record.titles)
   ....:
['Fire Line System']
['$93,668.90. One Mill Tax Apportioned by Various Ways Proposed']
['Broward NOW News: National Organization for Women, February 1987']

Creating a subject list

In [17]: for record in mods:
   ....:     for subject in record.subjects:
   ....:         print(subject.text)
   ....:
Concert halls
Architecture
Architectural drawings
Structural systems
Structural systems drawings
Structural drawings
Safety equipment
Construction
Mechanics
Structural optimization
Architectural design
Fire prevention--Safety measures
Taxes
Tax payers
Tax collection
Organizations
Feminism
Sex discrimination against women
Women's rights
Equal rights amendments
Women--Societies and clubs
National Organization for Women

More complex tasks

Creating a list of subject URI's only for LCSH subjects

In [18]: for record in mods:
   ....:     for subject in record.subjects:
   ....:         if 'lcsh' == subject.authority:
   ....:             print(subject.uri)
   ....:
http://id.loc.gov/authorities/subjects/sh85082767
http://id.loc.gov/authorities/subjects/sh88004614
http://id.loc.gov/authorities/subjects/sh85132810
http://id.loc.gov/authorities/subjects/sh85147343

Get URLs for objects using a No Copyright US rightsstatement.org URI

In [23]: for record in mods:
   ....:     for rights_elem in record.rights
   ....:         if rights_elem.uri == 'http://rightsstatements.org/vocab/NoC-US/1.0/':
   ....:             print(record.purl)
   ....:
http://purl.flvc.org/fsu/fd/FSU_MSS0204_B01_F10_09
http://purl.flvc.org/fsu/fd/FSU_MSS2008003_B18_F01_004

pymods's People

Contributors

Stargazers

Watchers

pymods's Issues

Element comparison errors when sorting NamedTuples

Some pymods.Record functions that return sorted lists of NamedTuples including lxml elements cause comparison error when there is a repeated element.

Given the structure:

<name>
  <namePart>Cash</namePart>
</name>
<name>
  <namePart>Cash</namePart>
</name>

pymods will create the tuples:
Name('Cash', '', '', '', '', '', lxml.Element pointer A)
Name('Cash', '', '', '', '', '', lxml.Element pointer B)

The only unique tuple element for sorting is the element pointer, but comparisons between lxml.Elements is not allowed:

TypeError: '<' not supported between instances of 'lxml.etree._Element' and 'lxml.etree._Element'

Really there's no reason for any pymods.Record function or property to return a sorted list. Removing the sort()'s would easily fix this.

Pass up @valueURI for subject/name

Should valueURI's from subject/elem@valueURI pass up to subject@valueURI when there isn't one there?

Example:

<subject>
  <name authority="lcnaf" 
              authorityURI="http://id.loc.gov/authorities/names" 
              type="personal" 
              valueURI="http://id.loc.gov/authorities/names/n79056767">
    <namePart type="date">1877-1956</namePart>
    <namePart type="given">Alben William</namePart>
    <namePart type="family">Barkley</namePart>
  </name>
</subject>

returns:

{ children: [ { type="name", valueURI="http://id.loc.gov/authorities/names/n79056767", ... }, ... ], ...}

when it might be more useful to return:

{ valueURI="http://id.loc.gov/authorities/names/n79056767", children= [...], ... }

DCRecord needs tests

Extend instance vars for more authority types

RDA content: <genre authority="rdacontent">text</genre>
RDA media: <form authority="rdamedia" type="RDA media terms">computer</form>
RDA carrier: <form authority="rdacarrier" type="RDA carrier terms">online resource</form>
COAR resource type: <genre authority="coar" authorityURI="http://purl.org/coar/resource_type" valueURI="http://purl.org/coar/resource_type/">bachelor thesis</genre>

URL search in record.collection can return an AttributeError

if related_item.find('./{0}location/{0}url'.format(NAMESPACES['mods'])).text is not None: (line 79) should test for elem.text only after testing for elem

OAIRecord.metadata.get_element needs to strip whitespace

.strip() list items before they are appended and returned.

DCRecord strip whitespace from delimited returns

Like so (lines 646-468):

        return [split_text.strip()
                for item in self.findall('{0}'.format(elem))
                for split_text in item.text.split(delimiter)]

Typo in pymods.DCRecord

line 647, in get_element
for item in self.finall('{0}'.format(elem))

lxml.etree persistence

This is a known issue with custom parsers: http://lxml.de/element_classes.html#element-initialization.

An internal container needs to be built.

names.role should act more link languages

roles=Role(text, code, authority) in the same tuple rather than a list of Role(text, type, ...) tuples.

Empty elements throwing TypeErrors

For example

        <subject>
          <name authority="lcnaf" authorityURI="http://id.loc.gov/authorities/names" type="personal" valueURI="http://id.loc.gov/authorities/names/n85385724">
            <namePart type="date"/>
            <namePart type="given">Benjamin V.</namePart>
            <namePart type="family">Cohen</namePart>
          </name>
        </subject>

returns
KeyError - sourceResource.subject: None, Can't convert 'NoneType' object to str implicitly

VRA Core support

Add support for VRA Core schema and elements: https://www.loc.gov/standards/vracore/

use MODSReader on string-Objects

Hello!

I'm trying to apply the MODSReader not to a xml-file (as in the examples provided) but rather on requests.get-responses I've tried transforming the xml-string into a file-like object using io.StringIO (which would be the usual way to deal with the issue in etree, I guess), but I'm getting a ValueError:

  File "mods_parse.py", line 6, in <module>
    MODSReader(io.StringIO(request_opac("pica.sys=j2017").text))
  File "/home/alex/.local/lib/python3.6/site-packages/pymods/reader.py", line 58, in __init__
    super(MODSReader, self).__init__(file_location, '{0}mods'.format(NAMESPACES['mods']), parser=mods_parser)
  File "/home/alex/.local/lib/python3.6/site-packages/pymods/reader.py", line 27, in __init__
    self.iterator = parse(file_location, parser=parser).iter(iter_elem)
  File "/home/alex/.local/lib/python3.6/site-packages/pymods/reader.py", line 8, in parse
    return etree.parse(source, parser=parser)
  File "src/lxml/etree.pyx", line 3469, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1856, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1871, in lxml.etree._parseMemoryDocument
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

Could you suggest me a way to pipe the xml-string directly into the parser?

Thank you very much!

Single records must be called with an iterator

In [2]: rec = pymods.MODSReader('FSU_ARHHouse_1018.xml')

In [3]: rec.
rec.close                      rec.makeelement
rec.copy                      rec.resolvers
rec.error_log                rec.setElementClassLookup
rec.feed                       rec.set_element_class_lookup
rec.feed_error_log        rec.target
rec.iterator                  rec.version

In [3]: mods = next(rec)

In [4]: mods.tag
Out[4]: '{http://www.loc.gov/mods/v3}mods'