oduwsdl / aiu Goto Github PK
View Code? Open in Web Editor NEWA library for interacting with web archive collections at Archive-It, Trove, Pandora, and more.
License: MIT License
A library for interacting with web archive collections at Archive-It, Trove, Pandora, and more.
License: MIT License
This may provide information that we cannot get through our screen scraping and other API access.
HAW has collections that can, at a minimum, be scraped by this library to produce a list of URI-Ms and metadata.
There are cases when non-memento relations show up in the Link header. The convert_LinkTimeMap_to_dict
function does not need to handle these relations, and currently ignores them.
In the case of https://ianmilligan.ca/, there is an item in the link header that causes this function to raise an exception because it is unquoted. The issue is present in https://homerfan.wordpress.com as well. Both of these appear to be WordPress sites.
In the example below, the relation "shortlink" is unquoted, which breaks the parser in this function.
# curl -I https://ianmilligan.ca/
HTTP/2 200
server: nginx
date: Thu, 06 Sep 2018 22:11:55 GMT
content-type: text/html; charset=UTF-8
strict-transport-security: max-age=86400
vary: Accept-Encoding
vary: Cookie
x-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
link: <https://wp.me/4cEB>; rel=shortlink
x-ac: 3.bur _bur
There is an extra slash before the identifier in some URIs returned from the code.
For subjects:
In [1]: ps = PandoraSubject(83)
In [2]: ps.subject_uri
Out[2]: 'http://pandora.nla.gov.au/subject//83'
For collections:
In [1]: pc = PandoraCollection(12022)
In [2]: pc.collection_uri
Out[2]: 'http://pandora.nla.gov.au/col//12022'
Some web clients and web servers will not redirect these correctly and calling code should not have to create a workaround.
The following function assumes that a user will have called function load_seed_metadata
before calling get_seed_metadata
.
If called in the wrong order, then get_seed_metadata throws the following exception.
Traceback (most recent call last):
File "/Users/smj/Unsynced-Projects/MementoEmbed/mementoembed/services/errors.py", line 27, in handle_errors
return function_name(urim, preferences)
File "/Users/smj/Unsynced-Projects/MementoEmbed/mementoembed/services/memento.py", line 195, in seeddata
output[\'metadata\'] = sr.seed_metadata()
File "/Users/smj/Unsynced-Projects/MementoEmbed/mementoembed/seedresource.py", line 152, in seed_metadata
metadata = self.aic.get_seed_metadata(self.urir)[\'collection_web_pages\']
File "/Users/smj/.virtualenvs/MementoEmbed/lib/python3.7/site-packages/aiu/archiveit_collection.py", line 506, in get_seed_metadata
d = self.seed_metadata["seeds"][uri]
KeyError: 'seeds'
I'm not sure which collections would be a good test. We would need to plan this before implementation.
Collections at NLA are stored at https://webarchive.nla.gov.au/collection
The can contain mementos or sub-collections. This makes getting metadata more complicated that getting it from Archive-It.
Each collection has a numeric identifier, such as 15003. The collection identifier exists at the end of the URL for that collection, e.g., https://webarchive.nla.gov.au/collection/15003. NLA collection 15003 has nine sub-collections.
Collection 13742 contains mementos instead of sub-collections. Our solution needs to be able to paginate and load additional content from the collection page. The "Show 10 More" button loads more content, but, to save on resources like RAM, we want to access the content without having to use a headless browser.
Fortunately, after some analysis with Chrome's developer tools, I've discovered that we can acquire a JSON representation of the collection via a URL like https://webarchive.nla.gov.au/bamboo-service/collection/13742 where we replace the last part of the path with the collection identifier.
Using ArchiveItCollection as an example, we will need to create another class that allows anyone to acquire this content via Python. Once this is done, it can be called from Hypercane or MementoEmbed, as needed.
I did not see a get_collectedby
(or equivalent) for PandoraCollection
or PandoraSubject
. Such a function would need to descend into TEPs and acquire the organization names that created these collections. This way a resulting story would have access to this metadata for visualization.
Ed Summers wrote code that discovers the Internet Archive collection based on a URI-M. Maybe we could do the reverse?
The README is not bad, but we need something with more examples, use cases, etc.
Any loop that examines TEP pages terminates if any page fails. It should recover gracefully if one of them fails. This occurs whenever get_metadata_from_tep
is called. Because of this Hypercane cannot process some Pandora Subjects, like 22.
Here is a minimal working example:
In [1]: from aiu import PandoraSubject
In [2]: ps = PandoraSubject(22)
In [3]: memento_list = ps.list_memento_urims()
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
~/.virtualenvs/hypercane-clean2/lib/python3.8/site-packages/aiu/pandora_collection.py in get_metadata_from_tep(res, data)
106 try:
--> 107 json_data = json.loads(res.text)
108 #print(json_data)
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
356 parse_constant is None and object_pairs_hook is None and not kw):
--> 357 return _default_decoder.decode(s)
358 if cls is None:
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py in decode(self, s, _w)
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
<ipython-input-3-69a1a780eaf3> in <module>
----> 1 memento_list = ps.list_memento_urims()
~/.virtualenvs/hypercane-clean2/lib/python3.8/site-packages/aiu/pandora_collection.py in list_memento_urims(self)
390 """Lists the memento URIMs of an NLA Trove collection."""
391
--> 392 self.load_subject_metadata()
393 #print(self.metadata["main"]["urims"])
394 return self.metadata["main"]["urims"]
~/.virtualenvs/hypercane-clean2/lib/python3.8/site-packages/aiu/pandora_collection.py in load_subject_metadata(self)
358 if not self.metadata_loaded:
359 soup = BeautifulSoup(self.firstpage_response.text, 'html5lib')
--> 360 self.metadata["main"] = extract_main_subject_data(soup,self.subject_id)
361 #self.metadata["optional"] = extract_optional_collection_data(self.session.get(self.collection_json_uri))
362 self.metadata_loaded = True
~/.virtualenvs/hypercane-clean2/lib/python3.8/site-packages/aiu/pandora_collection.py in extract_main_subject_data(soup, subject_id)
307 #print(tep_id)
308 tep_json_uri = tep_json_prefix + tep_id
--> 309 tep_dic = get_metadata_from_tep(requests.get(tep_json_uri),data)
310 #print(tep_json_uri)
311 #Some tep urls give server error subject 12, https://webarchive.nla.gov.au/bamboo-service/tep/75101
~/.virtualenvs/hypercane-clean2/lib/python3.8/site-packages/aiu/pandora_collection.py in get_metadata_from_tep(res, data)
115 return tep
116 else:
--> 117 raise ValueError("Could not find 'Problem accessing /bamboo-service/collection/' string in response")
118
119 except IndexError as e:
ValueError: Could not find 'Problem accessing /bamboo-service/collection/' string in response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.