cc-archive / cccatalog Goto Github PK

Soft Icons
Allow users to download free icons (for personal use).

Sample URL:

• www.softicons.com/system-icons/yuuminco-icons-by-dreamxis/pptx-icon
• www.softicons.com/toolbar-icons/silk-companion-icons-by-damien-guard/email-stop-icon
• www.softicons.com/toolbar-icons/black-wireframe-toolbar-icons-by-gentleface/globe-2-icon

familysearch.org/wiki/en/index.php?title=Bartow_County_Genealogical_Society_and_Family_Research_Library&oldid=1568090

NB: similar to issue: #31

Create aws glue job to move data to a new location and sync with raw data (on a monthly basis).

• sync previous crawls
  • filter temporary files
  • partition data
  • apply transformations
• triggered monthly at 2 AM UTC

McCord Museum
The McCord Museum collection celebrates Montreal history. The content is available in English and French; images are watermarked.

Sample URLs

• http://collections.musee-mccord.qc.ca/en/collection/artefacts/M19893
• http://collections.musee-mccord.qc.ca/en/collection/artifacts/I-14067.1
• http://collections.musee-mccord.qc.ca/en/collection/artifacts/II-138346/

Create a default parameter

Flora-On
Sample URLs
flora-on.pt/index.php?q=Crambe+hispanica
flora-on.pt/index.php?q=Potamogeton
flora-on.pt/index.php?q=Ornithogalum+narbonense

The meta data is in Spanish.
There are 4K unique URLs.
Each web page shows multiple images for the same plant family.

Shroomery
Information about 'magic' mushrooms.
The web pages are cc by-nc-sa (footer).

Sample URL
www.shroomery.org/6008/oldimages-3381-27787-jpg
www.shroomery.org/forums/showflat.php/Number/2913783
www.shroomery.org/forums/showflat.php/Number/3070156

NB: Low image count ~2200

Open Walls
Image sharing website. They allow uploads from both registered and anonymous users.
NB: Very little traceability.

Sample URL:

• openwalls.com/image?id=30044
• openwalls.com/image?id=41339
• openwalls.com/image?id=17695

Sync table when new partitions are added

• Scheduled to update monthly at 2:30 AM UTC (or on-demand)

• configure EMR cluster

• Add default execution flag

• Add optional user-defined crawl index

• Create data pipeline

• Test process-flow

• Schedule monthly execution

• adjust security groups
• create s3 gateway endpoint

Develop the framework for all providers based on the Met's ETL modules.

Sprint 3

• Thingiverse

Sprint 2

• Brooklyn Museum
• Pexels [On Hold]
• WoRMS
• Flickr

Sprint 1

• Met Museum
• Flickr
• Museum Victoria
• Deviantart
• Geograph.org.uk
• McCord Museum
• Animal Diversity Web
• Open Walls
• Soft Icons
• Flora-On
• IHA
• Behance
• Icon Finder
• Thingiverse
• EOL
• Science Museum
• Digitalt Museum
• Europeana

South African Climbing Wiki

Total links extracted: ~78,000

Sample URLs:

• www.climbing.co.za/w/index.php?title=Special:WhatLinksHere/Donker_Kloof&hidetrans=1
• www.climbing.co.za/w/index.php?title=Bronkies&diff=1671&oldid=1444
• www.climbing.co.za/wiki/index.php?title=Bobbejaans_Berg&diff=2770&oldid=1290

This provider will be challenging to parse because the additional meta data for each image is located in another URL (which may not be available in the current web crawl data).

For example:
The webpage below has six images:

• www.climbing.co.za/wiki/index.php?title=Bobbejaans_Berg&diff=2770&oldid=1290

The source/location for one of them is:

• http://www.climbing.co.za/wiki/File:Bb3s.jpg

If we extract the images, that are embedded in the html, the only meta data we will obtain are width and height (and any other attribute from the image tag). However, the additional meta data will not be available.

An alternative solution is to extract the images directly from their respective locations by using a regular expression:

• www.climbing.co.za/wiki/File.*?
  However, the total number of meta data URLs in our repository is currently very low.

Execute monthly at 2AM UTC.

Animal Diversity Web
This is the University of Michigan's online encyclopedia of animal history.

Sample URL:

• https://animaldiversity.org/accounts/Reptilia/pictures/collections/contributors/tanya_dewey/red_tail_juv/
• https://animaldiversity.org/collections/contributors/phil_myers/ADW_mammals/specimens/Artiodactyla/Bovidae/Neotragus_batesi/lower_lateral_1/
• https://animaldiversity.org/site/resources/anatomical_images/family_pages/sirenia/dugongidae.jpg/view.html/