cc-archive / cccatalog Goto Github PK
View Code? Open in Web Editor NEW[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.
Home Page: https://github.com/WordPress/openverse-catalog
License: MIT License
[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.
Home Page: https://github.com/WordPress/openverse-catalog
License: MIT License
Soft Icons
Allow users to download free icons (for personal use).
Sample URL:
familysearch.org/wiki/en/index.php?title=Bartow_County_Genealogical_Society_and_Family_Research_Library&oldid=1568090
NB: similar to issue: #31
Create aws glue job to move data to a new location and sync with raw data (on a monthly basis).
McCord Museum
The McCord Museum collection celebrates Montreal history. The content is available in English and French; images are watermarked.
Sample URLs
Create a default parameter
Flora-On
Sample URLs
flora-on.pt/index.php?q=Crambe+hispanica
flora-on.pt/index.php?q=Potamogeton
flora-on.pt/index.php?q=Ornithogalum+narbonense
The meta data is in Spanish.
There are 4K unique URLs.
Each web page shows multiple images for the same plant family.
Shroomery
Information about 'magic' mushrooms.
The web pages are cc by-nc-sa (footer).
Sample URL
www.shroomery.org/6008/oldimages-3381-27787-jpg
www.shroomery.org/forums/showflat.php/Number/2913783
www.shroomery.org/forums/showflat.php/Number/3070156
NB: Low image count ~2200
Open Walls
Image sharing website. They allow uploads from both registered and anonymous users.
NB: Very little traceability.
Sample URL:
Sync table when new partitions are added
configure EMR cluster
Add default execution flag
Add optional user-defined crawl index
Create data pipeline
Test process-flow
Schedule monthly execution
Develop the framework for all providers based on the Met's ETL modules.
Sprint 3
Sprint 2
Sprint 1
South African Climbing Wiki
Total links extracted: ~78,000
Sample URLs:
This provider will be challenging to parse because the additional meta data for each image is located in another URL (which may not be available in the current web crawl data).
For example:
The webpage below has six images:
The source/location for one of them is:
If we extract the images, that are embedded in the html, the only meta data we will obtain are width and height (and any other attribute from the image tag). However, the additional meta data will not be available.
An alternative solution is to extract the images directly from their respective locations by using a regular expression:
Execute monthly at 2AM UTC.
Animal Diversity Web
This is the University of Michigan's online encyclopedia of animal history.
Sample URL:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.