Scrapers

This is going to be a repository for random scrapers I've come to develop over the years. These scripts have been published in the intention of data preservation.

Description

All scrapers in this repository with a brief description

• artic - Grabs art along with metadata from artic.edu
• bensound - Grabs royalty music from bensound.com
• betterttv - Grabs emotes from betterttv.tv
• calibre-server - Grabs ebooks from a calibre-server
• dafont - Grabs fonts uploaded to dafont.com
• iconmonstr - Grabs icons in all formats from iconmonstr.com
• impawards - Grabs all movie posters from impawards.com
• linfoxdomain - Grabs all flash games from linfoxdomain.com
• memoryoftheworld - Grabs all the books from memoryoftheworld.org
• shzm - Grabs the top 50 songs by city from shazam.com
• thecoverproject - Grabs all the game posters from thecoverproject.net
• theoatmeal - Grabs comics from theoatmeal.com
• unicode-emoji-chart - Grabs emoji and metadata from unicode.org
• vulnhub - Downloads items and all metadata on each item from vulnhub.com
• wallhaven - Grabs all the images for a given search query from wallhaven.cc
• wallpaperflare - Grabs all or some wallpapers with search from wallpaperflare.com
• waset - Grabs papers published to waset.org
• xkcd - Grabs comics posted to xkcd.com
• zenpencil - Grabs all comics from zenpencils.com

Others

Scrapers in other repositories

• acloud-dl by r0oth3x49 - A cross-platform python based utility to download courses from acloud.guru for personal offline use

• allitebooks-downloader by thomasbrueggemann - 📚 Scrapes and downloads all IT eBooks http://allitebooks.com

• ArchiveBot by ArchiveTeam - ArchiveBot, an IRC bot for archiving websites

• archivebot-archives by nonPointer - This repository containes a list of files in the ArchiveBot Collection on the Internet Archive and the corresponding codes.

• archivers by nektro - A collection of scripts to mass download data from various sites

• ArchiveTools by recrm - A collection of tools for archiving and analysing the internet

• Automated-ISO-ripping by pascaldulieu - Short bash script to automatically rip ISOs

• awesome-dl by Kickball - This is a list of repositories and libraries that allow for scripted downloading of online content

• BBCSoundDownloader by FThompson - Bulk downloader for http://bbcsfx.acropolis.org.uk/.

• Bios-Archival-Standards by BiosPlus - A collection of gists and notes to help me [BiosPlus] standardize my archival efforts accross the board

• comics-downloader by Girbons - Command-line tool to download comics and manga in pdf/epub/cbr/cbz from a website

• Discord-Channel-scraper by simon987 - Scrapes an arbitrary number of lines from a Discord channel

• download_scholar_pdfs by bozelosp - Batch .PDF downloading from a list of DOIs and/or titles. PDF files are retrieved/download from libgen scholar archives.

• DownCloud by seru1us - Download SoundCloud tracks posted on targetted subreddits

• e621.net-file-downloader by OllieTails - Scrapes images from e621.net (nfsw)

• e621Crawler by fionera - Scrapes images from e621.net (nfsw)

• flickr-search-scraper by AlexOwen - Script to capture a JSON file of all search results along with the best version of each image in the search

• github-scraper - 🕷️ 🕸️ crawl GitHub web pages for insights we can't GET from the API... 💡

• GOGGames_Crawler by fionera - Hacky Crawler for GoodOldDownloads

• goodolddownloads-deobf by nabijaczleweli - Deobfuscate https://goggames.goodolddownloads.com/ links

• grab-site by ArchiveTeam - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

• humblebundle-downloader by diogogmt - Download book from Humble Bundles

• instagram-scraper by rarcega - Scrapes an instagram user's photos and videos

• InstaHoarder by beaston02 - Auto save instagram users stories and posts

• lazynlp by chiphuyen - Library to scrape and clean web pages to create massive datasets

• Lynx-Program-Downloader by spideyclick - This is a bash shell script that downloads programs from websites using Lynx.

• Mega.nz-IDM-downloader by CHEF-KOCH - How to download from Mega.nz with IDM - Unlimited

• Misc-Download-Scripts by simon987 - Miscellaneous download scripts

• Music-Hoarders-Bot by JPBotelho - Discord bot written in python for the music hoarders server

• PixivDownloader by nonPointer - Simple batch tool to download one's image from Pixiv

• RedditImageBackup by LameLemon - Grabs all images, gifs, videos and text posts from Reddit

• redditPostArchiver by pl77 - Easily archive important Reddit post threads onto your computer

• scrapereplacementdocs by Itxaka - Scrapy spider to download pdfs from replacementdocs.com

• Tidown by Transcodes - A simple, yet efficient, Tital downloader.

• tpget by 0x6a73 - Tutorialspoint downloader

• twitter-scraper by kennethreitz - Scrape the Twitter Frontend API without authentication.

• uDownloader by usbpc - 1Fichier Folder Downloader written in Kotlin with download rate limit and stuff

• wayback-machine-downloader by hartator - Download an entire website from the Wayback Machine