Awesome list of digital preservation tools

• Web Archiving
• Social networks
• Other digital objects
• Standards and specifications
• Organizations
• Major digital archives
• Knowledge bases
• Related lists

• Wget - a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.
• WPull - Wget-compatible web downloader and crawler.
• Conifer - collect and revisit web pages
• grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
• Heritrix3 - Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
• WAIL - Web Archiving Integration Layer: One-Click User Instigated Preservation

• Browsetrix Crawler - run a high-fidelity browser-based crawler in a single Docker container

• Archive Web.page - A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers
• Reply Web.page - Serverless Web Archive Replay directly in the browser
• pywb - Core Python Web Archiving Toolkit for replay and recording of web archives
• webrecorder-player - Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
• ipwb - InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

• AUT - The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
• AUT Notebooks - Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.
• WARCIO - Streaming WARC/ARC library for fast web archive IO
• Metawarc - Metadata extractor from WARC files
• WarcDB - WarcDB: Web crawl data as SQLite databases
• ArchiveSpark - An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
• CDX Toolkit - A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

• ArchiveBox - Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more
• Wayback - A self-hosted toolkit for archiving webpages to the Internet Archive, archive.today, IPFS, and local file systems
• Archivenow - A Tool To Push Web Resources Into Web Archives
• iagitup - A command line tool to archive a git repository from GitHub to the Internet Archive.

• ArchiveIt - web archiving online services

• twarc - A command line tool (and Python library) for archiving Twitter JSON

• instaloader - Download pictures (or videos) along with their captions and other metadata from Instagram.

• sfm-ui - Social Feed Manager user interface application.
• Media downloader - download Instagram Reels, Stories, Post, Stalk Instagram Profile, Facebook Public Videos, YouTube Videos and YouTube to MP3 converter, SoundCloud MP3 and Dailymotion videos. Made from Node JS Express JS, React JS and Rapid API.

• ydiskarc - command-line tool to backup public resources from Yandex.disk (disk.yandex.ru / yadi.sk) filestorage service
• filegetter - A command-line tool to collect files from public data sources using URL patterns and config files

• tgarc - A command line tool for archiving Telegram JSON

• wparc - Wordpress API data and files archival command line tool
• spcrawler - A command-line tool to backup Sharepoint public installations data from open API endpoint

• apibackuper - Python library and cmd tool to backup API calls

• The WARC Format 1.1 - The Web ARChive (WARC) archive format specifies a method for combining multiple digital resources into an aggregate archive file together with related information.
• CDX File format - format of CDX files, that are list of files in WARC files
• WARC Specifications - collection of WARC related specifications and formats
• The WACZ Format 1.1.1 - Web Archive Collection Zipped. WACZ is a media type that allows web archive collections to be packaged and shared on the web as a discrete file.

• Digital preservation coalition - The DPC is a not-for-profit company dedicated to digital preservation inititatives
• International Internet Preservation Consortium - Leading consortium for web archiving

• Archiveteam Wiki - Wiki about various archival topics and file formats

• Internet Archive - biggest digital archive with big web archives
• Common Crawl - open data search engine index crawled from whole Internet

• Awesome Web Archiving - An Awesome List for getting started with web archiving
• Awesome data takeout - An Awesome Data Takeout list of services to take out your personal data from major online services and providers