Code Monkey home page Code Monkey logo

suckit's Introduction

Build and test codecov Crates.io Docs Deps License License MSRV

SuckIT

SuckIT allows you to recursively visit and download a website's content to your disk.

SuckIT Logo

Features

  • Vacuums the entirety of a website recursively
  • Uses multithreading
  • Writes the website's content to your disk
  • Enables offline navigation
  • Offers random delays to avoid IP banning
  • Saves application state on CTRL-C for later pickup

Options

Option Behavior
-h, --help Displays help information
-v, --verbose Activate Verbose output
-d, --depth Specify the level of depth to go to when visiting the website. Default is -1 (infinity)
--ext-depth Specify the level of depth to go to when visiting websites that have a different domain name. Default is 0 (ignore external links), -1 is infinity
-j, --jobs Number of threads to use
-o, --output Output directory where the downloaded files are written
-t, --tries Number of times to retry when the downloading of a page fails
-u, --user-agent User agent to be used for sending requests
-i, --include Specify a regex to include pages that match this pattern
-e, --exclude Specify a regex to exclude pages that match this pattern
-a, --auth Provide usernames and passwords for the downloader to use
--dry-run Do everything without saving the files to the disk

Example

A common use case could be the following:

suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/

asciicast

Installation

As of right now, SuckIT does not work on Windows.

To install it, you need to have Rust installed.

  • Check out this link for instructions on how to install Rust.

  • If you just want to install the suckit executable, you can simply run cargo install --git https://github.com/skallwar/suckit

  • Now, run it from anywhere with the suckit command.

Arch Linux

suckit can be installed from available AUR packages using an AUR helper. For example,

yay -S suckit

Want to contribute ? Feel free to open an issue or submit a PR !

License

SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)

See LICENSE-APACHE and LICENSE-MIT for details.

suckit's People

Contributors

skallwar avatar cohenarthur avatar dependabot[bot] avatar pinkforest avatar untitaker avatar lhvy avatar atul9 avatar pjsier avatar creativcoder avatar spyrosroum avatar orhun avatar bastiengermond avatar waxymocha avatar jaslogic avatar marchellodev avatar mr-bo-jangles avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.