This script is for crawling multiple images on the internet and converting them into a single PDF file.
- Python3, pip
- (Recommended) Create a virtual environment & activate :
python3 -m venv venv
source venv/bin/activate
- Install dependencies with
pip
:
pip install -r requirements.txt
Edit config.yml
in /config
directory to suit your needs. Things you should change are:
-
URL_PREFIX: string URL of the website contains images you want to crawl. (everything except the images' name)
-
FILL_TO: number indicates how images' names are constructed. (for example: FILL_TO: 5 will convert index '1' -> '00001'. This is how the script is currently working. You can change how the script constructs the path to the images by modifying the
get_pic_url()
function incrawl_imgs.py
) -
IMG_EXTENSTION: default is '.jpg'.
-
START_INDEX: number indicates where to start iterating images' names. The default is 1.
-
END_INDEX: total number of images you want to crawl.
-
IMG_DIR: name of the directory to store your crawled images.
-
OUTPUT: name of the final pdf.
-
OUTPUT_DIR: name of the directory to store your final pdf.
python3 run_crawl_and_convert.py
- You can create another
.yml
file in/config
directory and pass it to the script by using the--cfg_file
flag :
python3 run_crawl_and_convert.py --cfg_file=[your-file-name].yml
Both
crawl_imgs.py
andconvert_to_pdf.py
can be run separately.run_crawl_and_convert.py
simply calls both of them.