This is a set of scripts that allows for an automatic collection of labelled images from websited supported by the Ripme application. For illustration purposes, the default categories are: dogs, cats and cows. See scripts/source_urls/. For your particular task, you will need to replace the plain text files in that directory with your own: give them approprite names (i.e. names of your classes/categories) and copy urls of image albums that belong to that class into each text file.

Here is what each script (located under scripts directory) does:

• 1_get_urls.sh - iterates through text files under scripts/source_urls downloading URLs of images for each of class/category. The Ripme application performs all the heavy lifting. The source URLs are mostly links to various subreddits, but could be any website supported by Ripme.
• 2_download_from_urls.sh - downloads image files from urls found in text files in raw_data directory.
• 3_create_train.sh - creates data/train directory and copies all *.jpg and *.jpeg files into it from raw_data. Also removes corrupted images.
• 4_create_test.sh - creates data/test directory and moves N=200 random files for each class from data/train to data/test (change this number inside the script if you need a different train/test split). Alternatively, you can run it multiple times, each time it will move N images for each class from data/train to data/test.

• Java runtime environment:
  • debian and ubuntu:sudo apt-get install default-jre
• Linux command line tools: wget, convert (imagemagick suite of tools), rsync, shuf

Change working directory to scripts and execute each script in the sequence indicated by the number in the file name, e.g.:

$ bash 1_get_urls.sh
$ find ../raw_data -name "urls_*.txt" -exec sh -c "echo Number of URLs in {}: ; cat {} | wc -l" \;
Number of URLs in ../raw_data/cows/urls_cows.txt:
739
Number of URLs in ../raw_data/dogs/urls_dogs.txt:
5884
Number of URLs in ../raw_data/cats/urls_cats.txt:
7446
$ bash 2_download_from_urls.sh
$ bash 3_create_train.sh
$ bash 4_create_test.sh
$ cd ../data
$ ls train
cats cows dogs
$ ls test
cats cows dogs