Image crawler

• Background service
• RESTful API
• Crawl Search URL for product URLs based on search criteria
• Actual images should not be downloaded, store all image URLs, remove any duplicate images
• Sets of images should be kept together
• URLs should be stored without duplicates
• Needs to scale across distributed hosts

1. go get github.com/PuerkitoBio/goquery
2. go get golang.org/x/net/html
3. go get github.com/go-redis/redis

go build

./crawd

Command line flags override crawld.json

There are 2 sets of URLs tracked:

1. product page URL
2. product image URL

URLs must be persisted such that multiple runs of crawld does not cause redundant crawling.

This essentially stores all the product URLs found by the crawler, this should be persisted over time so if this tool is run multiple times, state can be resumed. Lookups to see if this URL has been accessed already should be fast. The schema for this store should be simple, basically store the URL with no duplicates.

We use 2 datatypes for this, redis set and redis list.

This stores all the image URLs, there should be no duplicates, sets of images should be grouped together. Should differentiate thumbnail from full size images. Each image URL should allow a simple rating.

1. Highest rating, based on categories
2. Latest photos