This project crawls Dribbble continuously to keep record of popular designs and archive them by day. It also offers them over an API, RSS and daily newsletters.
The web interface runs at dailybbble.herokuapp.com on Heroku platform.
Crawler runs as an executable Python daemon at file fetcher.py
. It runs
continuously to retrieve data from Dribbble. You can use supervisor
to
keep this process alive.
In addition you can send daily/weekly emails newsletters by scheduling cron jobs (one runs every morning, one every Saturday noon) with commands
python -m dailybbble.emailer daily
python -m dailybbble.emailer weekly
See notes below installing these tasks on Heroku Scheduler.
Microsoft Azure Table Storage is used as database. Therefore you need to initialize enviornment variables
AZURE_ACCOUNT_NAME
AZURE_ACCOUNT_KEY
AZURE_TABLE_NAME
(where shots are going to be stored)
Build the Docker image:
$ docker build -t dailybbble-fetcher -f Dockerfile-fetcher .
Run the container:
$ docker run -d --restart=always \
-e AZURE_ACCOUNT_NAME='<paste_account_here>' \
-e AZURE_TABLE_NAME='dailybbble' \
-e AZURE_ACCOUNT_KEY='<paste_key_here>'
--name dailybbble_crawler \
dailybbble-fetcher
Check if it is running: docker logs -f dailybbble_crawler
.
In addition, for e-mail subscription the following environment variables are needed from SendGrid service:
SENDGRID_USERNAME
: account or API user name as in https://sendgrid.com/credentialsSENDGRID_PASSWORD
: account password or API keySENDGRID_LIST_NAMES
: comma separated names of 2 recipient lists for daily and weekly subscriptions (better you don't use commas while creating list names)SENDGRID_SENDER_NAME
: identity name of registered sender
To disable email sending (for instance if you ran out of money recently like I did), set environment variable:
DISABLE_EMAIL
: to1
and it will be hidden from the UI.
For making use of memcache
caching, configure the following
environment variables (auto-installed with Heroku Memcachier plugin):
MEMCACHIER_USERNAME
: SASL auth (if any)MEMCACHIER_PASSWORD
: SASL auth (if any)MEMCACHIER_SERVERS
: comma separated list of cache servers
You can use $ heroku config:set KEY=VALUE
to persistently set environment
on Heroku app.
For using Heroku Scheduler addon, here's the configuration:
Daily emails: every day at 9 AM PST:
Task: python -m dailybbble.emailer daily
Frequency: Daily
Next Run: 16:00 UTC
Weekly emails: every Saturday at 11 AM PST:
Task: ruby -e 'if Time.now.utc.wday != 6; exit 1; end' && python -m dailybbble.emailer weekly
Frequency: Daily
Next Run: 18:00 UTC
(To run scheduler dashboard, run heroku addons:open scheduler
.)
Copyright 2013, Ahmet Alp Balkan
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.