Copyright 2013 NPR. All rights reserved. No part of these materials may be reproduced, modified, stored in a retrieval system, or retransmitted, in any form or by any means, electronic, mechanical or otherwise, without prior written permission from NPR.
(Want to use this code? Send an email to [email protected]!)
- About this template
- Assumptions
- What's in here?
- Configure the project
- Tumblog setup
- Install requirements
- Project secrets
- Bootstrap issues
- Adding a template/view
- Run the project locally
- Editing workflow
- Run Javascript tests
- Run Python tests
- Compile static assets
- Test the rendered app
- Deploy the app
- Deploy to S3
- Deploy to EC2
- Install cron jobs
- Install web services
- Deploy Tumblr themes
This template provides a a project skeleton suitable for any project that is to be served entirely as flat files. Facilities are provided for rendering html from data, compiling LESS into CSS, deploying to S3, installing cron jobs on servers, copy-editing via Google Spreadsheets and a whole raft of other stuff.
Please note: This project is not intended to be a generic solution. We strongly encourage those who love the app-template to use it as a basis for their own project template. We have no plans to remove NPR-specific code from this project.
The init-tumblr
branch provides an HTML form for embedding and a mechanism to crowdsource user-submitted content via a Tumblog.
The following things are assumed to be true in this documentation.
- You are running OSX.
- You are using Python 2.7. (Probably the version that came OSX.)
- You have virtualenv and virtualenvwrapper installed and working.
The project contains the following folders and important files:
confs
-- Server configuration files for nginx and uwsgi. Edit the templates thenfab <ENV> render_confs
, don't edit anything inconfs/rendered
directly.data
-- Data files, such as those used to generate HTML.etc
-- Miscellaneous scripts and metadata for project bootstrapping.jst
-- Javascript (Underscore.js) templates.less
-- LESS files, will be compiled to CSS and concatenated for deployment.templates
-- HTML (Jinja2) templates, to be compiled locally.tests
-- Python unit tests.www
-- Static and compiled assets to be deployed. (a.k.a. "the output")www/live-data
-- "Live" data deployed to S3 via cron jobs or other mechanisms. (Not deployed with the rest of the project.)www/test
-- Javascript tests and supporting files.app.py
-- A Flask app for rendering the project locally.app_config.py
-- Global project configuration for scripts, deployment, etc.copytext.py
-- Code supporting the Editing workflowcrontab
-- Cron jobs to be installed as part of the project.fabfile.py
-- Fabric commands automating setup and deployment.public_app.py
-- A Flask app for running server-side code.render_utils.py
-- Code supporting template rendering.requirements.txt
-- Python requirements.
Edit app_config.py
and update PROJECT_NAME
, DEPLOYED_NAME
, REPOSITORY_NAME
any other relevant configuration details.
NPR apps-specific: Update relevant environment variables with API keys, etc., from Dropbox:
tumblog-setup
You can either source ~/Dropbox/nprapps/tumblr_oauth_keys.txt
(variables will only be set for the life of your current shell):
. ~/Dropbox/nprapps/tumblr_oauth_keys.txt
Or you can append its contents to your bash_profile|bashrc|zshrc:
cat ~/Dropbox/nprapps/tumblr_oauth_keys.txt >> ~/.zshrc
Log into our NPR apps tumblr account and create two new blogs:
- $NEW_TUMBLOG_NAME.tumblr.com (this will be our production blog)
- $NEW_TUMBLOG_NAME-staging.tumblr.com (this will be our staging blog)
Copy and paste our custom Tumblr theme into Tumblr's blog editor.
Navigate to "customize blog" and enter our Disqus shortname (found in Dropbox) under "Disqus shortname."
Node.js is required for the static asset pipeline. If you don't already have it, get it like this:
brew install node
curl https://npmjs.org/install.sh | sh
Then install the project requirements:
cd changinglives
npm install
mkvirtualenv changinglives
ln -s /usr/local/include/freetype2 /usr/local/include/freetype
pip install pil --allow-external pil --allow-unverified pil
pip install -r requirements.txt
Project secrets should never be stored in app_config.py
or anywhere else in the repository. They will be leaked to the client if you do. Instead, always store passwords, keys, etc. in environment variables and document that they are needed here in the README.
The app-template can automatically setup your Github repo with our default labels and tickets by running fab bootstrap_issues
. You will be prompted for your Github username and password.
A site can have any number of rendered templates (i.e. pages). Each will need a corresponding view. To create a new one:
- Add a template to the
templates
directory. Ensure it extends_base.html
. - Add a corresponding view function to
app.py
. Decorate it with a route to the page name, i.e.@app.route('/filename.html')
- By convention only views that end with
.html
and do not start with_
will automatically be rendered when you callfab render
.
A flask app is used to run the project locally. It will automatically recompile templates and assets on demand.
workon changinglives
python app.py
Visit localhost:8000 in your browser.
On the init-tumblr
branch, app.py
is used for the Tumblr form which will be baked out to a flat file that can then be embedded in the tumblog.
To run the simple app that will post to Tumblr (via an intermediate server for image uplaods):
python public_app.py
This will run the simple app on localhost:8001.
The app is rigged up to Google Docs for a simple key/value store that provides an editing workflow.
View the sample copy spreadsheet here. A few things to note:
- If there is a column called
key
, there is expected to be a column calledvalue
and rows will be accessed in templates as key/value pairs - Rows may also be accessed in templates by row index using iterators (see below)
- You may have any number of worksheets
- This document must be "published to the web" using Google Docs' interface
This document is specified in app_config
with the variable COPY_GOOGLE_DOC_KEY
. To use your own spreadsheet, change this value to reflect your document's key (found in the Google Docs URL after &key=
).
The app template is outfitted with a few fab
utility functions that make pulling changes and updating your local data easy.
To update the latest document, simply run:
fab update_copy
Note: update_copy
runs automatically whenever fab render
is called.
At the template level, Jinja maintains a COPY
object that you can use to access your values in the templates. Using our example sheet, to use the byline
key in templates/index.html
:
{{ COPY.attribution.byline }}
More generally, you can access anything defined in your Google Doc like so:
{{ COPY.sheet_name.key_name }}
You may also access rows using iterators. In this case, the column headers of the spreadsheet become keys and the row cells values. For example:
{% for row in COPY.sheet_name %}
{{ row.column_one_header }}
{{ row.column_two_header }}
{% endfor %}
With the project running, visit localhost:8000/test/SpecRunner.html.
Python unit tests are stored in the tests
directory. Run them with fab tests
.
Compile LESS to CSS, compile javascript templates to Javascript and minify all assets:
workon changinglives
fab render
(This is done automatically whenever you deploy to S3.)
If you want to test the app once you've rendered it out, just use the Python webserver:
cd www
python -m SimpleHTTPServer
There are two parts โ deploy the form to S3 and the simple app (public_app.py
) to EC2. See the next two sections for more details on S3 and EC2, but a high level workflow is:
- Run
fab <ENV> master setup
to configure the server (where<ENV>
is either staging or production). - Run
fab <ENV> master deploy
to deploy the app. - Run
fab <ENV> deploy_confs
to render the server conf files (nginx and uwsgi) and then deploy them to the server. This will also restart the app on the server.
fab staging master deploy
You can deploy to EC2 for a variety of reasons. We cover two cases: Running a dynamic Web application and executing cron jobs.
For running a Web application:
- In
fabfile.py
setenv.deploy_to_servers
toTrue
. - Also in
fabfile.py
setenv.deploy_web_services
toTrue
. - Run
fab staging master setup
to configure the server. - Run
fab staging master deploy
to deploy the app.
For running cron jobs:
- In
fabfile.py
setenv.deploy_to_servers
toTrue
. - Also in
fabfile.py
, setenv.install_crontab
toTrue
. - Run
fab staging master setup
to configure the server. - Run
fab staging master deploy
to deploy the app.
You can configure your EC2 instance to both run Web services and execute cron jobs; just set both environment variables in the fabfile.
Note: The Knockout-29.otf
font must be manually installed on the servers at /home/ubuntu/apps/changing-lives/repository/Knockout-29.otf
. It can not be installed automatically due to licensing restrictions.
Cron jobs are defined in the file crontab
. Each task should use the cron.sh
shim to ensure the project's virtualenv is properly activated prior to execution. For example:
* * * * * ubuntu bash /home/ubuntu/apps/$PROJECT_NAME/repository/cron.sh fab $DEPLOYMENT_TARGET cron_test
Note: In this example you will need to replace $PROJECT_NAME
with your actual deployed project name.
To install your crontab set env.install_crontab
to True
at the top of fabfile.py
. Cron jobs will be automatically installed each time you deploy to EC2.
Web services are configured in the confs/
folder. Currently, there are two: nginx.conf
and uwsgi.conf
.
Running fab setup
will deploy your confs if you have set env.deploy_to_servers
and env.deploy_web_services
both to True
at the top of fabfile.py
.
To check that these files are being properly rendered, you can render them locally and see the results in the confs/rendered/
directory.
fab render_confs
You can also deploy the configuration files independently of the setup command by running:
fab deploy_confs
The base Tumblr theme is in the tumblr/theme.html
file. There are included templates in templates/
: _form.html
, _prompt.html
and _social.html
. There are also two new constants in app_config.py
: PROJECT_CREDITS
and PROJECT_SHORTLINK
. These should be changed on a per-project basis.
To deploy a Tumblr theme with local URLs, type fab staging copy_theme
and the theme will be copied to your clipboard to paste into Tumblr. Note: This only works on Mac OSX. On Linux or Windows, use fab staging write_theme
and manually copy the theme HTML from tumblr/rendered-theme.html
.
To deploy a Tumblr theme with production URLs, type fab production copy_theme
and the theme will be copied to your clipbaord to paste into Tumblr. Note: This only works on Mac OSX. On Linux or Windows, use fab production write_theme
and manually copy the theme HTML from tumblr/rendered-theme.html
.