azavea / nyc-trees Goto Github PK

NYC Parks Trees Count! 2015 tree census

License: GNU Affero General Public License v3.0

Makefile 0.11% Python 43.79% Shell 1.93% JavaScript 23.98% HTML 16.95% CSS 12.58% PLpgSQL 0.35% CartoCSS 0.22% Ruby 0.09%

nyc-trees's Introduction

nyc-trees

NYC Parks Trees Count! 2015 tree census

Local Development

A combination of Vagrant 1.5+ and Ansible 1.8+ is used to setup the development environment for this project. It of the following virtual machines:

app
tiler
services

The app virtual machine contains an instance of the Django application, tiler contains a tiling server, and services contains shared resources of app and tiler:

PostgreSQL
Redis
Logstash
Kibana
Graphite
Statsite

Use the following command to bring up a local development environment:

$ vagrant up

After provisioning is complete, you can login to the application server to execute Django management commands:

$ vagrant ssh app
vagrant@app:~$ envdir /etc/nyc-trees.d/env /opt/app/manage.py test

Note: If you get an error that resembles the following, try logging into the app virtual machine again for the group permissions changes to take effect:

envdir: fatal: unable to switch to directory /etc/nyc-trees.d/env: access denied

Ports

The Vagrant configuration maps the following host ports to services running in the VMs. You can change the host port numbers by setting the environment variables listed in the Env Variable Override column.

Service	Port	URL
Django Web Application	8000	http://localhost:8000
Graphite Dashboard	8080	http://localhost:8080
Kibana Dashboard	15601	http://localhost:15601
Tasseo	15000	http://localhost:15000
PostgreSQL	15432
pgweb	15433	http://localhost:15433
Redis	16379	`redis-cli -h localhost 16379`
livereload	35729	(for gulp watch)
LiveServer Tests	9001	(for Sauce Labs)
Testem	7357

JavaScript and CSS

The main tool used is gulp, which is exposed through npm run to avoid version conflicts.

To create JS bundles, compile sass, and concatenate third-party css, minify CSS and JS, and version files using gulp-rev-all, use gulp or npm run build.
To do the above but skip minification and versioning, use gulp build --debug or npm run build-debug.
To watch files and automatically rebuild the JS or sass files when they change, use gulp watch --debug or npm run watch. This will also start a livereload server.

gulp build and gulp watch without the --debug flag are intentionally not exposed through npm run.

Caching

In order to speed up things up, you may want to consider leveraging the vagrant-cachier plugin. If installed, it is automatically used by Vagrant.

Data

See src/nyc_trees/apps/survey/fixtures/README.md for documentation on how the source data fixtures were created.

Testing

In order to simulate the testing environment used in CI, bring up the testing environment with:

$ VAGRANT_ENV="TEST" vagrant up

Once that is complete, execute the top-level test suite responsible for linting and unit testing the client and server-side components:

$ ./scripts/test.sh

If you want to run the integration tests, use the following command:

$ ./scripts/manage.sh selenium

In addition, other scripts exist if you want to test just one of the client or server-side components.

Deployment

For more details around the Amazon Web Services deployment process, please see the deployment README.

nyc-trees's People

Contributors

Stargazers

Watchers

Forkers

maurizi kdeloach strk 34tth3r1ch kunnareekr redbirdtaiwan

nyc-trees's Issues

How should we handle block segment expiration reminder emails

A few things to consider:

A single, unified method of sending email may preferable to different methods for different app features (account confirmation vs. event registration details vs. block segment expiration)
Reservations can be extended, so we can't just schedule an email to be sent in future and forget about it.

Slug field for Groups

Group profile pages will have unique URLs. Perhaps there should be a slug attribute on the Group model that census administrators can customize? Maybe default to a slugification of the group's name?

If ansible-galaxy is not available the vagrant provisioner should tell you to upgrade

$ vagrant up
Bringing machine 'app' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'services' up with 'virtualbox' provider...
==> app: Importing base box 'ubuntu/trusty64'...
==> app: Matching MAC address for NAT networking...
==> app: Checking if box 'ubuntu/trusty64' is up to date...
==> app: Setting the name of the VM: nyc-trees_app_1412951137456_56042
==> app: Clearing any previously set forwarded ports...
==> app: Fixed port collision for 22 => 2222. Now on port 2200.
==> app: Clearing any previously set network interfaces...
==> app: Preparing network interfaces based on configuration...
    app: Adapter 1: nat
    app: Adapter 2: hostonly
==> app: Forwarding ports...
    app: 80 => 8080 (adapter 1)
    app: 22 => 2200 (adapter 1)
==> app: Booting VM...
==> app: Waiting for machine to boot. This may take a few minutes...
    app: SSH address: 127.0.0.1:2200
    app: SSH username: vagrant
    app: SSH auth method: private key
    app: Warning: Connection timeout. Retrying...
    app: Warning: Remote connection disconnect. Retrying...
==> app: Machine booted and ready!
==> app: Checking for guest additions in VM...
==> app: Configuring and enabling network interfaces...
==> app: Mounting shared folders...
    app: /vagrant => /Users/jwalgran/Projects/nyc-trees
==> app: Running provisioner: ansible...
ERROR: cannot find role in /Users/jwalgran/Projects/nyc-trees/deployment/ansible/roles/azavea.ntp or /Users/jwalgran/Projects/nyc-trees/deployment/ansible/azavea.ntp
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Test runner fails with import error

$ vagrant ssh app -c "envdir /etc/nyc-trees.d/env /opt/app/manage.py test"
Traceback (most recent call last):
  File "/opt/app/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 50, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 284, in run_from_argv
    parser = self.create_parser(argv[0], argv[1])
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 53, in create_parser
    test_runner_class = get_runner(settings, self.test_runner)
  File "/usr/local/lib/python2.7/dist-packages/django/test/utils.py", line 144, in get_runner
    test_module = __import__(test_module_name, {}, {}, force_str(test_path[-1]))
ImportError: No module named discover_runner

User profile: what privacy setting should control visibility of "Tree Mapper"?

The user profile shows "Tree Mapper" if the user has "individual tree mapper" status.

I propose to show it if the user makes their "Achievements" public.

The other choice would be to show it if the user makes their profile public.

The services VM does not create a database during provisioning

Django attempts to connect to the nyc database (https://github.com/azavea/nyc-trees/blob/develop/src/nyc_trees/nyc_trees/settings/development.py#L26-L35), but it does not exist.

Add a dev provisioning option that allows for downloading and restoring a DB backup

Do we need to a the_geom_webmercator column with a 3857 SRID

If this is only a limitation of older windshaft versions, it would be preferable to store data in 4326 or maybe a NY state plane projection and have the tiler reproject to 3857.

Vagrant provisioning fails on initial vagrant up

Provisioning during the initial vagrant up fails due to SSH connection issues (possibly timeout related?).

Running vagrant provision afterwards successfully provisions all three VMs.

Add pgweb to the development services Vagrant VM

A nice GUI for adhoc data spelunking
https://github.com/sosedoff/pgweb

Add UI tests to verify that only group admins and census admins can edit group details

Blockface test data

When the time comes to load blockface test data, the data from the current TreeKit app can be downloaded (in shapefile, CSV, etc) here: https://treekit.cartodb.com/tables/blockface_live/public/map.

Note that the schema of this data may not match the final dataset.

Which browsers should we test with on Sauce Labs?

See https://saucelabs.com/platforms for the full list.

My suggestions:

Samsung Galaxy S3 (Android 4.3 + Android 4.4) (see http://www.appbrain.com/stats/top-android-phones)
iOS 7.1 + iOS 8.2
Windows 7 IE10 + IE11
Windows 7 Firefox 33
Windows 7 Chrome 38
OS X Safari 5

If we can run this but not fail the build:

Windows XP IE8

This list may be too long.
Ultimately we want our integration tests to actually finish within a reasonable amount of time.

vagrant up fails to install specified firefox version

Clone the repo
vagrant up

TASK: [nyc-trees.app | Install Firefox and Xvfb for headless UI tests] ******** 
failed: [app] => (item=firefox=33.0+build2-0ubuntu0.14.04.1) => {"failed": true, "item": "firefox=33.0+build2-0ubuntu0.14.04.1"}
stderr: E: Version '33.0+build2-0ubuntu0.14.04.1' for 'firefox' was not found

stdout: Reading package lists...
Building dependency tree...
Reading state information...

msg: 'apt-get install 'firefox=33.0+build2-0ubuntu0.14.04.1' ' failed: E: Version '33.0+build2-0ubuntu0.14.04.1' for 'firefox' was not found


FATAL: all hosts have already failed -- aborting

Tiler provision fails on nodejs

msg: 'apt-get install 'nodejs=0.10.32-1nodesource1~~trusty1' ' failed: E: Version '0.10.32-1nodesource1~~trusty1' for 'nodejs' was not found

Install psql on the App Vagrant VM

So manage.py dbshell will work.

Run flake8 on Jenkins, mark the build as unstable if it fails

Django dependencies need to be installed

I had to run sudo apt-get install binutils libproj-dev gdal-bin libpq-dev python-dev for Django to install and run

Prevent brute forcing the required quiz with a rate limit

Should we restrict the number of times a user can submit the required quiz in a given time period in order to prevent brute forcing the multiple choice questions?

Vagrant up cannot find nyc-trees.common

$ vagrant up
Bringing machine 'app' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'services' up with 'virtualbox' provider...
==> app: Importing base box 'ubuntu/trusty64'...
==> app: Matching MAC address for NAT networking...
==> app: Checking if box 'ubuntu/trusty64' is up to date...
==> app: Setting the name of the VM: nyc-trees_app_1412888071839_84283
==> app: Clearing any previously set forwarded ports...
==> app: Fixed port collision for 22 => 2222. Now on port 2200.
==> app: Clearing any previously set network interfaces...
==> app: Preparing network interfaces based on configuration...
    app: Adapter 1: nat
    app: Adapter 2: hostonly
==> app: Forwarding ports...
    app: 80 => 8080 (adapter 1)
    app: 22 => 2200 (adapter 1)
==> app: Booting VM...
==> app: Waiting for machine to boot. This may take a few minutes...
    app: SSH address: 127.0.0.1:2200
    app: SSH username: vagrant
    app: SSH auth method: private key
    app: Warning: Connection timeout. Retrying...
    app: Warning: Remote connection disconnect. Retrying...
==> app: Machine booted and ready!
==> app: Checking for guest additions in VM...
==> app: Configuring and enabling network interfaces...
==> app: Mounting shared folders...
    app: /vagrant => /Users/jwalgran/Projects/nyc-trees
==> app: Running provisioner: ansible...
ERROR: cannot find role in /Users/jwalgran/Projects/nyc-trees/deployment/ansible/roles/nyc-trees.common or /Users/jwalgran/Projects/nyc-trees/deployment/ansible/nyc-trees.common
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
[honey nyc-trees (develop)]$

Should we use a third-party Django forms library?

Downsides with the builtin Django forms:

You can't output with <div> easily
~~They use <input type="text" /> for everything, even when there are better HTML5 alternatives, like <input type="email" />~~ This appears to no longer be the case when it comes to email, at least.
They don't set the required attribute for required fields, so not filling in a field leads to a POST-back

Alternatives:

We don't have to make a decision immediately, but we should probably pick something before we start any significant styling work.

Should the data collection workflow include distance from the curb

UI tests are not serving static files

Add opt-in flags to User model

opted_in_to_parks_email boolean default to True
opted_in_to_event_email boolean default to True

Vagrant services VM does not install postgis

Static assets not working with gunicorn

Preferred backup and maintenance windows for data stores

Windows for backups are a range of 30 minutes that can happen on any day. Windows for maintenance are on specific days with an associated 30 minute time window. All times are in UTC.

API parameter examples:

PreferredBackupWindow='05:30-06:30',
PreferredMaintenanceWindow='sun:06:30-sun:07:30',

Add Document model

Just a container for uploading files to be linked to from training materials. Lean on Django Admin to show a copy-pastable link to the files.

Should we use the Django permissions system and admin flags?

We may need to create Permission objects if we want to use the Django Admin site.

We'll need to set either is_staff or is_superuser to allow Census admins access to the Django Admin site. We could use one of them to indicate whether a user is a census administrator and eliminate is_census_admin, to avoid the two flags from getting out of sync.

Periodic gulp-sass build failures

Attempts to build static assets for non-development environments periodically produces the following error:

TASK: [nyc-trees.app | Create static files and run collectstatic (staging/production)] *** 
�[0;31mfailed: [app] => {"changed": true, "cmd": ["npm", "run", "build"], "delta": "0:00:01.479992", "end": "2014-12-04 22:49:44.225961", "rc": 1, "start": "2014-12-04 22:49:42.745969"}�[0m
�[0;31mstderr: 
Error: /var/cache/nyc-trees/node_modules/gulp-sass/node_modules/node-sass/vendor/linux-x64/binding.node: file too short
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/var/cache/nyc-trees/node_modules/gulp-sass/node_modules/node-sass/lib/index.js:181:15)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

npm ERR! [email protected] build: `sudo -u nyc-trees ./node_modules/.bin/gulp`
npm ERR! Exit status 8
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is most likely a problem with the nyc-trees package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     sudo -u nyc-trees ./node_modules/.bin/gulp
npm ERR! You can get their info via:
npm ERR!     npm owner ls nyc-trees
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.13.0-39-generic
npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! cwd /opt/app
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /opt/app/npm-debug.log
npm ERR! not ok code 0�[0m
�[0;31mstdout: 
> [email protected] build /opt/app
> sudo -u nyc-trees ./node_modules/.bin/gulp�[0m
�[0;31m
FATAL: all hosts have already failed -- aborting�[0m

Add is_minor property to User model

If you are under a certain age, you cannot make personal info public.

Cannot install PostGIS

I received the following error trying to provision the services VM:

TASK: [azavea.postgis | Install PostGIS] ************************************** 
failed: [services] => (item=liblwgeom-2.1.2=2.1.2+dfsg-2,postgresql-9.3-postgis-2.1=2.1.2+dfsg-2) => {"failed": true, "item": "liblwgeom-2.1.2=2.1.2+dfsg-2,postgresql-9.3-postgis-2.1=2.1.2+dfsg-2"}
stderr: E: There are problems and -y was used without --force-yes

stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontenc1 libgl1-mesa-dri
  libgl1-mesa-glx libglapi-mesa libice6 libllvm3.4 libpciaccess0 libsm6
  libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0
  libxcb-glx0 libxcb-present0 libxcb-sync1 libxfont1 libxkbfile1 libxmu6
  libxshmfence1 libxt6 libxxf86vm1 x11-common x11-xkb-utils xfonts-base
  xfonts-encodings xfonts-utils xserver-common xserver-xorg-core
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  liblwgeom-2.1.2
The following packages will be DOWNGRADED:
  postgresql-9.3-postgis-2.1
0 upgraded, 1 newly installed, 1 downgraded, 0 to remove and 36 not upgraded.
Need to get 390 kB of archives.
After this operation, 700 kB of additional disk space will be used.

msg: 'apt-get install 'liblwgeom-2.1.2=2.1.2+dfsg-2' 'postgresql-9.3-postgis-2.1=2.1.2+dfsg-2' ' failed: E: There are problems and -y was used without --force-yes

FATAL: all hosts have already failed -- aborting

Feature-gating for end of census

Today's TSM reminded me that at the end of the Census, we'll want to turn off some features. Specifically:

New account registration
Event creation
Tree mapping

Those features should be designed with this in mind.

We'll also need to update the copy on the homepage once the census is over.

services play fails to extract and install Kibana

PLAY [services] *************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [services]

TASK: [azavea.ntp | Install NTP] ********************************************** 
changed: [services]

TASK: [azavea.ntp | Configure NTP] ******************************************** 
changed: [services]

TASK: [azavea.relp | Install RELP support for Rsyslog] ************************ 
changed: [services]

TASK: [azavea.relp | Configure RELP] ****************************************** 
changed: [services]

TASK: [nyc-trees.common | Check if Puppet service definition exists] ********** 
ok: [services]

TASK: [nyc-trees.common | Stop Puppet service] ******************************** 
changed: [services]

TASK: [nyc-trees.common | Remove Puppet service] ****************************** 
changed: [services]

TASK: [azavea.postgresql | Configure the PostgreSQL APT key] ****************** 
changed: [services]

TASK: [azavea.postgresql | Configure the PostgreSQL APT repositories] ********* 
changed: [services]

TASK: [azavea.postgresql | Install PostgreSQL] ******************************** 
changed: [services] => (item=postgresql-9.3=9.3.5-1.pgdg14.04+1,postgresql-server-dev-9.3=9.3.5-1.pgdg14.04+1)

TASK: [azavea.postgresql | Configure PostgreSQL host-based authentication] **** 
changed: [services]

TASK: [azavea.postgresql | Configure PostgreSQL] ****************************** 
changed: [services]

TASK: [azavea.redis | Install Redis] ****************************************** 
changed: [services]

TASK: [azavea.redis | Configure Redis] **************************************** 
changed: [services]

TASK: [azavea.unzip | Install unzip] ****************************************** 
changed: [services]

TASK: [azavea.nginx | Configure the Nginx (Stable) PPA] *********************** 
changed: [services]

TASK: [azavea.nginx | Install Nginx] ****************************************** 
changed: [services]

TASK: [azavea.nginx | Delete default site] ************************************ 
changed: [services]

TASK: [azavea.nginx | Configure Nginx] **************************************** 
changed: [services]

TASK: [azavea.kibana | Download Kibana] *************************************** 
changed: [services]

TASK: [azavea.kibana | Extract and install Kibana] **************************** 
fatal: [services] => input file not found at /usr/local/src/kibana-latest.zip or /usr/local/src/kibana-latest.zip

FATAL: all hosts have already failed -- aborting

Add privacy boolean flags to User model

profile_is_public
real_name_is_public
group_follows_are_public
contributions_are_public
achievements_are_public

Should be non-Null boolean fields that default to False

No pgweb service

I'm trying to use pgweb to look at my database but after reprovisioning I have no /etc/init/pgweb.conf.

Extract Auditable from OpenTreeMap for inclusion in this project

Vagrant app VM cannot access Postgres

django.db.utils.OperationalError: FATAL:  no pg_hba.conf entry for host "33.33.33.10", user "nyc", database "nyc", SSL on
FATAL:  no pg_hba.conf entry for host "33.33.33.10", user "nyc", database "nyc", SSL off

Ansible nginx install fails on brand new vagrant

TASK: [azavea.nginx | Install Nginx] ****************************************** 
failed: [services] => {"failed": true, "item": ""}
stderr: E: Version '1.6.2-2+trusty0' for 'nginx' was not found

stdout: Reading package lists...
Building dependency tree...
Reading state information...

msg: 'apt-get install 'nginx=1.6.2-2+trusty0' ' failed: E: Version '1.6.2-2+trusty0' for 'nginx' was not found


FATAL: all hosts have already failed -- aborting

Should the "turf" table have a different name

This is the colloquial name we have been using to talk about the blockfaces assigned to a group. It is a nice, short table name, but may be too "jargony"

``vagrant up`` fails when running a Postgres instance on the host

I was running Postgres.app on my MBP and I could not vagrant up until I quit.

$ vagrant up
Bringing machine 'services' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'app' up with 'virtualbox' provider...
==> services: Checking if box 'ubuntu/trusty64' is up to date...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 5432 is already in use
on the host machine.

To fix this, modify your current projects Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 5432, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding.

Event emails need an "Unsubscribe" link

Per Jeff, emails to users about events should have an "Unsubscribe to emails about this event" link. If the user clicks it they should be opted out of emails for that event and taken to their profile settings page "Emails" tab, which should display the "you have unsubscribed" message shown in the wireframes.

$ vagrant ssh app -c "envdir /etc/nyc-trees.d/env /opt/app/manage.py runserver 0.0.0.0:8000"
Traceback (most recent call last):
  File "/opt/app/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 46, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 94, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/opt/app/nyc_trees/settings/development.py", line 7, in <module>
    from base import *
  File "/opt/app/nyc_trees/settings/base.py", line 38, in <module>
    'NAME': environ.get('NYC_TREES_DB_NAME', 'nyc_trees'),
NameError: name 'environ' is not defined
Connection to 127.0.0.1 closed.

Model event integration: https://django-statsd.readthedocs.org/en/latest/#django-model-save-and-delete-integration

Depends on #15

Write an Omgeo adapter for the NYC geocoder, if NYC has a geocoder

https://github.com/azavea/python-omgeo