azavea / nyc-trees Goto Github PK
View Code? Open in Web Editor NEWNYC Parks Trees Count! 2015 tree census
License: GNU Affero General Public License v3.0
NYC Parks Trees Count! 2015 tree census
License: GNU Affero General Public License v3.0
$ vagrant up
Bringing machine 'app' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'services' up with 'virtualbox' provider...
==> app: Importing base box 'ubuntu/trusty64'...
==> app: Matching MAC address for NAT networking...
==> app: Checking if box 'ubuntu/trusty64' is up to date...
==> app: Setting the name of the VM: nyc-trees_app_1412951137456_56042
==> app: Clearing any previously set forwarded ports...
==> app: Fixed port collision for 22 => 2222. Now on port 2200.
==> app: Clearing any previously set network interfaces...
==> app: Preparing network interfaces based on configuration...
app: Adapter 1: nat
app: Adapter 2: hostonly
==> app: Forwarding ports...
app: 80 => 8080 (adapter 1)
app: 22 => 2200 (adapter 1)
==> app: Booting VM...
==> app: Waiting for machine to boot. This may take a few minutes...
app: SSH address: 127.0.0.1:2200
app: SSH username: vagrant
app: SSH auth method: private key
app: Warning: Connection timeout. Retrying...
app: Warning: Remote connection disconnect. Retrying...
==> app: Machine booted and ready!
==> app: Checking for guest additions in VM...
==> app: Configuring and enabling network interfaces...
==> app: Mounting shared folders...
app: /vagrant => /Users/jwalgran/Projects/nyc-trees
==> app: Running provisioner: ansible...
ERROR: cannot find role in /Users/jwalgran/Projects/nyc-trees/deployment/ansible/roles/azavea.ntp or /Users/jwalgran/Projects/nyc-trees/deployment/ansible/azavea.ntp
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
This is the colloquial name we have been using to talk about the blockfaces assigned to a group. It is a nice, short table name, but may be too "jargony"
I was running Postgres.app on my MBP and I could not vagrant up
until I quit.
$ vagrant up
Bringing machine 'services' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'app' up with 'virtualbox' provider...
==> services: Checking if box 'ubuntu/trusty64' is up to date...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 5432 is already in use
on the host machine.
To fix this, modify your current projects Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:
config.vm.network :forwarded_port, guest: 5432, host: 1234
Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding.
If this is only a limitation of older windshaft versions, it would be preferable to store data in 4326 or maybe a NY state plane projection and have the tiler reproject to 3857.
TASK: [azavea.nginx | Install Nginx] ******************************************
failed: [services] => {"failed": true, "item": ""}
stderr: E: Version '1.6.2-2+trusty0' for 'nginx' was not found
stdout: Reading package lists...
Building dependency tree...
Reading state information...
msg: 'apt-get install 'nginx=1.6.2-2+trusty0' ' failed: E: Version '1.6.2-2+trusty0' for 'nginx' was not found
FATAL: all hosts have already failed -- aborting
See https://saucelabs.com/platforms for the full list.
My suggestions:
If we can run this but not fail the build:
This list may be too long.
Ultimately we want our integration tests to actually finish within a reasonable amount of time.
Just a container for uploading files to be linked to from training materials. Lean on Django Admin to show a copy-pastable link to the files.
MultiPolyLine is safer, since it covers both cases, but if we know a blockface will always have one contiguous set of segments, PolyLine is the more correct column type.
Today's TSM reminded me that at the end of the Census, we'll want to turn off some features. Specifically:
Those features should be designed with this in mind.
We'll also need to update the copy on the homepage once the census is over.
opted_in_to_parks_email
boolean default to True
opted_in_to_event_email
boolean default to True
django.db.utils.OperationalError: FATAL: no pg_hba.conf entry for host "33.33.33.10", user "nyc", database "nyc", SSL on
FATAL: no pg_hba.conf entry for host "33.33.33.10", user "nyc", database "nyc", SSL off
$ vagrant up
Bringing machine 'app' up with 'virtualbox' provider...
Bringing machine 'tiler' up with 'virtualbox' provider...
Bringing machine 'services' up with 'virtualbox' provider...
==> app: Importing base box 'ubuntu/trusty64'...
==> app: Matching MAC address for NAT networking...
==> app: Checking if box 'ubuntu/trusty64' is up to date...
==> app: Setting the name of the VM: nyc-trees_app_1412888071839_84283
==> app: Clearing any previously set forwarded ports...
==> app: Fixed port collision for 22 => 2222. Now on port 2200.
==> app: Clearing any previously set network interfaces...
==> app: Preparing network interfaces based on configuration...
app: Adapter 1: nat
app: Adapter 2: hostonly
==> app: Forwarding ports...
app: 80 => 8080 (adapter 1)
app: 22 => 2200 (adapter 1)
==> app: Booting VM...
==> app: Waiting for machine to boot. This may take a few minutes...
app: SSH address: 127.0.0.1:2200
app: SSH username: vagrant
app: SSH auth method: private key
app: Warning: Connection timeout. Retrying...
app: Warning: Remote connection disconnect. Retrying...
==> app: Machine booted and ready!
==> app: Checking for guest additions in VM...
==> app: Configuring and enabling network interfaces...
==> app: Mounting shared folders...
app: /vagrant => /Users/jwalgran/Projects/nyc-trees
==> app: Running provisioner: ansible...
ERROR: cannot find role in /Users/jwalgran/Projects/nyc-trees/deployment/ansible/roles/nyc-trees.common or /Users/jwalgran/Projects/nyc-trees/deployment/ansible/nyc-trees.common
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
[honey nyc-trees (develop)]$
Attempts to build static assets for non-development environments periodically produces the following error:
TASK: [nyc-trees.app | Create static files and run collectstatic (staging/production)] ***
�[0;31mfailed: [app] => {"changed": true, "cmd": ["npm", "run", "build"], "delta": "0:00:01.479992", "end": "2014-12-04 22:49:44.225961", "rc": 1, "start": "2014-12-04 22:49:42.745969"}�[0m
�[0;31mstderr:
Error: /var/cache/nyc-trees/node_modules/gulp-sass/node_modules/node-sass/vendor/linux-x64/binding.node: file too short
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/var/cache/nyc-trees/node_modules/gulp-sass/node_modules/node-sass/lib/index.js:181:15)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
npm ERR! [email protected] build: `sudo -u nyc-trees ./node_modules/.bin/gulp`
npm ERR! Exit status 8
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is most likely a problem with the nyc-trees package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! sudo -u nyc-trees ./node_modules/.bin/gulp
npm ERR! You can get their info via:
npm ERR! npm owner ls nyc-trees
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.13.0-39-generic
npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! cwd /opt/app
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /opt/app/npm-debug.log
npm ERR! not ok code 0�[0m
�[0;31mstdout:
> [email protected] build /opt/app
> sudo -u nyc-trees ./node_modules/.bin/gulp�[0m
�[0;31m
FATAL: all hosts have already failed -- aborting�[0m
Group profile pages will have unique URLs. Perhaps there should be a slug
attribute on the Group
model that census administrators can customize? Maybe default to a slugification of the group's name?
I received the following error trying to provision the services VM:
TASK: [azavea.postgis | Install PostGIS] **************************************
failed: [services] => (item=liblwgeom-2.1.2=2.1.2+dfsg-2,postgresql-9.3-postgis-2.1=2.1.2+dfsg-2) => {"failed": true, "item": "liblwgeom-2.1.2=2.1.2+dfsg-2,postgresql-9.3-postgis-2.1=2.1.2+dfsg-2"}
stderr: E: There are problems and -y was used without --force-yes
stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontenc1 libgl1-mesa-dri
libgl1-mesa-glx libglapi-mesa libice6 libllvm3.4 libpciaccess0 libsm6
libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0
libxcb-glx0 libxcb-present0 libxcb-sync1 libxfont1 libxkbfile1 libxmu6
libxshmfence1 libxt6 libxxf86vm1 x11-common x11-xkb-utils xfonts-base
xfonts-encodings xfonts-utils xserver-common xserver-xorg-core
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
liblwgeom-2.1.2
The following packages will be DOWNGRADED:
postgresql-9.3-postgis-2.1
0 upgraded, 1 newly installed, 1 downgraded, 0 to remove and 36 not upgraded.
Need to get 390 kB of archives.
After this operation, 700 kB of additional disk space will be used.
msg: 'apt-get install 'liblwgeom-2.1.2=2.1.2+dfsg-2' 'postgresql-9.3-postgis-2.1=2.1.2+dfsg-2' ' failed: E: There are problems and -y was used without --force-yes
FATAL: all hosts have already failed -- aborting
We may need to create Permission
objects if we want to use the Django Admin site.
We'll need to set either is_staff
or is_superuser
to allow Census admins access to the Django Admin site. We could use one of them to indicate whether a user is a census administrator and eliminate is_census_admin
, to avoid the two flags from getting out of sync.
I had to run sudo apt-get install binutils libproj-dev gdal-bin libpq-dev python-dev
for Django to install and run
profile_is_public
real_name_is_public
group_follows_are_public
contributions_are_public
achievements_are_public
Should be non-Null boolean fields that default to False
The user profile shows "Tree Mapper" if the user has "individual tree mapper" status.
I propose to show it if the user makes their "Achievements" public.
The other choice would be to show it if the user makes their profile public.
$ vagrant ssh app -c "envdir /etc/nyc-trees.d/env /opt/app/manage.py runserver 0.0.0.0:8000"
Traceback (most recent call last):
File "/opt/app/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
settings.INSTALLED_APPS
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 46, in __getattr__
self._setup(name)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 94, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/opt/app/nyc_trees/settings/development.py", line 7, in <module>
from base import *
File "/opt/app/nyc_trees/settings/base.py", line 38, in <module>
'NAME': environ.get('NYC_TREES_DB_NAME', 'nyc_trees'),
NameError: name 'environ' is not defined
Connection to 127.0.0.1 closed.
Wire up Curator to purge ElasticSearch/Logstash indexes that are older than ~30 days.
$ vagrant ssh app -c "envdir /etc/nyc-trees.d/env /opt/app/manage.py test"
Traceback (most recent call last):
File "/opt/app/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 50, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 284, in run_from_argv
parser = self.create_parser(argv[0], argv[1])
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 53, in create_parser
test_runner_class = get_runner(settings, self.test_runner)
File "/usr/local/lib/python2.7/dist-packages/django/test/utils.py", line 144, in get_runner
test_module = __import__(test_module_name, {}, {}, force_str(test_path[-1]))
ImportError: No module named discover_runner
Installation: https://django-statsd.readthedocs.org/en/latest/#installation
Request timing middleware: https://django-statsd.readthedocs.org/en/latest/#usage
Model event integration: https://django-statsd.readthedocs.org/en/latest/#django-model-save-and-delete-integration
Depends on #15
If you are under a certain age, you cannot make personal info public.
Django attempts to connect to the nyc
database (https://github.com/azavea/nyc-trees/blob/develop/src/nyc_trees/nyc_trees/settings/development.py#L26-L35), but it does not exist.
I'm trying to use pgweb
to look at my database but after reprovisioning I have no /etc/init/pgweb.conf
.
PLAY [services] ***************************************************************
GATHERING FACTS ***************************************************************
ok: [services]
TASK: [azavea.ntp | Install NTP] **********************************************
changed: [services]
TASK: [azavea.ntp | Configure NTP] ********************************************
changed: [services]
TASK: [azavea.relp | Install RELP support for Rsyslog] ************************
changed: [services]
TASK: [azavea.relp | Configure RELP] ******************************************
changed: [services]
TASK: [nyc-trees.common | Check if Puppet service definition exists] **********
ok: [services]
TASK: [nyc-trees.common | Stop Puppet service] ********************************
changed: [services]
TASK: [nyc-trees.common | Remove Puppet service] ******************************
changed: [services]
TASK: [azavea.postgresql | Configure the PostgreSQL APT key] ******************
changed: [services]
TASK: [azavea.postgresql | Configure the PostgreSQL APT repositories] *********
changed: [services]
TASK: [azavea.postgresql | Install PostgreSQL] ********************************
changed: [services] => (item=postgresql-9.3=9.3.5-1.pgdg14.04+1,postgresql-server-dev-9.3=9.3.5-1.pgdg14.04+1)
TASK: [azavea.postgresql | Configure PostgreSQL host-based authentication] ****
changed: [services]
TASK: [azavea.postgresql | Configure PostgreSQL] ******************************
changed: [services]
TASK: [azavea.redis | Install Redis] ******************************************
changed: [services]
TASK: [azavea.redis | Configure Redis] ****************************************
changed: [services]
TASK: [azavea.unzip | Install unzip] ******************************************
changed: [services]
TASK: [azavea.nginx | Configure the Nginx (Stable) PPA] ***********************
changed: [services]
TASK: [azavea.nginx | Install Nginx] ******************************************
changed: [services]
TASK: [azavea.nginx | Delete default site] ************************************
changed: [services]
TASK: [azavea.nginx | Configure Nginx] ****************************************
changed: [services]
TASK: [azavea.kibana | Download Kibana] ***************************************
changed: [services]
TASK: [azavea.kibana | Extract and install Kibana] ****************************
fatal: [services] => input file not found at /usr/local/src/kibana-latest.zip or /usr/local/src/kibana-latest.zip
FATAL: all hosts have already failed -- aborting
Downsides with the builtin Django forms:
<div>
easily<input type="text" />
for everything, even when there are better HTML5 alternatives, like <input type="email" />
required
attribute for required fields, so not filling in a field leads to a POST-backAlternatives:
We don't have to make a decision immediately, but we should probably pick something before we start any significant styling work.
msg: 'apt-get install 'nodejs=0.10.32-1nodesource1trusty1' ' failed: E: Version '0.10.32-1nodesource1trusty1' for 'nodejs' was not found
Should we restrict the number of times a user can submit the required quiz in a given time period in order to prevent brute forcing the multiple choice questions?
vagrant up
TASK: [nyc-trees.app | Install Firefox and Xvfb for headless UI tests] ********
failed: [app] => (item=firefox=33.0+build2-0ubuntu0.14.04.1) => {"failed": true, "item": "firefox=33.0+build2-0ubuntu0.14.04.1"}
stderr: E: Version '33.0+build2-0ubuntu0.14.04.1' for 'firefox' was not found
stdout: Reading package lists...
Building dependency tree...
Reading state information...
msg: 'apt-get install 'firefox=33.0+build2-0ubuntu0.14.04.1' ' failed: E: Version '33.0+build2-0ubuntu0.14.04.1' for 'firefox' was not found
FATAL: all hosts have already failed -- aborting
Provisioning during the initial vagrant up
fails due to SSH connection issues (possibly timeout related?).
Running vagrant provision
afterwards successfully provisions all three VMs.
Windows for backups are a range of 30 minutes that can happen on any day. Windows for maintenance are on specific days with an associated 30 minute time window. All times are in UTC.
API parameter examples:
PreferredBackupWindow='05:30-06:30',
PreferredMaintenanceWindow='sun:06:30-sun:07:30',
Per Jeff, emails to users about events should have an "Unsubscribe to emails about this event" link. If the user clicks it they should be opted out of emails for that event and taken to their profile settings page "Emails" tab, which should display the "you have unsubscribed" message shown in the wireframes.
So manage.py dbshell
will work.
A nice GUI for adhoc data spelunking
https://github.com/sosedoff/pgweb
A few things to consider:
When the time comes to load blockface test data, the data from the current TreeKit app can be downloaded (in shapefile, CSV, etc) here: https://treekit.cartodb.com/tables/blockface_live/public/map.
Note that the schema of this data may not match the final dataset.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.