Code Monkey home page Code Monkey logo

friskby's Introduction

FriskBy build status

This web service is part of the FriskBy project.

Air quality measurement devices can POST measurements to the server. The server has a REST api which can be used to query the stored data.

The web service is based on the Python web framework Django: http://www.djangoproject.com.

Getting started

If you want to make modifications to the friskby web-server you must go through some initial setup before you can start hacking. The specific commandlines illustrated in this README assume that you are using a Debian based Linux distribution, but there is nothing Debian or even Linux specific to the software as such.

Installing the dependencies (I)

You need to install the following packages: git, postgresql, postgresql-server-dev-all, python-dev and python-pip:

bash% sudo apt-get install git postgresql postgresql-server-dev-all python-pip python-dev

After pip has been installed, we install all the required Python packages by giving the file requirements.txt to pip:

bash% sudo pip install -r requirements.txt

Setting up the source code

Development of the source code is done on GitHub using a model with personal forks. So to set up your environment for working with the friskby web service:

  1. Create an account on GitHub.

  2. Fork the friskby repository to your personal account. When you have done this you should have a personal repository: https://github.com/$USER/friskby.

  3. Clone your repository down to your personal computer:

    bash% git clone [email protected]:$USER/friskby
  4. Add the FriskByBergen/friskby repository as a remote repository:

    bash% git remote add upstream [email protected]:FriskByBergen/friskby

    This remote repository will be used when you should update your local git repository with the changes done by other developers.

The git/github workflows used by FriskBy are very common, and extensive explanations are only a google search away.

Installing the dependencies (II)

In addition to the binary packages listed above you must also install several Python packages. The Python packages can be installed with normal pip install:

bash% sudo pip install -r requirements.txt

Creating the database

A database for storing the measurements is an essential part of the friskby web service, and to develop on the source code for the web server you need to have your own local database. Observe that Django is quite database agnostic, and you could in principle use MySQL or Sqlite instead of postgres for your own personal development. However the friskby web server currently uses postgres in production, and there is also a possibility that we would like to use postgres extensions to Django in the future. The following guideline is therefor based on postgres:

  1. Change identity to posgtgres:

    bash% sudo su - postgres
  2. Create a new user (role):

    bash% createuser friskby-user -P
    Enter password for new role: <friskby-pwd>
    Enter it again: <friskby-pwd>

    As indicated the createuser program will prompt for a password.

  3. Create a new database - owned by the new user:

    bash% createdb friskby-db --owner=friskby-user

After these steps you should have made a database with name friskby-db and user with credentials (friskby-user, friskby-pwd). These three values should be part of the DATABASE_URL connection string - see the section about environment variables. Log out of the postgres account and test the connection:

bash% psql friskby-db -U friskby-user -h localhost

Environment variables

The configuration of the FriskBy web server is handled through the use of environment variables. In Django configuration settings are read from the settings namespace in the root of the project, in this project the settings/__init__.py file contains several calls of the type:

SETTING_VARIABLE = os.getenv("SETTING_ENV_VARIABLE")

The file init_env.sh.template is a template file which includes the environment variables you should set to run the FriskBy web server. Follow the instructions in this file and create a personal init_env.sh file, the init_env.sh file should not be under version control.

Testing the code

When you updated your environment you are ready to actually run the friskby code. Before starting Django, we need to migrate:

    bash% ./manage.py migrate

To run all the tests:

    bash% ./manage.py test

To start the development server:

   bash% ./manage.py runserver

Then you go to http://127.0.0.1:8000 in your browser and interact with your personal development copy of the FriskBy web server.

Developing and getting the code merged

When you are finished with your changes make a Pull Request on GitHub.

Deploying the code

The FriskBy web server is deployed on Heroku. It should be quite simple to deploy using an alternative platform.

Creating testdata

To get some testdata to work with there are management commands. To create three testsensors with random data:

   bash% ./manage.py add_testdevice TestDev1 TestDev2 TestDev3

This will by default add 100 random datapoints to each of the sensors, but by passing --num=nnn you can add a different number of points. The random devices can be removed with:

   bash% ./manage.py drop_testdevice TestDev1 TestDev3

which will remove the devices 'TestDev1' and 'TestDev3'. If you pass the special argument '--all' to the 'drop_testdevice' managament command all devices will be removed.

In addition to the 'add_testdevice' and 'drop_testdevice' commands there are commands 'add_testdata' and 'drop_testdata' which will only add or drop testdata, not the actual devices.

   bash% ./manage.py add_testdata

Will add 100 datapoints to each available sensor, by passing '--num=' you can control the number of points, and by passing '--device=' or '--sensor=' you can control which sensor gets the data.

   bash% ./manage.py drop_testdata

will drop all the testdata. Pass '--device' or '--sensor'.

Synchronizing with production data

To obtain the production data, follow these steps:

First we let the user friskby-user create databases:

$ sudo -iu postgres
$ psql
> ALTER USER "friskby-user" CREATEDB;
> \q

Then, whenever we want to replace our data with the production data:

$ sudo -iu postgres
$ dropdb friskby-db ; PGUSER=friskby-user PGPASSWORD=fby heroku pg:pull HEROKU_POSTGRESQL_SILVER_URL friskby-db --app friskby

where fby, <friskby-pwd> of course is replaced with your chosen 8+ character password.

friskby's People

Contributors

flikka avatar hjertnes avatar joakim-hove avatar mortalisk avatar njberland avatar oyta avatar pgdr avatar sch005 avatar semafor avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

friskby's Issues

API v1.0

This is a white document for the first attempt of a sane REST(?) API for the friskby data. This issue will be updated continuously as new ideas are proposed.

First, I want to be able to receive

  • latest observation (measurement like (arithmetic) mean, min max, p10, p90, median, std) on a location (e.g. Bryggen), and with one or both of measurement types PM25 and PM10
  • timeseries for the same above, but also with a accumulated period (granularity), e.g. every 3hrs, with possibly a start and end date
  • list of locations?
  • list of measurement types

So, maybe I want to write friskby/api/v1.0/timeseries or friskby/api/v1.0/observation/latest?

WIP Generating primary key for raw_data

WIP for the first step, adding an integerfield primary key to sensor_sensor:

Two steps:

  1. add auto-increment field (but must manually add unique numbers)
  2. move primary key from sensor_id to s_id (the integer field)

Model file:

class Sensor( Model ):
    IDPattern = "[-_:a-zA-Z0-9]+"

    s_id = IntegerField(  )
    sensor_id = CharField("Sensor ID" , max_length = 60 , primary_key = True ,
                          validators = [RegexValidator(regex = "^%s$" % IDPattern)])
    sensor_type = ForeignKey( SensorType )
    parent_device = ForeignKey( Device )
    data_type = ForeignKey( DataType , default = "TEST" )
    description = CharField("Description" , max_length = 256 )
    on_line = BooleanField( default = True )
    last_value = FloatField( null = True , blank = True)
    last_timestamp = DateTimeField( null = True , blank = True) 
...

Migration file:

# -*- coding: utf-8 -*-
# Generated by Django 1.10.3 on 2017-02-19 21:54
from __future__ import unicode_literals

from django.db import migrations, models


def forwards(apps, schema_editor):
    if not schema_editor.connection.alias == 'default':
        return
    sensor = apps.get_model('sensor', 'Sensor')
    s_id = 0
    for row in sensor.objects.all():
        s_id += 1
        row.s_id = s_id
        row.save()


class Migration(migrations.Migration):

    dependencies = [
        ('sensor', '0048_auto_20170219_1705'),
    ]

    operations = [
        migrations.AddField(
            model_name='sensor',
            name='s_id',
            field=models.IntegerField(default=1),
            preserve_default=False,
        ),
        migrations.RunPython(forwards)
    ]

API for new device, new location for given device

I need two things in the API:

  1. If I have COSMO_TOP_SECRET API key, I want to be able to create a new device that comes with two sensors, PM10 and PM25, and perhaps it can use a default Bergen location.
  2. If I have the API key for device d, I want to be able to, via the API, construct a new location based on (name, lat, long, elev) = L and give device d location L.

@joakim-hove could you give a hand here?

This is relating to #188

Attempting to test local friskby fails

Two tests fail:

  • test_get (plot.tests.test_url.PlotUrlTest)
  • ERROR: test_create (plot.tests.test_model.PlotTest)
ERROR: test_get (plot.tests.test_url.PlotUrlTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pgd/friskby/friskby/plot/tests/test_url.py", line 13, in setUp
    self.context = TestContext()
  File "/home/pgd/friskby/friskby/plot/tests/context.py", line 35, in __init__
    post_key = self.key)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 399, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 796, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 824, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 908, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 947, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1045, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 1054, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 337, in execute
    return Database.Cursor.execute(self, query, params)
IntegrityError: NOT NULL constraint failed: sensor_device.owner_id

Making the API easier to set up for users...

The API works nice and except for some confusion regarding what "company" means I find it to be understandable...

However - I guess that it will be very hard for students to set up their units and sensors...
Can we make a "configure your device" page to simplify registration for students?

Timestamps mandatory?

Are timestamps (suppose to be) mandatory? I vagely remember we discussed having the server add the timestamp if it was missing - and if I understand correctly this code tries to achieve it (?): https://github.com/FriskByBergen/friskby/blob/master/sensor/models.py#L25

In any case I would favor it being optional. It is mandatory now:

curl -i -H "Content-type: application/json" -X POST -d '{"sensorid":"ARUtest","value":11,"key":"d6b065e8-42a7-4bd8-90bc-df939d91de99"}' http://friskby.herokuapp.com/sensor/api/reading/

HTTP/1.1 400 BAD REQUEST
"Error: missing fields in payload: ['timestamp']"

Quick Highchart Safari support

Some issue on Safari on macOS and iOS. The map is skewed and there are no lines in the chart. I have not gotten around to finding the root cause yet.

Consider removing the TimeStamp model

It seems to me that we can remove some code and get some cleaner logic if we use raw timestamps.

Is there a good reason to keep the TimeStamp model?

Consistent sorting of devices?

If we sort the devices consistently, perhaps the same devices get more or less the same color each time?

I suggest we sort the devices alphabetically, but place the FriskPi devices above the BG devices.

It should be enough to sort device_rows before returning.

Migrate from heroku

Move the django site from Egypt to Canaan.

In no particular order:

  • Move DNS records (schedule update of clients)
  • Determine portability, e.g. find if postgres is strictly needed
  • Install service deps on new host
  • Figure out configuration (what goes where)
  • Figure out volume mappings
  • Stop old host
  • Install site on new host
  • Push update of clients

Accept posting without timestamp

If I submit the value (value, sensor_id, timestamp) with timestamp = null, I would like to have the server interpret it as now.

Then it is up to the client to not cache values, in a fire-and-forget style.

Remove space character from URLs

The timestamps in the URLs are currently using space between the date and time. Space is not considered valid I believe in the URL (however it is extensively used on the internet). We could change it to ISO8601-std with a T between the date and time? And should the time zone be a part of it?

http://friskby.herokuapp.com/?start=2017-02-23%2023:00:00&time=2017-03-02%2023:00:00&sensortype=PM10
Current URLs

http://friskby.herokuapp.com/?start=2017-02-23T23:00:00+00&time=2017-03-02T23:00:00+00&sensortype=PM10
Proposed format

Create custom raspbian-pi image with friskby pre-installed

Seems it would be nice for both first time setup of new devices, but also for fixing broken systems, if we had a custom raspbian-pi image that could just be downloaded and dd'd onto a flash device.

Seems we'll need to figure out how to:

  1. Build the image. Maybe using raspbian's own code? https://github.com/RPi-Distro/pi-gen
  2. What to include in the custom image, ideally pip and the rpiparticle pip/egg? Maybe some saltstack stuff for remote configuration?
  3. Ensure a transparent build and testing process to ensure the image is trusted by device owners.

Error when attempting to add testdata

After running ./manage.py add_testdata

django.db.utils.ProgrammingError: relation "sensor_device" does not exist
LINE 1: ...device"."locked", "sensor_device"."owner_id" FROM "sensor_de...

Merge friskby.no and http://friskby.herokuapp.com/

friskby.no is informative, but it ignores the fact that there are devices out there, sending data to a server with means for visualization of those data. http://friskby.herokuapp.com/ shows some data, and is super fast and well written, but it doesn't really say anything. If these are to be merged, you'll need to mix the information and data together in a carefully planned way.

Questions friskby.no should answer:

  • Is the Bergen air healthy right now? Could be answered using a trendy gauge where the particle count is normalized, where 100% is very bad and 0% is fine.
  • Is the air healthy, near where I live? Could be answered enumerating all the devices' readings using a smaller version of this gauge.
  • What is the trend like? Is it getting better or worse? Maybe a super simple chart for Bergen.
  • Where are sensors currently placed? Queue dynamic map.
  • What is friskby? Use a heroku-like, icon based grid?
  • How can I contribute? Step by step instructions right there on the front page.
  • Who contributes? Sponsor-grid?

I also think friskby.no should utilize the full width of the viewport, and be so called responsive.

I can implement these changes after a discussion or green light.

Move sensor status API from JS to Django (template)

The business logic part of the sensor status table should live completely in Django (by means of a template).

If we (for some reason) want to inject it into the webpage, we should still let Django create the table completely as HTML and then use JS only to put it into the DOM.

Make index on rawdata.timestamp

We currently select rows primarily based on a time range. It therefore makes sense to put an index on the column rawdata.timestamp_data. Something like this should do:

CREATE INDEX sensor_rawdata_timestamp_data_idx
  ON public.sensor_rawdata
  USING btree
  (timestamp_data DESC NULLS LAST);

This can probably speed up our main queries by up to 30%.

See Optimizing queries on a range of timestamps (two columns) (dba.sx) and
PostgreSQL index not used for query on range (so).

API for location creation needs to be CSRF exempt

Attempting to add locations from the dashboard results in

CSRF cookie not set.

The reason for this is that you should generally only be allowed to POST to friskby from friskby, not arbitrary URIs. However, for this API we need to trust the combination of the device ID and the API key, which means we need to do something like this: http://stackoverflow.com/a/30875830/538866 (trade session auth with the device ID and API key combination).

Dei offisielle sensorane har data fram i tid

Dei offisielle sensorane, t.d. BG_3_PM25, har data fram i tid.

Under er dei siste seks registrerte verdiane for BG_3_PM25. Dei er tatt ut kl 17:15 med GET https://friskby.herokuapp.com/sensor/api/reading/BG_3_PM25/?num=10

Sidan dette er zulu-tid er siste sanne verdi (so sant den er riktig) 2016-10-06T15:00:00Z. Deretter er det hoppa over fleire timar før neste usanne verdi er registrert på 2016-10-06T22:00:00Z.

[
    "2016-10-06T12:00:00Z",
    4.1
],
[
    "2016-10-06T13:00:00Z",
    12.0
],
[
    "2016-10-06T14:00:00Z",
    6.0
],
[
    "2016-10-06T15:00:00Z",
    5.5
],
[
    "2016-10-06T22:00:00Z",
    8.3
],
[
    "2016-10-06T23:00:00Z",
    6.2
]

]

Server error on front end trying to show PM2.5

Environment:

Request Method: GET
Request URL: http://friskby.herokuapp.com/?start=2017-02-24%2023:00:00&time=2017-03-03%2023:00:00&sensortype=PM25

Django Version: 1.10.3
Python Version: 2.7.12
Installed Applications:
('django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'corsheaders',
'rest_framework',
'git_version',
'api_key',
'sensor')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware')

Traceback:

File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
39. response = get_response(request)

File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
249. response = self._get_response(request)

File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)

File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/app/.heroku/python/lib/python2.7/site-packages/django/views/generic/base.py" in view
68. return self.dispatch(request, *args, **kwargs)

File "/app/.heroku/python/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
88. return handler(request, *args, **kwargs)

File "/app/friskby/views/quick.py" in get
87. sensor = Sensor.objects.get( parent_device=d, sensor_type__measurement_type = sensortype )

File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
85. return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py" in get
389. (self.model._meta.object_name, num)

Exception Type: MultipleObjectsReturned at /
Exception Value: get() returned more than one Sensor -- it returned 2!

Setup of new devices

I want to set up a new device, but I want to be inconvenienced as little as possible.

I have a rasperry-pi with a FriskbyImage installed, such that I only need to change one configuration file for “everything to work”.

Suggested solution:

  1. I go to friskby.no, and there's a menu item called "setup new device".
  2. I am taken to a registration page where I enter my email and password.
  3. I am logged in, and see a three step wizard labeled "setup new device".
  4. I enter name.
  5. I enter coordinates (captured from placing a needle on a map?).
  6. A configuration file is presented, which I am to install at /etc/friskby/<device name>.ini.
  7. I click "Register" and I am then taken to a “device page” where I can in some way confirm that my device is now successfully registered, and sending data to friskby.

Notes:

  • Writing the configuration file can be done either by SSH or by using the rasbian GUI, both can be documented.
  • Requires user registration in django, which isn't trivial. How to guard against spam/attacks?
  • New users are non-superusers, and they should only be allowed to view and manage their own sensors.

Would this be sufficient for setting up a new device (and sensors) and have its data logged correctly?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.