demarsch / gun-violence-dashboard Goto Github PK

Create main page of the application and add simple map of the US using Leaflet.js. Take a look at this tutorial. The map doesn't have to be a real map, a schematic is enough

Add a search bar to the main page, that will allow to search through the list of statistics. A statistic should consist of its name and source. Typing in the search bar should filter the list of available statistics

• Extract x-axis and z-axis data from DB and join it with regular data
• Add configuration of widget header
• Add filter by state
• Add filter by year
• Return 'Male'/ 'Female' for gender
• Add 'Top'/'Bottom' limits (N/A, not sure what to do with multipivots)
• Add extra statistics
• Scrape incident list from gun-violence archive
• Restyle check buttons
• Add large 'Add Widget' button if there are no widgets yet
• Sunburst chart OR x-y pivot chart

Right now if widget can't be built using the specified configuration, it will show a firebrick death sign with the explanation in the tooltip. We need to make widget types to react on the configuration change and report if they can be built using the specific configuration.

Implement this for each widget type - besides render method, widget type should have canRender method that returns object like this: { canRender: true } or { canRender:false, reason: 'Map chart can be built only if data is pivoted by state' }

Create new start page that will clarify the purpose of our site, describe the source of data we've used and allow user to go to the dashboard

Do it after #9 is closed. Create Python script data.py that will hold all the SQLAlchemy classes related to the database interaction. Add a function there that takes a dictionary of values (this dictionary will come either from Kaggle dataset CSV file or from our gunviolencearchive web scraper). This function should insert a record into the Incidents table and the required number of records into IncidentCategories (and Categories if some of them don't exist yet) and Participants.

Create a widget that will be used to show a vertical bar chart (single axis, one or two pivots)

Create an SQLIte database file and add the following tables to it:

1. Incidents

• id: integer, identity, primary key // It has nothing to do with the Id of the incident from Kaggle (we won't store it)
• date: date,
• state: varchar(2) //Keep the abbreviation
• city_or_county: varchar(100),
• n_killed: number,
• n_injured: number,
• latitude: number(7, 4),
• longitude: number(7, 4);

2. Participants

• id: integer, identity, primary key,
• incident_id: integer, foreign key for Incidents.id column
• age: varchar(10),
• gender: varchar(10),
• type: varchar(20),
• status: varchar(20)

3. Categories

• id: integer, identity, primary key,
• name: varchar(100)

4. IncidentCategories

• incident_id: integer, foreign key for Incidents.id column,
• category_id: integer, foreign key for Category.id column

5. Statistics

• id: integer, identity, primary key
• name: varchar(500),
• source: varchar(500)

6. StatisticsData

• id: integer, identity, primary key
• statistics_id: integer, foreign key for Statistics.id column
• state: varchar(2) //State abbreviation
• value: number(18, 6) //For now lets assume that all stats will be numbers (optionally, with fractional part)

Create a modal window to configure widget settings. At minimum this window should allow user to filter data by state and year

We need to introduce user sessions and add ability to login (at 1st iteration it may be some predefined list of users). Optionally we could allow to log in using OAuth providers from Facebook, Twitter and Google

Once #29 and #30 are completed, we need to prepare infrastructure to allows users to save, load, modify and delete their dashboard. For the 1st iteration we could allow 5 dashboard per user

Create a widget that will be used to show a pie chart for 2-axis and 2-axis multivalue data layouts

Create a widget that will be used to show a histogram for 2-axis multi value layout

We need to have a Python script that removes all columns we don't require from the Kaggle dataset CSV. The CSV can be downloaded from here.

1. Assume that this file is stored in the same directory as your script will be

2. Read the CSV using pandas

3. Leave only the columns date, state, n_killed, n_injured, incident_characteristics, latitude, longitude, participant_age, participant_gender, participant_status and participant_type

4. Save the clean dataframe to CSV clean_data.csv that is located inside the same directory as your script

Add widget configuration button in the widget header (to the left of 'Maximize' button). This button should bring on screen a widget settings dialog with all values set to those that were used for widget creation

Add an 'Export' button to the widget header that should allow user to download the data used for widget creation (allow to pick the format: JSON or CSV for now)

Add dropdown with checkbox list to the main page. The list should contain all the US states. When state checkbox is checked/unchecked the respective state on the map should be painted in white

The window will appear after clicking the button Add Widget. It should allow user to specify the following details:

• What numeric value will serve as an Y axis (Num. of Incidents, Num. of Killed, Num. of Injured)
• (Optional) Victims, grouped by gender or age
• (Optional) Suspects, grouped by gender or age
• Inclusive filter by category
• Exclusive filter by category
• Top X and Bottom X filters
• X axis selector (dropdown with the list of stats)
• Chart type selector (load the list of available widgets)

Create a widget that will be used to show a bubble chart (for 3-axis data layout) or a scatter plot (for 2-axis data layout). Use belly-button-biodiversity homework as a guide

We need to have a couple of CSV that will contain gun-legislation related data grouped by year (2014-2018) and state. The data should have the following columns:

• State //Abbreviation only
• Year
• Value //See notes below

If you can make a Jupyter Notebook that builds this dataset on the fly from some API, it would be preferable, but if not, just download CSV and save it to the data folder in the repository.

Notes: any kind of data per state/year can be useful. Some candidates are:

• Gun friendliness rating
• Number of legally owned/registered guns
• CCW allowed/restricted (1 for allowed, 0 for restricted)
• Background check required for gun purchase (1 for required, 0 for not required)
• etc

If a data doesn't change from year to year, then keep the same values

Right now all our data is located right in the repository and as long as our database is read-only it is fine. But if we plan to add data in live mode, the database should be located somewhere else. Check if there are free options to host MySQL database somewhere and if there are some, use it and create a Jupyter notebook to export data from our SQLite file and import it into MySQL.

The MySQL heroku add-ons likely are not viable options because their free version only allow to store up to 5 MB of data while our incidents DB is about 30 MB

We need to have a CSV that will contain poverty/median income (or both) data grouped by year (2014-2018) and state. The data should have the following columns:

• State //Abbreviation only
• Year
• Poverty % / Median Income
  If you can make a Jupyter Notebook that builds this dataset on the fly from some API, it would be preferrable, but if not, just download CSV and save it to the data folder in the repository

Use PEG.js library to generate a simple grammar parser that should be able to parse simple arythmetic expressions performed on variables and numeric constants e.g. a / b * 100

For the first version, we need to have RESTful service with two endpoints:

• api/v1/stats which should return the JSON containing the list of all stats in the following format

[{
    'id': 123456,
    'name': 'Population'
    'source' : {
        'id': 123,
        'name' : 'Census'
    },
    'years': [
        2010,
        2011,
        2012,
        2013,
        2014
    ]
}]

• api/v1/data?stat=stat_id&state=states&year=years that returns the data for specific query where:
  stat_id - integer Id of stat
  states - string that contains a single state code (e.g. CA) or a comma-separated list of state codes (e.g. CA,TX,PA)
  year - 4-digit integer that represents the year (e.g. 2018), or a comma-separated list of 4-digit integers (e.g. 2015,2018) or two increasing 4-digit integers separated with hyphen (e.g. 2015-2018)
  The response should be the JSON, example is below:

{
    'id': 123456,
    'name': 'Population',,
    'source' : {
        'id': 123,
        'name' : 'Census'
    },
    'states': [
        {
            'code': 'CA',
            'name': 'California',
            'data': [
                {
                    'year': 2014,
                    'value': 100500
                }
            ]
        }
    ]
}

• Add filter by state
• Add filter by year
• Add widget title

Add Category to the list of possible break down properties and implement grouping by category server-side

We need to have a CSV that will contain population data grouped by year (2014-2018) and state. The data should have the following columns:

• State //Abbreviation only
• Year
• Population
  If you can make a Jupyter Notebook that builds this dataset on the fly from some API, it would be preferrable, but if not, just download CSV and save it to the data folder in the repository

We need to have a script that will scrape gunviolencearchive.org web site and save the data on gun incidents to CSV. Note that huge part of the data is already available on Kaggle, so start scraping from the first incident that is not present in that Kaggle dataset