Code Monkey home page Code Monkey logo

openness-project-nmid's Introduction

The Openness Project - New Mexico Campaign Finance Site

This site brings greater transparency to New Mexico politics by providing an overview of campaign contributions, expenditures, and the people, businesses, and organizations involved.

Development

Development requires a local installation of Docker and Docker Compose.

To run the application, first build the app image:

docker-compose build

Next, load in data and build your search index:

docker-compose run --rm app python make nightly 
docker-compose run --rm app python manage.py make_search_index

Finally, run the app:

docker-compose up

The app will be available at http://localhost:8000. The database will be exposed on port 32001.

Resetting the database

The data import scripts for this app will automatically recognize if you have data imported, and add or update new data accordingly. However, if you'd like to start over from scratch but don't want to delete your user and page data, you can start by running the flush_camp_fin command to flush campaign finance data from the database:

docker-compose run --rm app python manage.py flush_camp_fin

ETL

The nightly and quarterly ETL scripts are run in a separate repo, through github actions: https://github.com/datamade/nmid-scrapers

Errors / Bugs

If something is not behaving intuitively, it is a bug, and should be reported. Report it here: https://github.com/datamade/openness-project-nmid/issues

openness-project-nmid's People

Contributors

derekeder avatar evz avatar fgregg avatar hancush avatar jeancochrane avatar reginafcompton avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

Forkers

ws-pittman

openness-project-nmid's Issues

Treasurers

It appears you're linking to a search of people when i click on a treasurer but with Thomas Lawley of CVNM Verde Voters Fund it doesn't show any treasurer things for him.

Same with other treasurer's clicked on for PAC. Some clicking reveals it's just a problem with them, it appears, not with candidate treasurers.

Group contributions and expenditures by contact type

It looks like the "contact type" table has some interesting information about what kind of organization is giving money to a candidate or committee or where the candidate or committee is spending money. It might be cool to show totals by election cycle for each of these organization types in a chart on the detail pages. The types that we have are:

State Party Committee
Legislative District Party Committee
Financial Institution
Minor Party Committee
Food/Catering Services
Business
Telecommunication
Office Supplies
Other
County Party Committee
Political Action Committee
Individual
Professional Services
Advertising
Legal
Native American Tribe
Union

content placeholders for CMS

Create static CMS placeholders for the following sections:

Static pages

  • about

Home page

  • intro text
  • top earners (requires #61)
  • largest donations
  • top candidates
  • top committees

Other pages

  • candidates
  • committees

Related to #42

Missing table inventory

Things that we should get:

  • District table (I'm guessing this is the district for which the candidate is running)
  • County table
  • Contact table (This is referred to from the Candidate and PAC tables so I'm guessing it's the contact person)
  • Treasurer table (This is referred to from the PAC and Campaign tables so is probably the treasurer that is listed on the filings)
  • Political Party table
  • Address table
  • A mysterious table that I think is probably just called Status. There are several tables which record some kind of status. This ranges from just "status" to "primary election winner status" and "transaction status". I'm not sure if all these values are stored in one table but that's what it seems like. If nothing else, it'd be nice to know what those status IDs refer to.
  • Election Season table. (This is referred to from the campaign table and might have information about which election the campaign is active for. This will make it possible to line up races between candidates, etc and make it possible to group things for a given election season)

Things that would be nice to have:

  • Division table (perhaps another geography? Not really sure)
  • Filing type table (This is referred to from the filing period table and might have info about the kinds of filings that they get)

Other tables that seem to exist but we don't have:

  • Regular filing period table (This is referred to from the filing period table. Not sure what it refers to)
  • Entity table (This may or may not exist. It seems like this is a unique ID that is unique across PACs, Candidates and Campaigns and allows for joining tables that have info about all three kinds of "entities". It might be cool to see what's in there, though, if they do have that table)

add CSV downloads

CSV downloads should be made available for

  • all donations & expenditures
  • committee & candidate views
  • search result views

site launch checklist

Framing / Call to Action

  • Make sure that the site has a clear call to action. This should not be 'hey look at this cool tool'. Rather, it should be a way for someone to engage in a meaningful way on the issue being presented.

Some examples:

  • Join a mailing list (which means you need to create one)
  • Attend an event
  • Share on Facebook/Twitter
  • Donate to a particular organization
  • Create a Tumblr for people to share interesting findings (if it's a data access site)

Web Search Indexing

  • If you have a staging site, tell the search engine robots not to index you with a robots.txt
  • Make sure you allow indexing when you are ready to launch.

Google Analytics

  • Create Google Analytics account
  • Hook up the GA Tracking Code (typically in our analytics_lib.js file)
  • Set up relevant Goals and Funnels
  • Set up Google Webmaster Tools
  • Verify site in Webmaster Tools with DNS TXT record
  • Link Webmaster Tools to Google Analytics

Sharing & Rich Snippets

  • Set up general meta tags
    • <meta name=“description”>
    • <meta name=“author”>
  • Set up Facebook meta tags & validate here
    • <meta property=“og:site_name”>
    • <meta property=“og:title”>
    • <meta property=“og:type”>
    • <meta property=“og:description”>
    • <meta property=“og:url”>
    • <meta property=“og:image”>
  • Set up Twitter meta tags & validate here
    • <meta name=“twitter:card”>
    • <meta name=“twitter:site”>
    • <meta name=“twitter:creator”>
    • <meta name=“twitter:description”> (note that this needs to be under 200 characters)
    • <meta name=“twitter:title”>
    • <meta name=“twitter:url”>
    • <meta name=“twitter:image:src”>
  • Create 2-5 meme images using Canva or a similar tool

Mobile Friendliness

Test on various mobile devices:

  • scrolling is easy
  • nav bar works
  • hoverable things are tappable
  • charts/maps look ok

Page Speed

Miscellaneous Polish

Load Testing

If your site relies on a database or server-side code, it should use caching and be load tested. If it's a static HTML or Jekyll site, you can skip this section.

Testing

  • Plan for a day of bugfixing. This day should happen after you have added the last features you plan on adding.

GitHub Readme

If the site is open source, make sure the Readme.md is complete and accurate.

  • open source this repo

Here's a few good examples:

The Readme should have the following sections:

  • Overview
  • Setup
  • Running locally
  • Team
  • Errors / Bugs
  • Pull Requests
  • Copyright and License

Outreach

  • Write up press release
  • Identify who will be the primary person of contact and make their info prominent on the website and all PR
  • Identify relevant media contacts to email
  • Identify relevant social news sites (reddit, listservs, slack channels)
  • Divide & conquer - send out the media blitz!

make home page more attractive

The current home page is pretty simple and lifeless and also derivative of Illinois Sunshine.

I'd like to try out adding an image with some text on the home page similar to https://chicago.councilmatic.org/

Here's a few good pictures I've found of the NM State Capitol:

home page content

Home page has been stubbed out with the following sections:

  • Search form
  • Largest donations in the last 90 days
  • Top candidates
  • Top Committees
  • About

Any others we should add?

merge committee and candidate templates

These pages seem to be very similar, despite being about different things.

Currently, the only difference I can see between them is the candidate page has a separate list of offices the person has run for. If this is the case, it would be much more DRY to merge these templates into one.

About/FAQ

Create an enhanced about page that's more FAQ like.

Fish created here

Still needs editing/approval by Trip and Marjorie, suggestions welcome from Datamade folk

Have net funds start with 0 when we have the whole history

For many candidates we have the full history of the campaign contributions. When have this, let's start the net funds from 0 instead of after their first filing.

Here's an example of campaign where we see the first filing. https://nmid.datamade.us/candidates/sarah-maestas-barnes-3080/

Not exactly sure where the 0 should be place on the timeline. Some options

  • The 0 should be placed at the beginning of the filing period of the first recorded filing. Ms. Barnes first report was probably a biannual report made in April 2014, so start put the 0 at October 2013
  • If there is any filing for the creation of the campaign committee, use that.
  • Date of first recorded contribution

Top 25 donors vs. top donations

When i'm looking at Gov. Martinez's page, the top donors for 2014 are ranked (the first year she had donation limits). But if click on largest donations, it gets those big $200K things from 2010. It seems like they should be in the top 25?

widget for top earners

Create a widget for top earners (both candidates and committees) in the last 90 days.

Candidates is misnomer

The listing that we are calling candidates are not really candidates, because we are listing lots of people who are not running for anything, right now. This should be called something like "Politicians" instead.

Show multiple active committees for one candidate

Since it is possible for a candidate to have more than one active committee at a time, let's make sure that we show all of the relevant latest filings. To accomplish this we'll:

If there is more than one filing for the most recent reporting period, show all of them.
Also show the most recent report for all non-final committees.

wrong balances

NM's Future Leaders is shown with a $149.26 balance here but a zero balance here
same with this PAC
PAC West
this guy
this group

anyway if you want a longer list i can do that - i suspect it's something to do with substituting the total expenditures for closing balance when the latter is 0, though it doesn't appear to be the case with all the 0 balance committees.

remove public financing from top donations

screen shot 2016-09-26 at 2 55 12 pm

Public Election Fund and New Mexico Department of Finance indicate public financing. These and entries like them should be removed from the top donations page on the home page.

Older PAC donations

Looking at an older, closed PAC and when i click on largest or all donations getting a spinning wheel. There are no donations shown under the top 25, either, so presumably all donations are pre-2010? Same goes for some other old committees.

Make search do something

I'm going to start with just implementing the things that we had for Illinois Sunshine. If there are any new things that we want to add or old things to modify, lets discuss.

Problem in `import_data`

Importing Candidates.csv
Found 3374 records in Candidates.csv
Updated 0 records in camp_fin_candidate
Traceback (most recent call last):
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: null value in column "full_name" violates not-null constraint
DETAIL:  Failing row contains (57, null, A., B., Valdez, null, null, (505) 753-4018, 2010-01-01 00:00:00-06, dvaldez@cybermesa.com, null, null, 0, 3149, 2478, 1824, 1, null, null).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/django/core/management/base.py", line 305, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/django/core/management/base.py", line 356, in execute
    output = self.handle(*args, **options)
  File "/home/fgregg/public/openness-project-nmid/camp_fin/management/commands/import_data.py", line 112, in handle
    self.doETL(entity_type)
  File "/home/fgregg/public/openness-project-nmid/camp_fin/management/commands/import_data.py", line 162, in doETL
    self.addNewRecords()
  File "/home/fgregg/public/openness-project-nmid/camp_fin/management/commands/import_data.py", line 520, in addNewRecords
    self.executeTransaction(insert_new)
  File "/home/fgregg/public/openness-project-nmid/camp_fin/management/commands/import_data.py", line 668, in executeTransaction
    self.connection.execute(query, *args)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 906, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
    statement, parameters
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/fgregg/public/openness-project-nmid/.env/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in column "full_name" violates not-null constraint
DETAIL:  Failing row contains (57, null, A., B., Valdez, null, null, (505) 753-4018, 2010-01-01 00:00:00-06, dvaldez@cybermesa.com, null, null, 0, 3149, 2478, 1824, 1, null, null).
 [SQL: ' \n            INSERT INTO camp_fin_candidate (\n              id, entity_id, prefix, first_name, last_name, middle_name, suffix, business_phone, home_phone, address_id, status_id, date_added, contact_id, email, date_updated, qual_candidate_id, deceased\n            )\n              SELECT raw."candidateid"::bigint AS id, raw."entityid"::bigint AS entity_id, raw."prefix"::varchar AS prefix, raw."firstname"::varchar AS first_name, raw."lastname"::varchar AS last_name, raw."middlename"::varchar AS middle_name, raw."suffix"::varchar AS suffix, raw."businessphone"::varchar AS business_phone, raw."homephone"::varchar AS home_phone, raw."addressid"::bigint AS address_id, raw."statusid"::bigint AS status_id, raw."dateadded"::date AS date_added, raw."contactid"::bigint AS contact_id, raw."emailaccount"::varchar AS email, raw."datelastupdated"::timestamp with time zone AS date_updated, raw."qualcandidateid"::bigint AS qual_candidate_id, raw."deceased"::varchar AS deceased\n              FROM raw_candidate AS raw\n              JOIN new_candidate AS new\n                ON raw."candidateid" = new.id\n        ']

Improvements for donations and expenditures chart.

  • The hover should just on the center of a period
  • The tooltip should show the entire amount of donations and expenditures in the period (We will need to kind of fudge this when there are overlapping reporting periods.

Data questions (and some answers)

  • Are there unique identifiers for candidates or committees?
  • Are there unique identifiers for transactions?
  • Is there candidate committee information? (the contributions appear to be just made directly to candidates but if you look at the more detailed reports, it's clear that, at least sometimes, they are actually going through a committee) (Answer from @sandrafish: i think there are committee names but they list the candidate stuff by candidates - some candidates also have PACs, which are listed under PAC filings)
  • What about inter-committee transfers?
  • If a candidate makes a mistake and files an amended report, how is that reflected in the data? (Answer from @sandrafish: we should double check with them, but my understanding is that the old data is replaced with the new - there is no indication on what they’re giving us) Discovery from @evz It appears that corrected entries are just appended to the listing of transactions. This means that, if the update is made to a field that we are using to figure out if a record is unique, we'll end up with a duplicate entry. The way this was handled in Illinois is that the entire filing was repeated even if a single correction was made someplace. This added a lot of records that we basically duplicates but it made finding corrections a bit more straightforward.
  • Is there a notion of "filing period" reflected in the data but no notion of a filing with things like beginning and ending balance, debts, investments, etc. (Answer from @sandrafish We need to ask if they could include that in a separate download - it IS included on individual PDFs, but the cash on hand is only available there that i can tell - and the totals that are listed on pages like this one are often incorrect - they sometimes appear not to include in-kind, for example)
  • What about committee officers and other information about committees (address, supported candidate(s), etc)? (Answer from @sandrafish i believe they fill out paper forms to create the committee, tho they could be digital - they are not located only. the PDFs include info about committee treasurers)
  • Is additional information about candidates downloadable? (address, political party, office sought, election cycle, incumbent status, committees, etc)
  • When an individual runs for more than on office in their lifetime, is there any way of telling if they are the same person from the data?

Would be nice to have some reference lines in the visualisations

It could be very nice to have some reference lines in the visualisations to indicate if a person has a lot or a little amount of money for the race they are running for.

Not exactly sure what the right reference point would be. Here are some ideas

  • Median amount of money among other people running for that office now
  • Median amount of money among incumbents for that office
  • Median peak amount of money for people running for that office in last cycle

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.