leemendelowitz / dcmetrometrics Goto Github PK

Create a data visualization of escalator outages where we color code a day red if there was an escalator outage that day, green otherwise.

We've created snapshots of the dynamic pages for google to index. The links in the snapshots do not work however.

Solution would be to add a rewrite rule to nginx to add the .html if it's missing to any location which points to the snapshots directory.

This hot car report was ignored because it had the word "cool":

https://twitter.com/thisisbossi/status/570940112697233408

We should create a utility for manually adding hot car reports to the database.

Use logrotate to rotate the dcmetrometrics logs.

Units should be sorted by unit id on a station page.

For example, it's currently out of order for this station:
http://www.dcmetrometrics.com/#/stations/Greensboro

A unit which transitioned to Modernization status did not get an update type properly assigned:

http://www.dcmetrometrics.com/#/units/E05X01ESCALATOR

Sparkline

Each station should have a list of recent updates to its escalators/elevators, so we can quickly see what has changed recently at a glance without having to go to each units individual page.

This is similar to the list of recent updates that appears on the homepage, but filtered down to a particular station.

It would be really cool to create an outage animation where we dynamically show outages on a map. We could create an animation for outages using d3 similar to this flight visualization:

http://www.tnoda.com/blog/2014-04-02

Every break, fix, off, on status could emit a growing, fading colored circle at a station (red, green, orange, blue) for example.

A civic.json file provides information about this project that's not included in the Github metadata. The full details of the format are here. Just put it in the root of your repo and we'll automatically display that info on the projects page.

Code for DC also has a tool to help put the file together here, and a validator to make sure everything's in the right place here.

(Also, since Code for DC is tracking this project, any issues tagged with help wanted will show up on http://codefordc.org/projects/helpwanted.html. If you are stuck on something, try that!)

Add a toggle which shows or hides escalators descriptions.

Also, a tooltip over the escalator id which shows it's currrent status would be awesome.

While iterating over a cursor, writes can result in documents being returned multiple times. When computing performance summary, we might want to isolate cursors by taking snapshots on statuses because it would be safer.

Investigate using snapshots at various points in the application.

http://docs.mongodb.org/manual/faq/developers/#faq-developers-isolate-cursors

Add a "active streaks" to the escalators/elevators page. This would be summaries of the longest current outages.

It should include the total time of the outage, not just of the most recent status. For example, "Callback/Repair -> minor repair" should be one outage comprising multiple statuses.

Add support for MetroHotCars to accept 7000 series cars.

Need to edit this:
https://github.com/LeeMendelowitz/DCMetroMetrics/blob/master/dcmetrometrics/hotcars/process_tweets.py#L141

Outage class does not generate the full list of outage days when the outage is still ongoing. The generated json for the unit's performance summary does not include the outage days of an active outage, and as a result the "Daily Escalator Outages" visualization has some stale information.

For example, if an escalator breaks on Jan 1 and is fixed Jan 5th, the outage days of Jan 1, 2, 3, 4, 5 don't appear in the visualization until the outage is over.

http://www.washingtonian.com/2016/03/01/washington-dc-metro-escalator-outages/

This tweet from @MetroTAG was a manual RT which added hashtags #wmata #hotcar:
https://twitter.com/MetroTAG/status/369791434234134528

However, it was automatically ignored because it had the word "RT".

However, the original tweet from @OrangleLineInfo was not captured by @MetroHotCars because it was missing the #wmata #hotcar hashtags:
https://twitter.com/OrangeLineInfo/status/369789591953616896

Adding such a filter would allow us to search for street escalators for example

Each station page should have a list of elevators.

We captured this as a hot car report, but it's not.

https://twitter.com/heatherwgaona/status/525450603217309696

This is important for the ELES App, where we have separate documents for the Unit and UnitStatus.
In particular, the Unit's KeyStatuses embedded document references UnitStatus, and they get saved separately and can get out of sync potentially.

http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/

The Angular time filter which should say "1 m" or "2h 5m" doesn't properly work for times less than 1 minutes.

Add this to the press page:
http://fivethirtyeight.com/features/washington-d-c-s-metro-catches-fire-more-than-four-times-a-week/

The end_time is set on some unit's latest status. The end_time should only be set when the status is followed by another status.

Add code to the utils to recompute key statuses, and then remove the end time from any lastStatus if end_time has been set.

It's not clear how the end_time on the lastStatus was set in the first place, but it may have happened when bad code was committed and deployed, and an Exception was thrown on a status change when the old status's end_time was set but before properly saving the new status.

The time stamp on the recent outages does not work on Firefox. Instead of "20m ago" it just says "ago".

Move KeyStatuses and UnitPerformanceSummary into the Unit object. A unit can only have a single KeyStatuses and a single UnitPerformanceSummary document, so it is probably easier to embed them in a single document.

The UnitStatuses should probably remain separate from the Unit document, because occasionally we query for UnitStatuses by time period across all units.

It's worth considering this.

A search for "Dupont street" for dupont street escalators will return no results right now. Using or would allow us to return Dupont street escalators!

We need to start using Angular UI-router for nested views. Some examples of use cases:

• On outages page, show the escalator outages or elevator outages pane.
• On station page, show the escalator, elevator, or recent changes pane.
• On a unit page, it's summary statistics, or it's full status history.

This way we have a url for the outages page, or a station page

https://github.com/angular-ui/ui-router

This means the search order will be in the URL, and this will help share the rankings page by link.

There is an issue with timezones in Ubuntu linux.

dateutil.zoneinfo.gettz("America/New_York")

returns None. This needs to be investigated further.

Running the utils/generateAllWebPages.py script produces this issue.

This exception is being thrown when parsing outage data. Perhaps WMATA is providing malformed data about a station.

We should better handle bad quality data coming from WMATA API.

2015-10-08 07:37:43,254 - ELESApp - ERROR - Traceback:
Traceback (most recent call last):
  File "/home/lmendelo/dcmetrometrics/repo/ELESAppRunner.py", line 66, in _run
    self.app.tick()
  File "/home/lmendelo/dcmetrometrics/repo/dcmetrometrics/eles/ELESApp.py", line 165, in tick
    incidents = getELESIncidents()
  File "/home/lmendelo/dcmetrometrics/repo/dcmetrometrics/eles/ELESApp.py", line 91, in getELESIncidents
    incidents = [Incident(i) for i in incidents]
  File "/home/lmendelo/dcmetrometrics/repo/dcmetrometrics/eles/Incident.py", line 51, in __init__
    self.addAttr()
  File "/home/lmendelo/dcmetrometrics/repo/dcmetrometrics/eles/Incident.py", line 68, in addAttr
    if ',' in stationName:
TypeError: argument of type 'NoneType' is not iterable

run_daily_service_reports.py does more work than necessary for units whose most recent status is a long running operational status. This should be fixed to only go back in time for units whose most recent status is a long running non-operational status.