Code Monkey home page Code Monkey logo

ha-gtfs-rt-v2's Introduction

  • 👋 Hi, I’m @nocalla
  • 👀 I’m interested in ...
  • 🌱 I’m currently learning ...
  • 💞️ I’m looking to collaborate on ...
  • 📫 How to reach me ...

ha-gtfs-rt-v2's People

Contributors

mark1foley avatar nocalla avatar phardy avatar tdickman avatar zacs avatar

ha-gtfs-rt-v2's Issues

Failing to get route details from National Transport Ireland API

The following config gets the INFO:root:Input file configuration is valid. message when running test.py, but the sensor is not getting the details correctly. I've confirmed that the route and stopid occur in the log file.

trip_update_url: "https://api.nationaltransport.ie/gtfsr/v2/TripUpdates"
vehicle_position_url: "https://api.nationaltransport.ie/gtfsr/v2/Vehicles"
x_api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
departures:
  - name: "test 1"
    route: "2974_45181"
    stopid: "8370B2434301"
    directionid: "0"
    service_type: Bus
  - name: "test 2"
    route: "2974_45181"
    stopid: "8370B2432301"
    directionid: "1"
    service_type: Bus

Anatomy of the log file

Initial setup data

INFO:root:Input file configuration is valid.
INFO:__main__:Adding Sensor: Name: test 1, route id: 2974_45181, direction id: 8370B2434301
INFO:__main__:trip_update_url: https://api.nationaltransport.ie/gtfsr/v2/TripUpdates
INFO:__main__:vehicle_position_url: https://api.nationaltransport.ie/gtfsr/v2/Vehicles
INFO:__main__:route_delimiter: None
INFO:__main__:header: {'x-api-key': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.nationaltransport.ie:443
DEBUG:urllib3.connectionpool:https://api.nationaltransport.ie:443 "GET /gtfsr/v2/Vehicles HTTP/1.1" 200 88131

Vehicle position data

This shows the current position of each individual bus (no route or stop data).

INFO:__main__:Successfully updated vehicle positions - 200
DEBUG:__main__:......Adding position for trip id 2961_143 position latitude 53.33244323730469 longitude -6.567302227020264
DEBUG:__main__:......Adding position for trip id 2961_442 position latitude 53.34862518310547 longitude -6.983066082000732
DEBUG:__main__:......Adding position for trip id 2799_2405 position latitude 52.24595260620117 longitude -7.143592834472656
...

Trip Updates

This seems to show at what point in the trip each bus is at. I'm not sure what "Stop Time" means - is that the delay?

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.nationaltransport.ie:443
DEBUG:urllib3.connectionpool:https://api.nationaltransport.ie:443 "GET /gtfsr/v2/TripUpdates HTTP/1.1" 200 905006
INFO:__main__:Successfully updated trip data - 200
DEBUG:__main__:...Received Trip Id 2974_32055 Route Id: 2974_45138 direction id 1 Start Time: 14:10:00 Start Date: 20230320
DEBUG:__main__:......Stop: 8530B6358301 Stop Sequence: 25 Stop Time: 0
DEBUG:__main__:......Stop: 8530B158221 Stop Sequence: 26 Stop Time: 0
DEBUG:__main__:......Stop: 8530B1558401 Stop Sequence: 27 Stop Time: 0
DEBUG:__main__:......Stop: 8530B1559601 Stop Sequence: 28 Stop Time: 0
DEBUG:__main__:......Stop: 8530B158211 Stop Sequence: 29 Stop Time: 0
DEBUG:__main__:......Stop: 8530B1581501 Stop Sequence: 30 Stop Time: 0
DEBUG:__main__:...Received Trip Id 2974_31539 Route Id: 2974_45135 direction id 0 Start Time: 14:25:00 Start Date: 20230320
DEBUG:__main__:......Stop: 8470B556161 Stop Sequence: 21 Stop Time: 0
DEBUG:__main__:...Received Trip Id 2974_19916 Route Id: 2974_45132 direction id 1 Start Time: 14:30:00 Start Date: 20230320
DEBUG:__main__:......Stop: 8240B111911 Stop Sequence: 11 Stop Time: 0
DEBUG:__main__:......Stop: 8220B135001 Stop Sequence: 12 Stop Time: 0
...

There are very occasionally routes with non-zero Stop Times, but the vast majority are 0. I suspect that this may be the cause of this issue as per mark1foley#8.

DEBUG:__main__:...Received Trip Id  Route Id: 2974_45265 direction id 0 Start Time: 21:30:00 Start Date: 20230320
DEBUG:__main__:......Stop: 8460B525641 Stop Sequence: 1 Stop Time: 1679347800
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B525641, stop time 1679347800
DEBUG:__main__:......Stop: 8460B5232101 Stop Sequence: 2 Stop Time: 1679347875
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5232101, stop time 1679347875
DEBUG:__main__:......Stop: 8460B5232301 Stop Sequence: 3 Stop Time: 1679347965
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5232301, stop time 1679347965
DEBUG:__main__:......Stop: 8460B5232401 Stop Sequence: 4 Stop Time: 1679348010
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5232401, stop time 1679348010
DEBUG:__main__:......Stop: 8460B5232501 Stop Sequence: 5 Stop Time: 1679348025
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5232501, stop time 1679348025
DEBUG:__main__:......Stop: 8460B5226901 Stop Sequence: 6 Stop Time: 1679348160
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5226901, stop time 1679348160
DEBUG:__main__:......Stop: 8460B5227001 Stop Sequence: 7 Stop Time: 1679348220
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5227001, stop time 1679348220
DEBUG:__main__:......Stop: 8460B5241301 Stop Sequence: 8 Stop Time: 1679348295
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241301, stop time 1679348295
DEBUG:__main__:......Stop: 8460B5241401 Stop Sequence: 9 Stop Time: 1679348370
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241401, stop time 1679348370
DEBUG:__main__:......Stop: 8460B5228101 Stop Sequence: 10 Stop Time: 1679348415
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5228101, stop time 1679348415
DEBUG:__main__:......Stop: 8460B5241501 Stop Sequence: 11 Stop Time: 1679348550
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241501, stop time 1679348550
DEBUG:__main__:......Stop: 8460B5241701 Stop Sequence: 12 Stop Time: 1679348625
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241701, stop time 1679348625
DEBUG:__main__:......Stop: 8460B5241801 Stop Sequence: 13 Stop Time: 1679348670
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241801, stop time 1679348670
DEBUG:__main__:......Stop: 8460B5241901 Stop Sequence: 14 Stop Time: 1679348775
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5241901, stop time 1679348775
DEBUG:__main__:......Stop: 8460B5242001 Stop Sequence: 15 Stop Time: 1679348820
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5242001, stop time 1679348820
DEBUG:__main__:......Stop: 8460B5242101 Stop Sequence: 16 Stop Time: 1679348850
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5242101, stop time 1679348850
DEBUG:__main__:......Stop: 8460B5242201 Stop Sequence: 17 Stop Time: 1679348880
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5242201, stop time 1679348880
DEBUG:__main__:......Stop: 8460B5242301 Stop Sequence: 18 Stop Time: 1679348925
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5242301, stop time 1679348925
DEBUG:__main__:......Stop: 8460B5242401 Stop Sequence: 19 Stop Time: 1679348955
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5242401, stop time 1679348955
DEBUG:__main__:......Stop: 8460B5234201 Stop Sequence: 20 Stop Time: 1679349135
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5234201, stop time 1679349135
DEBUG:__main__:......Stop: 8460B5234301 Stop Sequence: 21 Stop Time: 1679349180
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B5234301, stop time 1679349180
DEBUG:__main__:......Stop: 8460B555431 Stop Sequence: 22 Stop Time: 1679349210
DEBUG:__main__:.........Adding route id 2974_45265, trip id , direction id 0, stop id 8460B555431, stop time 1679349210

Sensor Status

All of the parameters are marked as undefined.

INFO:__main__:Sensor Update:
INFO:__main__:...Name: test 1
INFO:__main__:...Route: 2974_45181
INFO:__main__:...Stop ID: 8370B2434301
INFO:__main__:...Direction ID: 0
INFO:__main__:...Icon: mdi:bus
INFO:__main__:...Service Type: Bus
INFO:__main__:...unit_of_measurement: min
INFO:__main__:...Due in: -
INFO:__main__:...Due at not defined
INFO:__main__:...Latitude not defined
INFO:__main__:...Longitude not defined
INFO:__main__:...Next Bus not defined

Debugging

  • I added a debug print to the _get_next_services function and it returned the following dictionary of Stop IDs mapped to empty lists:
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
{'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '838370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}   {'1': {'8380B109711': [], '8380B351691': [], '8370B2847901': [], '8370B2406901': [], '8370B243881': [], '8370B240181': [], '8370B248041': [], '8370B241701': [], '8380B109741': [], '8380B2433801': [], '8380B2433901': [], '8370B2965001': [], '8370B244491': [], '8370B244501': [], '8370B243341': [], '8380B2433601': []}, '0': {'8370B2432501': [], '8380B2432901': [], '8380B2433001': [], '8380B5590901': [], '8380B237781': [], '8380B237791': [], '8380B288861': [], '8370B2431701': [], '8370B243491': [], '8370B2430601': [], '8370B2899701': [], '8370B2432401': [], '8380B2432801': [], '8380B109741': [], '8380B109761': [], '8380B212051': [], '8370B2126301': [], '8370B2405001': [], '8370B4289701': [], '8370B2430501': []}}
  • I tried using one of the routes with published Stop Times, but the result was unchanged.
  • I went directly to the API endpoint test page to see what came back in the JSON data - it looks like there are Stop Times present there. Perhaps there's an issue with how the API formats Stop Times? There's reference to a "delay" parameter and there is also a "timestamp" on every Trip entry.
HTTP/1.1 200 OK

content-length: 3932784
content-type: application/json
vary: Origin

{
    "header": {
        "gtfs_realtime_version": "2.0",
        "incrementality": "FULL_DATASET",
        "timestamp": "1679345003"
    },
    "entity": [{
            "id": "T1",
            "trip_update": {
                "trip": {
                    "start_time": "14:45:00",
                    "start_date": "20230320",
                    "schedule_relationship": "ADDED",
                    "route_id": "2974_45133",
                    "direction_id": 0
                },
                "stop_time_update": [{
                    "stop_sequence": 16,
                    "arrival": {
                        "time": "1679337900"
                    },
                    "stop_id": "8530B1558401"
                }],
                "timestamp": "1679345003"
            }
        },
        {
            "id": "T2",
            "trip_update": {
                "trip": {
                    "trip_id": "2974_23566",
                    "start_time": "15:20:00",
                    "start_date": "20230320",
                    "schedule_relationship": "SCHEDULED",
                    "route_id": "2974_45133",
                    "direction_id": 1
                },
                "stop_time_update": [{
                    "stop_sequence": 16,
                    "arrival": {
                        "delay": 4126
                    },
                    "stop_id": "8220B135001",
                    "schedule_relationship": "SCHEDULED"
                }],
                "vehicle": {
                    "id": "1280"
                },
                "timestamp": "1679345003"
            }
        },
        {
            "id": "T3",
            "trip_update": {
                "trip": {
                    "trip_id": "2974_31546",
                    "start_time": "15:25:00",
                    "start_date": "20230320",
                    "schedule_relationship": "SCHEDULED",
                    "route_id": "2974_45135",
                    "direction_id": 0
                },
                "stop_time_update": [{
                    "stop_sequence": 22,
                    "arrival": {
                        "delay": 1407
                    },
                    "stop_id": "8460B5228401",
                    "schedule_relationship": "SCHEDULED"
                }],
                "vehicle": {
                    "id": "1729"
                },
                "timestamp": "1679345003"
            }
        },
        {
            "id": "T4",
            "trip_update": {
                "trip": {
                    "trip_id": "2974_31958",
                    "start_time": "15:30:00",
                    "start_date": "20230320",
                    "schedule_relationship": "SCHEDULED",
                    "route_id": "2974_45138",
                    "direction_id": 0
                },
                "stop_time_update": [{
                        "stop_sequence": 27,
                        "arrival": {
                            "delay": 780
                        },
                        "departure": {
                            "delay": 850
                        },
                        "stop_id": "8470B532331",
                        "schedule_relationship": "SCHEDULED"
                    },
                    {
                        "stop_sequence": 28,
                        "arrival": {
                            "delay": 777
                        },
                        "departure": {
                            "delay": 777
                        },
                        "stop_id": "8470B521231",
                        "schedule_relationship": "SCHEDULED"
                    },
                    {
                        "stop_sequence": 29,
                        "arrival": {
                            "delay": 1043
                        },
                        "departure": {
                            "delay": 1043
                        },
                        "stop_id": "8470B530311",
                        "schedule_relationship": "SCHEDULED"
                    },
                    {
                        "stop_sequence": 30,
                        "arrival": {
                            "delay": 838
                        },
                        "stop_id": "8460B5228401",
                        "schedule_relationship": "SCHEDULED"
                    }
                ],
                "vehicle": {
                    "id": "1808"
                },
                "timestamp": "1679345003"
            }
        },

Implement reading of timetable GTFS data provided by NTA

See here for an example of how this is implemented. This project uses the old API, however, so cannot use this implementation directly.

Reference: the PDF presentation at the bottom of this page explains the static timetable files. There is a warning that these files may update once daily.

I'm leaning towards using something like Pandas here rather than maintaining an ongoing database, given that most people will only be tracking a small number of routes/stops. (And also I just like Pandas).

Hypothesised workflow:

  1. Once daily automatically download the files from here (I think the one marked Realtime Operators should do it.)
  2. Daily parse the files using the target route and stop numbers as filters to assemble a schedule.
  3. Ongoing API calls will pull the full trip, stop, and vehicle location dataset. Map the API results to the assembled static schedule and amend the stop time based on the result.
  4. The data should be output in a way that matches the current sensor configuration.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.