gtracy / madison-transit-api-gtfs Goto Github PK

The docs still live in the old GAE/python repo. Now that the subdomain has been switched, the docs are no longer accessible.

see developer docs for API definition - https://api.smsmybus.com/locations.html#getvehicles

The old API implementation tracked metrics for usage of the API. Need to implement that here as well.

    "url": "/v1/getvehicles?key=xxx&routeID=2"

console data is getting pushed into logs. Let's clean this up.

These debug errors are rare but come in bursts. We probably can't control the missing properties from Metro, but we need to make sure we are handling the situation correction.

/v1/getarrivals?key=XXX&stopID=533

{
"level": 50,
"time": 1649559190797,
"pid": 13,
"hostname": "169.254.186.185",
"obj": null,
"property": "id",
"msg": "missing feed property"
}

we only store dev keys in lowercase. so no matter what the client is trying to use, convert it to lowercase before validating it.

The end user experience appears graceful but this error is being thrown on the requests.

{
"level": 50,
"time": 1662035462719,
"pid": 15,
"hostname": "169.254.5.97",
"err": {
"type": "TypeError",
"message": "Cannot read property 'route_id' of undefined",
"stack": "TypeError: Cannot read property 'route_id' of undefined\n at /var/task/api/fetch.js:44:40\n at Array.forEach ()\n at Object.module.exports.fetch_trips (/var/task/api/fetch.js:42:21)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async /var/task/api/schedule.js:33:23"
},
"msg": "Cannot read property 'route_id' of undefined"
}

it's currently using stop_id as the unique key so we'e only storing one request per dev key per stop.

what a dope.

implement a unique id for this table.

It should be very rare that there is missing trip details in Dynamo/GTFS. Raise an error when it occurs so we can create alerting around it for more investigation.

The project started as a simple prototype and quickly blossomed into an API replacement.

I got in the bad habit of using console logging everywhere which makes monitoring and alerting way harder than it should be.

Implement a better logging framework such as logme or pino throughout the entire codebase.

Update the Lambda configuration to remove 'kiosk' from the cached key list.

There is an abuser among us...

These debug errors are rare but come in bursts. We probably can't control the missing properties from Metro, but we need to make sure we are handling the situation correction.

/v1/getarrivals?key=wisc81jw&stopID=971

{
"level": 50,
"time": 1649558980128,
"pid": 13,
"hostname": "169.254.186.185",
"obj": null,
"property": "label",
"msg": "missing feed property"
}

GTFS supports the notion of different services which are activated based on the calendar. This means that the GTFS file definitions can list out multiple definitions for the same routes, stops and trips in a single GTFS file. Choosing the correct one is dependent on the date.

The current implementation is dynamic based on the date. Instead, it depends on the correct data being loaded into the production database and code.

There's an opportunity to be smarter about choosing the right service based on the calendar making it far easier to load data and not worrying about rotating the GTFS data on the exact date the changes go into effect.

This requires that we port its use over to ESM.

https://www.npmjs.com/package/got

There is too much detail in this repo and it belongs in the repo that actually manages the dynamo table

{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "TypeError: Cannot read property 'trip_headsign' of undefined",
"reason": {
"errorType": "TypeError",
"errorMessage": "Cannot read property 'trip_headsign' of undefined",
"stack": [
"TypeError: Cannot read property 'trip_headsign' of undefined",
" at /var/task/api/schedule.js:49:43",
" at processTicksAndRejections (internal/process/task_queues.js:95:5)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: TypeError: Cannot read property 'trip_headsign' of undefined",
" at process. (/var/runtime/index.js:35:15)",
" at process.emit (events.js:400:28)",
" at process.emit (domain.js:475:12)",
" at processPromiseRejections (internal/process/promises.js:245:33)",
" at processTicksAndRejections (internal/process/task_queues.js:96:32)"
]
}

{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "TypeError: Cannot read property 'route_short_name' of undefined",
"reason": {
"errorType": "TypeError",
"errorMessage": "Cannot read property 'route_short_name' of undefined",
"stack": [
"TypeError: Cannot read property 'route_short_name' of undefined",
" at /var/task/api/schedule.js:70:77",
" at runMicrotasks ()",
" at processTicksAndRejections (internal/process/task_queues.js:95:5)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: TypeError: Cannot read property 'route_short_name' of undefined",
" at process. (/var/runtime/index.js:35:15)",
" at process.emit (events.js:400:28)",
" at process.emit (domain.js:475:12)",
" at processPromiseRejections (internal/process/promises.js:245:33)",
" at processTicksAndRejections (internal/process/task_queues.js:96:32)"
]
}

items in Stops_gtfs and Trips_gtfs get rebuilt on every import which means items can become stale and invalid.

start adding a TTL property in order to leverage the built in delete function of Dynamo

It's not clear that this is even possible because we don't know when Metro updates its GTFS data relative to the expiration date.

{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "TypeError: req.log_error is not a function",
"reason": {
"errorType": "TypeError",
"errorMessage": "req.log_error is not a function",
"stack": [
"TypeError: req.log_error is not a function",
" at /var/task/api/schedule.js:70:29",
" at processTicksAndRejections (internal/process/task_queues.js:95:5)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: TypeError: req.log_error is not a function",
" at process. (/var/runtime/index.js:35:15)",
" at process.emit (events.js:400:28)",
" at process.emit (domain.js:475:12)",
" at processPromiseRejections (internal/process/promises.js:245:33)",
" at processTicksAndRejections (internal/process/task_queues.js:96:32)"
]
}

this is not a valid error message. the stop ID may be accurate, but we're missing trip details from GTFS.

todo : cleanup the error message

Every time we refresh the GTFS data with Metro changes, we leave behind a trail of GTFS carcasses that should just be deleted.

Let's write a script to clean it up.

[ { service_id: '103', start_date: 2022-12-05T04:59:59.999Z, end_date: 2023-03-04T05:00:00.000Z }, { service_id: '104', start_date: 2023-03-06T04:59:59.999Z, end_date: 2023-06-10T04:00:00.000Z } ]

there is a 48 hour gap between those two because the the first service should run through the end of 03-04 and the second service should start at the beginning of 03-05.

{
"level": 50,
"time": 1645024238157,
"pid": 22,
"hostname": "169.254.194.57",
"err": {
"type": "TypeError",
"message": "Cannot read property 'id' of null",
"stack": "TypeError: Cannot read property 'id' of null\n at Object.module.exports.getValue (/var/task/api/utils.js:67:19)\n at /var/task/api/fetch.js:54:53\n at Array.forEach ()\n at /var/task/api/fetch.js:46:54\n at Array.forEach ()\n at Object.module.exports.fetch_trips (/var/task/api/fetch.js:43:21)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async /var/task/api/schedule.js:33:23"
},
"msg": "Cannot read property 'id' of null"
}

{
"level": 50,
"time": 1644291303153,
"pid": 15,
"hostname": "169.254.194.205",
"err": {
"type": "Error",
"message": "Message (user) attribute 'stopid' must contain a non-empty value of type 'String'.",
"stack": "InvalidParameterValue: Message (user) attribute 'stopid' must contain a non-empty value of type 'String'.\n at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/query.js:50:29)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request. (/var/task/node_modules/aws-sdk/lib/request.js:38:9)\n at Request. (/var/task/node_modules/aws-sdk/lib/request.js:688:12)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:116:18)",
"code": "InvalidParameterValue",
"time": "2022-02-08T03:35:03.153Z",
"requestId": "857ca241-3944-5db2-b1a9-f83ade634c7f",
"statusCode": 400,
"retryable": false,
"retryDelay": 9.972165703280101
},
"msg": "queue.push() exception"
}

https://api.smsmybus.com/v1/getarrivals?key=nomar&stopID=1100&routeid=07

that should only return trips from route 7 traveling through stop 100. it's returning all routes currently.

{
"level": 50,
"time": 1643398528520,
"pid": 14,
"hostname": "169.254.50.249",
"err": {
"type": "TypeError",
"message": "Cannot read property 'id' of null",
"stack": "TypeError: Cannot read property 'id' of null\n at /var/task/api/fetch.js:53:73\n at Array.forEach ()\n at /var/task/api/fetch.js:45:54\n at Array.forEach ()\n at Object.module.exports.fetch_trips (/var/task/api/fetch.js:42:21)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async /var/task/api/schedule.js:30:23"
},
"msg": "Cannot read property 'id' of null"
}

{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "TypeError: Cannot read property 'toLowerCase' of undefined",
"reason": {
"errorType": "TypeError",
"errorMessage": "Cannot read property 'toLowerCase' of undefined",
"stack": [
"TypeError: Cannot read property 'toLowerCase' of undefined",
" at module.exports.validateDevKey (/var/task/api/devkey.js:21:39)",
" at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)",
" at next (/var/task/node_modules/express/lib/router/route.js:137:13)",
" at module.exports.afterHours (/var/task/api/utils.js:21:9)",
" at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)",
" at next (/var/task/node_modules/express/lib/router/route.js:137:13)",
" at Route.dispatch (/var/task/node_modules/express/lib/router/route.js:112:3)",
" at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)",
" at /var/task/node_modules/express/lib/router/index.js:281:22",
" at Function.process_params (/var/task/node_modules/express/lib/router/index.js:341:12)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: TypeError: Cannot read property 'toLowerCase' of undefined",
" at process. (/var/runtime/index.js:35:15)",
" at process.emit (events.js:400:28)",
" at process.emit (domain.js:475:12)",
" at processPromiseRejections (internal/process/promises.js:245:33)",
" at processTicksAndRejections (internal/process/task_queues.js:96:32)"
]
}

{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "TypeError: Cannot read property 'bikes_allowed' of undefined",
"reason": {
"errorType": "TypeError",
"errorMessage": "Cannot read property 'bikes_allowed' of undefined",
"stack": [
"TypeError: Cannot read property 'bikes_allowed' of undefined",
" at /var/task/api/schedule.js:84:54",
" at runMicrotasks ()",
" at processTicksAndRejections (internal/process/task_queues.js:95:5)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: TypeError: Cannot read property 'bikes_allowed' of undefined",
" at process. (/var/runtime/index.js:35:15)",
" at process.emit (events.js:400:28)",
" at process.emit (domain.js:475:12)",
" at processPromiseRejections (internal/process/promises.js:245:33)",
" at processTicksAndRejections (internal/process/task_queues.js:96:32)"
]
}

https://gtfs.smsmybus.com/v1/getarrivals?key=nomar&stopID=0178

If you call this without the zeros, it works correctly

https://gtfs.smsmybus.com/v1/getarrivals?key=nomar&stopID=178