Code Monkey home page Code Monkey logo

curlbus's Introduction

curlbus ๐Ÿš‰๐Ÿš๐ŸšŒ

It's like a catbus, but uses a different command :)

curlbus provides easy terminal based UI (or JSON) abstraction to the Israeli Mininstry of Transportation realtime public transit API.

Support for other countries / transit operators is theoretically possible to implement, if you want that feature - let me know.

Inspired by wttr.in and mapscii.

Usage

Terminal UI

curl https://curlbus.app/<stop_code>

To find the stop code, use OpenStreetMap, Google Maps, or look at the sign on the stop itself.

Other endpoints are also available:

  • /operators - Get a list of known transit operator names
  • /<operator>/<route_number> - Get information for a specific route from a specific operator, for example /dan/1.
  • /<operator>/<route_number>/<alternative> - Get information for a specific route alternative

You can set up convinent shell aliases to quickly query a line or a stop you care about, for example put this in your ~/.bashrc:

alias bus="curl https://curlbus.app/36601"

And now you just need to type bus to get live ETAs for your bus home :)

HTML-based terminal-like UI

curlbus is also accessible via a browser, but curl is still the recommended method of interaction.

JSON API

curlbus also has a JSON based API. Send a request with the header Accept: application/json to get the output in a JSON format. All endpoints support JSON.

Installing the Server

If you want to run it yourself, you'll need access to the Israeli MoT realtime "SIRI-SM" API.

Follow the instructions on the Ministry of Transportation website to apply for API access.

Note that having your application reviewed can take very long time, and that the API access requires your IP to be added to the whitelist, which means you have to have a static IP. Not great for cloud deployments.

After you have API access (if you haven't given up on this stage), install the required dependencies:

pip3 install --user -r requirements.txt

And edit config.ini.example to fill in the required values. Remeber to point it to the MoT server if you have API access.

You'll need Postgresql for the GTFS database. Other databases are not supported.

You need to use ./update_feed.sh to load the GTFS database, and ./load_cities.py to download the city name database.

The GTFS feed updates nightly, but update_feed.sh currently can only load it into an empty database. For now, the way to do updates is manual (once a week or so):

Change the configuration file to point to a different postgres database (I've been using two databases, "gtfs" and "gtfs2". when "gtfs2" is active I switch to "gtfs" and vise-versa), connect to it and make sure to drop all tables, then run ./update_feed.sh and ./load_cities.py. Afte they're done, restart the service. Doing it this way ensures the update is atomic and there are no inconsistencies while the update is running, and allows you to roll back in case of a problematic update (by just changing the config file back to the previous database).

I don't particularly like this process being manual, but I didn't have time to automate it.

Note that the GTFS feed can be quite big (more than 2GB of csv files), and loading it into your database can take a while, so be patient and make sure to have plenty of free disk space.

Development Server

Running mock_siri_server.py and pointing your config file for it will allow you to run curlbus locally without access to the SIRI API. it still requires PostgreSQL.

mock_siri_server.py will make up random arrival times for random routes when queried, but all the routes would be valid ones that actually stop on the requested stop according to the GTFS database. Make sure to run update_feed.sh before running the mock server.

Deploy with Docker

  1. Create a file docker-compose.yml:
version: '3.1'
  
services:
  curlbus:
    container_name: curlbus
    image:  guysoft/curlbus
    volumes:
      - ./config.ini:/curlbus/config.ini:ro
    tty: true
    links:
      - "db:postgres"
    ports:
      - 8080:80

  db:
    image: postgres
    restart: always
    container_name: curlbus-db
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: curlbus
    volumes:
      - curlbus-data:/var/lib/postgresql/data

volumes:
  curlbus-data:
wget https://raw.githubusercontent.com/elad661/curlbus/master/config.ini.docker -O config.ini
  1. Edit config.ini to include your SIRI username.

sudo docker-compose up -d
sudo docker exec -it curlbus /curlbus/update_feed.sh
sudo docker exec -it curlbus /curlbus/load_cities.py -c /curlbus/config.ini
sudo docker exec -it curlbus /curlbus/main.py -c /curlbus/config.ini
  1. Your curlbus server is avilable at port 8080

curlbus's People

Contributors

danyshaanan avatar elad661 avatar guysoft avatar salty-horse avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

curlbus's Issues

Mobile browser compatible CSS

The current CSS for the HTML version of curlbus doesn't play nice with smaller screens (such as mobile phones). It would be nice to somehow make it look nicer on these screens.

Mock server not handling routes queries?

I have executed update_feed.sh to fill my local database.

After making a call to /10000, curlbus fails on:

Traceback (most recent call last):
  File "/home/ori/.virtualenvs/curlbus/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 378, in start
    resp = await self._request_handler(request)
  File "/home/ori/.virtualenvs/curlbus/lib/python3.6/site-packages/aiohttp/web_app.py", line 341, in _handle
    resp = await handler(request)
  File "/home/ori/.virtualenvs/curlbus/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 88, in impl
    return await handler(request)
  File "/home/ori/.virtualenvs/curlbus/lib/python3.6/site-packages/gino/ext/aiohttp.py", line 109, in _middleware
    return await handler(request)
  File "/home/ori/devel/curlbus/curlbus/server.py", line 122, in handle_station
    response = await client.request([stop_code])
  File "/home/ori/devel/curlbus/curlbus/siri.py", line 192, in request
    response = SIRIResponse(text, group, self.verbose)
  File "/home/ori/devel/curlbus/curlbus/siri.py", line 104, in __init__
    for delivery in _listify(answer['ns3:StopMonitoringDelivery']):
KeyError: 'ns3:StopMonitoringDelivery'

I printed answer on the line where it fails:

OrderedDict([('ns3:ResponseTimestamp', '2018-06-16T20:04:29.187+03:00'),
             ('ns3:ProducerRef', 'ISR Siri Server (141.11)'),
             ('ns3:ResponseMessageIdentifier', '16503786'),
             ('ns3:RequestMessageRef', 'user id from MoT:1529168696.3143766'),
             ('ns3:Status', 'false'),
             ('ns3:ErrorCondition',
              OrderedDict([('ns3:OtherError',
                            OrderedDict([('ns3:ErrorText', '720')])),
                           ('ns3:Description',
                            'User authentication failed for user id from MoT '
                            'from MY_IP_IS_HERE')]))])

BTW, why is my internet-facing IP address in the answer, instead of the localhost address from which I made the request?

Missing packages in requirements.txt - psycopg2?

Running the mock server failed due to missing psycopg2.

After installing psycopg2==2.7.4 it failed with:

  File "/lib/python3.6/site-packages/gino/strategies.py", line 42, in create
    pool = await dialect.init_pool(u, loop)
AttributeError: 'PGDialect_psycopg2' object has no attribute 'init_pool'

Is it because I have the wrong version of some packages?

problem in get route info when adding the json header

hi,
whenever i try getting route info via the API (by adding the json header), i get the following message:
"500 Internal Server Error

Server got itself in trouble"
the endpoint works fine when querying without the json header

Random feature ideas

  • Fuzzy stop name search, curlbus.app/~<something> to find stop codes matching a specific name
  • Direct routing (no transfers!), curlbus.app/<stop_code>/to/<other_stop_code> will show only lines that will take you from the first stop to the other stop.
  • Static schedule for stop? Frequencies? this needs some thought.
  • Filtering using the URL, to only see lines you care about for a specific stop. This needs thought about syntax (perhaps curlbus.app/<stop_code>|dan1|dan51 or something similar)
    • Alternatively, curlbus.app/<operator>/<route>/<alternative>/<stop_code>, but that doesn't allow chaining. Would be nice to support both as the latter is more discoverable, but the first is more useful.
  • Show status for trains? (delayed/on-time)
  • Show distance on-route (using shapes.txt) of the nearest bus from the current stop. Calculating this is expensive in terms of CPU time, is it worth it?

update readme to clarfiy how to update the GTFS feed

Calling update_feed.sh when you already have the database won't work. The way I've been doing it on the server, to save time, is having two databases, gtfs and gtfs2.

When it's time to update the GTFS feed, I modify the config to point to the other database, run update_feed.sh, and then when it's done - restart curlbus. This way curlbus can continue working while the feed is updated, there's no need for complicated GTFS feed update logic, and I can revert to the older GTFS copy in an atomic operation.

I should update the README to reflect that, and also write a script to automatically do the update&switch.

Several improvements ideas

Hi Elad!
wttr.in creator here.

I was really glad to find your beautiful service.

I tried several queries but many of them seem to be broken:

$ curl https://curlbus.app/citypass
(after several minutes)
^C

Or this:

$ curl https://curlbus.app/galim
(after several minutes)
^C

What I would suggest to do:

  1. create a help page (/:help)
  2. create a list page for each section (/citypass/:list), to get the list of all available items
  3. search of nearest bus stops (you can use "wttr.in" resolving mechanism for that, if you want).

The service seems to be very cool and promising!! I like it a lot.
If you need my help, I'll be happy to help

special-casing for Israel Railways

Israel Railways lines only have long_name, and no short_name, so there aren't really any meaningful numbers to be used for filtering on /rail/{route_number}

I need to come up with a system to assign numbers/short English names to the rail route numbers, and to have an index of them in /rail or /rail/routes.

[enhancement] Find nearby stations

Is this a request for help?: yes


Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) None

Anything else we need to know:
I want to build a kiosk that would display nearby buses in a given radius. For that I need a way to get GPS coordinates, and from them return a list of bus stations. Then I can pull them with curlbus.

The stations gps data is available here and is updated daily. The reason I think this should be in curlbus is because its 177MB to download and update, and it seems an external service should do it, or at least an instance of curlbus (if you want your privacy).

Mock data for local development

A mock SIRI server that would generate random responses for a stop code would help contributors with no access to the MoT API to contribute to curlbus

Some stations do not have a full address, only a street name

Not sure if its a parsing issue so reporting.
I am trying to group stations that are close to each other, I found that he best way so far was to group them according to street address (street name + number).
The reason is that it lets you also figure out which direction the bus is going.

However, it seems that not all addresses have a number, for example

station id: 21472

Address field:

{'name': {'HE': 'ืงื ื™ื•ืŸ ืขื–ืจื™ืืœื™/ื“ืจืš ื‘ื’ื™ืŸ', 'EN': 'Azrieli Mall/Begin Road', 'AR': 'ูƒู†ูŠูˆู† ุนุฒุฑูŠุฆูŠู„ูŠ/ุฏูŠุฑูŠุฎ ุจูŠุฌูŠู†'}, 'address': {'street': 'ื“ืจืš ืžื ื—ื ื‘ื’ื™ืŸ', 'city': 'Tel Aviv Yafo', 'platform': '', 'floor': ''}, 'location': {'lat': 32.074914, 'lon': 34.790734}}

Note address': {'street': 'ื“ืจืš ืžื ื—ื ื‘ื’ื™ืŸ',
And they did not give a number.

Its sister station is (one going the other way):

Station id: 21669

{'name': {'HE': 'ืงื ื™ื•ืŸ ืขื–ืจื™ืืœื™/ื“ืจืš ื‘ื’ื™ืŸ', 'EN': 'Azrieli Mall/Begin Road', 'AR': 'ูƒู†ูŠูˆู† ุนุฒุฑูŠุฆูŠู„ูŠ/ุฏูŠุฑูŠุฎ ุจูŠุฌูŠู†'}, 'address': {'street': 'ื“ืจืš ืžื ื—ื ื‘ื’ื™ืŸ', 'city': 'Tel Aviv Yafo', 'platform': '', 'floor': ''}, 'location': {'lat': 32.07569, 'lon': 34.791667}}

Just making sure you didn't miss some data from the original call.
Also any suggestions how to group stations would be helpful.
Using GPS would not help because it could catch stations from either side of the road.

Missing `app` category in config.ini.example

When running main.py, it tries to read the app section:

server.run(config_dict['app'])

The sample file has a server section with the fields it seems to look for. Was it renamed, and the example config wasn't updated?

Showing gibberish in Windows terminal

Trying to curl through Windows cmd gives the following screen:
capture
I'm not sure if this issue is on my end or in the code. I would like some input on that, if possible.

The HTML-based terminal-like UI works well.
Hebrew installed on the computer. Tested on Windows 10 Home.

asyncpg.exceptions.UndefinedTableError: relation "stops" does not exist

curl localhost:8081/1 output:

curlbus    | (Press CTRL+C to quit)
curlbus    | Error handling request
curlbus    | Traceback (most recent call last):
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/aiohttp/web_protocol.py", line 433, in _handle_request
curlbus    |     resp = await request_handler(request)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/aiohttp/web_app.py", line 504, in _handle
curlbus    |     resp = await handler(request)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/aiohttp/web_middlewares.py", line 117, in impl
curlbus    |     return await handler(request)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/gino_aiohttp.py", line 114, in _middleware
curlbus    |     return await handler(request)
curlbus    |   File "/curlbus/curlbus/server.py", line 149, in handle_station
curlbus    |     stop_info = await get_stop_info(db, stop_code)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/aiocache/decorators.py", line 109, in wrapper
curlbus    |     return await self.decorator(f, *args, **kwargs)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/aiocache/decorators.py", line 124, in decorator
curlbus    |     result = await f(*args, **kwargs)
curlbus    |   File "/curlbus/curlbus/gtfs/utils.py", line 126, in get_stop_info
curlbus    |     stop = await query.gino.first()
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/gino/api.py", line 137, in first
curlbus    |     return await self._query.bind.first(self._query, *multiparams, **params)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/gino/engine.py", line 328, in first
curlbus    |     return await result.execute(one=True)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/gino/dialects/base.py", line 214, in execute
curlbus    |     rows = await cursor.async_execute(
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/gino/dialects/asyncpg.py", line 184, in async_execute
curlbus    |     result, stmt = await getattr(conn, "_do_execute")(query, executor, timeout)
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/asyncpg/connection.py", line 1711, in _do_execute
curlbus    |     stmt = await self._get_statement(
curlbus    |   File "/usr/local/lib/python3.10/dist-packages/asyncpg/connection.py", line 397, in _get_statement
curlbus    |     statement = await self._protocol.prepare(
curlbus    |   File "asyncpg/protocol/protocol.pyx", line 168, in prepare
curlbus    | asyncpg.exceptions.UndefinedTableError: relation "stops" does not exist

postgres 14

hardcoded database settings in update_feed.sh

The script update_feed.sh connects to the database with psql -d gtfs. This uses the default user, and hard-codes the database name.

I have set a different username and database in config.ini's gino.dsn. It should use this value.
(Perhaps by extracting it reliably with an inline Python script that uses the configparser module.)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.