grote / osm2gtfs Goto Github PK
View Code? Open in Web Editor NEWTurn OpenStreetMap data and schedule information into GTFS
License: GNU General Public License v3.0
Turn OpenStreetMap data and schedule information into GTFS
License: GNU General Public License v3.0
We need a way to make sure our changes and refactors do not break the current creators.
Maybe a shell script ...
The process could be, for each creator :
Hi,
I think the current bounding box approach is not ideal. Transit networks can overlap and do not end necessarily at squared boxes.
I'd suggest to run a query to obtain relations with
(and probably maintain the bounding box optional to just make the overpass api query more performant).
Does this make sense? I'd happy to provide a pull request for this. But I also think it's best to get feedback before. Thanks!
google/transitfeed#394 has been merged and released as 1.2.16
.The workaround in the code to add the feed_info.txt
manually should be removed now.
Also, Accra regression tests need to be fixed. There are different stop times now.
I'm testing Transportr (a locally compiled version) with incofer data and I notice that some stops are avoided as line change points. Example: instead of changing from line 2 to line 4 in stop A is being done in stop B affecting times of the total trip. I suspect maybe that non grouped stops could be the problem, as stop A of line 2 and stop A of line 4 aren't exactly in the same point but are in the same station:
-84.0693107,Estación Atlántico,9.9349735,4310018677,0
-84.0686079,Estación Atlántico,9.9346699,4628999925,0
Adding support for grouping stops (stations) in the GTFS file seems simple. The GTFS part is solved using the parent_station and location_type of the stops.txt (https://developers.google.com/transit/gtfs/reference/stops-file). And from OSM we could check for the stop_area relation the contains both stop_position. At the end we should have something like this:
stop_lon,stop_name,stop_lat,stop_id,location_type, parent_station
-84.0693107,Estación Atlántico,9.9349735,4310018677,0, ID
-84.0686079,Estación Atlántico,9.9346699,4628999925,0, ID
lon, Estación Atlántico, lat, ID, 1,
The OsmHelper is the part of the script that retrieves data from OpenStreetMap and puts it into the own class/object data structure (to be) discussed in #30. It has included a simple caching functionality that saves data obtained from OpenStreetMap into files and grabs them, if present, to avoid unnecessary hits to the OSM's Overpass API.
I think the current structure of the OsmHelper can be improved and I'm doing a very first draft here, so we have a base for discussion:
OsmHelper
- config
- routes {route_id, RouteMaster}
- stops {osm_id, Stop}
-------
+ get_routes(): Dic of RouteMaster
+ get_route(String route_id): RouteMaster
+ get_stops(): Dic of Stop
+ get_stop(String osm_id): Stop
- query_routes(): Dic of RouteMaster
- query_route(ref): RouteMaster
- query_stops(): List of Stop
- query_stop(osm_id): Stop
+ cache_refresh(name): void
- cache_write(name, content): void
- cache_read(name): content
Some thoughts:
Hi,
I'm writing here not sure if it is the right place, so sorry in advance for that. I've installed osm2gtfs because I need to generate GTFS data from Birmingham and Newcastle (uk), but there is no GTFS information anywhere.
My question is if it is possible to create GTFS data for these cities only having OSM data. If it is possible, there is information/guide to follow?
Thank you,
Proma
I've been working on the script and building the creators for my city and i noticed that if i run the
script with the :
--refresh-all flags it work fine
but if i try to run it with out the flag the script doesn't work and i raise the " Should have implemented this"
like in the route_creator and in trips_creator.
so that tells me the script is not reading my creators and in my config.json i did specify the selector
"selector" : "irtramma"
any ideas why is this happening? i did run incofer with the data that james has in one of his repos and works fine.
There is a validation error when start_date is automatically generated:
transitfeed.problems.InvalidValue: Invalid value 2017301 in field start_date
This is because datetime.datetime() returns the numer of the month as one digit (3) instead of two (03).
We need documentation on how the script is technically structured, how to extend the script and how to make it work for other cities and other type of schedule information. I suggest to use the wiki here in this github repo.
Currently the stop_id
for the GTFS is extracted based on the id
coming from OpenStreetMap. This is nice, because ideally it allows mapping between a stop and its object in OpenStreetMap.
Some context: OSM has three types of elements: node, way, relation. They have separate spaces for IDs:
Element types have their own ID space, so there could be a node with id=100 and a way with id=100, which are unlikely to be related or geographically near to each other.
Currently it seems to work like this (in the standard stop creator): if the stop is a node or a way it gets it's ID (even though they are not the same). If it is a relation of type stop_area
, then a SA
gets prepended to the ID.
This is not ideal, because IDs should be solid (and not change over time), and they should not conflictive (like IDs between nodes and ways).
My local company Expresso Lorenzutti have made time tables available in PDF format. I understand it might be difficult to accommodate all forms of publishing time tables, and have therefor started to write a scraper for this. I would like to save it in a simple, common format, that osm2gtfs can read directly. One easy solution would be to save it as JSON. If you have any suggestions to formats I am open.
Currently the filename and path of the output file is hard-coded into the script. We should handle this more generic. Either as another argument on the command line or inside the config file (#3). I prefer the first solution, but before implementing, I wanted to check with you guys.
Cheers!
The data = OsmHelper()
object initiation queries and creates the whole data structure of all routes and stops already in the constructor.
This is not very efficient, because even when updating only the routes (or even one route) or only the stops, everything get's queried and built anyway.
I guess, we should move those queries and data tree builders into the respective functions, instead of building the whole data tree in the constructor function.
If the json config file includes in the feed info only one of the dates: "start_date" or "end_date" then the other is generated automatically. But when this happen the object type for the generated date is datetime.date and the type of the date from config file is datetime.datetime. When the dates validation is done an TypeError occurs: «can't compare datetime.datetime to datetime.date»
This could be fixed using datetime.datetime in both cases. I'll create a PR is you agree.
Hello, I hope you don't mind me asking some questions. if this is not the correct place for this, feel free to close this issue.
I'm working on a personal project (mostly for learning) building an open source progressive web app that'll show you bus routes between two points, since I've found no free alternatives that work well in Argentina.
What I have so far is a basic frontend interface and a python backend that runs querys on a local postgis db to which I have loaded osm data for my region, here is the code. These seems to kinda work but I'm having some trouble with osm2pgsql splitting bus routes into chunks when the route makes a circle and determining the direction so that I only get the A -> B routes.
what do you think of this approach?
yesterday I saw your youtube video about transportr and I wonder if using gtfs data is strictly necessary in order to search routes (if I don't care about arrival and departure times yet) ?
another question would be if you could provide me some basic info about how I would adapt this library so I can try to get data from Argentina and contribute that to navitia. There are some closed source apps like Moovit and mapa.buenosaires.gob.ar that have departure and arrival times, so I'm trying to figure out how to get them as well. (the latter being a government website, so I send some emails but they have not answered yet)
Thanks in advance
Currently, the data about the schedule information is getting pulled in a very custom way into the script.
Others would like to use the script. For now I see different ways for feeding the script:
I suggest moving different ways on pulling schedule data into different files, which could be handled similar to small plugins. Basically all of them then have to output the same information to the main script: Stop times of the first and last stops (mostly terminals, I guess).
Probably we could specify in the config file (#3) the name of the schedule information plugin. Do you have any suggestions on how to implement this properly?
This is an important step in order to make this tool generically usable.
@grote worte:
For the dates, we need a more flexible solution, because when used with Navitia, the difference between start and end date can never be more than a year and ideally, for this use-case, I like these dates to automatically move forward when I run the script again without needing to update them somewhere manually.
Great idea. Maybe we can make the script to understand syntax like now
and +1year
, etc...
Hello,
currently a lot of custom configuration data is stored in code. I'd like to push it to be more generic.
I'd suggest to rely on a config.json file similar to geom2gtfs does: https://github.com/conveyal/geom2gtfs/blob/master/kingco.json.example
We could start with a simple structure, based on the general information used for querying from OSM and from GTFS' agency.txt and start and end date from calendar.txt:
{
"query":{
"network": "",
"bbox": {"e": "-48.2711", "n": "-27.2155", "s": "-27.9410", "w": "-49.0155"},
},
"agency":{
"agency_id": "",
"agency_name": "",
"agency_url": "",
"agency_timezone": "",
"agency_lang": "",
"agency_phone": "",
"agency_fare_url": "",
},
"feed_info":{
"publisher_name": "",
"publisher_url": "",
"licence_name": "",
"licence_url": "",
"version": "",
},
"gtfs_mode":3,
"start_date": "",
"end_date":"",
"schedule_information": "",
"output": "",
}
What do you think? Is this a way to go?
The current stops creation is very much custom for Florianópolis. After the abstraction work happening in (#9) I'd suggest working on a default implementation for stops generation based on OSM's Public Transport Schema 2.
Managua can be a good start as the data quality is very consistent and complying the standard data structure without exceptions.
I'm creating this issue just to keep issue#55 with the original theme.
In my opinion, all other creators should ideally be merged here. This makes sense for example when >we refactor the code. If the creators live in other repositories, they need to be manually adjusted to
whatever changed in the upstream repository.
Make sense to keep the creators in the same repo, also if we try to make generic default creators will be very useful to have them together.
And what about config files? A config folder to keep them could be a solution. What you think?
When building a stops list i\m getting the following error:
Traceback (most recent call last):
File "osm2gtfs.py", line 101, in
main()
File "osm2gtfs.py", line 67, in main
stops_creator.add_stops_to_schedule(schedule, data)
File "C:\Projecten\osm2gtfs\creators\stops_creator.py", line 20, in add_stops_to_schedule
stops = data.get_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 239, in get_stops
self._get_names_for_unnamed_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 488, in _get_names_for_unnamed_stops
self._find_best_name_for_unnamed_stop(stop)
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 518, in find_best_name_for_unnamed_stop
""" % (stop.lat, stop.lon, stop.lat, stop.lon))
File "C:\Python27\lib\site-packages\overpy_init.py", line 123, in query
raise exception.OverpassTooManyRequests
overpy.exception.OverpassTooManyRequests: Too many requests
or
Traceback (most recent call last):
File "osm2gtfs.py", line 101, in
main()
File "osm2gtfs.py", line 67, in main
stops_creator.add_stops_to_schedule(schedule, data)
File "C:\Projecten\osm2gtfs\creators\stops_creator.py", line 20, in add_stops_to_schedule
stops = data.get_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 211, in get_stops
self._get_names_for_unnamed_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 488, in _get_names_for_unnamed_stops
self._find_best_name_for_unnamed_stop(stop)
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 545, in find_best_name_for_unnamed_stop
candidate.get_nodes(resolve_missing=True))
File "C:\Python27\lib\site-packages\overpy_init.py", line 860, in get_nodes
tmp_result = self.result.api.query(query)
File "C:\Python27\lib\site-packages\overpy_init.py", line 126, in query
raise exception.OverpassGatewayTimeout
overpy.exception.OverpassGatewayTimeout: Server load too high
Is there a way to avoid it?
After creating a new route or stop from the API, the new object should be written to the file right away.
If there's an error (such as two many requests), you can resume right where you left off instead of starting from scratch again.
Currently, this repository includes different implementations for cities. Particularly the trip_creators are very much exclusively targeted and programmed for a special data source. However for any new person coming to this script it is almost impossible to find out where to get those data sources, in order to run osm2gtfs with the existing implementations, for example for the sake of testing. As they are so heavily connected, the data (source) on the schedule should be provided, too.
Instead of adding the actual data, I'd suggest to include a source
field to the configuration file where a valid url to the file containing the time information should be specified. The script would then download the data, save it into the data
directory and use it. Probably also an argument (similar to the --refresh-
ones should be implemented, e.g. --refresh-time-table
).
When generating an end_date
with a start_date
in January or running the script in January without a specified start_date
the script throws the following error: calendar.IllegalMonthError: bad month number 0; must be 1-12
. A special check needs to be introduced for this particular case.
Right now the code includes Portuguese language. This is probably not very good and we should think about how to localize it. Potfiles are probably the way to go!?
osm2gtfs downloads data from OpenStreetMap (Public Transport Schema version 2) and puts it into transitfeed's structure to generate GTFS.
Currently there is a parallel data structure (mainly living in the classes RouteMaster
, Route
, Stop
) with some regional touch. We already started some discussion about this in #26, but I think it's worth to discuss this separately.
Resuming the past comments on this:
We were considering to use directly transitfeed's (routes, stops and trips objects) instead of a parallel data structure. The downside of this would be that we would make ourselves totally dependent on transitfeed's classes which might change without further notice.
I further think it's not really possible because stops associated to routes (and variants) are connected in transitfeed's objects through the trips. Something osm2gtfs generates at the very end based on stops over routes. So this would just not work, as far as I understand.
Ergo, we need a parallel structure(!?). So, let's think about this and how this would look like.
Some ideas:
find_best_name_for_unnamed_stop()
, if possibleHi there,
the script obtains some attributes from the route from OpenStreetMap, such as reference number (ref
) and the name. And then there are additional attributes hard-coded in the script, these are:
Those should be pulled from OpenStreetMap and not be hard coded.
The tagging schema provides tags on bus routes for description
, colour
and we can also add website
. For the text color we could either come up with a new tag in OSM or define it in the script's config file (#3)
In code currently stops without names get assigned "[" + self.stop_no_name + "]"
and after trying to find a good name (OsmConnector._find_best_name_for_unnamed_stop()
) it just uses stop_no_name
without brackets to not query again and again. We probably should move to None
.
We need documentation on how the script works, which functionalities it has and how the config file is structured. I suggest to use the wiki here in this github repo.
Debugging, such as screen output and special functions like Route.print_shape_for_leaflet()
are currently within regular code. I'd like to optimize this. Right now I'm thinking about:
Debug
class for special functionsModules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability.
Class names should normally use the CapWords convention.
Our creator modules (filenames) however are currently using CapWords convention. This should be changed to fit PEP 8.
The stops
parameter of Route Class
is a list the is being assigned with two different object types during the default execution of the script.
In osm_connector
it is a list of strings, where each string corresponds to the node ID of the stop in OSM. In the default creator
the stop parameter is changed to a list of osm2gtfs Stop Objects
.
Providing transfers information could enhance the use of the generated GTFS for journeys computation.
There could be several ways of doing it :
Currently, all the requirements are just listed in the README.md, but they should probably be moved into a requirements.txt
file for easy installation.
Making the config file argument required is not ideal. I suggest to have config.json
as a default argument for the config parameter, but don't include this file in the repository and even add it to .gitignore
.
This enables people to have their local config file and play with it without it being tracked by version control and without having to specify the file every time they want to run the script. If people are happy with the official file in the repository, they can symlink config-fenix.json
to config.json
.
The current route creation is very much custom for Florianópolis. After the abstraction work happening in (#9) I'd suggest working on a default implementation for route generation based on OSM's Public Transport Schema 2.
Managua can be a good start as the data quality is very consistent and complying the standard data structure without exceptions.
Please add a (open) licence to the repository, to enable collaboration. Thanks!
There are some function calls and variable names in Portuguese language. I have spotted them primarily in the osm_routes.py
file. These should be cleaned up.
What is the difference between travel_time
and duration
attributes of Route
object ?
(Sorry for the delayed question related with PR #66.)
Hello to all!
I´m trying to run osm2gtfs using ubuntu 14 64 bits.
Steps i have made:
**Obtaining file:///opt/osm2gtfs
Running setup.py (path:/opt/osm2gtfs/setup.py) egg_info for package from file:///opt/osm2gtfs
Downloading/unpacking overpy>=0.4 (from osm2gtfs==0.0.1)
Downloading overpy-0.4.tar.gz (41kB): 41kB downloaded
Running setup.py (path:/tmp/pip_build_root/overpy/setup.py) egg_info for package overpy
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
your setuptools is too old (<12)
setuptools_scm functionality is degraded
zip_safe flag not set; analyzing archive contents...
Installed /tmp/pip_build_root/overpy/pytest_runner-3.0-py2.7.egg
warning: no files found matching 'docs.Makefile'
Downloading/unpacking transitfeed (from osm2gtfs==0.0.1)
Downloading transitfeed-1.2.14.tar.gz (347kB): 347kB downloaded
Running setup.py (path:/tmp/pip_build_root/transitfeed/setup.py) egg_info for package transitfeed
Installing collected packages: overpy, transitfeed, osm2gtfs
Running setup.py install for overpy
warning: no files found matching 'docs.Makefile'
Running setup.py install for transitfeed
changing mode of build/scripts-2.7/feedvalidator.py from 644 to 755
changing mode of build/scripts-2.7/schedule_viewer.py from 644 to 755
changing mode of build/scripts-2.7/kmlparser.py from 644 to 755
changing mode of build/scripts-2.7/kmlwriter.py from 644 to 755
changing mode of build/scripts-2.7/merge.py from 644 to 755
changing mode of build/scripts-2.7/unusual_trip_filter.py from 644 to 755
changing mode of build/scripts-2.7/location_editor.py from 644 to 755
changing mode of build/scripts-2.7/feedvalidator_googletransit.py from 644 to 755
changing mode of build/scripts-2.7/shape_importer.py from 644 to 755
changing mode of /usr/local/bin/feedvalidator.py to 755
changing mode of /usr/local/bin/kmlparser.py to 755
changing mode of /usr/local/bin/schedule_viewer.py to 755
changing mode of /usr/local/bin/location_editor.py to 755
changing mode of /usr/local/bin/shape_importer.py to 755
changing mode of /usr/local/bin/feedvalidator_googletransit.py to 755
changing mode of /usr/local/bin/unusual_trip_filter.py to 755
changing mode of /usr/local/bin/kmlwriter.py to 755
changing mode of /usr/local/bin/merge.py to 755
Running setup.py develop for osm2gtfs
Creating /usr/local/lib/python2.7/dist-packages/osm2gtfs.egg-link (link to .)
Adding osm2gtfs 0.0.1 to easy-install.pth file
Installing osm2gtfs script to /usr/local/bin
Installed /opt/osm2gtfs
Successfully installed overpy transitfeed osm2gtfs
Cleaning up...**
Check if transitfeed is installed
pip install transitfeed
Requirement already satisfied (use --upgrade to upgrade): transitfeed in /usr/local/lib/python2.7/dist-packages
Check if overpy is installed
pip install overpy
Requirement already satisfied (use --upgrade to upgrade): overpy in /usr/local/lib/python2.7/dist-packages
Finally i run osm2gtfs command
osm2gtfs --config incofer.json --output testinconfer
But i get an error
Traceback (most recent call last):
File "/usr/local/bin/osm2gtfs", line 9, in
load_entry_point('osm2gtfs==0.0.1', 'console_scripts', 'osm2gtfs')()
File "/opt/osm2gtfs/osm2gtfs/osm2gtfs.py", line 35, in main
config = Configuration(args)
File "/opt/osm2gtfs/osm2gtfs/core/configuration.py", line 32, in init
self._prepare_dates()
File "/opt/osm2gtfs/osm2gtfs/core/configuration.py", line 98, in _prepare_dates
if 'start_date' in config['feed_info']:
KeyError: 'feed_info'
What i doing wrong? Some one knows how to solve it?
Thanks
The current trips creation is very much custom for Florianópolis. This issue here is the place to discuss how a default implementation for trips creation could and should look like. Probably it's going to be in the direction of @jamescr Incofer trips creation, as he is using a standard json containing schedule information for all stops. Fenix and Managua is doing more fancy and custom stuff to create a schedule, which the can live (partially) in the respective individual creators.
Hello to all,
I'm very interested in this project. So thanks to the authors.
Can u upload again the json files, for example incofer.json for costa rica?
I have checked all links but are dead
Thanks
A load_agency(config) method returning a transitfeed.Agency object created using data from the json config file. In the json file under the "agency" object, the valid keys will be any member of the transitfeed.Agency._FIELD_NAMES array.
Hi,
in code there is a blacklist of some routes hard-coded into the script. Are these from a different network
? Or is there any other reason they are excluded. I'd suggest to mark them in the data of OSM to be able to sort them out, and removing the blacklist in code.
Originally the script was created with bus routes in mind but this could also work for train routes (as the case of incofer train routes)
The idea is to add an optional argumnet: "--route-type". the default value will be "bus".
Currently we have the following folder (package) structure
creators
(extendible part of the application)factory
(handles logic to for the creators)osmhelper
(everything else: OsmConnector, Cache, data structure)data
(data caching, input and output)src
(external python packages)I'm proposing to merge factory
and osmhelper
into one single package called core
. What do you think?
Instead of printing out random messages, it would be nice to get a kind of "report" of the data it ran on, things to improve on OpenStreetMap, etc. These information should be grouped.
In order to generate proper GTFS files good data quality in OSM is recommended, if not required.A little writeup/manual would help users how to assure data quality.
I think the wiki here would be the best place to live for this information. Maybe once it's getting to be more complete we could integrate it into learnOSM.org
Just a rough brainstorm list of the contents I suggest to include:
public_transport:version=2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.