In Italy there are no available official Open Data about the performance (delays, cancellations, ...) of the italian public rail transport. This project offers a tool which allows anyone to gather it and run some stats and visualizations.

flowchart TB

S[Scraper] --> |Downloads data| D("ViaggiaTreno and Trenord APIs")
S -->|Produces| P[(Daily .pickle dumps)]
E[Extractor] -->|Reads| P
E[Extractor] -->|Produces| C[(Daily .CSV dumps)]
A2["(BYOD Analyzer)"] -.->|Reads| C
A[Analyzer] -->|Reads| C
A[Analyzer] -->|Produces| K(Stats, visualizations, etc...)

The application is composed by multiple modules, accessible via CLI:

• scraper: unattended script to incrementally download and preserve the current status of the italian railway network. If run constantly (e.g. ~every hour using cron) all trains will be captured and saved in data/%Y-%m-%d/trains.pickle.
• train-extractor and station-extractor: converts raw scraped data to usable .csv files;
• analyze : shows reproducible stats and visualizations.

The project is written in Python and it uses modern typing annotations, so Python >= 3.11 is needed.

A Dockerfile is available to avoid installing the dependencies manually. You can use the automatically updated ghcr.io/marcobuster/railway-opendata:latest Docker image if you want the latest version available on the master branch.

For instance, the following command will start the scraper on your machine.

$ docker run -v ./data:/app/data ghcr.io/marcobuster/railway-opendata:latest scraper

⚠️ WARNING: this project currently uses the builtin hash(...) function to quickly index objects. To ensure reproducibility between runs, you need to disable Python's hash seed randomization by setting the PYTHONHASHSEED=0 environment variable. If you fail to do so, the software will refuse to start.

$ export PYTHONHASHSEED=0
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r requirements.txt
$ python main.py ...

• Start the scraper. For continuos data collection, it should be run every ~hour.

  $ python main.py scraper

• Extract train data from a pickle file and save it in CSV.

  $ python main.py train-extractor -o data/2023/04-29/trains.csv data/2023-04-29/trains.pickle

• Extract station data from a pickle file and save it in GeoJSON.

  $ python main.py station-extractor -f geojson data/stations.pickle

• Describe a dataset and filter observation by date.

  $ python main.py analyze --start-date 2023-05-01 --end-date today data/stations.csv data/2023-05-*/trains.csv --stat describe

• Show delay stats of the last stop.

  $ python main.py analyze --group-by train_hash --agg-func last [..]/stations.csv [..]/trains.csv --stat delay_box_plot

• Show daily train count grouped by railway companies.

  $ python main.py analyze --group-by client_code [..]/stations.csv [..]/trains.csv --stat day_train_count

• Display an interactive map and open it in the web browser.

  $ python main.py analyze [..]/stations.csv [..]/trains.csv --stat trajectories_map

• Display a timetable graph.

  $ python main.py analyze [..]/stations.csv [..]/trains.csv --stat timetable --timetable-collapse

In the extracted trains CSV, each line is a train stop (not station nor train). Many fields are actually duplicated.

See CONTRIBUTING.md.

The ViaggiaTreno APIs are known to be buggy and unreliable. As stated before, many fields (like departure_expected and arrival_expected) are not always guaranteed to be present and some concepts are counter-intuitive (a train number is not an unique identifier nor are station codes).

ViaggiaTreno is the main source of truth for many final user applications (like Trenìt! or Orario Treni) and is itself linked on the Trenitalia official website. For instance, if the API does not return information for a train stop, no other application will display it: the data simply does not exists online. The scraper always tries to save as much data as possible ("best effort") even when is probably incomplete; in those cases, proper flags (like phantom and trenord_phantom) are activated so the developer can choose for themselves.

Copyright (c) 2023 Marco Aceti. Some rights reserved (see LICENSE).

Terms and conditions of the ViaggiaTreno web portal state that copying is prohibited (except for personal use) as all rights for the content are reserved to the original owner (Trenitalia or Gruppo FS). In July 2019 Trenitalia sued Trenìt for using train data in its app, but partially lost. I think data about the performance of public transport should be open as well, but I'm not a lawyer and I'm not willing to risk lawsuits redistributing data; if someone wants to, the tool is now available.

BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.