POC to generate consolidated statistics from json files (generated by navitia-stat-logger or navitia-stat-exporter)

• Spark 1.5+ (may work with previous versions, but untested)
• A repository with exported statistics files where stat files are stored in a tree like

  |
  \- <year>
      |
      \- <month>
          |
          \- <day>
              |
              \- <files>.json.log(.gz)

The files are json logs (one json per line). The files may be compressed using gzip.

• For requests_calls consolidation

<path/to/spark>/bin/spark-submit  --conf spark.ui.showConsoleProgress=true --master='local[3]' requests_calls.py <your_export_directory> <start_date> <end_date>

where:

• start_date and end_date is in YYYY-MM-DD format

• For coverage_journeys consolidation

<path/to/spark>/bin/spark-submit  --conf spark.ui.showConsoleProgress=true --master='local[3]' coverage_journeys.py <your_export_directory> <start_date> <end_date>

where:

• start_date and end_date is in YYYY-MM-DD format

Note that the results are stored in the export dir

Launching 'spark-submit' command , if you get error :

zsh: no matches found

You can replace in command line --master='local[3]' by --master=local, or use bash.