In the same vein as my previous #82 post, I think you should "make visible" the PostgreSQL database installed by the Nominatim stack.
That would allow users to rely on their own database (in our case, one that is installed on another server) instead of deploying a database local to Headway.
That would also allow a better use of already existing resources, such as database backups, replication, etc.

Offline geocoding would probably be best accomplished by getting Carmen to run in the browser, per #50. This seems like it will require removing rayon and rocksdb from carmen-core, rewriting or repackaging vtquery and probably also other things. I'm guessing Carmen itself will need to be ported over to use local storage instead of the node filesystem APIs too. After the dust settles performance will also need to be evaluated in terms of response time, latency from a cold cache, and subjective quality of the results.

Steps to reproduce

• Visit
• Enter "pitt street, sydney" into the Where To? Box

Expected Result

Each result is unique

Actual Result

There are 3 different results with exactly the same text and subtext.

Currently headway is set up with a single area/city which decides which data is pulled in from all sources.

One common scenario I think should be considered: Single country, with transit direction for 1-3 cities inside that country (for smaller countries this can still be less resources than some of the larger cities)

So we may have a larger map but only transit integration for a subset of it.

Note that this is not the same as specifying arbitrary areas or combinations thereof (implementing that would solve this though),which would great but may not be as trivial to implement

It would be nice if there was a way to update them too but that sounds like a can of worms so it probably just makes sense to track that separately.

Hi and thank you for this great project!

Unfortunately for me, I can't go through with the building of NewYork base data.

The make NewYork command begins failing at 16:53:00.408 with this message:

ERROR (OTPMain.java:55) The provided transit date have no trips within the configured transit service period. See build config 'transitServiceStart' and 'transitServiceEnd': [2012-07-28, 2025-07-28]
16:53:00.415 INFO (OtpStartupInfo.java:47) OTP SHUTTING DOWN (version: 2.1.0, ser.ver.id: 2.1.0, commit: 0d24cc0a3f7ddddf1db87d805e09fd0ca70eb5b5, branch: master)
The command '/bin/sh -c java ${JAVA_MEM_ARGS} -jar /otp/otp-shaded.jar --build --save .' returned a non-zero code: 100
make: *** [Makefile:101: /mnt/cru/kuq/headway-main/data/NewYork.graph.obj] Error 100

As a result, the transportation data is not buit and the whole stack fails after Valhalla fails when launched.
Here is the (expected) error message in Valhalla's docker console:

cp: cannot stat '/data_mount/NewYork.valhalla.tar': No such file or directory

Please help.
Thank you. Stephen

Use of OpenStreetMap data requires attribution to be provided when data from the project is utilised.

See https://www.openstreetmap.org/copyright

Now that custom extracts are supported, we really need viewport biasing for autocomplete and (once it's implemented) the search results page.

Hi Ellen,
Thank you for the great work on NYC! I have tried installing the stack and it passed the error point I had mentioned.
However, I get yet another blocking error further down, this time at ITAG=headway_build_fonts, when installing fontnik.

Here is the trace in the console:

Step 2/23 : RUN npm install fontnik
 ---> Running in a8d8a75e27b6
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm ERR! network request to https://registry.npmjs.org/minimist failed, reason: getaddrinfo ENOTFOUND registry.npmjs.org
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-08-11T18_21_08_481Z-debug.log
The command '/bin/sh -c npm install fontnik' returned a non-zero code: 1
make: *** [Makefile:151: /mnt/cru/kuq/hwk/data/fonts.tar] Error 

As I understood it is an error while connecting to the NPM registry, I tried to connect manually using curl:

curl https://registry.npmjs.org/rc

And I got a correct response.

So I assume there is still an error there.
S.

Originally posted by @asitemade4u in #66 (comment)

I want to make a website for the project that captures the convenience of all those sites out there that tell you to pipe curl into bash to install their software. I own the domain maps.earth and want to use it for this site.

What I'm imagining is a world map rendered with pmtiles, and a UI on top of it that lets you search for an administrative area that may not be in the BBBike extract list, e.g. Atlanta, Georgia. The UI will let people draw a bounding box over an area they care about and prepare an OSM extract for them. (tbd how to make this scale, but I have some ideas). It would also collect GTFS feeds for the bounding box, and maybe even let people disable some of them. Once everything is selected, a make command could be generated with a token for that user's selections.

Running the command would kick off the OSM extract and spin wait on the user's machine while it's running, then download the extract and GTFS feeds and kick off the rest of the build per normal.

This is a long-term suggestion but I've looked at the logs for the demo server and almost all of the geocoder requests are for cities outside of the Seattle metro area. It turns out people really just want to see their own cities on the map before they get really interested.

This may improve search result quality on https://maps.earth/

• openaddresses: #289
• geonames

I don't want people getting the wrong impression about Headway's transit capabilities now that we don't build with OTP by default.

I just found this: graphhopper/graphhopper#1743 and seems like this might not be a priority for the graphhopper team. I know OTP supports transit shapes. I was also not as impressed with Graphhopper for bicycle routing as I have been with Valhalla.

What I'm envisioning is that the user's location could be projected onto the route polyline, and the scroll position along the route timeline set appropriately. Bonus points to adjust the ETA based on current location and the splits of the steps remaining. Extra bonus points to adjust the split of the current step based on how far through it the user is.

Super bonus points to adjust the polyline style for the portion of the polyline that's already been traveled.

TBD whether Valhalla's GPS tracking service makes sense to use here. The privacy implications of that are pretty bad if the server is untrusted, but implementing that wouldn't necessarily block implementation of offline routing because Valhalla does compile to WASM.

Right now we only hit the pelias autocomplete endpoint which doesn't perform interpolation, among other issues. Letting users press enter to see an actual search result page would give us the opportunity to hit the search endpoint and get additional address coverage that way.

I just got an API key for the OneBusAway real-time API in my area, so now I have access to some real-time transit feeds. I think it would be very cool to plot transit vehicle locations on the map, and I believe OpenTripPlanner can incorporate real-time delay information into its routing which would be very cool to implement.

OTP right now is the biggest memory hog by far, so it would be nice to let people disable it if they don't have memory to spare.

Just splitting this out into its own issue to prevent clutter in #50.

Valhalla is currently building with emscripten in https://github.com/headwaymaps/valhalla but a lot of the porting process remains. Tile fetches and tile caching are the most obvious examples, requiring the use of emscripten's networking capabilities and filesystem capabilities respectively.

This should solve the issues that show up when you click on a result that is an OSM way (or maybe also a relation) and the rich display name is replaced with the name of the neighborhood or suburb.

I had this idea while talking with some folks in the OpenStreetMap slack. I think it would be really cool to kick people back to OSM to map their own local places, improving OSM, headway and maps.earth for everybody.

I like the way it's explained here. Specific to pelias but still. https://geocode.earth/blog/2020/virtuous-cycle

It would be really nice to have a good kubernetes config or config template for Headway. See #39 for additional information.

Looks like bash variables aren't being assigned properly in the makefile.

Steps to reproduce:

git clone $HEADWAY_GIT_URL
cd headway
cp .env.example .env
make Amsterdam

Expected: Build succeeds.

Actual:

ITAG=headway_build_gtfs_$(echo Amsterdam | tr '[:upper:]' '[:lower:]')
HEADWAY_BBOX=$(grep "Amsterdam:" web/bboxes.csv | cut -d':' -f2)
docker build ./gtfs_build --build-arg HEADWAY_BBOX="${HEADWAY_BBOX}" --tag ${ITAG}
flag needs an argument: --tag
See 'docker build --help'.
make: *** [/Users/evinsellin/code/headway/data/Amsterdam.gtfs.tar] Error 125

I'll have a fix for this shortly.

So:

1. I have deployed the NewYork stack
2. using BBox Finder, I have defined a BBox for the map
3. I have added this entry in the .env file: HEADWAY_FORCE_BBOX="-74.051971,40.622813,-73.893013,40.751418"
4. But nothing special happens, except that the error message displayed when there is no BBox defined does not show any longer.

I tried HEADWAY_FORCE_BBOX="-74.051971 40.622813 -73.893013 40.751418" and it still does not work

It would be really nice to have continuous integration that clones the repo, builds it for a small city and at least does health checks for all the services we bring up. There's a lot of churn happening right now but when the dust settles that seems like a good goal.

I also want to do more code review. I've sworn off committing directly to main because it's clear that other people actually care about this project now, but just now with #23 I realized there isn't really anyone around to review anything except one person, and I don't want to shove all the review work onto one person. So if you want to help out with that leave a comment here!

Probably OSM Bright, because if memory serves it's easier to self-host than some of the others.

This was tracked in my head but it fell off my radar so it makes sense to track it explicitly I think. The trip planner frontend is the only mode that is completely unusable right now. Whereas the others are just missing ETAs and a decent nav experience, the trip planner is completely lacking because it doesn't display steps at all, only polylines representing each sub-trip.

For a long time I didn't think I wanted to support the full planet use-case but the fact of the matter is that's what everyone wants, and I think it would be really exciting for people and maybe hopefully convince some folks to help me work on the frontend. :)

• Make sure everything is usable with large areas (see: #78 maybe others)
• Add timeouts to prevent queries of death, especially to Valhalla.
• Rate limit requests to geospatial services in nginx based on IP address to prevent people from easily using us for batch queries.

Also consider switching to Pelias? I don't want to cause a bunch of churn but it will be extremely expensive to run a full planet installation of nominatim, and imports are very slow. Plus we currently store about 2x as much geocoding data as we really need to since we have two geocoders. Pelias has search & autocomplete, and a places API that could replace nominatim.

Currently Headway uses bright style but it would be nice to allow the style to be changed from the docker-compose.yml file, as an environment entry.

Headway is GREAT and works fast.
I was wondering if it was possible to add one or several personal layers of POIs on top of the OSM tiles.
It would be useful, for example, to add GPS tracks, geolocate photos taken, etc.

OpenTripPlanner would be a much better choice for transit than valhalla. It's unclear if valhalla's transit support is even maintained.

See: #82 (comment)

We should push images in the CI config to the GitHub container registry, then headway users can pull those images instead of having to build their own. Care will need to be taken to ensure that images can be overridden for local development.

I went to ToorCamp 2022 recently and had a conversation with someone who suggested I use Gradle for the build system of Headway instead of Make. They mentioned a few benefits, including the fact that if a build step fails then the artifacts from that build step aren't cached or written to disk. It would also cache build artifacts in a smarter way, meaning things don't get evicted from the cache needlessly.

In other words if I ran:

make Adelaide
make Seattle
make Adelaide

In step 3, Gradle would use the cached Adelaide artifacts from step 1.

Both of these seem huge to me, because they would completely negate the benefits of using docker build instead of docker run, reduce docker image clutter, and reduce wasted effort spent on updating Makefile style to work on various platforms/versions.

If nobody objects to the use of Gradle in the near future, I think I'll begin work on this. I'm entirely open to using just about anything that provides the cache and reproducibility-on-error benefits that Gradle seems to, just want to evaluate everything on its merits. :)

It would be nice to have the option of clicking/tapping a spot on the map to enter that as the start (or end) point of the route.

Just wanted to put it out there, because, as other tickets and HN comments have mentioned, the address autocomplete isn't super sophisticated yet, so adding this functionality would make a it just a bit easier to use the routing capabilities in the meantime.

For example, my home address doesn't appear in the autocomplete at all, so I typed in a nearby restaurant which did pop up. With that location set as the destination, having the option to click near my house would be super convenient. For a variety of reasons, I find that I use that functionality on Google Maps quite a lot, actually.

Thanks for this ambitious project! I'm looking forward to following it :)

When hosting behind a reverse proxy using https, tiles cannot be loaded.

I was able to track it down to the magic_string_magic dummy host that is defined in https://github.com/headwaymaps/headway/blob/main/web/nginx.conf.template#L53.

It seems to only redirect accesses to http://magic_string_magic, but when the whole page is https, also this call will be https://magic_string_magic.

I was able to fix it by manually changing both occurrences of http://magic_string_magic to https://magic_string_magic (but of course this only works temporary until the container gets re-created).

I think a proper fix would be to have an ENV var defining whether the whole stack is hosted behind https and changing the entries in nginx.conf.template accordingly (could maybe even be dynamically parsed from HEADWAY_PUBLIC_URL).

#21 nominally added this feature but when I ran it I found the following error message:

rm -rf /home/ellen/dev/headway/data/*.mbtiles
rm -rf /home/ellen/dev/headway/data/*.nominatim.sql
rm -rf /home/ellen/dev/headway/data/*.osm.pbf
rm -rf /home/ellen/dev/headway/data/*.photon
rm -rf /home/ellen/dev/headway/data/bbox.txt
rm -rf /home/ellen/dev/headway/data/sources
rm -rf /home/ellen/dev/headway/data/nominatim_flatnode
rm -rf /home/ellen/dev/headway/data/nominatim_pg
docker images | grep ^headway_build_ | tr -s ' ' | cut -d' ' -f3 | xargs docker rmi
Error response from daemon: conflict: unable to delete f7354fa160e3 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 32a0db2e6aa4 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 7687b7da0d06 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 7f5243e4dfa0 (must be forced) - image is referenced in multiple repositories
make: *** [Makefile:150: clean_all] Error 123

I would be thrilled to help finance the project but would prefer to donate a one time donation.
I know that is possible with GitHub: please allow it.

Hi Ellen,

1. My goal is to manage several cities using Headway, notably NYC and Paris.
2. Of course I have tried and succeeded in installing several different "stacks" in different folders.
3. Now I would like to share the "application" part of each stack and have only the data specific to each city.
4. As we have here a local docker Registry, I decided I would rewrite the docker-compose.yml file so that each stack fetches a common image from the Registry instead of fetching it from the local folder -- which is the way docker stacks are supposed to work.

It did not work because Headway saved dependencies within the containers instead of externalizing them as data.
So here is what happens:

1. I saved each container in NewYork as an image to our Registry
2. I then stopped the NYC stack
3. I docker-pruned the system on the server to be sure the slate would be cleaned for the test
4. I modified the docker-compose.yml file of Philadelphia so that the image underlying each container is fetched from our Registry
5. then launched it using docker-compose up -d as usual
6. Photon failed because it could not find the /data/Philadelphia.photon.tar.bz2 file, and for a good reason: the image had been created for NewYork
7. As a result Headway did not work at all.

Here is the trace from the console:

Extracting photon index
+ echo 'Extracting photon index'
+ cd /
+ pbzip2 -d
+ cat /data/Philadelphia.photon.tar.bz2
+ tar x
cat: /data/Philadelphia.photon.tar.bz2: No such file or directory
pbzip2: producer_decompress: *ERROR: when reading bzip2 input stream
Terminator thread: premature exit requested - quitting...

Here is a simple fix: load the file from a mounted volume instead, eg. from the /data folder mounted local volume.

I think the reason that Headway got a lot of traction on HN is because it currently, right now, solves a problem that people have. It provides maps for everyday use in a chosen metro area, without compromising privacy. If the project moves towards decentralized or federated hosting of an entire world map, I want to make absolutely certain that I'm personally happy with the level of privacy it provides (see: #50). But I also want to make sure that I'm not forgetting that a lot of people don't want to make requests to servers they don't control, at all, period, end of story. Headway must continue to be self-hostable. This issue exists to track that. The default should be for headway to bring up a self-hosted maps stack. We may choose to add a make planet target or some environment variables that control federation, but I don't want to lose track of what made people so excited about Headway in the first place.

Bottom line: Headway instances should never call out to external servers at runtime unless specifically configured to do so. Nor should they ever serve a page that solicits clients to call out to any external servers unless specifically configured to do so. At least, that's my opinion.

Right now I know of these cartography issues:

• Fix highway sprites (like interstate icons in the US) not showing up on the map
• Fix the missing pedestrian_polygon sprite (see: https://github.com/ellenhp/headway/pull/45#issuecomment-1144710951)
• Identify which openmaptiles layer contains park information and change the style to point to the correct layer (see: https://github.com/ellenhp/headway/pull/45#issuecomment-1144710951)

But there may be others.

Do a bbox intersection with the feeds from mobilitydata.org, then download the ones that intersect, I guess? That makes sense for now, but if people do larger installations they may want to pick and choose which ones they download.

Getting the following errors in browser console after running make Sucre.up.

GET /static/[email protected] 404 (Not Found)
GET /static/[email protected] 404 (Not Found)
Error: Source layer "park" does not exist on source "openmaptiles" as specified by style layer "park".
Error: Source layer "park" does not exist on source "openmaptiles" as specified by style layer "park_outline".

So, I was able to build Philadelphia and to launch the docker stack successfully.
But when I access the server on port 8080:

• the interface shows,
• the routing engine works
• but the map (tiles) itself does not show.

I tried (on Debian) with Firefox and Brave browsers and the same.

I'm noticing that since nominatim is not installed on the (container local) postgres server on the photon import step, the following error is yielded:

ERROR:  could not access file "/usr/local/lib/nominatim/module//nominatim.so": No such file or directory
ERROR:  function public.transliteration(text) does not exist

Could be resolved by either installing the module inside the photon container, or use a shared postgres server (either independent or the one bundled in the nominatim container)

Or maybe this is a non-issue?

Pelias doesn't offer a url param to control how strongly to bias the results towards the focus point, so in many cases I've found that it doesn't bias strongly enough. Look into getting this implemented upstream?

Right now nominatim is just used as a source of truth for OSM data. Only the lookup endpoint is exposed. There's no reason we couldn't replace it with something more lightweight built on top of postgis. That could drastically reduce the memory requirements for the full Headway stack, potentially even to the point that the whole planet could be hosted with 64GB of RAM which is about the point where it becomes feasible to do.

Lots of users have asked for this, and it could alleviate the need for a really slick bootstrapping UI (see #49). There are a number of blockers, mostly privacy related:

• Base map
  • Investigate offline downloads to help with privacy (I don't want the server to know where in the basemap I pan/zoom to at any given moment)
• Geocoding
  • Get Mapbox Carmen working in the browser
  • Integrate Carmen into the headway frontend for offline geocoding
  • Confirm granularity of Carmen tiles isn't too fine for privacy's sake
• Routing
  • Get valhalla compiling to wasm
  • Integrate valhala into the headway frontend for offline routing
  • Serve raw valhalla tiles
  • Investigate the privacy issues inherent in sending ad-hoc requests for valhalla tiles to an untrusted server (can they reliably determine my future location information?)
  • Investigate whether offline downloads could help with this privacy issue

edit: These are just the hard blockers, there's a lot of architectural stuff that would need to happen before this becomes feasible.

This init container thing we do right now doesn't scale well to cover the planet. It's not reasonable to have one copy of each artifact per pod when the artifacts are tens or hundreds of gigabytes. My current plan is to create a longhorn RWX volume claim and share that between all of the pods. Each pod will mount the volume claim as read-only and serve data out of it. I'll have to write a new service to manage the data that's kept in the pvc and help do the following:

1. download new data when it becomes available
2. restart the deployments as necessary when they need to start serving new data
3. determine when it's safe to delete old data by examining running pods
4. actually delete the old data once it's safe

That service will need to be able to access the cluster's api (kubelet? still getting up to speed on terminology) for items 2 and 3.