agrenott / pyhgtmap Goto Github PK

First of all. I'm happy that someone continues with pyhgtmap, would be a pitty.

Today a lot of countries provide better source date for elevation lines than viewfinderpanoramas or srtm type data as open data. We want to use this more precise and open data for creating better elevations.

But having to fiddle arround and put that into a so called 'cache' directory is sort of difficult and I was also not able to recreate the needed index.
If I handover all needed hgt files (or geotiff files) on the command line the elevation lines are calculated for everything handed over.
I want to be able to apply there a polygon with the argument --polygon as possible when running (after downloading) against the cache directory.

In my opinion (and especially for our need) this would be a great enhancement of the tool.

Cheers
Patrik

As I couldn't get in touch with the original author, I'm going to officially fork this project under a new name, and release it on PyPi.
It will also be the opportunity to move to more modern packaging.

New name will probably be "pyhgtmap" (notice the "ght" -> "hgt"), as I never understood why it wasn't the original name anyway...
I'll probably start numbering at v3.0.

Phyghtmap used to output to a single -pbf file. Would it be possible to have the same behaviour again instead of one output file per input file?
I know I could use osmconvert to do this - but I think it would be easier if pyhgtmap can output directly to a single file.

And yeah - great improvements overall and much faster now! Is the jobs parameter doing anything? In phyghtmap it used to be broken that I could max a value of 2 but no more (didn't check this for a long time - so maybe was solved at some point).

While trying to setup data for rendering OpenTopoMaps I tried to replace the no longer maintained phyghtmap by this project. First of all thank you very much for further maintaining this fork :)

While following the guide in OpenTopoMaps, I encountered a problem with a tif in EPSG:3857. It produced an invalid OSM XML with coordinates in EPSG:3857 instead of the expected EPSG:4326. See an example below.

<?xml version="1.0" encoding="utf-8"?>
<osm version="0.6" generator="pyhgtmap 3.6">
<bounds minlat="48.0001861" minlon="5.9999876" maxlat="52.0001678" maxlon="17.9995035"/>
<node id="10000000" lat="6608469.6395790" lon="723805.5616385" version="1"/>
<node id="10000001" lat="6608455.7934252" lon="723769.5616385" version="1"/>

I had a look at the implementation and I think the issue came up with the simplification in 8fc9edd. Here we removed the clipPath method and with that, we also removed the transformation. The old comment actually stated This method also does a potentially needed transformation of the projection and as a first step the projection was applied.

298>        # do the transform if necessary
299>        if self.transform != None:
300>            path = numpy.array(self.transform(path))

After removing the clipPath method also the transformation is no longer applied, in fact the self.transformation is no longer used at all as far as I can see. I think we should add again the snippet from above to the trace method? I already quickly tried to do something like this (although I am not 100% sure how to correctly handle the numpy.ndarray) and it produced better looking coordinates.

GDAL comes with painful dependencies, and is used only for GeoTiff processing.
Would be nice to make it an optional dependency.

hgt file /home/contourlines/hgt/VIEW1/N38E012.hgt: 3601 x 3601 points, bbox: (12.00000, 38.00000, 13.00000, 39.00000)
Traceback (most recent call last):
File "/home/contourlines/my_venv/bin/pyhgtmap", line 8, in
sys.exit(main())
File "/home/contourlines/my_venv/lib/python3.10/site-packages/pyhgtmap/main.py", line 589, in main
HgtFilesProcessor(opts.nJobs, opts.startId, opts.startWayId).process_files(
File "/home/contourlines/my_venv/lib/python3.10/site-packages/pyhgtmap/hgt/processor.py", line 177, in process_files
self.process_file(file_name, check_poly, options)
File "/home/contourlines/my_venv/lib/python3.10/site-packages/pyhgtmap/hgt/processor.py", line 173, in process_file
self.process_tile(file_name, tile, options)
File "/home/contourlines/my_venv/lib/python3.10/site-packages/pyhgtmap/hgt/processor.py", line 143, in process_tile
p.start()
File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 65, in _launch
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

Trying pyhgtmap on Europe with 10m inverval (view1) it crashed out like this. I will try again with jobs=2 instead of jobs=12 but I'm pretty sure it did not run out of memory. Asia would be a whole different animal - took me many tries on old phyghtmap to get it working (though I think the main problem was splitting the outpup file with mkgmap splitter)

When I run
pip install -U pyhgtmap

nothing happens - because I think the main version is still 3.0.0. Could you please add an explanation on how to update to latest sourcode from Github? Where do I need to move this code and what to run then?

Happy New Year !

I've run into a problem while trying to build contour file from source sonn1 with a polygon file that contains a multipolygon.
The errormessage points me into the direction that ithas to do with the multipolygon, but we have other multipolygon cases that didn't fail (at least I didn't notice it) and the final map was built correctly.
Possibly there is something very specific with this poly file or pyhgtmap runs through other code branches than with other poly files..... or we have a problem with all multipolygon files and didn't notice it yet.

I will try to build other contour files with other multipolygon to find out more from my end too.

The actual error I get ist:

  File "/home/xxx/venv_pyhgtmap/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 593, in main
    main_internal(sys.argv[1:])
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 585, in main_internal
    HgtFilesProcessor(opts.nJobs, opts.startId, opts.startWayId, opts).process_files(
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/hgt/processor.py", line 268, in process_files
    self.process_file(file_name, check_poly)
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/hgt/processor.py", line 222, in process_file
    hgt_tiles = hgt_file.makeTiles(self.options)
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/hgt/file.py", line 668, in makeTiles
    chopData(bbox, truncatedData)
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/hgt/file.py", line 640, in chopData
    tileMask = polygon_mask(
  File "/home/xxx/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/hgt/file.py", line 275, in polygon_mask
    clipped_polygons.append([(x, y) for x, y in clipped_p.exterior.coords])
AttributeError: 'GeometryCollection' object has no attribute 'exterior'

I found interesting information at the following link:
https://stackoverflow.com/questions/64099107/convert-multipolygon-geometry-into-list

As I can not attach the poly file here directly, please find it at:
https://github.com/freizeitkarte/fzk-mde-garmin/blob/c99ffbb32d7439e5d9c7d4288b76054d11405f8b/Freizeitkarte-Entwicklung/poly/Freizeitkarte_GBR%2B.poly

Hope this helps to fix it.

Cheers Patrik

I'm not sure if the downloadpart from Sonny is already running or if I cought it in an intermediate state.

It looks like I got the google drive authentication running, at least I do not get any errormessages related to this.

But when running the tool I get the following errormessages:

(venv_pyhgtmap) xyz@myhostname:~/develop/fzk-utils/elevation$ pyhgtmap --step=20 --osm-version=0.6 --jobs=1 --line-cat=500,100 --start-node-id=750000000000 --start-way-id=750000000000 --max-nodes-per-tile=0 --pbf --polygon=./poly.work/Freizeitkarte_LUX.poly --source=SONN3 --write-timestamp --output-prefix=./pbf/ele_20_100_500/Hoehendaten_Freizeitkarte_LUX
N49E005: trying sonn3 ...
Traceback (most recent call last):
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/__init__.py", line 68, in get_file
    self.check_cached_file(file_name, resolution)
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/__init__.py", line 49, in check_cached_file
    found_size: int = os.path.getsize(file_name)
  File "/usr/lib/python3.8/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: 'hgt/SONN3/N49E005.hgt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xyz/venv_pyhgtmap/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 593, in main
    main_internal(sys.argv[1:])
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 574, in main_internal
    hgtDataFiles = NASASRTMUtil.getFiles(
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/NASASRTMUtil.py", line 915, in getFiles
    saveFilename = sources_pool.get_file(opener, area, source)
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/NASASRTMUtil.py", line 886, in get_file
    file_name = self._real_pool.get_source("sonn").get_file(
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/__init__.py", line 77, in get_file
    self.download_missing_file(area, resolution, file_name)
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/sonny.py", line 75, in download_missing_file
    files: List[GoogleDriveFile] = self.gdrive.ListFile(
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/sonny.py", line 61, in gdrive
    gauth.CommandLineAuth()
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 122, in _decorated
    self.LoadCredentials()
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 391, in LoadCredentials
    self.LoadCredentialsFile()
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 418, in LoadCredentialsFile
    self.credentials = self._default_storage.get()
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/client.py", line 407, in get
    return self.locked_get()
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/file.py", line 54, in locked_get
    credentials = client.Credentials.new_from_json(content)
  File "/home/xyz/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/client.py", line 302, in new_from_json
    module_name = data['_module']
KeyError: '_module'
(venv_pyhgtmap) xyz@myhostname:~/develop/fzk-utils/elevation$

Can you tell me if and what I'm doing wrong ? ....

Cheers Patrik

Hi

I tried generating contours from a geotiff file that had a Lambert2008 projection.
They are wrong however, and do not match up with the ones (correctly) generated from the same geotiff in EPSG:3857.
I've included a screenshot from QGIS showing OSM tiles, with the Lambert2008 geotiff on top. The pink contours are generated by pyhgtmap from the Lambert2008 tiff file, with the brown ones generated from the EPSG:3857 geotiff file.

Here's the listgeo output of that Lambert2008 geotiff file:

   Version: 1
   Key_Revision: 1.0
   Tagged_Information:
      ModelTiepointTag (2,3):
         0                 0                 0                
         323891.028851897  1030258.88565706  0                
      ModelPixelScaleTag (1,3):
         90                90                0                
      End_Of_Tags.
   Keyed_Information:
      GTModelTypeGeoKey (Short,1): ModelTypeProjected
      GTRasterTypeGeoKey (Short,1): RasterPixelIsArea
      GTCitationGeoKey (Ascii,8): "unknown"
      GeographicTypeGeoKey (Short,1): User-Defined
      GeogCitationGeoKey (Ascii,111): "GCS Name = unknown|Datum = Unknown based on GRS 1980 ellipsoid using towgs84=0,0,0,0,0,0,0|Primem = Greenwich|"
      GeogGeodeticDatumGeoKey (Short,1): User-Defined
      GeogAngularUnitsGeoKey (Short,1): Angular_Degree
      GeogEllipsoidGeoKey (Short,1): Ellipse_GRS_1980
      GeogSemiMajorAxisGeoKey (Double,1): 6378137          
      GeogInvFlatteningGeoKey (Double,1): 298.257222101    
      GeogPrimeMeridianLongGeoKey (Double,1): 0                
      GeogTOWGS84GeoKey (Double,3): 0                0                0                
      ProjectedCSTypeGeoKey (Short,1): User-Defined
      ProjectionGeoKey (Short,1): User-Defined
      ProjCoordTransGeoKey (Short,1): CT_LambertConfConic_2SP
      ProjLinearUnitsGeoKey (Short,1): Linear_Meter
      ProjStdParallel1GeoKey (Double,1): 49.8333333333333 
      ProjStdParallel2GeoKey (Double,1): 51.1666666666667 
      ProjFalseOriginLongGeoKey (Double,1): 4.35921583333333 
      ProjFalseOriginLatGeoKey (Double,1): 50.797815        
      ProjFalseOriginEastingGeoKey (Double,1): 649328           
      ProjFalseOriginNorthingGeoKey (Double,1): 665262           
      End_Of_Keys.
   End_Of_Geotiff.

Projection Method: CT_LambertConfConic_2SP
   ProjFalseOriginLatGeoKey: 50.797815 ( 50d47'52.13"N)
   ProjFalseOriginLongGeoKey: 4.359216 (  4d21'33.18"E)
   ProjStdParallel1GeoKey: 49.833333 ( 49d50' 0.00"N)
   ProjStdParallel2GeoKey: 51.166667 ( 51d10' 0.00"N)
   ProjFalseEastingGeoKey: 649328.000000 m
   ProjFalseNorthingGeoKey: 665262.000000 m
Ellipsoid: 7019/GRS 1980 (6378137.00,6356752.31)
TOWGS84: 0,0,0
Projection Linear Units: 9001/metre (1.000000m)

Corner Coordinates:
Upper Left    (  323891.029, 1030258.886)  (  0d35'44.32"W, 53d58'42.19"N)
Lower Left    (  323891.029,  353998.886)  (  0d 0'25.44"E, 47d54'50.38"N)
Upper Right   (  771911.029, 1030258.886)  (  6d13'40.63"E, 54d 3'43.06"N)
Lower Right   (  771911.029,  353998.886)  (  6d 0' 0.58"E, 47d59'15.17"N)
Center        (  547901.029,  692128.886)  (  2d54'47.76"E, 51d 1'49.64"N)

I've asked for this already on Panarchos discussion page - not sure how difficult it would be. But in very mountaineous/steep areas this would be the biggest possible improvment for phyghtmap.

OSM pbf data for contourlines created with gnuplot and b-spline interpolation looks much cleaner and in Garmin format also consumes less size.

The contourlines map of kleineisel.de for example have this. The difference is really big when zoomed in very close in mountaineous areas.

--simplifyContoursEpsilon=? helps a bit too - but is super slow and hence really impossible to use say on a Europe polygon - even on a very fast server with loads of RAM.

Not sure if gnuplot would be faster too.
https://wiki.openstreetmap.org/wiki/Srtm2osm_perl is not working anymore - but that script was editable to also use b-spline interpolation with gnuplot and it required ASC data as input... I'm not sure if there is another way instead of gnuplot to use b-spline interpolation instead of linear interpolation with phyghtmap.

I've noticed that you added the possibility to directly download hgt from Sonny. Helpful feature...

Unfortunately I've tried it out on Ubuntu 20.04 and did get the following error message:
ERROR: Package 'pyhgtmap-3.5.3.dev3-g192dadb' requires a different Python: 3.8.10 not in '>=3.9'

Did you drop Python 3.8 support or is this just an intermediate issue/problem ?

Thanks for clarifying.
Patrik

          More details concerning this point:

Concerning the file generation, it's not the IO taking time (it's actually using another thread with pyosmium, and is done in batches), but the computing. Pyosmium interface requires a function call per node, and for millions of nodes this takes a lot of CPU. I think in the latest profiling I did this now more than half of the total processing time.

Profiling the generation of a single output from 2 view1 local files (with python -m yappi -f callgrind -o yappi_ex1.out ../../pyhgtmap/main.py --pbf --log=DEBUG --max-nodes-per-tile=0 /mnt/g/git/garmin_mtb/work/hgt/VIEW1/N46E014.hgt /mnt/g/git/garmin_mtb/work/hgt/VIEW1/N46E015.hgt):

1. is the time spent writing NODES to PBF output (11215767 nodes in this example)
2. is the time spent actually generating contours
3. is the time spent writing WAYS to PBF output (50796 ways in this example)

At best, parallelization could allow processing 2 in parallel of (1+3), which would be ~25% improvement of the overall elapsed time. Not really worth the added complexity until one find a way to optimize the actual PBF output part.

Originally posted by @agrenott in #8 (comment)

I was testing SONN1 Download last few hours/days.
It works properly so far for:

• Switzerland
• Luxembourg

But unfortunately it does not work for Germany, for some files pyhgtmap tells me that the file does not exist on the server, but if I check directly on the google drive of Sonny all is there.

It complains about the following files (ordered alphabetically):

N47E013: no file found on server.
N48E010: no file found on server.
N48E011: no file found on server.
N48E012: no file found on server.
N48E013: no file found on server.
N49E010: no file found on server.
N49E011: no file found on server.
N49E012: no file found on server.
N50E007: no file found on server.
N50E008: no file found on server.
N50E010: no file found on server.
N51E007: no file found on server.
N51E008: no file found on server.
N51E009: no file found on server.
N51E010: no file found on server.
N51E011: no file found on server.
N52E008: no file found on server.
N52E009: no file found on server.
N54E006: no file found on server.
N55E007: no file found on server.

But when checking on Sonny's Google Drive (Folder 'DTM Europe 1asec v22 by Sonny' they're existing.

Is the problem on my end, or on the tools end (pyhgtmap) ? Or probably on the providing end (Google drive) ?

Thanks for checking.
Patrik

Hi,
allthough I didn't try it with the very latest sources looking at the commits I assume the following problem exists also in the last commit:

I've tried to create contours from SONNY1 today and ran into following error message:

N54E005: trying sonn1 ...
    self.check_cached_file(file_name, resolution)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/__init__.py", line 49, in check_cached_file
    found_size: int = os.path.getsize(file_name)
  File "/usr/lib/python3.8/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: './hgt/SONN1/N54E005.hgt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 673, in Refresh
    self.credentials.refresh(self.http)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/client.py", line 545, in refresh
    self._refresh(http)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/client.py", line 761, in _refresh
    self._do_refresh_request(http)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/oauth2client/client.py", line 819, in _do_refresh_request
    raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
oauth2client.client.HttpAccessTokenRefreshError: invalid_grant: Token has been expired or revoked.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/XYZ/venv_pyhgtmap/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 593, in main
    main_internal(sys.argv[1:])
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/main.py", line 574, in main_internal
    hgtDataFiles = NASASRTMUtil.getFiles(
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/NASASRTMUtil.py", line 915, in getFiles
    saveFilename = sources_pool.get_file(opener, area, source)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/NASASRTMUtil.py", line 886, in get_file
    file_name = self._real_pool.get_source("sonn").get_file(
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/__init__.py", line 77, in get_file
    self.download_missing_file(area, resolution, file_name)
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/sonny.py", line 75, in download_missing_file
    files: List[GoogleDriveFile] = self.gdrive.ListFile(
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pyhgtmap/sources/sonny.py", line 61, in gdrive
    gauth.CommandLineAuth()
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 131, in _decorated
    self.Refresh()
  File "/home/XYZ/venv_pyhgtmap/lib/python3.8/site-packages/pydrive2/auth.py", line 675, in Refresh
    raise RefreshError("Access token refresh failed: %s" % error)
pydrive2.auth.RefreshError: Access token refresh failed: invalid_grant: Token has been expired or revoked.

``

I think that's probably normal, but I hat do delete .pyhgtmap/gdrive-credentials.json manually to get it requesting a new Code and running again. Not sure if this is the intention, but it probably could be handled bit more user friendly and (re)move the old credentials file automatically and request new authorisation.

Cheers
Patrik

New multiprocessing usage is not compatible with native Windows Python.

Current work-around is to rely on Python inside WSL.

First of all Merry Christmas.

In order to have proper worldwide coverage for 1 asec tiles (also very north and very south) it would be great to have direct support for ALOS data source.

It looks that the download could be integrated quite easily as after the user registration you just have to know the path of the needed tile. The hierarchy for a 1 asec tile is something like
.../version>/<Folder_of_5x5_degrees>/<tile>

You could also download a complete collection of 5x5 degrees (25 tiles in different version) as a zip, but I think that's not really helpful for pyhgtmap.

Cheers
Patrik

ALOS Global Digital Surface Model "ALOS World 3D - 30m (AW3D30)"

Below the details about data source and example download links:

Entry point with instructions (including user registration):
https://www.eorc.jaxa.jp/ALOS/en/dataset/aw3d30/aw3d30_e.htm

Acess to map for tile download:
https://www.eorc.jaxa.jp/ALOS/en/aw3d30/data/index.htm

direct download link for a single 1asec tile:
https://www.eorc.jaxa.jp/ALOS/aw3d30/data/release_v2303/N045E005/N046E007.zip

direct download link for 5 asec collection of 1asec tiles
https://www.eorc.jaxa.jp/ALOS/aw3d30/data/release_v2303/N045E005_N050E010.zip

Hello, not really an issue, I'm just trying to make sure I'm doing things as efficiently as possible.

So I'm trying to generate hillshade, colored relief files as well as import contours into the database.

In the begining I had:

pyhgtmap \
  -o contour \
  --step=10 \
  -0 \
  --pbf \
  --hgtdir=/mnt/data/hgt \
  -a $LEFT:$BOTTOM:$RIGHT:$TOP \
  --earthexplorer-user=$EE_USER \
  --earthexplorer-password=$EE_PASSWORD

Then for the relief & hillshade I would do:

# merge everything into one big file
gdal_merge.py -n 32767 -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -o /mnt/data/hgt/raw.tif /mnt/data/hgt/SRTM3v3.0/*.tif

# warp example
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 500 500 /mnt/data/hgt/raw.tif warp-500.tif

# relief example
gdaldem color-relief -co COMPRESS=LZW -co PREDICTOR=2 -alpha warp-500.tif /home/importer/misc/relief_color.txt $RELIEF_DIR/relief-500.tif

# hillshade example
gdaldem hillshade -z 5 -compute_edges -co BIGTIFF=YES -co COMPRESS=JPEG -co TILED=YES warp-500.tif $HILLSHADE_DIR/hillshade-500.tif

And all this worked fine but when I tried to import the contours into the database:

osm2pgsql \
  --slim \
  -d contours \
  -C $MEMORY \
  --number-processes $CORES \
  --style /home/importer/styles/contours.style \
  contour_*.pbf

It would crash with a core dump.

After some investigation I figured that this is caused by files without any data, and indeed, if I removed the small contour PBF files the command would run without issues.

So I changed the first invocation of the pyhgtmap command to do --download-only:

pyhgtmap \
  --download-only \
  --hgtdir=/mnt/data/hgt \
  -a $LEFT:$BOTTOM:$RIGHT:$TOP \
  --earthexplorer-user=$EE_USER \
  --earthexplorer-password=$EE_PASSWORD

The relief and hillshade commands above still worked fine, and then I would do:

cd /mnt/data/hgt
for hgtfile in *.tif; do gdal_fillnodata.py $hgtfile /mnt/data/hgt/filled/$hgtfile; done
cd /mnt/data/hgt/filled
pyhgtmap -o contour --step=10 -0 --pbf *.tif

And then run the above osm2pgsql command which would work fine.

My question is, if there's a way to instruct the pyhgtmap command to do a "fill no data", or some other way so that there's no need to run the command twice?

hgt file /home/contourlines/hgt/SRTM1v3.0/N25E003.hgt: 3601 x 3601 points, bbox: (3.00000, 25.00000, 4.00000, 26.00000)
Killed

not sure why this is happening - command was:
nice -n 19 pyhgtmap --earthexplorer-user=test--earthexplorer-password=Test--jobs=10 --polygon=/home/contourlines/bounds/africa.poly --step=10 --no-zero-contour --void-range-max=-420 --output-prefix=africa --line-cat=$detail --start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf --hgtdir=/home/contourlines/hgt -j16

With around 62GB of available RAM. Did id run out of memory? I will rerun with --jobs=2 but a bit better error message why it was killed would be great. Europe was killed as well. with above command.

For eg. lon-5.00_-4.00lat43.25_43.28_view1.o5m.
Need checking if the bounding box is invalid as well, or only the file name.

I'm trying to use pyhgtmap in a Docker container starting FROM debian:bookworm-slim with the following packages installed through apt:

python3
python3-pip
python3-venv
python3-gdal

However I was getting the following error:

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?

So I added:

python3-lxml

to the list of apt packages to be installed. Now I'm getting the following warning:

/opt/venv/lib/python3.11/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
  warnings.warn(

After which it proceeds to download the files.

Is this warning normal or have I made a mistake?

          Well I now have another problem with the newest version - seems there are problems with order - when trying to split the output with mkgmap splitter - I get the error:

Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
This is not supported with keep-complete=true or --problem-list
uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted

Phygtmap produced data never had that problem. Running osmosis on this data to sort it - would again fail due to size constraints.

I tried if it's related to a map being south of the Equator - and used Fiji which was fine except for getting this error message although I don't see a problem with the data output on first glance:

hgt file /home/contourlines/hgt/SRTM1v3.0/S20W180.hgt: 3601 x 3601 points, bbox: (-180.00000, -20.00000, -179.00000, -19.00000), checking polygon borders
/usr/local/lib/python3.10/dist-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)

Not sure how what this means. Maybe input data is problematic on that tile?

I then tried Ecuador to have a map passing the equator. Also no problem.
France neither. Seems to affect only very big countries - and I think that problem already has been there with pyhgtmap v 3.1 (but I don't know right now - just know that splitting Europe with mkgmap splitter failed which was produced by v 3.1

Originally posted by @extremecarver in #16 (comment)

Thanks a lot for fixing the matplotlib issue - just as an info here the install on building/installing phyghtmap from source (current master version):

running install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
warnings.warn(

....... cut as it's not needed / working ----

Installed /usr/local/lib/python3.10/dist-packages/phyghtmap-2.23-py3.10.egg
Processing dependencies for phyghtmap==2.23
Searching for numpy>=1.24.2
Reading https://pypi.org/simple/numpy/
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: is an invalid version and will not be supported in a future release
warnings.warn(
Downloading https://files.pythonhosted.org/packages/c5/21/275cfa7731ee2e121b1bf85ddb21b8712fe2f409f02a8b61521af6e4993d/numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=a51725a815a6188c662fb66fb32077709a9ca38053f0274640293a14fdd22978
Best match: numpy 1.24.2
Processing numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to /usr/local/lib/python3.10/dist-packages
Adding numpy 1.24.2 to easy-install.pth file
Installing f2py script to /usr/local/bin
Installing f2py3 script to /usr/local/bin
Installing f2py3.10 script to /usr/local/bin

Installed /usr/local/lib/python3.10/dist-packages/numpy-1.24.2-py3.10-linux-x86_64.egg
Searching for contourpy>=1.0.7
Reading https://pypi.org/simple/contourpy/
Downloading https://files.pythonhosted.org/packages/ec/59/5eac40e348a7bf803cea221bcd27f74a49cb81667b400fdfbb680e86e7bb/contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=87f4d8941a9564cda3f7fa6a6cd9b32ec575830780677932abdec7bcb61717b0
Best match: contourpy 1.0.7
Processing contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to /usr/local/lib/python3.10/dist-packages
Adding contourpy 1.0.7 to easy-install.pth file

Installed /usr/local/lib/python3.10/dist-packages/contourpy-1.0.7-py3.10-linux-x86_64.egg
Searching for bs4>=0.0.1
Reading https://pypi.org/simple/bs4/
Downloading https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz#sha256=36ecea1fd7cc5c0c6e4a1ff075df26d50da647b75376626cc186e2212886dd3a
Best match: bs4 0.0.1
Processing bs4-0.0.1.tar.gz
Writing /tmp/easy_install-z83e9uaq/bs4-0.0.1/setup.cfg
Running bs4-0.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-z83e9uaq/bs4-0.0.1/egg-dist-tmp-yo2_otlo
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

zip_safe flag not set; analyzing archive contents...
Moving bs4-0.0.1-py3.10.egg to /usr/local/lib/python3.10/dist-packages
Adding bs4 0.0.1 to easy-install.pth file

Installed /usr/local/lib/python3.10/dist-packages/bs4-0.0.1-py3.10.egg
Searching for matplotlib==3.5.1
Best match: matplotlib 3.5.1
Adding matplotlib 3.5.1 to easy-install.pth file

Using /usr/lib/python3/dist-packages
Searching for GDAL==3.4.1
Best match: GDAL 3.4.1
Adding GDAL 3.4.1 to easy-install.pth file

Hey,

I had no issue ;-) Just want to say thank you for your bugfixes and your version! It works like a charme on Linux Mint 21.1 while the original one fails due to the changes of the matplotlib.

I would prefer to make your Git Repo the standard source for phyghtmap... https://wiki.openstreetmap.org/wiki/User_talk:Panarchos#ValueError:_too_many_values_to_unpack_-_Solution

Thanks again!

Hello,

I'm not in the field and I'm having trouble knowing whether what I'm trying to do is feasible or not, or simply useless.

Let me explain.

I made a rendering for Garmin https://ravenfeld.gitlab.io/open-garmin-map/ devices thanks to your program.

Except that I find that the contour lines are not very smooth and that this may be linked to the file I'm using. (Sonny 1° or Nasa 1° for countries outside Europe).

I thought that for the map of France I could use a better file by using the RGEalti files https://www.geoportail.gouv.fr/donnees/rge-alti-sources, but the files are in .asc and not hgt.

I've tried using gdal to convert them but without success so far.

With a bit of digging, I realized that hgt has an accuracy of 30m, so converting from .asc to .hgt would be pointless.
What's the difference between all the formats? .hgt, .tiff, .asc, .dem and maybe others?

Maybe I'm using pyhgtmap wrong?
--smooth How much should we put it on?
Is there any point in trying to use RGEalti files?

If you have any documents or just want to explain to me why these different formats and if it's feasible what I'm trying to do, thank you in advance.

In French (native language)

Bonjour,

Je ne suis pas dans le domaine et j'ai du mal a savoir si ce que je cherche a faire est réalisable ou pas ou tout simplement inutile.

Je m'explique.

Je réalisé un rendu pour les appareils Garmin https://ravenfeld.gitlab.io/open-garmin-map/ grace a votre programme, encore merci.

Sauf que voila je trouve que les courbes de niveaux ne sont pas très lissé et que c'est peut etre lié au fichier que j'utiise. (Sonny 1° ou Nasa 1° pour les pays hors Europe).

Je me suis dit que pour la carte de la France je pourrais utiliser de meilleur fichier en utilisant les fichier RGEalti https://www.geoportail.gouv.fr/donnees/rge-alti-sources, mais les fichiers sont en .asc et non hgt.

J'ai essayé d'utiliser gdal pour les convertir mais sans réussite pour le moment.

En creusant un peu j'ai pu comprendre que le hgt avait une précision de 30m et que du coup passer d'un .asc a .hgt cela ne servirait a rien.
Deja c'est quoi la différence entre tout les formats ? .hgt,.tiff,.asc, .dem et peut etre d'autres ?

C'est peut être pyhgtmap que j'utilise mal ?
--smooth on doit le mettre sur combien a 3 j'ai eu des soucis. 2 me semble bien non ?
Est ce que pour vous cela a un interet d'essayer d'utiliser les fichiers RGEalti ?

Si vous avez documents ou tout simplement serait m'expliquer pourquoi ses différents format et si c'est réalisable ce que je cherche a faire merci d'avance et merci de m'avoir lu jusqu'ici ;)

After the tool downloads 5 files I get an authentication error.

N44E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N44E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N44E021.tif ...
N44E021: using file /mnt/data/hgt/SRTM3v3.0/N44E021.tif.
N45E021: trying srtm3v3.0 ...
N45E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N45E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N45E021.tif ...
N45E021: using file /mnt/data/hgt/SRTM3v3.0/N45E021.tif.
N46E021: trying srtm3v3.0 ...
N46E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N46E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N46E021.tif ...
N46E021: using file /mnt/data/hgt/SRTM3v3.0/N46E021.tif.
N47E021: trying srtm3v3.0 ...
N47E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N47E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N47E021.tif ...
N47E021: using file /mnt/data/hgt/SRTM3v3.0/N47E021.tif.
N48E021: trying srtm3v3.0 ...
N48E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N48E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N48E021.tif ...
N48E021: using file /mnt/data/hgt/SRTM3v3.0/N48E021.tif.
N49E021: trying srtm3v3.0 ...
N49E021: downloading file https://earthexplorer.usgs.gov/download/5e83a43cb348f8ec/SRTM3N49E021V2/EE to /mnt/data/hgt/SRTM3v3.0/N49E021.tif ...
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 779, in downloadAndUnzip_Tif
    os.stat(saveFilename)
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data/hgt/SRTM3v3.0/N49E021.tif'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/main.py", line 561, in main
    hgtDataFiles = NASASRTMUtil.getFiles(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 887, in getFiles
    saveFilename = getFile(area, source)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 876, in getFile
    return downloadAndUnzip(url, area, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 762, in downloadAndUnzip
    return downloadAndUnzip_Tif(url, area, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 784, in downloadAndUnzip_Tif
    downloadToFile(url, saveFilename, source)
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 756, in downloadToFile
    return downloadToFile_SRTMv3(url, filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 731, in downloadToFile_SRTMv3
    opener = earthexplorerLogin()
             ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pyhgtmap/NASASRTMUtil.py", line 726, in earthexplorerLogin
    res2 = opener.open(req2)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

So 5 files are downloaded correctly and the 6th file gives an error. I have to wait a few minutes and then try again. I also tried, I am not able to login in the web browser either during this time.

The solution could be to to move the call to earthExplorerLogin() from downloadToFile_SRTMv3 somewhere higher in the stack like in getFiles so the login is only called once per all downloads (if there are srtm sources), but I'm not sure I know enough Python to do that.

Hi,

after successful downloading and building elevation for Luxembourg (small and handy) I tried today with really needed areas, where we do not find any SRTM1 data: enlarged Finland and northern part of enlarged Norway.

Unfortunately pyhgtmap bails out already while trying to download data from the server. Without looking deeper into it I think it has to do that some files are not found on the server and that pyhgtmap stops execution with an exception.

I'm not sure if the file would really be needed or is somwhere, where anyway no data exists due to no landmass or similar. But in my opinion the script should warn as with other sources something like 'file not found on server' instead of stopping execution.

Could you please have a look at it ? Below you find part (in my opinion the important part) of pyhgtmap output. And attached you find two poly files (in the zip) that didn't build proper maps but just have thrown exceptions.

If you need more details or further tests, just let me know.

Thanks and regards
Patrik

Output of pyhgtmap:

...
checking if area N70E032 intersects with polygon ... no
N59E018: trying alos1 ...
N60E018: trying alos1 ...
N61E018: trying alos1 ...
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/sources/__init__.py", line 111, in get_file
    self.check_cached_file(file_name, resolution)
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/sources/__init__.py", line 94, in check_cached_file
    raise FileNotFoundError(f"File {file_name} not found")
FileNotFoundError: File ./hgt/ALOS1/N61E018.tif not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pab/miniconda3/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/main.py", line 67, in main
    main_internal(sys.argv[1:])
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/main.py", line 45, in main_internal
    hgtDataFiles = NASASRTMUtil.getFiles(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/NASASRTMUtil.py", line 762, in getFiles
    saveFilename = sources_pool.get_file(opener, area, source)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/NASASRTMUtil.py", line 731, in get_file
    file_name = self._real_pool.get_source(source[0:4]).get_file(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/sources/__init__.py", line 122, in get_file
    self.download_missing_file(area, resolution, file_name)
  File "/home/pab/miniconda3/lib/python3.12/site-packages/pyhgtmap/sources/alos.py", line 103, in download_missing_file
    with ZipFile(
         ^^^^^^^^
  File "/home/pab/miniconda3/lib/python3.12/zipfile/__init__.py", line 1341, in __init__
    self._RealGetContents()
  File "/home/pab/miniconda3/lib/python3.12/zipfile/__init__.py", line 1408, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

problematic-poly.zip