codeforberlin / wfs-downloader Goto Github PK

View Code? Open in Web Editor NEW

26.0 8.0 5.0 17 KB

Downloads GML files from a set of WFS service in a pseudo-paginated way using bounding boxes.

Home Page: https://github.com/codeforberlin/wfs-downloader

License: MIT License

Python 100.00%

wfs ogc python gml boundingbox webfeatureservice download

wfs-downloader's Introduction

WFS Downloader

Downloads GML files from a set of WFS service in a pseudo-paginated way using bounding boxes and combine them again to one file.

Install

pip install wfs-downloader

Usage

Create a config.yml specifying your setup like this:

url: http://fbinter.stadt-berlin.de/fb/wfs/data/senstadt/s_wfs_baumbestand_an
layer: fis:s_wfs_baumbestand_an

bbox:
  west:   370000.0
  south: 5800000.0
  east:   415000.0
  north: 5837000.0

size: 10000
outputfile: strassenbaeume.xml
projection: EPSG:25833
tmpdir: /tmp

where:

url is the url of the WFS Service,
layer is the name of the Layer,
bbox is the bounding box for th objects you want to retrieve,
size is the extend of a single request (or page),
outputfile is the name of the resulting GML file,
projection is the used projection, and
tmpfile is the path to the directory to store temporary files for each request.

Then run the script with the config.yml as argument:

wfs-downloader config.yml

Help

$ wfs-downloader --help
usage: Downloads GML files from a set of WFS service in a pseudo-paginated way using bounding boxes and combine them again to one file. The WFS services are specified in settings.py.

positional arguments:
  config         config file

optional arguments:
  -h, --help     show this help message and exit
  --no-download  skip the download
  --no-combine   skip the combine

wfs-downloader's People

Contributors

Stargazers

Watchers

Forkers

knudmoeller chaostheorie dncpax geosmrob omkurz

wfs-downloader's Issues

manage throttle or closing connections by server

when too many requests are made some servers close connection. may be use a delay between requests? also handle this kind of exceptions.

remove duplicates

some how remove duplicates, since bbox will sometimes download the same features, with lines or polygons.

urllib.error.HTTPError: HTTP Error 500: Internal Server Error

config.yml:
url: https://inspire-hessen.de/ows/services/org.2.dd63cf0f-cf8a-4a69-be4c-b5c056b8528b_wfs
layer: cp:CadastralParcel

bbox:
west: 777247.0
south: 4939527.0
east: 1023641.0
north: 5165779.0

size: 10000
outputfile: flurstck_hessen.xml
projection: EPSG:25832
tmpdir: C:/Users/user/Programmieren/Pycharm

I'll get the following error:

(venv) PS C:\Users\user\Programmieren\Pycharm> wfs-downloader config.yml

fetching C:/Users/user/Programmieren/Pycharm\flurstck_hessen_777247.0_4939527.0.xml
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\user\Programmieren\Pycharm\venv\Scripts\wfs-downloader.exe_main.py", line 7, in
File "C:\Users\user\Programmieren\Pycharm\venv\Lib\site-packages\wfs_downloader\download.py", line 32, in main
download_files(config)
File "C:\Users\user\Programmieren\Pycharm\venv\Lib\site-packages\wfs_downloader\download.py", line 73, in download_files
urlretrieve(url, filename)
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 240, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 215, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\userAppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 521, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 630, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 559, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 492, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 639, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

Ease usage: Print help by default

If someone invokes the script without parameters the help should be automatically printed as in:

$ ./download.py

change axis order in config when EPSG 4326

For some reason, in config file I have to switch south/north to longitude and east/west to longitude to get data from a wfs service. Have to try other services to understande if it's just my service. For instance:
bbox:
west: 37.621
south: -8.410
east: 38.602
north: -7.311

When in fact this is latlon notation, not west/south (lonlat).

The real coordinates are: -8.410 37.621 for minx miny, and -7.311 38.602 for maxx maxy.

Otherwise I get no data...