Code Monkey home page Code Monkey logo

kartverket_storm_surge_data's Introduction

kartverket_storm_surge_data

Interact with storm surge data from the kartverket API. This implements a simple interaction to the Kartverket API, as it is described in the tideapi_protocol.pdf. All data generated through this module are directly provided by Kartverket. Their license is available at: https://www.kartverket.no/data/lisens/. For more information about the specific Open tidal and water level data from Kartverket, see: https://www.kartverket.no/en/sehavniva/api-for-water-level-data-and-widget/. We gratefully thank Kartverket for making these data freely available.

Installation

The package should be used with Python 3.7 or higher, on Linux platforms. It has been developed and tested on Ubuntu 20.04. Platforms other than Linux may fail to run the package. Tests can be run with pytest, i.e. from this location in terminal: pytest -v .. They may take quite a while, as a full water level dataset will be built and checked in the process.

To install: add to your pythonpath. For example on my machine, this folder is at location /home/jrmet/Desktop/Git/kartverket_storm_surge_data/, and the command to add to my .bashrc is:

  • export PYTHONPATH="${PYTHONPATH}:/home/jrmet/Desktop/Git/kartverket_storm_surge_data"

You will need a few packages installed locally on your machine to be able to use the code:

  • numpy
  • matplotlib
  • netCDF4
  • tqdm
  • bs4
  • cartopy

Description

The code in the module implements a simple interaction with Kartvekets web API, following the specification in the tideapi_protocol.pdf. The code makes sure that we are 'nice' to the API, by limiting request rate to max 1 request / second, limiting the size of each individual request, and caching data for later use. Building the full dataset on the period 1970-2020 can take up to a couple of days as a consequence. Once the dataset has been built and dumped as .nc4, data access is of course very quick. When building the dataset, up to typically 5 GB of http request data are cached. The final .nc4 file containing all data in binary format is about 500MB in size.

The recommended workflow is the following (all refering to the content of the kartverket_stormsurge module):

  • Generate a netCDF4 dataset from API requests, using the script_generate_nc4_dataset.py.
  • If wanted, check perform a few random checks of the dataset generated (this was created to make sure that there were no mistakes in the netCDF4 dataset generation): script_check_dataset.py.
  • After this point, the data can be accessed directly from the netCDF4 dataset. For an example of interaction, see script_access_dataset.py.

Feedback / discussion / bugs

Please use the Github issue tracker at https://github.com/jerabaul29/kartverket_storm_surge_data/issues to report any issue, discuss, etc.

Notes

  • This small module was developed in the context of the 'MachineOcean' project, Norwegian Meteorological Institute: https://machineocean.met.no/.
  • Kartverket does some maintenance on the last Thursday of each month, and it would be wise to avoid querying if possible then. A warning will be issued if any API request is performed on the last Thursday of a month.

Example of resuts

The script_access_dataset.py will typically produce figures (or / and data for any use) from the Kartverket data about:

  • station location:

location

  • time extend over which the data are available for each station:

time span

  • data dump for any station over the time range spanned by the .nc4 dataset generated:

example data

Useful links

kartverket_storm_surge_data's People

Contributors

jerabaul29 avatar

Watchers

 avatar

kartverket_storm_surge_data's Issues

to report to Kartverket: some new stations are reported by the API but further API queries fail

URL: http://api.sehavniva.no/tideapi.php?tide_request=stationlist&type=perm

returns:

<tide xsi:noNamespaceSchemaLocation="http://api.sehavniva.no/stationinfo.xsd">
<meta licenseurl="http://kartverket.no/data/lisens/"/>
<stationinfo>
<location name="Andenes" code="ANX" latitude="69.326067" longitude="16.134848" type="PERM"/>
<location name="Bergen" code="BGO" latitude="60.398046" longitude="5.320487" type="PERM"/>
<location name="Bodø" code="BOO" latitude="67.292330" longitude="14.399770" type="PERM"/>
<location name="Bruravik" code="BRJ" latitude="60.492094" longitude="6.893949" type="PERM"/>
<location name="Hammerfest" code="HFT" latitude="70.664750" longitude="23.678690" type="PERM"/>
<location name="Harstad" code="HAR" latitude="68.801261" longitude="16.548236" type="PERM"/>
<location name="Heimsjøen" code="HEI" latitude="63.425224" longitude="9.101504" type="PERM"/>
<location name="Helgeroa" code="HRO" latitude="58.995212" longitude="9.856379" type="PERM"/>
<location name="Honningsvåg" code="HVG" latitude="70.980318" longitude="25.972697" type="PERM"/>
<location name="Kabelvåg" code="KAB" latitude="68.212639" longitude="14.482149" type="PERM"/>
<location name="Kristiansund" code="KSU" latitude="63.113859" longitude="7.734352" type="PERM"/>
<location name="Leirvik" code="LEH" latitude="59.766394" longitude="5.503670" type="PERM"/>
<location name="Mausund" code="MSU" latitude="63.869331" longitude="8.665231" type="PERM"/>
<location name="Måløy" code="MAY" latitude="61.933776" longitude="5.113310" type="PERM"/>
<location name="Narvik" code="NVK" latitude="68.428286" longitude="17.425759" type="PERM"/>
<location name="Ny-Ålesund" code="NYA" latitude="78.928545" longitude="11.938015" type="PERM"/>
<location name="Oscarsborg" code="OSC" latitude="59.678073" longitude="10.604861" type="PERM"/>
<location name="Oslo" code="OSL" latitude="59.908559" longitude="10.734510" type="PERM"/>
<location name="Rørvik" code="RVK" latitude="64.859456" longitude="11.230107" type="PERM"/>
<location name="Sandnes" code="SBG" latitude="58.868232" longitude="5.746613" type="PERM"/>
<location name="Sirevåg" code="SIE" latitude="58.505200" longitude="5.791602" type="PERM"/>
<location name="Stavanger" code="SVG" latitude="58.974339" longitude="5.730121" type="PERM"/>
<location name="Tregde" code="TRG" latitude="58.006377" longitude="7.554759" type="PERM"/>
<location name="Tromsø" code="TOS" latitude="69.646110" longitude="18.954790" type="PERM"/>
<location name="Trondheim" code="TRD" latitude="63.436484" longitude="10.391669" type="PERM"/>
<location name="Vardø" code="VAW" latitude="70.374978" longitude="31.104015" type="PERM"/>
<location name="Viker" code="VIK" latitude="59.036046" longitude="10.949769" type="PERM"/>
<location name="Ålesund" code="AES" latitude="62.469414" longitude="6.151946" type="PERM"/>
</stationinfo>
</tide>

but then the following stations seem invalid: BRJ LEH SBG SIE

For example:

http://api.sehavniva.no/tideapi.php?tide_request=obstime&stationcode=BRJ

returns

<error>No valid data series or station code</error>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.