avwx-rest / avwx-engine Goto Github PK

View Code? Open in Web Editor NEW

88.0 13.0 30.0 68.29 MB

Aviation Weather parsing engine. METAR & TAF

License: MIT License

Python 100.00%

avwx-engine's Introduction

AVWX

Documentation: https://engine.avwx.rest

Source Code: https://github.com/avwx-rest/avwx-engine

PyPI: https://pypi.org/project/avwx-engine/

AVWX is a global aviation weather fetching and parsing engine. It sources reports from a variety of government sources, parses individual elements, and calculates additional information like flight rules and time range interpolation.

AVWX currently supports:

Station data and search
METAR
TAF
PIREP
AIRMET / SIGMET
NOTAM
NBM (NBH, NBS, NBE)
GFS (MAV, MEX)

Install

The easiest way to get started is to download the library from pypi using pip:

python -m pip install avwx-engine

Basic Usage

Reports use ICAO, IATA, or GPS idents when specifying the desired station. Exceptions are thrown if a potentially invalid ident is given.

>>> import avwx
>>>
>>> metar = avwx.Metar('KJFK')
>>> metar.station.name
'John F Kennedy International Airport'
>>> metar.update()
True
>>> metar.data.flight_rules
'IFR'

You can learn more by reading the project documentation

Note: This library requires Python 3.10 or above

Development

Download and install the source code and its development dependencies:

Clone this repository

git clone https://github.com/avwx-rest/avwx-engine
cd avwx-engine

Requirements:
- Hatch
- Python 3.10+
Create a virtual environment and install the dependencies

hatch env create

Activate the virtual environment

hatch shell

Formatting and Code Checks

hatch handles all of the formatting and linting for us. The library and test suite are fully typed and formatted. Make sure to run these checks before submitting PRs because the workflows will fail if errors are found.

Typing with mypy:

hatch run types:check

Code formatting and linting:

hatch fmt

Testing

Testing is managed by hatch which uses pytest and coverage under the hood.

hatch test

The end-to-end test files were generated using util/build_tests.py and placed into tests/{report}/data. Because Timestamp generation interprets the text based on the current date, Timestamp objects are nullified in the end-to-end tests.

Documentation

The documentation is automatically generated from the content of the docs directory and from the docstrings of the public signatures of the source code. The documentation is updated and published to engine.avwx.rest automatically as part each release.

You can also preview local changes during development:

hatch run docs:serve

Releasing

Trigger the Draft release workflow (press Run workflow). This will update the changelog & version and create a GitHub release which is in Draft state.

Find the draft release from the GitHub releases and publish it. When a release is published, it'll trigger release workflow which creates PyPI release and deploys updated documentation.

avwx-engine's People

Contributors

Stargazers

Watchers

avwx-engine's Issues

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

Hi there,

I was trying to follow the basic example on the README of how to use this package. But I got the error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

(Full stack trace below)

Steps to Reproduce

Created a new directory using Git Bash on Windows 10
python -m venv venv
source ./venv/Scripts/activate
pip install avwx-engine
started a python repl with python

typed:

>>> import avwx
>>> metar = avwx.Metar('KJFK')

Immediately got an error

Full Stack Trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\current\base.py", line 65, in __init__
    super().__init__(icao)
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\base.py", line 61, in __init__
    self.station = Station.from_icao(icao)
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\station.py", line 158, in from_icao
    info = copy(_STATIONS[ident.upper()])
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 29, in __getitem__
    self._load()
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 25, in _load
    self.data = json.load(self.source.open())
  File "C:\Python38\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

Am I doing something wrong?

lookup station by IATA code?

i maintain a discord bot that uses this library, and it would be really usful for there to be a way to lookup a station by the iata code, instead of just by the icao code.

Add avt7 service for china

Example:
http://www.avt7.com/Home/AirportMetarInfo?airport4Code=ZJQH

empty responses

Hi,

I use the API from an Azure Function (similar to AWS Lambda) to provide my travelling colleagues with some flight weather info before they drive to the airport.

However, like 4 out of 10 times the API comes back with an empty response and only after retrying several times I get an actual response with weather info.
As this is used as a Slackbot, retrying multiple times is obviously not great as it destroys user experience.

Invoke-RestMethod -Method Get -Uri "http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info" -Verbose
VERBOSE: GET http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info with 0-byte payload
VERBOSE: received 1114-byte response of content type application/json

This is a successful response, but then like 40% of all times I get a 0-byte response back.

Any idea as to why this happens or what I can do to prevent this?

METAR with no TAF

Hi, when retrieving a METAR for AU BOM station YCNK, there is no TAF issued as this is not an airport with a published approach. Hence there is only a METAR. How can we retrieve just the metar data only when no TAF exists? I see from service.py that the BOM service looks for TAF to then pull the METAR data. Can this dig a bit deeper?

Allowing for NOTAM body with field delimiters, like `C)`

I came across a NOTAM that was throwing an error:

A3475/22 NOTAMN
Q) LIMM/QFAXX/IV/NBO/A/000/999/4537N00843E005
A) LIMC B) 2205182200 C) PERM
E) REF AIP AD 2 LIMC 1-12 ITEM 20 'LOCAL TRAFFIC REGULATIONS'
BOX 2 'APRON' PARAGRAPH 2.1 'ORDERLY MOVEMENT OF AIRCRAFT ON
APRONS' INDENT 4 'SERVICES PROVIDED' POINT C) 'FOLLOW-ME ASSISTANCE
PROVIDED ON PILOT'S REQUEST AND MANDATORY IN CASE' ADD THE FOLLOWING
IN CASE:
- GENERAL AVIATION AIRCRAFT UP TO ICAO CODE B (MAXIMUM WINGSPAN 24
METERS) AND HELICOPTERS ARRIVING AND DEPARTING FROM STANDS 301 TO 320
AND FROM 330 TO 336.
ARR TAXI ROUTE: AFTER TWR INSTRUCTIONS VIA APN TAXIWAY P-K TO
INTERMEDIATE HOLDING POSITION (IHP) K9 WHERE FOLLOW-ME CAR WILL
BE WAITING.
DEP TAXI ROUTE: AFTER TWR INSTRUCTIONS AND WITH FOLLOW-ME
ASSISTANCE VIA APN TAXIWAY N-K TO IHP K8

The body of the NOTAM (under tag E)) contains text that is the same as a tag, the C) in POINT C) 'FOLLOW-ME.
avwx/current/notam.py doesn't allow for this to happen.

I solved the error by checking that the tag's content is being set for the first time (in theory, arbitrary text should only appear after the header, so this should be relatively safe).

My proposed edit below:

        if tag == "Q":
            if qualifiers is None:
                qualifiers = _qualifiers(item, units)
        elif tag == "A":
            if station is None:
                station = item
        elif tag == "B":
            if start_text == "":
                start_text = item
        elif tag == "C":
            if end_text == "":
                end_text = item
        elif tag == "D":
            if schedule is None:
                schedule = item
        elif tag == "E":
            if body == "":
                body = item
        elif tag == "F":
            if lower is None:
                lower = core.make_altitude(item.split()[0], units, repr=item)[0]
        elif tag == "G":
            if upper is None:
                upper = core.make_altitude(item.split()[0], units, repr=item)[0]

However, one issue remains because the original string is being sliced every time the tags are found, the NOTAM body is cut just before the C) in the text.

body='REF AIP AD 2 LIMC 1-12 ITEM 20 &apos;LOCAL TRAFFIC REGULATIONS&apos;\nBOX 2 &apos;APRON&apos; PARAGRAPH 2.1 &apos;ORDERLY MOVEMENT OF AIRCRAFT ON\nAPRONS&apos; INDENT 4 &apos;SERVICES PROVIDED&apos; POINT'

I'm no RegEx wizard, so there might be a more succinct way of solving this problem by altering the RegEx that matches the tags.

ZBAD airport

Looking for METAR or TAF for airport with ICAO "ZBAD" the API returns an error: "ZBAD does not publish reports"

In stations.json everything seems correct:

"ZBAD": {"city": "Beijing", "country": "CN", "elevation_ft": 98, "elevation_m": 30, "iata": "PKX", "icao": "ZBAD", "latitude": 39.509945, "longitude": 116.41092, "name": "Beijing Daxing International Airport", "note": null, "reporting": true, "runways": [{"ident1": "11L", "ident2": "29R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17L", "ident2": "35R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17R", "ident2": "35L", "length_ft": 12467, "width_ft": 0}, {"ident1": "01L", "ident2": "19R", "length_ft": 11155, "width_ft": 0}], "state": "13", "type": "large_airport", "website": null, "wiki": "https://en.wikipedia.org/wiki/Beijing_Daxing_International_Airport"}

And on ADDS I can retrieve the METAR and TAF just fine: https://aviationweather.gov/metar/data?ids=zbad&format=raw&hours=0&taf=on&layout=on

Add support for NOAA NBM NBX files

Currently only "NBH", "NBS", and "NBE" are supported. These three support forecasts out to 192 hours. Adding support for "NBX" would support forecasts out to 264 hours.

The NBX format is documented here: https://vlab.noaa.gov/web/mdl/nbm-textcard-v4.1

Incorrect airport names

KSFO = "International Airport", so does KLAX. They look to be defined wrong in stations.json.

Print METAR to webhook

Hiya!

Not really an issue, more of a query.

Could I use my API key to print a METAR to a Discord webhook daily?

Replace validators with voluptuous

Current validation via Flask-RESTful are deprecated. They need to be replaced. Voluptuous has worked well in the passed for input and output validation.

Secondary reason is Flask-RESTful extension is not compatible with Quart conversion

Support for Australian METARs

On http://www.bom.gov.au/aviation/forecasts/taf/#20 I can get Australian METARs/TAFs, but unfortunately they're not available trough NOAA.
I couldn't find a clear documented API (they have 'data feeds', and most probably have an API to retrieve METAR/TAF via the page above).

I couldn't quickly figure out what the internal structure for AVWX-Engine is like, so after I get to solving that puzzle for myself, I might be able to send a PR. In the mean time, there's this issue ;)

SIGMET error

example: "WSMV31 VRMM 020506 VRMF SIGMET 2 VALID 020500/020900 VRMM- VRMF MALE FIR EMBD TS FCST WI N0524 E07800-S0215 E07748-S0600 E07500-S0600 E06807-S0007 E06800-N0457 E07000-N0524 E07800 TOP FL420 MOV E2KT INTSF="

error: "invalid literal for int() with base 10: 'E2KT'"

Error scrapper

Hello, I have the following error using the TAF Scrapper: NOAA_Scrape server returned 404. Is it an error of the library?

Crash when overriding normal METAR fetch

When attempting to manually update a station's METAR, as can be seen in the example from the documentation site, a crash occurs.

>>> from avwx import Metar
>>> ksfo = Metar('KSFO')
>>> report = 'KSFO 031254Z 36024G55KT 320V040 1/8SM R06/0200D +TS VCFC OVC050 BKN040TCU 14/10 A2978 RMK AIRPORT CLOSED'
>>> ksfo.update(report)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/avwx/base.py", line 120, in update
    report = self.service.fetch(self.icao, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 101, in fetch
    return aio.run(self.async_fetch(station, timeout))
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 109, in async_fetch
    return await self._fetch(station, url, params, timeout)
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 79, in _fetch
    resp = await client.get(url, params=params)
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1456, in get
    timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1282, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1313, in send
    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1342, in _send_handling_redirects
    request, auth=auth, timeout=timeout, history=history
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1378, in _send_handling_auth
    response = await self._send_single_request(request, timeout)
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1414, in _send_single_request
    timeout=timeout.as_dict(),
  File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 183, in request
    await self._add_to_pool(connection, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 309, in _add_to_pool
    await self._connection_semaphore.acquire(timeout=timeout.get("pool", None))
  File "/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py", line 203, in acquire
    await asyncio.wait_for(self.semaphore.acquire(), timeout)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    if timeout <= 0:
TypeError: '<=' not supported between instances of 'str' and 'int'

I am getting this crash on both MacOS and Raspberry Pi OS after a fresh install of avwx.

Metars for airports without constant observation

When a smaller airport like KSQL is closed for the night, the METARs are aging beyond the query window avwx-engine uses (2 hours I believe). E.g. at this point KSQL's last METAR is 149 minutes old and is not returned because it is outside the query window.

Suggested behavior would be to get the last available METAR, even if older than 2 hours.

File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/init.py", line 89, in update
report = self.service.fetch(self.station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 56, in fetch
report = self._extract(resp.read().decode('utf-8'), station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 101, in _extract
raise self.make_err(raw)
avwx.exceptions.InvalidRequest: Could not find report path in NOAA response

74332561 6

Refactor parsing

I think a quick(ish) refactor of the parsing modules could pay dividends in the future. Currently the parsing logic looks through a string and extracts based on string manipulation. Then in the same context, the parsing function determines what type of translation should be used for each of the plucked strings (lets call them ‘atoms’).

What I propose is a lightweight framework that predefines ‘atoms’ as a regular expressions. This allows for multi-worded atoms to be searched for with the same amount of effort as any normal encoded/non-encoded atom. Each atom can be predefined and a translator is responsible for translating/decoding each atom into whatever. This way, each module can just build the atoms it expects to find in whatever it’s going to translate, add them to translators that decode even the most complicated ones (this will allow for those awful ones like tornados and what not), and then in the parser modules, you’d never have to change anything and can add as many handlers as you want.

An example in the remarks could be

PEAK_WIND = r”””
\bPK WND 
(?<direction>\d{3})
(?<velocity>\d{2,3})
/
(?<hours>\d{2})?  # rarely present
(?<minutes>\d{2})
\b”””

Using that, some class...

class PeakWindAtom(Atom):
    pattern = PEAK_WIND
    search(str) -> match or None
    __contains__(str) -> bool

would be used by

class PeakWindHandler(AtomHandler):
    atom: some Atom
    <<abstract>> can_translate(str) -> bool : atom in is string
    <<abstract>> translate(str) -> str: decode the data

would be used by

class RemarksParser(Parser):
    - handlers: List[handlers]
    + parse(remarks_string) -> Dict[str, str]: raw-decoded pairs
    + add_handler

The patterns and atoms could be reused. And simple ones like ACFT MSHP could be build using a simple factory (so the API doesn’t have to change for simple ones or complex ones). All of your existing decoding logic could stay in place, or slowly be changed to regex group dicts for easier changes.

No tests would need to be changed as far as I can tell. Added tests will be in their own module that you can place wherever you like best before any merge.

I think starting with the remarks module is a good place to start. Before I’d get started and submit a PR, is this something you’d be open to?

TAF not available at many airports

Many European airports do not have TAF available using avwx-engine and avwx.rest.

For example:

>>> import avwx
>>> taf = avwx.Taf('EDDM')
>>> taf.update()
False

and

curl --request GET \
  --url 'https://avwx.rest/api/taf/EDDM?options=&airport=true&reporting=true&format=json&onfail=cache' \
  --header 'authorization: Bearer *******************'

Returns a cached report from 2020-03-30T17:51:54.476000Z.

The same result has been tested on EGLL, EDDF, EBBR, LFPG, LEMD.

I get up to date reports for these locations using NOAA's https://aviationweather.gov/taf/data?format=raw&ids=EDDM

LLJR

any ICAO code starting with "LL" is = Israel
LLJR is not an exception.
See more at : https://en.wikipedia.org/wiki/Atarot_Airport