Code Monkey home page Code Monkey logo

arlreader's Introduction

ARLreader

Python only library to read the NOAA ARLs packed format for HYSPLIT (https://ready.arl.noaa.gov/HYSPLIT.php). Currently only wokring for the GDAS1 assimilation data (https://www.ready.noaa.gov/gdas1.php), which is also available from ARL (ftp://arlftp.arlhq.noaa.gov/pub/archives/gdas1). A more extensive description of the format is provided in: Things to know when working with the ARL binary fomat

Currently only GDAS1 and the profiles of GDAS0p5 and GDAS0p25 are working.

Requirements

Either python3 or Anaconda3 with the libraries specified in requirements.txt.

Installation

Installation within a new conda virtual environment

create a new virtual environment (here for anaconda)

conda create -n ARLreader   # you can choose other names as well, 
                            # but using the consistent name during the installation.
activate ARLreader   # activate the virtual environement

conda install python=3.6   # install python, shipped with `pip`, `setuptools`...
pip install git+https://github.com/martin-rdz/ARLreader.git

Usage

python interface

Reading a 2d Field:

import ARLreader as Ar

gdas = Ar.reader('data/gdas1.apr14.w1')
print('indexinfo ', gdas.indexinfo)
print('headerinfo ', gdas.headerinfo)
for i, v in gdas.levels.items():
    print(i, ' level ', v['level'], list(map(lambda x: x[0], v['vars'])))
# load_heightlevel(day, houer, level, variable)
recinfo, grid, data = gdas.load_heightlevel(2, 3, 0, 'RH2M')

Read the profile at a given location an write it to a text file with load_profile(day, hour, (lat, lon)):

profile, sfcdata, indexinfo, ind = Ar.reader('data/gdas1.apr14.w1').load_profile(2, 3, (51.3, 12.4))
print(profile)
Ar.write_profile('testfile.txt', indexinfo, ind, (51.3, 12.4), profile, sfcdata)

Get the filename from a datetime Ar.fname_from_date(datetime.datetime(2014, 4, 3)).

example

command line interface

ARLreader -h   # prompt up the help

Below is the help messages for using ARLreader:

usage: ARLreader [-h] [-s START_TIME] [-e END_TIME] [--latitude LATITUDE]
                 [--longitude LONGITUDE] [--station STATION]
                 [-f GLOBAL_FOLDER] [-o PROFILE_FOLDER] [--create_subfolder]
                 [--interp] [--force]

extract the GDAS1 profile from GDAS1 global binary data.

optional arguments:
  -h, --help            show this help message and exit
  -s START_TIME, --start_time START_TIME
                        start time of your interests (yyyymmdd-HHMMSS).
                        e.g.20151010-120000
  -e END_TIME, --end_time END_TIME
                        stop time of your interests (yyyymmdd-HHMMSS).
                        e.g.20151010-120000
  --latitude LATITUDE   latitude of your station. (-90, 90).
                        Default: 30.533721
  --longitude LONGITUDE
                        longitude of your station. (0, 360).
                        Default: 114.367216
  --station STATION     station name.
                        Default: wuhan
  -f GLOBAL_FOLDER, --global_folder GLOBAL_FOLDER
                        folder for saving the GDAS1 global files.
                        e.g., 'C:\Users\zhenping\Desktop\global'
  -o PROFILE_FOLDER, --profile_folder PROFILE_FOLDER
                        folder for saving the extracted profiles
                        e.g., 'C:\Users\zhenping\Desktop\wuhan'
  --create_subfolder    create subfolder structure to save the profiles.(yyyy/mm/profiles)
  --interp              interpolate the data with bi-linear interpolation.
  --force               force to download the GDAS1 global dataset (not suggested)

setup the reader for a new station

ARLreader -s 20190920-000000 -e 20190923-000000 --latitude 51.35 --longitude 12.35 --station leipzig -f <data_folder> -o <output_folder>

Tests

python3 -m pytest -v

License

The code is partly based on a prior implementation in IDL by Patric Seifert.

Copyright 2017, Martin Radenz, Yin Zhenping MIT License

arlreader's People

Contributors

zpyin avatar martin-rdz avatar

Stargazers

Kecheng Fu avatar  avatar Katherine (Katie) Ackerman avatar Vardhan Agnihotri avatar  avatar jhzhao@qibebt avatar  avatar  avatar  avatar  avatar sytang-astro avatar Radulescu Iulia Maria avatar liuc avatar Hale Ruhland avatar Pankaj Kumar avatar  avatar  avatar  avatar A. Crawford avatar  avatar

Watchers

James Cloos avatar  avatar A. Crawford avatar

arlreader's Issues

Wrong indexing in after data extraction

Hi Martin,

Bug Report

The 2-D data has very limited length after data extraction from certain level. It can be 2x2 if certain latitude and longitude is given (see below).

def unpack_data(binarray, nx, ny, initval, exp, prec, stopat=(-1, -1)):
"""
unpack the binary data with the option to stop at a given index
"""
# Reshape the array
binarray = np.reshape(binarray,(ny,nx))
# Calculate the array difference matrix
e = (2 ** float(7 - exp))
binarray = (binarray - 127) / e
# Calculate the first row using the initial value
first_row = np.cumsum(binarray[:, 0]) + initval
# Asign the values to the array
binarray[:, 0] = first_row
# Calculate all the rows using the first row. Transpose in order to have an (nx,ny) array
data = np.cumsum(binarray, axis=1).T
data[np.abs(data) < prec] = 0
if stopat != (-1, -1):
nx_min = max(stopat[1] - 1, 0)
nx_max = min(stopat[1] + 1, nx)
ny_min = max(stopat[0] - 1, 0)
ny_max = min(stopat[0] + 1, ny)
else:
nx_min = 0
nx_max = nx
ny_min = 0
ny_max = ny
return data[nx_min:nx_max, ny_min:ny_max]

But the indexing is still going under the index of original laitude and longitude, which can be values between [0,180] and [0-360]. This will throw error.

val = data[lonindex, latindex]

Bug Fixes

Change line 931

val = data[lonindex, latindex]

into

val = data[0, 0]

I don't know whether this fix will trigger other issues (Please check it).
For my purpose of extraction GDAS1 profile, it works OK.

Greetings from China!

calling function fname_from_date()

Solution:
Adding new lines from After currentDate, line 234
week_no_start = ((currentDate.day - 1) // 7) + 1
currentday_start = (week_no_start - 1) * 7 + 1

image

Possible performance upgrade

Firstly, I would like to thank you for such a useful library.

I have not been able to use the fast_funcs but after working with the library, I came with a performance upgrade in the "standard" unpack_data function that could be of your interest:

def unpack_data2(binarray, nx, ny, initval, exp, prec, stopat=(-1, -1)):
    """
    unpack the binary data with the option to stop at a given index
    """

    # Reshape the array
    binarray  = np.reshape(binarray,(ny,nx))

    # Calculate the array difference matrix
    e = (2 ** float(7 - exp))
    binarray = (binarray - 127) / e

    # Calculate the first row using the initial value
    first_row = np.cumsum(binarray[:, 0]) + initval

    # Asign the values to the array
    binarray[:, 0] = first_row

    # Calculate all the rows using the first row. Transpose in order to have an (nx,ny) array
    data = np.cumsum(binarray, axis=1).T

    data[np.abs(data) < prec] = 0

    if stopat != (-1, -1):
        nx_min = max(stopat[1] - 1, 0)
        nx_max = min(stopat[1] + 1, nx)
        ny_min = max(stopat[0] - 1, 0)
        ny_max = min(stopat[0] + 1, ny)
    else:
        nx_min = 0
        nx_max = nx
        ny_min = 0
        ny_max = ny

    return data[nx_min:nx_max, ny_min:ny_max]

The following screenshot shows the difference in the time required to load a single variable from a 0.25 gfs model:Screenshot 2021-04-29 at 15 49 28

Once again, thanks.

IndexError on calling load_profile with GFS data

Hi, thank you for this package.

I am using ARLreader to parse GFS 0.25 files that I downloaded from ARL's FTP server (ftp://arlftp.arlhq.noaa.gov/archives/gfs0p25/). When I try to load the profile for a location, I get an IndexError. Here's a simple example and the resultant error:

import ARLreader as Ar
gfs = Ar.reader('/path/to/file/20180222_gfs0p25')
profile, sfcdata, indexinfo, ind = gfs.load_profile(22, 3, (28.62, 77.22)) # Delhi

Error log:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\username\Anaconda3\envs\arlenv\lib\site-packages\ARLreader\__init__.py", line 940, in load_profile
    calc_p_from_sigma(profile["SIGMA"], sfc_pres)
IndexError: tuple index out of range

On some digging, I find that the string formatter in logger.info() expects two return values from calc_p_from_sigma() which returns only one. An easy fix is to remove the second set of empty braces ({}) from line 938 in the snippet below.

logger.info("p from sigma {} {}".format(
calc_p_from_sigma(profile["SIGMA"], sfc_pres)
))

installation packages compatibility issue

Hey @ZPYin,

Is there a specific reason for the requirement of certain version numbers of the required packages?

I wish to use it with an existing python env I am already running (e.g. python > 3.8).

These are the installation steps I tried:

conda install altgraph atomicwrites attrs colorama Cython future importlib-metadata more-itertools packaging pefile pluggy py pyparsing pytest pywin32-ctypes six wcwidth wincertstore zipp numpy
pip install git+https://github.com/martin-rdz/ARLreader.git
However, the last command failed due to dependencies.

Then I solved it as follows:

  1. Unhighlighting the versions number written in the requirments. E.g. altgraph==0.16.1 ---> altgraph # ==0.16.1
  2. git clone https://github.com/martin-rdz/ARLreader.git ARLreader
  3. cd ARLreader
  4. python setup.py install

Currently, it seems that the installation worked and it's possible to import the package.

For future users, I guess that it will be better to drop the versions number requirements (unless you find it important for package functionality)

tnx

Adi

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.