Code Monkey home page Code Monkey logo

eccodes's Introduction

eccodes

Github Repository Crates.io License dependency status Crates.io MSRV ecCodes version

This crate contains (mostly) safe high-level bindings for ecCodes library. Bindings can be considered safe mainly because all crate structures will take ownership of the data in memory before passing the raw pointer to ecCodes.

Currently only reading of GRIB files is supported.

Because of the ecCodes library API characteristics theses bindings are rather thick wrapper to make this crate safe and convenient to use.

This crate officially supports mainly Linux platforms same as the ecCodes library. But it is possible to install ecCodes on MacOS and this crate successfully compiles and all tests pass.

If you want to see more features released quicker do not hesitate to contribute and check out Github repository.

ecCodes is an open-source library for reading and writing GRIB and BUFR files developed by European Centre for Medium-Range Weather Forecasts.

Usage

ecCodes installation

This crate uses eccodes-sys with default options to link ecCodes. Check eccodes-sys website for more details on how it links the library.

The recommended way to install ecCodes on your computer is using your package manager. For example, on Ubuntu you can use apt-get:

sudo apt-get install libeccodes-dev

or brew on MacOS:

brew install eccodes

Alternatively, you can install the library manually from source in suitable directory following this instructions.

Then add the lib/pkgconfig directory from your ecCodes installation directory to the PKG_CONFIG_PATH environmental variable. If ecCodes have been compiled as shared library you will also need to specify LD_LIBRARY_PATH. For example:

export PKG_CONFIG_PATH=<your_eccodes_path>/lib/pkgconfig
export LD_LIBRARY_PATH=<your_eccodes_path>/lib

Working with GRIB files

To access a GRIB file you need to create CodesHandle with one of provided constructors.

GRIB files consist of messages which represent data fields at specific time and level. Messages are represented by the KeyedMessage structure.

CodesHandle implements FallibleStreamingIterator which allows you to iterate over messages in the file. The iterator returns &KeyedMessage which valid until next iteration. KeyedMessage implements several methods to access the data as needed, most of those can be called directly on &KeyedMessage. You can also use try_clone() to clone the message and prolong its lifetime.

Data defining and contained by KeyedMessage is represented by Keys. You can read them directly with read_key(), use KeysIterator to iterate over them or use CodesNearest to get the values of four nearest gridpoints for given coordinates.

You can also modify the message with write_key() and write it to a new file with write_to_file().

Example

// We are reading the mean sea level pressure for 4 gridpoints
// nearest to Reykjavik (64.13N, -21.89E) for 1st June 2021 00:00 UTC
// from ERA5 Climate Reanalysis

use eccodes::{ProductKind, CodesHandle, KeyType};
use eccodes::FallibleStreamingIterator;

// Open the GRIB file and create the CodesHandle
let file_path = Path::new("./data/iceland.grib");
let product_kind = ProductKind::GRIB;
let mut handle = CodesHandle::new_from_file(file_path, product_kind)?;

// Use iterator to find a message with shortName "msl" and typeOfLevel "surface"
// We can use while let or for_each() to iterate over the messages
while let Some(msg) = handle.next()? {
    if msg.read_key("shortName")?.value == KeyType::Str("msl".to_string())
        && msg.read_key("typeOfLevel")?.value == KeyType::Str("surface".to_string()) {
       
        // Create CodesNearest for given message
        let nearest_gridpoints = msg.codes_nearest()?
            // Find the nearest gridpoints to Reykjavik
            .find_nearest(64.13, -21.89)?;
        // Print value and distance of the nearest gridpoint
        println!("value: {}, distance: {}",
            nearest_gridpoints[3].value,
            nearest_gridpoints[3].distance);
    }
}

Writing GRIB files

The crate provides a basic support for setting KeyedMessage keys and writing GRIB files. The easiest (and safest) way to create a new custom message is to copy existing one from other GRIB file, modify the keys and write to new file.

You can find a detailed example of setting keys and writing message to file in the documentation.

Errors and panics

This crate aims to return error whenever possible, even if the error is caused by implementation bug. As ecCodes is often used in scientific applications with long and extensive jobs, this allows the user to handle the error in the way that suits them best and not risk crashes.

All error descriptions are provided in the errors module. Destructors, which cannot panic, report errors through the log crate.

None of the functions in this crate explicitly panics. However, users should not that dependencies might panic in some edge cases.

Safety

This crate aims to be as safe as possible and a lot of effort has been put into testing its safety. Moreover, pointers are always checked for null before being dereferenced.

That said, neither main developer nor contributors have expertise in unsafe Rust and bugs might have slipped through. We are also not responsible for bugs in the ecCodes library.

If you find a bug or have a suggestion, feel free to discuss it on Github.

Features

  • message_ndarray - enables support for converting KeyedMessage to ndarray::Array. This feature is enabled by default. It is currently tested only with simple lat-lon grids.

  • experimental_index - enables support for creating and using index files for GRIB files. This feature experimental and disabled by default. If you want to use it, please read the information provided in codes_index documentation.

  • docs - builds the crate without linking ecCodes, particularly useful when building the documentation on docs.rs. For more details check documentation of eccodes-sys.

To build your own crate with this crate as dependency on docs.rs without linking ecCodes add following lines to your Cargo.toml

[package.metadata.docs.rs]
features = ["eccodes/docs"]

License

The ecCodes library and these bindings are licensed under the Apache License Version 2.0

eccodes's People

Contributors

quba1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

eccodes's Issues

Read from index

Is your feature request related to a problem? Please describe.
As GRIB-files are binary dumps of message-sections index-files are used to encode the precalculated binary location of messages matching a set of keys. Some GRIB-files contains thousands of messages and to facilitate fast reading of messages index-files are delivered with the files and use to query the binary location of a message and receive handles. Loading GRIB-files by reading an index-file is required for performance in certain situations.

Describe the solution you'd like
I would like safe rust bindings for reading from GRIB index-files.

Additional context
I would like to use this crate to eventually migrate from C++ for working with GRIB and this is a required feature. I wouldn't mind trying to implement this feature myself but I'm still inexperienced working with rust and starting off with unsafe rust seems a bit daunting.

Index-files are supported by the eccodes C-api. It's also possible to create an index file quite simply with eccodes CLI-tools to get something to experiment with.

I believe this is the C-interface for reading from an existing index
https://github.com/onyb/eccodes/blob/master/src/eccodes.h#L242

codes_index* codes_index_read(codes_context* c,const char* filename,int *err);

The workflow is to index a file using a set of predetermined keys, such as ["shortName","typeOfLevel","level","stepType"]. You then read the index-file, selecting a value for each of the keys the index was created with, e.g. selecting

codes_index_select_string(index,"shortName","t");
codes_index_select_string(index,"typeOfLevel","heightAboveGround");
codes_index_select_string(index,"level",2);
codes_index_select_string(index,"stepType","instantaneous");

and then retreiving the grib-handle from the index using codes_handle_new_from_index

`codes_index` is potentially not thread safe

ecCodes 2.33.0 manual build with POSIX-threads enabled.

Minimal example:

use std::{ffi::CString, thread};

fn main() {
    let t1 = thread::spawn(|| thread1("T1"));
    let t2 = thread::spawn(|| thread1("T3"));

    t1.join().unwrap();
    t2.join().unwrap();
}

fn thread1(name: &str) {
    let keys = CString::new("shortName, typeOfLevel").unwrap();
    let grib_file = CString::new("./data/iceland-surface.grib").unwrap();

    let mut err = 0;
    let ci;
    unsafe {
        let ctx = eccodes_sys::codes_context_get_default();

        println!("{name}: index_new");
        ci = eccodes_sys::codes_index_new(ctx, keys.as_ptr(), &mut err);
        println!("{name}: index_new: {err:?}");

        println!("{name}: index_add_file");
        err = eccodes_sys::codes_index_add_file(ci, grib_file.as_ptr());
        println!("{name}: index_add_file: {err:?}");

        thread::sleep(std::time::Duration::from_secs(3));

        println!("{name}: index_delete");
        eccodes_sys::codes_index_delete(ci);
        println!("{name}: index deleted");
    }
}

With eventually fail like so:

 cargo run                                                                                                                                
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/ec-test`
T1: index_new
T3: index_new
T1: index_new: 0
T1: index_add_file
T3: index_new: 0
T3: index_add_file
T1: index_add_file: -11
[1]    112709 segmentation fault (core dumped)  cargo run

Cannot compile `eccodes`

Describe the bug
After following documentation, command cargo build crashed with the following error:

error[E0432]: unresolved import `eccodes_sys::_IO_FILE`
  --> /Users/bastienlecorre/.cargo/registry/src/index.crates.io-6f17d22bba15001f/eccodes-0.6.9/src/intermediate_bindings.rs:17:27
   |
   |     CODES_TYPE_UNDEFINED, _IO_FILE,
   |                           ^^^^^^^^ no `_IO_FILE` in the root

To Reproduce
Steps to reproduce the behavior:

  1. cargo add eccodes
  2. cargo build

Expected behavior
The compile process should finish successfully.

Screenshots

Screenshot 2023-10-19 at 23 51 43

Desktop

  • OS: MacOS
  • Version: Sonoma 14.0

Additional context
New to Rust, maybe an error on my part?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.