Comments (2)
the above doesn't work as such, the times columns is a list, so you'll need to do check if the entry bein processed has the 'values' key.
from fmiopendata.
Here's what I've come up with:
args = ['timeseries=True', f"starttime={start_time}", f"endtime={end_time}"]
obs = download_stored_query(query, args=args)
cols = set([v for p in obs.data for v in obs.data[p]])
cols.remove('times')
dfs = []
for name in obs.data:
data = {k: obs.data[name][k]['values'] for k in cols}
idx = pd.DatetimeIndex(name='hour', data=obs.data[name]['times'])
idx0 = pd.CategoricalIndex(name='place', data=[name]*idx.size)
df = pd.DataFrame(data=data, index=[idx0, idx], columns=cols, dtype='float64')
dfs.append(df)
df = pd.concat(dfs)
df.attrs.update({'location_metadata': obs.location_metadata})
df.to_parquet('data/airquality.parquet')
If you wanted to compress this, unfortunately Pandas and numpy don't offer compressed fixed point formats, but you can use int16
and multiply the values by 10 to achieve that.
Alternative indexing:
mi = pd.MultiIndex.from_product([[name], obs.data[name]['times']], names=['place', 'hour'])
df = pd.DataFrame(data=data, index=mi, columns=cols, dtype='float64')
from fmiopendata.
Related Issues (20)
- How to parse netcdf to use forecast::enfuser::airquality? HOT 2
- download_stored_query & timeseries - KeyError HOT 3
- Non-timeseries data & HTTP Error 400: Bad Request HOT 4
- Possible bug when fetching data before January 1st 1970 HOT 1
- Possible bug when fetching data before January 1st 1970 HOT 2
- dropsonde data ei tule? HOT 1
- Forecast data download - ModuleNotFoundError: No module named 'eccodes' " HOT 6
- Use GitHub Actions instead of Travis for unit tests
- Fix deprecation warnings in WFS XML handling HOT 1
- Mast data parsing doesn't work
- Add more details to WFS html listing HOT 1
- requests module to "install_requires" HOT 1
- Available parameters for fmiopendata.wfs download_stored_query HOT 1
- Add support to the FMI timeseries endpoint, which provides CSV, JSON and XML formats HOT 2
- half an hour time shift? HOT 1
- example does not work? HOT 2
- grid data fails due to "AttributeError: module 'eccodes' has no attribute 'GribFile'" HOT 4
- Re-run examples and update README accordingly
- Download and calibrate latest radar reflectivity (dBZ) composite fails HOT 2
- TypeError in downloading and parsing the lightning data HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fmiopendata.