Comments (2)
JackKelly
commented on May 31, 2024
Definitely agree!
UPDATE: Ignore the rest of this comment! Please see the next comment!...
The Met Office NetCDF files are about 100 MBytes.
100 MB maybe feels a little large to put into our git repo. What do you think?
We could put a handful of testing NetCDF files in an OCF public S3 bucket (which would also help us test the code that reads from S3. But that would slow the tests down (when running locally, at least)). Or we could reduce the filesize by throwing away most of the data, and then saving the NetCDF in git. Or we could do both?!? Or something else? What would you prefer?
from metoffice_ec2.
JackKelly
commented on May 31, 2024
Scrap what I said above! Some of the NetCDF files are less than a MB (so we can probably just stick them into git).
For reference, here's the mean size (in MB) of MOGREPS-UK NetCDF files from the Met Office:
air_pressure 229.484640
air_pressure_at_sea_level 3.359660
air_temperature 21.432869
atmosphere_boundary_layer_thickness 3.463518
cloud_area_fraction 2.139006
cloud_area_fraction_assuming_only_consider_surface_to_1000_feet_asl 1.191044
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_2p5_oktas 1.198854
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_4p5_oktas 1.142050
cloud_volume_fraction_in_atmosphere_layer 31.550393
dew_point_temperature 4.072501
fog_area_fraction 0.287845
freezing_level_altitude 4.102886
high_type_cloud_area_fraction 1.457913
land_binary_mask 0.209811
low_type_cloud_area_fraction 2.073978
lwe_graupel_and_hail_fall_rate 0.145724
lwe_snowfall_rate 0.142409
lwe_thickness_of_graupel_and_hail_fall_amount 0.145957
lwe_thickness_of_snowfall_amount 0.143660
lwe_thickness_of_surface_snow_amount 0.368421
medium_type_cloud_area_fraction 1.509519
number_of_lightning_flashes_per_unit_area 0.139348
rainfall_rate 0.545779
relative_humidity 70.426761
soil_temperature 5.251025
surface_air_pressure 7.087338
surface_altitude 2.405621
surface_diffusive_downwelling_shortwave_flux_in_air 4.592893
surface_direct_downwelling_shortwave_flux_in_air 4.460242
surface_downwelling_longwave_flux_in_air 6.106000
surface_downwelling_shortwave_flux_in_air 5.235519
surface_downwelling_ultraviolet_flux_in_air 1.054016
surface_temperature 2.256696
surface_upward_sensible_heat_flux 4.923685
surface_upwelling_ultraviolet_flux_in_air 0.400496
thickness_of_rainfall_amount 0.620976
total_radar_reflectivity 24.705189
total_radar_reflectivity_max_in_column 3.865009
upward_air_velocity 112.851796
visibility_in_air 7.156086
wet_bulb_freezing_level_altitude 4.419323
wet_bulb_potential_temperature 4.963114
wind_from_direction 81.070012
wind_speed 33.663551
wind_speed_of_gust 3.214056
from metoffice_ec2.
Related Issues (20)
- Use zstd instead of LZMA HOT 1
- Script uses increasingly large amounts of memory on EC2 HOT 5
- Avoid using h5netcdf? HOT 9
- rename MetOfficeMessage.load_netcdf to open_netcdf
- More unit tests :) HOT 1
- Run creating output images for FE on AWS
- Investigate why script doesn't output files HOT 2
- Add Sentry Error Logging
- Predict PV yield using baseline NWP model
- Deal with increasing S3 cost HOT 3
- Look into running workload on Spot instead of Fargate HOT 2
- Autoscale Compute Fleet based on SQS HOT 1
- Automatically build docker images HOT 1
- Find out which irradiance field to use HOT 2
- load_subset_and_save_data() missing 1 required positional argument
- Use irradiance data for predictions HOT 1
- float() argument must be a string or a number, not 'list' HOT 5
- dimensions or multi-index levels ['height'] do not exist HOT 2
- Run inference on AWS
- Various NaN values in predictions HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metoffice_ec2.