Comments (13)
@huard and @Zeitsperre, the above test runs to completion. It asserts that clisops.subset
can return a valid xarray.Dataset
that has a coordinate with ZERO values in it. If you specify the lat_bnds
the wrong way around it returns an object with a shape of (1032, 0, 384)
?
Do you agree that we should change this behaviour to one of the following?
- A: raise an Exception such as: "you provided latitude bounds that are incompatible with the range of the data latitude values which are in the range: -".
- B: detect that the user has sent the
lat_bnds
in the wrong order, then reverse them and return the correct data within that selection.
Tagging @cehbrecht @ellesmith88
from clisops.
Oh yeah, that's definitely not working as intended. If we wanted a quick fix, adding this check to the check_lons
decorator would be simple to perform (rename it to check_lons_lats
?).
In terms of the behaviour, I would lean towards raising an error, as the same coordinates fed into other GIS libraries would be seen as an error in most cases. Sending back the allowed range values would be a bit more overhead (needs to examine the object to get them), but that's fine if the process is stopped.
from clisops.
Since the bounds are within the range, I would suggest this is not an error.
This is possibly a user wanting to reverse the order of the coordinates, at least, this seems to be how xarray
interprets a negative slice. So another option to consider is to do slice(start, stop, -1)
when bounds are in decreasing order. This will reverse the output.
from clisops.
If xarray
handles negative slices without error, then the way forward is pretty clear. Raising a UserWarning
(from the warnings
library) works fine for me. the check can be added to the same decorator I mentioned above.
from clisops.
@Zeitsperre: xarray
doesn't return an error - but the resulting DataArray
object is kind of meaningless:
>>> print(ds1.tas.size)
0
I don't believe that this is ever useful. I would be in favour of clisops
fixing the issue as follows:
- identify the bounds are in reverse order
- warn the user that we have reversed the bounds order, but
- return the correct subset
What do you think?
@huard, if the user wants to reverse an axis then I think we should leave it to them to do:
ds.tas.reindex(lat=ds.tas.lat[::-1])
from clisops.
@agstephens Ok, I would go further and suggest documenting the fact that the order of bounds is irrelevant and not even raise a warning.
from clisops.
@huard and @Zeitsperre Coming back to this issue. I like David's idea of not even telling the user. So, we just document it clearly:
Note about selection ranges in the opposite direction to the coordinate values
If you request a selection range (such as level, latitude or longitude) that specifies the lower and upper bounds in the opposite direction to the actual coordinate values then clisops
will detect this issue and reverse your selection before returning the data subset.
from clisops.
What do you think? I don't think there will be much code to write.
The only issue is time - can we do the same with the time axis - or is that too dangerous?
from clisops.
Not sure about time. Again, I would look to how xarray
or pandas
handles slices or intervals that go backwards in time.
One thing that could also be an issue is longitude. If we want to grab a bounding box that crosses the date-line or the meridional line, how do we go about doing that? Asking for the bounding box with lons that start in eastern Russia (175) and ending in Alaska/Yukon (-175) would select the inverse in that case. This issue might still be a problem in the code base (I can't recall if we addressed it already for bounding boxes; for polygons the code splits the features and extracts the mask), but the case here makes me question performing the same adjustments for other coordinates.
For latitude, I don't see a problem though.
from clisops.
I suppose that we assume that the data clisops will operate on is all standard ? That there will be no cases where some models have ascending order, while others have descending order?
In that case, I suggest that for all coordinates, we reorder the coordinate bounds to match the order of the original coordinate, whether it's ascending or descending (e.g. ocean depth). If someone needs to reorder the array, then I agree that this should be done explicitly.
If on the other hand we expect that in some use cases model A has ascending order, and model B descending order, then it could make sense for clisops to impose the alignment requested by the coordinate bounds. This would guarantee that model differences are automatically handled by the subsetting operation, and that outputs are comparable after subsetting.
from clisops.
Another point:
- if a web-form created the request from a widget: it would always be (for example) low to high
- if some datasets were ordered from high to low: then the standard xarray approach would never allow the user to get back the data - that is a problem
As mentioned by @huard:
model A has ascending order, and model B descending order
- this is really something that requires checking and a
daops.fix
should be applied.
So, actions:
- we need a standard check that all lats/lons/levels are
positive
in their ordering - we need to make sure, in the C3S case, that all our datasets have low to high values for their coordinates. In this case, the best approach is to always return data - which means it might return data that is ordered differently to the bounds given in the request.
from clisops.
@ellesmith88 please check if this already covered in clisops.core.subset
.
from clisops.
Fixed.
from clisops.
Related Issues (20)
- New release HOT 5
- Prototype adding time bounds to Datasets returned by time averaging
- Review the regrid-main branch ready for merger HOT 1
- Encountering problem with `clisops.ops.subset` related to `clisops.utils.dataset_utils.check_lon_alignment` HOT 2
- inconsistent bounds (lat_bnds etc) after subset operation HOT 7
- subset with time_components fails for day values which are not in the netcdf file HOT 2
- Notebooks are not being served on ReadTheDocs
- InvalidParameterValue in average notebook
- subset_bbox_indexer broken with latest xarray HOT 2
- Add a 'intersects' option to subset bbox and shape HOT 2
- Problem when lon_bnds set to None in subset_bbox
- use a test data generator for specfic test cases?
- Migrate to shapely 2.0 HOT 2
- shape_bbox_indexer seems fragile with some geometries HOT 1
- Support for points in shape_bbox_indexer HOT 1
- ⚠️ Nightly upstream-dev CI failed ⚠️
- Error: The defined _FillValue and missing_value are not the same HOT 5
- Problems with ATLAS datasets HOT 8
- PermissionError upon importing clisops.core in some Windows environments HOT 5
- Error when performingsubset_bbox on dataset that contains multiple timesteps and rotated coordinates HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clisops.