Comments (16)
aidanheerdegen
commented on September 26, 2024
This may seem trivial, but I'm using num2date and date2num to rebase the time axes in netCDF files. When I do this operation it doesn't round trip. When I rebase the time dimension of a netCDF file and then rebase back to the original time units they are no longer the same.
So in the case below, ds2 is ds rebased, and then the reverse rebase applied, so they should be the same, and they appear to be on casual inspection:
(Pdb) ds.time
<xarray.DataArray 'time' (time: 120)>
array([693150.5, 693180. , 693209.5, 693240. , 693270.5, 693301. , 693331.5,
693362.5, 693393. , 693423.5, 693454. , 693484.5, 693515.5, 693545. ,
693574.5, 693605. , 693635.5, 693666. , 693696.5, 693727.5, 693758. ,
693788.5, 693819. , 693849.5, 693880.5, 693910. , 693939.5, 693970. ,
694000.5, 694031. , 694061.5, 694092.5, 694123. , 694153.5, 694184. ,
694214.5, 694245.5, 694275. , 694304.5, 694335. , 694365.5, 694396. ,
694426.5, 694457.5, 694488. , 694518.5, 694549. , 694579.5, 694610.5,
694640. , 694669.5, 694700. , 694730.5, 694761. , 694791.5, 694822.5,
694853. , 694883.5, 694914. , 694944.5, 694975.5, 695005. , 695034.5,
695065. , 695095.5, 695126. , 695156.5, 695187.5, 695218. , 695248.5,
695279. , 695309.5, 695340.5, 695370. , 695399.5, 695430. , 695460.5,
695491. , 695521.5, 695552.5, 695583. , 695613.5, 695644. , 695674.5,
695705.5, 695735. , 695764.5, 695795. , 695825.5, 695856. , 695886.5,
695917.5, 695948. , 695978.5, 696009. , 696039.5, 696070.5, 696100. ,
696129.5, 696160. , 696190.5, 696221. , 696251.5, 696282.5, 696313. ,
696343.5, 696374. , 696404.5, 696435.5, 696465. , 696494.5, 696525. ,
696555.5, 696586. , 696616.5, 696647.5, 696678. , 696708.5, 696739. ,
696769.5])
Coordinates:
* time (time) float64 6.932e+05 6.932e+05 ... 6.967e+05 6.968e+05
Attributes:
long_name: time
units: days since 0001-01-01 00:00:00
cartesian_axis: T
calendar_type: NOLEAP
calendar: NOLEAP
bounds: time_bounds
(Pdb) ds2.time
<xarray.DataArray 'time' (time: 120)>
array([693150.5, 693180. , 693209.5, 693240. , 693270.5, 693301. , 693331.5,
693362.5, 693393. , 693423.5, 693454. , 693484.5, 693515.5, 693545. ,
693574.5, 693605. , 693635.5, 693666. , 693696.5, 693727.5, 693758. ,
693788.5, 693819. , 693849.5, 693880.5, 693910. , 693939.5, 693970. ,
694000.5, 694031. , 694061.5, 694092.5, 694123. , 694153.5, 694184. ,
694214.5, 694245.5, 694275. , 694304.5, 694335. , 694365.5, 694396. ,
694426.5, 694457.5, 694488. , 694518.5, 694549. , 694579.5, 694610.5,
694640. , 694669.5, 694700. , 694730.5, 694761. , 694791.5, 694822.5,
694853. , 694883.5, 694914. , 694944.5, 694975.5, 695005. , 695034.5,
695065. , 695095.5, 695126. , 695156.5, 695187.5, 695218. , 695248.5,
695279. , 695309.5, 695340.5, 695370. , 695399.5, 695430. , 695460.5,
695491. , 695521.5, 695552.5, 695583. , 695613.5, 695644. , 695674.5,
695705.5, 695735. , 695764.5, 695795. , 695825.5, 695856. , 695886.5,
695917.5, 695948. , 695978.5, 696009. , 696039.5, 696070.5, 696100. ,
696129.5, 696160. , 696190.5, 696221. , 696251.5, 696282.5, 696313. ,
696343.5, 696374. , 696404.5, 696435.5, 696465. , 696494.5, 696525. ,
696555.5, 696586. , 696616.5, 696647.5, 696678. , 696708.5, 696739. ,
696769.5])
Coordinates:
* time (time) float64 6.932e+05 6.932e+05 ... 6.967e+05 6.968e+05
Attributes:
long_name: time
units: days since 0001-01-01 00:00:00
cartesian_axis: T
calendar_type: NOLEAP
calendar: NOLEAP
bounds: time_boundsbut
(Pdb) ds2.equals(ds1)
False
(Pdb) ds.time.values - ds2.time.values
array([-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10,
-1.16415322e-10, -1.16415322e-10, -1.16415322e-10, -1.16415322e-10])My cftime and python versions:
$ conda list cftime
cftime 1.0.2.1 py36h7eb728f_0 conda-forge
$ python --version
Python 3.6.6
from cftime.
aidanheerdegen
commented on September 26, 2024
Another simple example (suitable for testing?)
(Pdb) cftime.date2num(cftime.datetime(1, 12, 1, 0, 0, 0, 0, -1, 1), units='days since 01-01-01',calendar='noleap')
334.00000000000006
from cftime.
jswhit
commented on September 26, 2024
Pull request #79 improves the precision, so now you will get 3650.0000000000007958 instead of 3650.000000000001. I don't see what I can do about those extra annoying digits - that's just the nature of floating point arithmetic.
from cftime.
aidanheerdegen
commented on September 26, 2024
I was hoping there might some clever way to enforce the same precision on the number output from date2num as the precision of the date itself. In the example above:
cftime.date2num(cftime.datetime(1, 12, 1, 0, 0, 0, 0, -1, 1), units='days since 01-01-01',calendar='noleap')
the precision is microseconds. So we need less precision, not more. Or at least, specified precision.
I think it is an error that this doesn't roundtrip:
>>> day=1
>>> roundtripday = cftime.date2num(cftime.num2date(day,units='days since 01-01-01',calendar='noleap'),units='days since 01-01-01',calendar='noleap')
>>> roundtripday
1.0000000000000002
>>> assert(day == roundtripday)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError
Do you agree? In which case, is there a solution?
from cftime.
jswhit
commented on September 26, 2024
I'm working on splitting the Julian day into an integer and a floating point remainder, which will reduce round-off error. There won't be a general solution though - there will always be cases where the round-off error prevents the roundtrip from matching.
from cftime.
aidanheerdegen
commented on September 26, 2024
Thanks for the feedback.
Now I can't get it to reproduce the above behaviour. In my python3 environment it will roundtrip ok:
$ conda list cftime
# Name Version Build Channel
cftime 1.0.2.1 py36h7eb728f_0 conda-forge
$ python
Python 3.6.6 | packaged by conda-forge | (default, Oct 12 2018, 14:08:43)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cftime
>>> day=1
>>> print(cftime.date2num(cftime.num2date(day,units='days since 01-01-01',calendar='noleap'),units='days since 01-01-01',calendar='noleap'))
1.0
but not python2:
$ conda list cftime
# Name Version Build Channel
cftime 1.0.2.1 py27h7eb728f_0 conda-forge
$ python
Python 2.7.15 | packaged by conda-forge | (default, Oct 12 2018, 14:10:50)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cftime
>>> day=1
>>> print(cftime.date2num(cftime.num2date(day,units='days since 01-01-01',calendar='noleap'),units='days since 01-01-01',calendar='noleap'))
1.0000000000000002
>>>
But yesterday I am sure it was failing in all environments (the python3 environment has been updated today). I am mystified.
from cftime.
jswhit
commented on September 26, 2024
In this particular case the failure in the roundtrip is caused by Unidata/netcdf4-python#433. If you comment out the code in cftime/_cftime.pyx after
# Add a small offset (proportional to Julian date) for correct re-conversion.
and
# remove the offset from the microsecond calculation.
you will get exactly 3650.0. However, many of the tests will then fail (for reasons described in that pull request).
from cftime.
jswhit
commented on September 26, 2024
I think pull request #80 addresses this issue (at least for all the cases you presented). @aidanheerdegen, could you test this (branch msround)?
from cftime.
aidanheerdegen
commented on September 26, 2024
Hi Jeff, thanks for looking into this so promptly and providing a potential solution. I won't be able to test this until Monday (Australian time), but will get back to you when I have. Cheers
from cftime.
aidanheerdegen
commented on September 26, 2024
Hi Jeff,
I'm having issues building the package
gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march -DNDEBUG -D_FORTIFY_SOURCE -fPIC -I/g/data3/hh5/public/apps/miniconda3/envs/analysis27-18.10/include/python2.7 -c cftime/_cftime.c -o build/temp.linux-x86_64-2.7/cftime/_cftime.o
gcc: error: unrecognized command line option '-march'; did you mean '-march='?
error: command 'gcc' failed with exit status 1
Some limited googling didn't come up with a quick solution. Do you have any idea of the best way to overcome this?
from cftime.
jswhit
commented on September 26, 2024
Something is amiss with your build environment. Are you using 'python setup.py build' or conda?
from cftime.
jswhit
commented on September 26, 2024
python -c 'import sysconfig; print(sysconfig.get_config_var("CFLAGS"))' will tell you what your default python compiler build flags are.
from cftime.
aidanheerdegen
commented on September 26, 2024
I am using a conda python and
python setup.py build
Is this a bad combination?
$ python -c 'import sysconfig; print(sysconfig.get_config_var("CFLAGS"))'
-fno-strict-aliasing -m64 -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
from cftime.
jswhit
commented on September 26, 2024
No, that looks fine. The problem is the empty '-march', it should be '-march=platform'. Don't know where conda is getting that from then - maybe from your environment? Do you have a CPPFLAGS env var set?
from cftime.
aidanheerdegen
commented on September 26, 2024
Thanks Jeff, that was the answer. My module load conda is putting all sorts of frankly "interesting" things in my environment:
$ env | grep -i FLAGS
LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now
CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE
DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE
CFLAGS=-march
DEBUG_CFLAGS=-march
I 've switched to another conda environment which doesn't do this, and it is compiling ok, thanks for the assistance.
And yes, that does seem to behave as I would wish. Round-tripping works as do other whole number issues I had before.
Thanks for the fix. Does that break anything else?
from cftime.
jswhit
commented on September 26, 2024
No, all the tests are passing. I will merge the pull request now.
from cftime.
Related Issues (20)
- Dealing with netcdf time:units with unexpected characters HOT 5
- Future plans for the `longdouble` argument HOT 2
- Support time units that are a multiple of second, hour, or day (with a reference date) HOT 5
- Unexpected results when not using separators in format code with cftime.datetime.strptime HOT 7
- Matplotlib date formatter and cftime : incompatible ? HOT 8
- Parsing "months since ..." dates HOT 4
- Build `musllinux` wheels HOT 4
- Missing 1.6.3 sdist on pypi
- reformat axis, calendar, units or all? Repeating time ranges HOT 1
- Passing empty array to date2num raises `IndexError: index 0 is out of bounds for axis 0 with size 0` HOT 1
- Numpy 2.0 incompatibility
- BUG: Not NumPy 2.0 compatible HOT 9
- PyPI release pulls numpy 2.0.0rc1 for python 3.12 HOT 4
- [Bug]: `TypeError: unsupported operand type(s) for +: 'cftime._cftime.DatetimeNoLeap' and 'NoneType'` on Linux HOT 13
- No `aarch64` wheels on PyPI HOT 3
- PyPI publish release action failing HOT 1
- Missing sdist for 1.6.4 HOT 2
- sdist not uploaded to pypi by github action HOT 1
- Error message recommends unsupported option HOT 4
- Usage examples other than API docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cftime.