Comments (6)
There are probably some tricks we can try like using a static lookup table instead of branches:
I tried that, but it did not improve the situation on my machine.
I did some other refactoring that brought down the benchmark's runtime by 20% on my machine for s/us/ns.
There are also code paths that the benchmark is not really testing (those I saw a similar improvement from my changes): s/ms/us/ns values so large that the "day" parsing part of the code gets triggered as well as negative offsets, both don't get checked because the benchmark runs only with random numbers between 0 and 10.
from pandas.
I now have a version of the date time conversion that is correct and tested and runs up to 2.5x faster than main and the previous version before the regression (at least on my machine, M2 ARM). Interestingly, I do not see large differences in runtimes between the pre regression commit and main.
I would suggest changing the benchmark inputs to negative values that are at least on the order of one day to make most of the logic in the function actually show up in the runtime of the benchmark. What do you think? Is there a procedure to follow to update the history of benchmark runtimes?
To back that up, here are some of my benchmark results for all three versions, I will update my PR soon:
input range | version | main | main pre regression | fix |
---|---|---|---|
+30±10ns | 4.9 | 4.8 | 3.9 |
-30±10ns | 7.9 | 7.2 | 5.2 |
±30ns | 8.4 | 8.5 | 7.1 |
+4d±100s | 11.6 | 11.4 | 5.2 |
-4d±100s | 12.7 | 12.7 | 5.3 |
±4d | 9.7 | 9.4 | 7.5 |
(Runtimes are in ms, runs are for 10^6 random values)
The current versions of the function have branch prediction and variable dependency issues that only show up when using larger ranges of inputs and negative values. The previously mentioned checks against the base unit size are of very low cost in reality since the unit size does not change (in the benchmark, but also in real world) and get branch predicted without failure.
Randomly negative and positive inputs still do not do all too well in my version, but realistically timestamps would not have this property.
from pandas.
Possibly related?: #57035
Good find, but the regression here should be unrelated to that plotting issue. Converting 1M timestamps takes some milliseconds, but the referenced regression is in the seconds range.
from pandas.
I think the issue here is that we combined a few case statements into one:
case NPY_FR_s:
case NPY_FR_ms:
case NPY_FR_us:
case NPY_FR_ns: {
npy_int64 per_sec;
if (base == NPY_FR_s) {
per_sec = 1;
} else if (base == NPY_FR_ms) {
per_sec = 1000;
} else if (base == NPY_FR_us) {
per_sec = 1000000;
} else {
per_sec = 1000000000;
}
...
Whereas previously the code was copy/pasted and slightly tweaked for every case statement.
There are probably some tricks we can try like using a static lookup table instead of branches:
case NPY_FR_s:
case NPY_FR_ms:
case NPY_FR_us:
case NPY_FR_ns: {
const npy_int64 sec_per_day = 86400;
static npy_int64 per_secs[] = {
1, // NPY_FR_s
1000, // NPY_FR_ms
1000000, // NPY_FR_us
1000000000 // NPY_FR_ns
};
const npy_int64 per_sec = per_secs[base - NPY_FR_s];
but I'm not really sure the tricks are worth it. And I think the code de-duplication is probably the increased timing here
from pandas.
@dontgoto thanks for the great research. If you have improvements to the benchmark I would just go ahead and submit it - we don't manage the history of changes to a benchmark itself that strictly
from pandas.
Possibly related?: #57035
from pandas.
Related Issues (20)
- BUG: merging DataFrames on a column containing just NaN values triggers address violation in `safe_sort` HOT 4
- BUG: UnboundLocalError when full outer merging two dataframes HOT 3
- Python 3.13 wheel builds are failing HOT 2
- BUG: Inconsistent behavior between `.any` and `.all` with missing values HOT 1
- BUG: stacked bar graphs show invalid label position due to invalid rectangle bottom when data is 0 HOT 9
- BUG: `isna` on pyarrow backed Series is returning Series with `bool` dtype instead of `bool[pyarrow]` HOT 9
- BUG: dropna in pivot_table is not affecting the outcome
- BUG: pandas.DataFrame.plot crashes, if subplots argument receives a touple. HOT 3
- BUG: assert_frame_equal does not include the obj parameter in error when a MultiIndex is different HOT 4
- QST: How to use 'numba' for group by sum HOT 1
- BUG: Large XML files on Windows trigger false Encoding error HOT 2
- BUG: Series.replace(dict-like, dict-like) raises uninformative AttributeError HOT 1
- BUG: escapechar=',' Causes Double Commas in Output in Pandas 2.2.2 HOT 5
- DOC: fix docstring validation errors for `pandas.Timestamp` HOT 15
- DOC: Development on Gitpod have problems HOT 1
- BUG: Ability to set both color and style in pandas plotting HOT 2
- Title: Feature Request: Improve diff Function to Support Forward and Backward CompletionENH: HOT 2
- Title: Feature Request: Improve diff Function to Support Forward and Backward CompletionENH: HOT 2
- Bug in Chunk Processing: Non-NULL IDs Become NULL During IterationBUG: HOT 2
- ENH: Add Float128 support for groupby. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandas.