.over() performs quite slow in given sample about polars HOT 4 CLOSED

@avimallu Weird. Its a 14 core processor (13th gen i7)

from polars.

.over() performs a group_by e.g.

(df2.group_by("group")
    .agg(
       pl.when(pl.col("random_value")==1)
         .then(2)
         .otherwise(pl.col("random_value"))
         .alias("result")
    )
)

But the lists would be exploded and the results joined back in the correct row order.

from polars.

I find it strange that it doesnt try to do 1 operation across all the groups simultaneously.

Why do you find it strange? You're explicitly asking to do a group-by (using over) operation, and it's doing that.

What you seem to be asking for is for Polars to ignore that you've asked for an over calculation, because the calculation's result is the same regardless of whether or not the group is present. That's something that could be achieved using LazyFrames, but I don't think it's implemented right now.

I'm also not able to reproduce your 20x slower timings, so there seems to be some optimization going on - in my case, it could be that it's parallelizing across cores efficiently - your recent issue (#15480) might point to having a processor with fewer available cores? 🤔

24.8 ms ± 886 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
107 ms ± 569 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
384 ms ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

from polars.

Hmm, forgot to turn off battery saver. Turned everything to max and activated all cores manually and got this

44.2 ms ± 2.74 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
24.8 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
477 ms ± 15.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Idk. Maybe i just dont understand the underlying process of .over(). Feels like it still has potential to do it better in the given example

from polars.