`filter` slows down on large datasets about datar HOT 6 CLOSED

Closing this as datar.dplyr.filter() now achieves similar performance as DataFrame.loc[].

from datar.

There could be cython functions called inside DataFrame.query(), but I need to confirm.
However, before that, we can confirm whether it is because of the optimization inside DataFrame.query() or somewhere else. Could you try this:

conditions = (df_test.Description =="WHITE HANGING HEART T-LIGHT HOLDER" ) | (df_test.Description =="REGENCY CAKESTAN
D 3 TIER")

%%time
filter(df_test, conditions)

%%time
df_test[conditions]

from datar.

Here is the output

from datar.

Looks like that 20x time was optimized inside .query(). One of the reasons might be that the data in the conditions are from inside the data frame itself.

What if you do:

descr = df_test.Description.copy()
df_test.query('(@descr == "WHITE HANGING HEART T-LIGHT HOLDER") | (@descr == "REGENCY CAKESTAND 3 TIER")')

from datar.

The condition aggregation is also eating the time up. There is an extra calculation:

# check condition, conditions
    condition = Array([True] * _data.shape[0])
    for cond in conditions:
        if is_scalar(cond):
            cond = Array([cond] * _data.shape[0])
        condition = condition & cond

    condition[is_null(condition)] = False
    try:
        condition = condition.values.flatten()
    except AttributeError:
        ...

    out = _data[condition]

that I used an initial array with all Trues. Let's try to use the first condition passed in as the initial condition and see if it helps.

Might need a prun to profile which steps are eating up the time.

from datar.

See: https://pwwang.github.io/datar/performance/#datar.dplyr.filter()-vs-DataFrame.loc%5B%5D-vs-DataFrame.query()

from datar.