detection is really slow in some cases about clevercsv HOT 5 CLOSED

Thanks again @kaskawu for reporting this issue. I've updated CleverCSV using the unix_path regex you suggested above (diving into it, that regex seemed to be the problem). I'm preparing an updated release of the package now. Thanks also @lmmentel for confirming!

from clevercsv.

Hi @kaskawu! Thanks for your interest in the package and for reporting this issue. Strangely, I have a hard time replicating your results:

$ python3 -m timeit -- "from clevercsv import Detector; Detector().detect('fileurl="file://$PROJECT_DIR$/../aaaaaa_aaaaaaa_aaaaa/.aaa/." filepath=$')"
500 loops, best of 5: 721 usec per loop

and with the change you propose:

$ python3 -m timeit -- "from clevercsv import Detector; Detector().detect('fileurl="file://$PROJECT_DIR$/../aaaaaa_aaaaaaa_aaaaa/.aaa/." filepath=$')"
1 loop, best of 5: 638 usec per loop

What version of the regex package are you using?

That said, it does seem to make a massive difference on your system, so I'm certainly open to making this change. I do however want to make sure I fully understand the cause before implementing any changes. Thanks!

from clevercsv.

> pip3 freeze | grep regex
regex==2020.5.7
> 

That said, I tested across multiple python versions. I tried python 3.7 and 3.8, and the slowdown only happens on 3.8:

Python 3.7:

> python3 --version
Python 3.7.7
> python3 -m timeit -- "from clevercsv import Detector; Detector().detect('fileurl="file://$PROJECT_DIR$/../aaaaaa_aaaaaaa_aaaaa/.aaa/." filepath=$')"
1 loop, best of 5: 5.75 msec per loop

Python 3.8:

> python3 --version
Python 3.8.2
> python3 -m timeit -n 1 -r 1 -- "from clevercsv import Detector; Detector().detect('fileurl="file://$PROJECT_DIR$/../aaaaaa_aaaaaaa_aaaaa/.aaa/." filepath=$')"
1 loop, best of 1: 19.7 sec per loop

from clevercsv.

Wow that's very interesting! Thanks for doing some more digging. I'll take a more detailed look at this soon, hopefully I can reproduce it in someway and figure out a good solution. Thanks again for reporting it!

from clevercsv.

Same here, performance drops with python3.8

python --version
Python 3.8.1
python -m timeit -n 1 -r 1 -- "from clevercsv import Detector; Detector().detect('fileurl="file://$PROJECT_DIR$/../aaaaaa_aaaaaaa_aaaaa/.aaa/." filepath=$')" 
1 loop, best of 1: 8.34 sec per loop

from clevercsv.