Improve `--ignore` about pytest-watch HOT 11 CLOSED

@aldanor, any more thoughts on this?

from pytest-watch.

My point was this (it's easier to illustrate it with an example):

/foo
/bar
/foo/bar
/foo/foo
/foo/foo/hello
/foo/foo/world

Let's say with want to ignore bar. Because watchdog doesn't allow us to plug into observer's recursion logic unless we reimplement it, we have to walk the entire directory tree oursevels (just ONCE though) and figure out the folder we want to watch: if all of the subfolders in a folder are watched, then watch this folder; if at least one of the subfolders is ignored, watch all of the non-ignored subfolder, e.g. in the example above it would be

/foo
/foo/foo

(note the observers here are NON recursive, i.e. /foo matches only /foo/* and not /foo/**)

This could be easily implemented and it would work fine imo, chances are you're not going to have 15-level deep source tree with hundred thousand files you need to watch in it.

from pytest-watch.

@aldanor Yeah, that makes sense.

My point was that this is a Linux-only thing, since Windows and Mac both have methods to watch /foo at the operating system level and allow us to efficiently filter ignored paths in the callback. (Because watchdog can put a single watcher at the root instead of walking the full directory tree.) So let's try to keep things simple. Or see if there's another solution altogether like fixing this in watchdog (i.e. fixing the root of the problem of inefficient deep directory watches on Linux), since this would introduce a good bit of complexity to the project.

from pytest-watch.

allow us to efficiently filter ignored paths in the callback

This just doesn't work at all on Linux -- as I said, if you have a virtualenv (or .tox, quite commonly) folder in the source tree, the handler is never triggered at all -- it just gets completely swamped by thousands of files and folders (and that's on a production grade 48-core server with a raid array, lol).

The choice here is between something that works on all platforms but could work more optimally on some and something that just plain doesn't work on some platforms. Plus, I don't think there's such a huge overhead on watching e.g. 10 folders? For a typical source tree, my method (even a full-depth implementation and not a top-level one) would only make you watch several folders. Give me a single example of a repo where that wouldn't be true and there would be a noticeable overhead of launching multiple observers?

The worst case scenario for my method is when you have a source tree 15 levels deep and a single folder all the way down at the 15-th level that you have to ignore. Then you sort of have to subscribe to all folders on all 14 levels. But is that at all likely?

The best case scenario is when you have to ignore a few folders at the root level, and that's most often the case.

from pytest-watch.

@joeyespo

Now that I think about, this could be related to me having to use polling observer on Linux; maybe we could ignore events at the handler level for non-polling observers and at the observer level for polling (because then it has to keep track of snapshots which makes it blow up)?

This will need to be confirmed though.

from pytest-watch.

The choice here is between something that works on all platforms but could work more optimally on some and something that just plain doesn't work on some platforms

Well, we could special-case the Linux implementation so it's optimized for all three major platforms. My point was that we should look for the right abstraction so we don't have to hand-code the special case. Whether that be watchdog or another library. "Should I take advantage of the OS, or manually wire-up watches to all subdirectories" ideally shouldn't be ptw's concern.

Plus, I don't think there's such a huge overhead on watching

Just like you don't want to optimized this project for Mac / Windows, I don't really want to give up the OS-level optimizations of non-Linux. (I don't want to exclude Linux either.) So let's keep those optimizations in place and do the custom thing when using observers that don't take advantage of the underlying OS (namely, inotify, kqueues, and polling). I'm suggesting that we find a way to defer this to a lower abstraction library like watchdog or a watchdog plugin. This allows all projects that monitor a directory tree to take advantage of this optimization, not just ptw.

from pytest-watch.

Now that I think about, this could be related to me having to use polling observer on Linux

Why are you using a polling observer? (Are you using --poll?)

from pytest-watch.

I had to use --poll on the Linux box at work because it just wouldn't work without it for some reason.

from pytest-watch.

That's odd. Want to get to the bottom of that then?

Are you working in a virtual machine? That's the primary use case for --poll right now.

from pytest-watch.

Nope, it's remote server with an NFS mount. I'm actually editing files from Windows and it picks it up from Linux remotely.

from pytest-watch.

@aldanor Is this still an issue for you? v4.0 was just published, which forwards --ignore to pytest and has improved internals that might help.

I'm closing as an effort to clean up issues, but feel free to re-open or open a new issue with some more specifics from 4.0 onward.

from pytest-watch.