roma-glushko / notifykit Goto Github PK

Returns events in batches in order we have detected them.

We are using sync approach to listening to events which works but there is one caveat. We had to interrupt after some time to see if we had not received a termination signal yet:

https://github.com/roma-glushko/inotifykit/blob/e3097ce41826908ec8f33c51c5b27afb33f02567/src/lib.rs#L38

So essentially, we are polling signal info instead of being notified when it happened.

Let's try to explore asynchronous approach to event watching/signal handling to save even more CPU cycles.

Here is a way to reproduce that on MacOS:

touch vi.com && mv vi.com vi.co

Instead of correctly stitching two rename events, debouncer produces two Rename(Any) events.

I'm thinking of incorporating the full debouncer logic to extend it a way notifykit usecases require.

For now, there are a few benefits from that:

• keep one single thread to process raw notify-rs events
• bring better async support
• ability to address bugs like #10
• expose more information in notifykit events potentially useful for end consumers (e.g. detection time, inode ID, etc.)

Right now, it's not match statement ignores Event classes when trying to match them there.

Also, we need to allowlist attributes like path, access_mode, etc, to be matched by the statement. It's not disallowed by default.

A quickest possible approach to reuse the current sync API in asyncio-based apps

I tested out notifykit as it seems to promising alternative to watchdog (which has all sorts of issues).

Alas, I run into an issue in one of my first experiments:

#!/usr/bin/env python
import asyncio
import os

from notifykit import Notifier


async def manage_watches(notifier):
    for i in range(10):
        path = f"./subdir{i:04d}"
        if not os.path.isdir(path):
            os.mkdir(path)
        print("BEFORE", path)
        notifier.watch([path])
        print("AFTER", path)
        await asyncio.sleep(0.01)


async def watch() -> None:
    notifier = Notifier(debounce_ms=200, debug=True)
    task = asyncio.create_task(manage_watches(notifier))
    async for event in notifier:
        print(event)
    await task


if __name__ == "__main__":
    asyncio.run(watch())

This example simulates the use case of a more complex asynchrounous program, where one task is responsible for adding (and removing watches) while another task is responsible for acting upon events.

When running this example, I get the following output before the script blocks:

BEFORE ./subdir0000
watcher: INotifyWatcher { channel: Sender { .. }, waker: Waker { inner: Waker { waker: WakerInternal { fd: File { fd: 8, path: "anon_inode:[eventfd]", read: true, write: true } } } } }
AFTER ./subdir0000
BEFORE ./subdir0001

It seems that adding watches while iterating over the notifier is not allowed. Adding the watches before is not a problem, but that is not my use case. (The recursive option is also not useful for the case on which this example is based, because it has a more complicated set of rules to decide which directories to (un)watch.)

Event though we are using notify-rs (that supports inotify-like watches across various platforms like macOS, Linux, Windows, etc) library in inotifykit, it's still tricky to get fully cross-platform API.

The problem is in events/information that different platform support. For example,

• some platforms support file metadata change events and some not
• some platforms emits rename as a separate event and some emits two events instead (old file deletion + new file creation)
• etc.

Because of this discrepancies in how events are dispatched, it still requires some work to smooth that experience to make it resemble a real platform-agnostic API.

It would be great to put that heavy-lifting on inotifykit, so our users could really focus on their business logic.

Solution

It seems like a way to go is to find a common ground between all supported file watcher's API and unify it on the library level.

One ramification of this is that we would need to hide some of the events or information that is not widely supported by all watchers. it seems like a reasonable limitation as if you had an access to that information, it would be impossible to run that code on the platform that didn't support it (so we would break our cross-platform promise).

This behavior should be optional, so people that needs missing info and don't need true cross-platform support would be able to have that.

Reference:

• https://pyo3.rs/v0.20.0/ecosystem/logging