I have a server for file uploads. With low latency I need to trigger some python code that reads the incoming files and...

I have an issue right now though. Some times the files are empty and I think it’s because the upload is not done yet. How can I determine if the file is done being written to?

I think inotify have this functionality but I agree with you that it is nice to have it platform independent.

Do you have a proposal for on how to handle this?

I think I'll like this package.

One problem I had using is that Emacs autosave files are detected. These files start with .#

I worked around this problem by changing the PythonWatcher like this:

class PythonWatcher(DefaultDirWatcher):
    def should_watch_file(self, entry):
        return entry.name.endswith(('.py', '.pyx', '.pyd')) and not entry.name.startswith(".#")

Of course there should be a more general way to do something like this.

Thanks for providing this package.

Samual,

I am learning python free style so I will apologize in advance because this is probably a complete misunderstanding on my part. I used your code below and it works great the only question I have is why this would return two lines. I only copied the superstore xls file into the directory once. Thanks!

import asyncio
from watchgod import awatch

async def main():
    async for changes in awatch('/Users/tgaluska/File'):
        print(changes)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

There are cases where bits of an application are under multiple directories. If watchgod's path param accepted a PATH_STRING, then it could support apps that are distributed between multiple disparate places in the file_system.

e.g.

$ watchgod mymodule.main "/foo/abc:/bar/cde"

Cases where this is relevant are:

1. where PYTHON_PATH is being used for module resolution
2. pip install -e . or python setup.py develop are being used to work on code in multiple unrelated dirs

The implementation could just split the path on : and then create watchers for each. If no seperator, exists it would behave equivalently with single path invocation.

Hi

• I tested the program written with the watchgod package on the win server 2012r2, and found that when monitoring the network mapped disk, when manually disconnecting the network mapped disk, watchgod will monitor the deletion change that does not actually exist, when reconnecting the network When the disk is mapped, it will generate additional changes to all files in the monitored directory. These false changes will cause unintended actions. Is this reasonable? Thank you!

The result is shown below:

Dear Samuel,

I am writing to you because of an issue I mentioned previously.
What I have been bumped into is that watchgod reports old files wich aren't modified during the "monitoring time".

It happens when the connection to a remote (network) folder is lost, but after that it is reconnected. Afterwards it has happened, it begins to report all the files being in that certain directory.

The os is win7.

How could I help? What information do you need?

What is your advice?

Best Regard,

P.S.: the network was wireless one.

I was hoping to use this library to build a development server that restarts itself whenever it detects a code change, in much the same way that Django or Flask do. Unfortunately, the following two problems prevent it from functioning as I would expect:

1. The first time I change a file, the server appears to restart as expected. Unfortunately, no code is reloaded, meaning my changes are not applied.
2. The second time I change a file, the change is not picked up on at all.

The test_awatch_log is flaky and fails on slow systems and/or systems under heavy load. I can reproduce it by running two games (Krunker and SuperTuxKart) while simultaneously running the test on my laptop. What happens is that the number of messages containing "DEBUG" goes below 4 and the test thus fails.

You might wonder if this really is a problem - after all, you don't usually run multiple games while testing your code. The problem is that while packaging watchgod for Alpine Linux I experienced this test randomly failing on their continuous integration (CI) on certain arches (armhf, aarch64, s390x), presumably as a result of me not being the only person who uses these CI runners and the systems thus being under heavy load.

I don't have a proposed way to fix this, and I understand if it's something you don't want to fix, but I thought I would report it nonetheless.

Just to let you know that watchfiles is now available on conda-forge. It can be installed using mamba or conda:

mamba install -c conda-forge watchfiles

Let me know if you want to be part of the package maintainers, and I'll add you to the list.

I need to monitor a string in a logfile and then react to it like insert some data to a database.
Can watchgod watch for a specific string or pattern on a specific log file rather than directory?

say I have a console script installed like so (setup.cfg):

[options.entry_points]
console_scripts =
    myservice=myservice._cli:main

rather than running watchgod as such:
watchgod myservice._cli.main --args run --host 0.0.0.0 --port 50051

I would like to run watchgod like:
watchgod myservice --args run --host 0.0.0.0 --port 50051

so that consistency can be maintained in respect to how its run in a docker entrypoint:

ENTRYPOINT ["myservice run --host 0.0.0.0 --port 50051"]

which would also have the added benefit of not having to expose implementation details

is this doable easily?

anyio provides an abstraction over the various async python frameworks: https://github.com/agronholm/anyio

Hi,

Thanks for another cool library.

I have a question though. I have a script like this

async def handle_change(file_path: str):
    await asyncio.sleep(3)


async def main():
    logging.info("Starting watchgod to observe '\\home'")
    async for changes in awatch("/home"):
        logger.info(f"Received {len(changes):,d} changes")
        for change in changes:
            logger.info(f"Handling {change}")
            if not change[0] == Change.deleted:
                await handle_change(change[1])


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

but it puzzles me that every change are handled one by one.

2020-03-30 12:33:52 - Starting watchgod to observe '\home'
2020-03-30 12:34:01 - Received 12 changes
2020-03-30 12:34:01 - Handling (<Change.modified: 2>, '/home/test/202003061000322536-en_9999999_0.xml')
2020-03-30 12:34:04 - Handling (<Change.modified: 2>, '/home/test/202003131000324250-en_9999999_0.xml')
2020-03-30 12:34:07 - Handling (<Change.modified: 2>, '/home/test/202002281000321102-en_9999999_0.xml')
2020-03-30 12:34:10 - Handling (<Change.modified: 2>, '/home/test/03107643.xml')
2020-03-30 12:34:13 - Handling (<Change.modified: 2>, '/home/test/03021521.xml')
2020-03-30 12:34:16 - Handling (<Change.modified: 2>, '/home/test/202003041000322054-en_9999999_0.xml')
2020-03-30 12:34:19 - Handling (<Change.modified: 2>, '/home/test/03043688.xml')
2020-03-30 12:34:22 - Handling (<Change.modified: 2>, '/home/test/02289691.xml')
2020-03-30 12:34:25 - Handling (<Change.modified: 2>, '/home/test/202003031000321883-en_9999999_0.xml')
2020-03-30 12:34:28 - Handling (<Change.modified: 2>, '/home/test/202002261000320605-en_9999999_0.xml')
2020-03-30 12:34:31 - Handling (<Change.modified: 2>, '/home/test/02267232.xml')
2020-03-30 12:34:34 - Handling (<Change.modified: 2>, '/home/test/202002271000320770-en_9999999_0.xml')

What am I missing here?

Hi,

I have just discovered your project.

I am trying to watch an elm source directory with watchgod and build js from elm whenever something change.

As soon as save the file it works perfectly but if I edit a file from emacs a backup file is created and en error is generate until I save the file:

error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/dev_js/dev_elm/contacts/src/.#ListContacts.elm'
...
error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/dev_js/dev_elm/contacts/src/.#ListContacts.elm'
error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/dev_js/dev_elm/contacts/src/.#ListContacts.elm'
{(<Change.added: 1>, '/dev_js/dev_elm/contacts/src/ListContacts.elm')}

This is my script, perhaps I do something wrong I am not used with asyncio.

import asyncio
from os import chdir
from time import perf_counter
from pathlib import PurePath

from watchgod import awatch


ELM_PATH = PurePath('/dev_js/dev_elm/contacts/')
JS_PATH = PurePath('/dev_python/contacts_proto/js')


async def build_from_src(src_file, dest_file):
    print("build", src_file)
    start = perf_counter()
    dst_path = JS_PATH.joinpath(dest_file)
    cmd = f"elm make src/{src_file} --output={dst_path}"
    await asyncio.create_subprocess_shell(cmd, stdout=asyncio.subprocess.DEVNULL)
    timing = (perf_counter() - start)
    print(f"{dest_file} built from {src_file} in {timing:.3}s")


FILES = (
    ("NewContact.elm", "new_contact.js"),
    ("ListContacts.elm", "list_contacts.js"),
    ("Main.elm", "main.js"),
)


async def main_loop():
    print("start watcher")
    async for changes in awatch(f"{ELM_PATH}/src"):
        print(changes)
        asyncio.gather(
            *(build_from_src(src, dest) for (src, dest) in FILES))

chdir(str(ELM_PATH))
loop = asyncio.get_event_loop()
loop.run_until_complete(main_loop())
loop.close()

I am looking into the code where is this error and how to deal with this.

I have this error:

RuntimeError: 
            Attempt to start a new process before the current process
            has finished its bootstrapping phase.
            This probably means that you are on Windows and you have
            forgotten to use the proper idiom in the main module:
                if __name__ == '__main__':
                    freeze_support()
                    ...
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce a Windows executable.

I am running a subprocess for every file watched. But if i change the status of files continously in a 1 second delay, only 2 of the batches are getting processed, while i am doing 3 batch changes.

Code

import random
import asyncio
import pathlib
from watchgod import awatch

count = 0
async def exec_(path):
    global count
    seconds = random.randrange(1, 10)
    cmd = f'sleep {seconds};echo {seconds}'
    print(f'[ executing process with delay {seconds} seconds ]')
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await proc.communicate()
    print(f'[ executed process with delay {seconds} seconds ] ')
    count += 1
    print(f'Number of files processed: {count}')
    return stdout, stderr

async def process(status, path):
    if status.value == 1:
        print(f'{path} has been created')
        path = pathlib.Path(path).resolve()
        #await exec_(path)
    elif status.value == 2:
        print(f'{path} has been modified')
    await exec_(path)

async def main():
    async for changes in awatch('foo/'):
        print(changes, len(changes))
        tasks = []
        for change in changes:
            t = asyncio.Task(process(*change))
            tasks.append(t)
        await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

To Reproduce:

 $ python <this-script>.py <dir>
 $ for i in 1 2 3; do $ touch <dir>/{1..10}; sleep 1; done

Expected

...
...
Number of files processed: 30

Current output

...
...
[ executed process with delay 9 seconds ] 
Number of files processed: 20

Please let me know what i am doing wrong. Appreciate the work.
Thank you

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create separate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Update: see #102 which supercedes this issue.

See this thread for a discussion of the arguments, merits & drawbacks of both renaming packages, and the name "watchgod".

TL;DR - I'm gong to rename this package to watchman (update: something else) to avoid confusion with the "watchdog" package.

I wanted a new name that was fairly close to the current name (still had the "watch" idea) but wasn't as easily confused. I emailed @alephmelo about a package he had released a few years ago (but was no longer actively maintaining) under the name "watchman", he kindly agreed to remove his package so I could use the name. Big thanks to @alephmelo for being so responsive and generous. I've uploaded a holding package to secure that name.

The process for switching will be roughly as follows (see the twitter thread above for a discussion of how to go about this):

1. Make this announcement, respond to any feedback, wait a while - there's no big rush
2. Rename the github repo, include a notice in the readme about the "old" name
3. Rename the codebase, effectively *.replace('god', 'man')
4. release the new codebase to pypi under the name watchman
5. release a final version of watchgod with only the following changes:

• prominent notice in the readme about the new name
• deprecation warnings on the main public functions explaining about the renaming
• if this gets approved before the release, add a pip announcement

I want to make some significant improvements to this package over the coming months, in particular #25. But I don't think it's a good idea to make any big, potentially breaking changes during the migration to a new name - we should try to keep the process of switching over as simple as possible - just changing the dependency and renaming methods/imports.

Hi, I just tried a simple test (from the pypy example: https://pypi.org/project/watchdog/). and on osx it kinda bahaves strange. so I run it on a non existing file:
python3 test_watchdog.py test
then I update that file by moving another file over it:
touch test2 && mv test2 test
and I want to be able to do that multiple times. thing is on the first chagne I get this event:

2021-04-07 09:23:24 - queue_event <DirDeletedEvent: event_type=deleted, src_path='/Users/daniel/Documents/git/game-on-server/python/tests/test', is_directory=True>
2021-04-07 09:23:24 - Deleted directory: /Users/daniel/Documents/git/game-on-server/python/tests/test

whis is wrong, and then after subsequent updates I get no events at all.
can watchdog only watch directories?

Hi, @samuelcolvin @pyup-bot

I want print(filename) if a file added in directory using WatchGod.

Like this -

if Change.status == added :
print(Changes.filename)

How can I coding this ?

Thanks in advance ~

Best,
@bemoregt.

If I try and run the watchgod CLI with my app in a docker-compose environment, it will fail due to the lack of /dev/tty being present:

web_1          | [02:48:45] watching "/app/" and reloading "app.main" on changes...
web_1          | Process Process-1:
web_1          | Traceback (most recent call last):
web_1          |   File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
web_1          |     self.run()
web_1          |   File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
web_1          |     self._target(*self._args, **self._kwargs)
web_1          |   File "/usr/local/lib/python3.7/site-packages/watchgod/cli.py", line 45, in run_function
web_1          |     with set_tty(tty_path):
web_1          |   File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
web_1          |     return next(self.gen)
web_1          |   File "/usr/local/lib/python3.7/site-packages/watchgod/cli.py", line 36, in set_tty
web_1          |     with open(tty_path) as tty:
web_1          | OSError: [Errno 6] No such device or address: '/dev/tty'

By default docker-compose up doesn't configure a TTY (but docker-compose run does) and while this can be configured does watchgod really need to require a TTY?

For example:

import argparse
import time

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--foo')
    parser.add_argument('--bar', '-b', action='store_true')

    args = parser.parse_args()

    while True:
        print(args)
        time.sleep(5)

watchgod foo.main --foo "bar" -b

Right now this will not work because sys.args has too many parameters. But in theory we can patch sys.args by removing parsed watchgod's args.

I struggled a bit with different Watchers to detect the creation of directories, but failed to get it to work.

I understood eventually that I had to also add a file to the newly created directory before an event was emitted. Is this intentional (if so, it might be relevant for the documentation)?

https://github.com/passcod/notify

^ Yup, another team's work in a heavy lifting language.

Ruby, Java (etc) folks should also depend on notify then we get decent effort to performance tune and bugfix once and bind to more elegant HLLs like Python multiple times. The alternative is to code it yourself and end up with Watchdog's 137 open issues and 20 unconsumed pull requests.

I just try to run example form docs.

1. create foobar.py file with aiohttp code

2. run watchgod

$ watchgod foobar.main
ImportError: No module named 'foobar'

https://github.com/samuelcolvin/watchgod#cli

Could we have a way to reload when it encounters an error?

The problem is, I am doing development in a container, if I save something with an error it will crash out and I have to restart the container.

Ideally it would be nice to reload and even have an option on number second to wait for the reload on error.

Hi,

It would be nice to have pre-build binaries for ARM Linux on PyPI.

The thing is that by default docker for mac on Apple Silicon macs runs containers and image builds for linux/arm64 platform and to install the package one needs to either install a rust compilator in container/image or run containers/builds under linux/amd64 emulation, which is significantly slower than the default platform.

This is most noticeable when running the watchgod CLI command as part of a docker-compose setup to run an aiohttp app. Even if you set stop_signal: SIGINT on your container, the container will fail to shut-down properly and result in sending a SIGKILL to halt your App. You can work around this by using tini or some other init system to send the signal to the process group.

The "KeyboardInterrupt" gets logged but the process doesn't terminate until the next file system event happens and RustNotify.watch() returns.

This is because windows doesn't have signals, so open_signal_receiver

Doesn't work and there's no way to tell the rust code to terminate.

Potential solution is to return from RustNotify.watch() regularly, and thereby check if an error has been raised. Seems a bit ugly but I can't think of a better solution.

Let's see if people actually suffer from this?

Context

I have an eventstream on Vue to feed with SSE information from a small XML file whenever it changes. My backend is a Flask app, although I'm upgrading it to FastAPI.

I have a watchdog solution that uses its standard observer.start(), which is a thread. This allows me to run concurrently. I'm trying to replace watchdog with watchgod.

The watchgod solution is async coroutines, which seem to be using ThreadPoolExecutor. Here's what I'm trying to do:

handler()

This is the function that is supposed to be looking nfor changes:

async def handler():
    file = os.path.join(path, xml_file)
    async for changes in awatch(path, watcher_cls=RegExpWatcher, watcher_kwargs={'re_files': re_files}):
        change = changes.pop()[0]
        if change.name in ('added', 'modified'):
            patient_data = ET.parse(file)
            patient_dict = parse_patient(patient_data)
            patient_id = registration_controller.save_patient(**patient_dict)
            patient_data = fix_patient_id(patient_data, patient_id)
            patient_data.write(vx_xml_file, encoding='UTF-8', xml_declaration=True, short_empty_elements=False)
            patient_dict['birth_date'] = patient_dict.pop('birthday')
            patient_queue.append(patient_dict)

Endpoint

This is the function that is holding the SSE eventstream:

app.route("/auto_register")
def emr_xml():
    def eventStream():
        while True:
            time.sleep(1)
            if patient_queue != []:
                patient_dict = patient_queue.pop()
                yield f"data: {json.dumps(patient_dict)}\n\n"
            else:
                yield 'retry: 1000\n\n'

    resp = Response(eventStream(), mimetype="text/event-stream")
    resp.headers["Access-Control-Allow-Origin"] = "*"
    resp.headers["Cache-Control"] = "no-cache"
    return resp

As you can see, I need to regularly "push forward" the connection by feeding it the retry bit, so I can't use either async for changes in awatch or for changes in watch, otherwise it will block until something changes in the file and I might timeout the SSE connection. In order to decouple this, I have the handler feeding a queue (which is just a list) and the endpoint checking it asynchronously every second.

Kick-off

My issue is with starting the watcher. All the examples mention either doing it synch or starting the async version with:

loop = asyncio.get_event_loop()
loop.run_until_complete(handler())

I also tried:

asyncio.run(handler())

The effect is always the same: it will block my endpoint and, in fact, the whole application.

What I can't do

I don't think I can go on and async thw whole thing. Flask won't work with that. I thought of having the eventStream function doing the await asyncio.gather(handler(), reset()) where reset would just await asyncio.sleep(1) 1 second and return the retry string, but this also did not work.

What I can do

Please tell me.

This issue replaces #25 and #96 as my plan has change significantly.

I've wanted to rename watchgod for a while, I've also wanted to migrate from file scanning/polling to a rust based file system notification backend. Instead of doing that as two separate (disruptive) steps, I'm going to do them together.

TL;DR; watchgod is being renamed to watchfiles and rewritten in rust. watchgod will remain available in pypi (with a message about migrating).

• watchgod will remain unchanged with the file scanning python code
• watchfiles will use the new rust backend from it's first release

See this thread for a discussion of the arguments, merits & drawbacks of both renaming packages, and the name "watchgod".

I wanted a new name that was fairly close to the current name (still had the "watch" idea) but wasn't as easily confused. I originally thought about "watchman", but there's already a package by that name maintained by facebook. @thornycrackers was kind enough to release the name watchfiles in pypi so I could use it for this package. Big thanks to @thornycrackers for being so responsive and generous. I've uploaded a holding package to secure that name.

The process for switching will be roughly as follows (see the twitter thread above for a discussion of how to go about this):

1. Rename in the codebase, effectively *.replace('god', 'files'), include a notice in the readme about the "old" name
2. Rename the github repo
3. release the new codebase to pypi under the name watchfiles, version will be v0.10 to avoid confusion with watchgod versions
4. release a version of watchgod with a prominent notice in the readme about the new name and migration, add the Development Status :: 7 - Inactive pypi classifier
5. Perhaps in future release a final version of watchgod with:

• deprecation warnings on the main public functions explaining about the renaming
• if this gets approved before the release, add a pip announcement

I run my project inside a docker container with a volume binding to the source dir of the project. I have a virtualenv (created with direnv) inside the project's .direnv. Some links in the venv point to places of the host system, not the container system, so the links are broken in the container.

watchgod spams errors like this:

django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
celeryworker    | [08:28:36] error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/lib/python3.7/struct.py'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
celeryworker    | [08:28:36] error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/lib/python3.7/struct.py'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
celeryworker    | [08:28:37] error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/lib/python3.7/struct.py'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
celeryworker    | [08:28:37] error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/lib/python3.7/struct.py'
django          | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/include/python3.7m'
celeryworker    | [08:28:37] error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/.direnv/python-3.7.3/lib/python3.7/struct.py'

Not only don't I need to follow the .direnv dir, and I believe it's futile to spam an error message.
But I also think watchgod should NEVER follow symlinks, because there are good chances they point to stuff outside the scope of what you'd like to watch.

#31

Should accept either argument and not require both.

A problem with watchgod was discovered when a business program was running. When a batch file to delete files was started in the background, the information monitored by watchgod seemed to have some problems, including some Change.added actions that did not actually exist. What is going on? Thank you.

The batch file is as follows:
@echo off
forfiles -p "E:\GDS\TEST" -s -m . -d -1 -c "cmd /c del @path"
for /f "delims=" %%a in ('dir /ad /b /s E:\GDS\TEST^|sort /r') do (
rd "%%a">nul 2>nul &&echo emptydir"%%a"deleted!
)
exit

The monitoring information is as follows:

2020-06-04 09:06:03,350 - main.Watcher - INFO - E:\GDS\TEST monitoring... - [watcher.py:92]
2020-06-04 09:06:03,379 - main.Watcher - INFO - E:\CODES\Python\transfer\test1 monitoring... - [watcher.py:92]
2020-06-04 09:06:15,900 - main.Watcher - INFO - Received E:\GDS\TEST 4859 changes - [watcher.py:58]
2020-06-04 09:06:15,902 - main.Watcher - INFO - Received 52 Change.added changes - [watcher.py:61]
2020-06-04 09:06:15,903 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:15,933 - main.Watcher - INFO - Received 15 Change.added changes - [watcher.py:61]
2020-06-04 09:06:15,961 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:15,963 - main.Watcher - INFO - Received 121 Change.added changes - [watcher.py:61]
2020-06-04 09:06:15,964 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:15,966 - main.Watcher - INFO - Received 10 Change.added changes - [watcher.py:61]
2020-06-04 09:06:15,967 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:15,969 - main.Watcher - INFO - Received 2 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,025 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,052 - main.Watcher - INFO - Received 60 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,054 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,055 - main.Watcher - INFO - Received 19 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,056 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,057 - main.Watcher - INFO - Received 47 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,058 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,059 - main.Watcher - INFO - Received 22 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,060 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,061 - main.Watcher - INFO - Received 2 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,062 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,064 - main.Watcher - INFO - Received 11 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,065 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,066 - main.Watcher - INFO - Received 7 Change.added changes - [watcher.py:61]
2020-06-04 09:06:16,067 - main.Watcher - INFO - Received 1 Change.deleted changes - [watcher.py:61]
2020-06-04 09:06:16,105 - main.Watcher - INFO - Received 8 Change.added changes - [watcher.py:61]

How to monitor 2 or more directories? I use a multi-process way to call a single directory monitoring, I feel this implementation is not very elegant, is there a more elegant way to achieve it? Has anyone achieved it? Thank you! The following is part of my multi-process code:

    ...
    def loop_watcher(self, directory):
        self.loop = asyncio.get_event_loop()
        print("add_signal_handler.")
        signal.signal(signal.SIGINT, self.raise_graceful_exit)
        signal.signal(signal.SIGTERM, self.raise_graceful_exit)
        print(" Ctrl+C 停止运行.")
        print(f"pid {os.getpid()}: 发送 SIGINT 和 SIGTERM 信号退出循环")
        try:
            self.loop.run_until_complete(self.watching(directory))
        except GracefulExit:
            pass
        finally:
            if self.loop is not None:
                self.loop.close()  

    def main(self):
        pool = Pool(processes=len(self.paths), initializer=self.init_worker)
        for directory in self.paths:
            if os.path.isdir(directory):
                pool.apply_async(self.loop_watcher, (directory, ))
        try:
            print("Waiting forever.")
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            print("Caught KeyboardInterrupt, terminating workers")
            pool.terminate()
            pool.join()
            sys.exit(1)
        else:
            print("Quitting normally")
            pool.close()
            pool.join()
   ...

Do you plan to add (emulation of) file system events and event handlers?

It seems like changes to the file content aren't propagated by watchfiles, however, they seem to be noticed by Notify.

Running watch with debug=True showed the following output when changing the content of a file in the watched directory:

raw-event: Event { kind: Modify(Any), paths: ["..././watched\\test"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None }
raw-event: Event { kind: Modify(Any), paths: ["..././watched\\test"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None }

Note that I just made slight changes to the paths part of the raw event to strip out the full path on my machine.

I also tried adding files, deleting files, and renaming files. These were propagated correctly.

After upgrading to 0.8 I got the following error

[15:25:04] watching "/app" and reloading "celery.__main__.main" on changes...
'import warnings' failed; traceback:
KeyboardInterrupt
/venv/lib/python3.10/site-packages/kombu/utils/compat.py:82: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for ep in importlib_metadata.entry_points().get(namespace, [])
[15:25:09] process has not terminated, sending SIGKILL
Traceback (most recent call last):
  File "/venv/bin/watchgod", line 8, in <module>
    sys.exit(cli())
  File "/venv/lib/python3.10/site-packages/watchgod/cli.py", line 139, in cli
    run_process(
  File "/venv/lib/python3.10/site-packages/watchgod/main.py", line 188, in run_process
    for changes in watch(
  File "/venv/lib/python3.10/site-packages/watchgod/main.py", line 43, in watch
    yield anyio.run(_awatch.__anext__)
  File "/venv/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run
    return asynclib.run(func, *args, **backend_options)
  File "/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
    return await func(*args)
  File "/venv/lib/python3.10/site-packages/watchgod/main.py", line 99, in __anext__
    watcher = self._w = await self.run_in_executor(
  File "/venv/lib/python3.10/site-packages/watchgod/main.py", line 143, in run_in_executor
    return await anyio.to_thread.run_sync(func, *args, limiter=self._thread_limiter)
  File "/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 754, in run
    result = context.run(func, *args)
TypeError: PythonWatcher.__init__() got an unexpected keyword argument 'ignore_paths'
someservice exited with code 1

To me it seems this is due to misspell on line:
https://github.com/samuelcolvin/watchgod/blob/63a49658f8ded07c7e79278d20a71398e7a762e2/watchgod/cli.py#L133

Specifically, if you look at the diff v0.7...v0.8#diff-716db2f16f170e5b4160957ed276c8768ce804fb889f26c2c3fd48b08a551062R133 you see the watcher kwargs previously was ignored_paths (note the d in ignored)

Could you verify if this is a bug, or if the CLI for the --ignore-paths option has changed?

This is my code
Code File location: /home/kamrul/cc_v13/apps/file_transfer/commands/job_watcher.py
Code: async for changes in awatch('/home/kamrul/cc_v13/sites/zenith.loca/private/files/Job'):
But shwiing error this.

Error:
error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/kamrul/cc_v13/sites/zenith.loca/private/files/Job/etc/systemd/system/timers.target.wants/logrotate.timer'

where they got this part of path. /etc/systemd/system/timers.target.wants/logrotate.timer ?

It's can be file location and watch location different issue?

There was a bug reported in Debian bug tracker which I fixed with this patch against version 0.8.2. Please note that I did not found a solution for version 0.12. where the issue can be reproduced as well.
Kind regards, Andreas.

In

for changes in watch('./path/to/dir'):
print(changes)

the watch() function should accept string as well as pathlib.Path().
It's not a big deal, but it would be more pythonic.

To me it's not clear how awatch exactly works. Let's say I'd like to watch a single file with a polling period of 200ms. How do I have to call awatch?

Hi!

Thanks for an alternative for watchdog! What is the proper way to make it work with things like celery?

Normally celery starts this way: celery -A config.celery_app worker -l INFO
so my guess was: watchgod celery --args -A config.celery_app worker -l INFO or watchgod /opt/venv/bin/celery --args -A config.celery_app worker -l INFO since celery itself is just a python file:

#!/opt/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys

from celery.__main__ import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

But it fails with:

"/opt/venv/bin/celery" doesn't look like a module path

The example in the readme indicated there is a process_args lkeyword argument, but when I try the example I get the following error:
TypeError: run_process() got an unexpected keyword argument 'process_args'

Did I miss anything?

Hi!
I encountered the following with watchfiles 0.12

Error:
Upon deletion changes, watchfiles is reporting addition changes.

Code used:

async def main():
    async for changes in awatch("some dir"):
        for change_type, change_file in changes:
            print(change_type, change_file)
asyncio.run(main())

Steps to reproduce:

• Do a batch delete (multiselect files & then delete)
  
• The following changes are being reported

Change.added path-to-img1.bmp
Change.added path-to-img3.bmp
Change.deleted path-to-img0.bmp
Change.deleted path-to-img2.bmp
Change.deleted path-to-img4.bmp

Dear Samuel Colvin,

I am writing to you to ask for your kind help regarding your awesome watchgod.

What I want is to use the „watch” with a "RegExpWatcher" watcher.

for changes in watch('./path/to/dir'):
    print(changes)

The problem that I have not managed to use the RegExpWatcher.

I tried the watcher_cls=watchgod.RegExpWatcher argument, but it did not help.

Another thing, yesterday when I was monitoring a directory with watchgod, I noticed that it suddenly began to report all the files being in that certain directory. But those files were not modificated, created, copied, etc during the monitoring time. It was strange, so I decided, that I set up another filter concerning the reported file modification datetime.

Here you can see the snippet.

def generate_filenames(path):

    for changes in watchgod.watch(path=path):
        for status, location in changes:
            if (
                status == watchgod.Change.added
                and len(pathlib.Path(location).stem) == 14
                and "WD" in pathlib.Path(location).stem
                and datetime.datetime.now()
                - datetime.datetime.fromtimestamp(
                    pathlib.Path(location).stat().st_mtime
                )
                >= datetime.timedelta(seconds=3600)
            ):
                yield location

Would you be so kind as to give me the right direction.

I really appreciate your kind help.

Best Regards,

For some reason we are getting an attribute error in Python 3.10:

  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/watchgod/main.py", line 145, in __del__
    self._executor.shutdown()
AttributeError: 'awatch' object has no attribute '_executor'

This seems a little bizarre to me, I can't see any reason the attribute would no longer exist.

See: aio-libs/aiohttp-devtools#376

Hi,

I was just testing Watchgod as an alternative to Watchdog when I ran into this bug which I'm not sure if it's intended behaviour, but I find important to report.

The program returns an error whenever it comes across broken links:
error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/tmp/.com.google.Chrome.NdlWPj/SingletonCookie'

Apparently, it gets stuck and stops processing newer changes. Also the CPU goes sky high when it's happening.

Steps to reproduce:

1. Create test file
   $ touch /tmp/test
2. Create symbolic link to test file on desired folder:
   $ ln -s /tmp/test /opt/testfile
3. Remove original file
   $ rm /tmp/test
4. Run sample code on /opt

PyPi .tar.gz release should contain a license file to support adding it as a Yocto .bb file.