Need bundle markers (latency has no effect on the FSEvents monitor). about fswatch HOT 22 CLOSED

There's already a concept of latency but it has no effect on the FSEvents monitor. If you are using that monitor, you are out of luck: the event stream is scheduled in the run loop of the calling thread, which is the main thread, so that it's a blocking call that never returns. Just look at the CFRunLoopRun(); instruction at the end of fsevents_monitor::run.

Sometimes I thought about implementing a multi-threaded FSEvents monitor but I felt it comparatively adds more complexity than benefits.

Furthermore, fswatch is an observer: you can hack it to pause but the file system activity will not. If I wanted to buffer events and process them, I'd do that after I process the output of fswatch.

from fswatch.

I'm renaming and leaving this issue open for reference.

from fswatch.

I forgot an important thing: the FSEvents event stream is currently configured with the user-specified latency. That's another reason why I'd rather keep fswatch simple in this respect.

from fswatch.

So here’s the use case. Various files are developed for clients and are dropped into the filesystem structure. If I create 12 PNG files for them I want to send them a single message that lists the 12 files. I don’t want to send them 12 messages.

So while the filesystem event is sent, I want fswatch to only send me those notifications about the file creations every 2 minutes, for example. That way they’ve had time to drop everything there and then I get a list of those 12 files and I process them as a single “thing”. By wrapping it like this:

— Start of batch
file 1
file 2
file 3
— End of batch

I now see that end of batch marker and then send out my updates as appropriate. If I just get it line by line, I don’t know where a batch starts and ends. Does that make sense?

On Aug 28, 2014, at 12:21 PM, Enrico Maria Crisostomo [email protected] wrote:

There's already a concept of latency but it has no effect on the FSEvents monitor. If you are using that monitor, you are out of luck: the event stream is scheduled in the run loop of the calling thread, which is the main thread, so that it's a blocking call that never returns. Just look at the CFRunLoopRun(); instruction at the end of fsevents_monitor::run.

Sometimes I thought about implementing a multi-threaded FSEvents monitor but I felt it comparatively adds more complexity than benefits.

Furthermore, fswatch is an observer: you can hack it to pause but the file system activity will not. If I wanted to buffer events and process them, I'd do that after I process the output of fswatch.

—
Reply to this email directly or view it on GitHub.

from fswatch.

Hi @grosch,

Your use case perfectly makes sense to me, but I think it's not fswatch's job to do it. Here's my reasoning.

First of all, you're relying on "lucky" timing between the I/O activity and fswatch. What happens then if fswatch catches events right in the middle and returns you two batches (the worst thing being: the second batch is delivered two minutes late)?

Secondly, if you want to wait on fswatch output, buffering input and timing out every now and then (2 minutes, for example), you can do it when reading from fswatch output.

Rather, I'd try to solve the problem at the source and to me it seems there's a lack of synchronisation. Let's say an external batch job creates x files in a client-specific directory. Why don't you "signal" the job end with a special file, with a lock file, or whatever? You'll get (per client directory) a stream of events you'd buffer and if and only if you receive the event signaling the end (e.g.: the myjob.lockfile is removed) then you proceed sending the message with the buffered content.

from fswatch.

It's people dropping files into place not a script.

But I can work around it by watching all your output and keeping a timer to make sure I've had no other output for two minutes for example.

My only block right now is the clean way to know what is file name and what are flags.

On Aug 28, 2014, at 12:41 PM, Enrico Maria Crisostomo [email protected] wrote:

Hi @grosch,

Your use case perfectly makes sense to me, but I think it's not fswatch's job to do it. Here's my reasoning.

First of all, you're relying on "lucky" timing between the I/O activity and fswatch. What happens then if fswatch catches events right in the middle and returns you two batches (the worst thing being: the second batch is delivered two minutes late).

Secondly, if you want to wait on fswatch output, buffering input and timing out every now and then (2 minutes, for example), you can do it when reading from fswatch output.

Rather, I'd try to solve the problem at the source and to me it seems there's a lack of synchronisation. Let's say an external batch job creates x files in a client-specific directory. Why don't you "signal" the job end with a special file, with a lock file, or whatever? You'll get (per client directory) a stream of events you'd buffer and if and only if you receive the event signaling the end (e.g.: the myjob.lockfile is removed) then you proceed sending the message with the buffered content?

—
Reply to this email directly or view it on GitHub.

from fswatch.

@grosch, then I think your latency problem would be solved by the -l option, depending on the monitor you are using. Which OS are you running this fswatch instance on? Did you try invoking fswatch like this:

$ fswatch -l 120 [path]

from fswatch.

Yes. But since it runs continuously I have to know when that bundle of output came in. That's what the markers were for

On Aug 28, 2014, at 12:54 PM, Enrico Maria Crisostomo [email protected] wrote:

@grosch, then I think your latency problem would be solved by the -l option, depending on the monitor you are using. Which OS are you running this fswatch instance on? Did you try invoking fswatch like this:

$ fswatch -l 120 [path]
—
Reply to this email directly or view it on GitHub.

from fswatch.

True. Currently you could use -l with -1 so that fswatch exits. You know it finished and you restart it immediately. In this case you have a race condition: you could lose events between the time fswatch returns and the time it starts again.

If you did not need the path of the changed file you could also you -o but I guess it's not feasible given your use case.

from fswatch.

As per the flags, when I'm parsing the output of fswatch I'm not using the textual representation of the flags, but the numeric mask. This way the record format can be tokenized as (assuming you are using -0 which you should):

path space number\0

You split the records using NUL, then extract the number: everything else is the path.

from fswatch.

I’m not seeing the option to print the numeric masks.

On Aug 28, 2014, at 1:12 PM, Enrico Maria Crisostomo [email protected] wrote:

As per the flags, when I'm parsing the output of fswatch I'm not using the textual representation of the flags, but the numeric mask. This way the record format can be tokenized as (assuming you are using -0 which you should):

path space number\0
You split the records using NUL, then extract the number: everything else is the path.

—
Reply to this email directly or view it on GitHub.

from fswatch.

Can you see -n?

 -n, --numeric         Print a numeric event mask.

from fswatch.

I'm blind :)

On Aug 28, 2014, at 3:11 PM, Enrico Maria Crisostomo [email protected] wrote:

Can you see -n?

-n, --numeric Print a numeric event mask.
—
Reply to this email directly or view it on GitHub.

from fswatch.

Don't worry, I experience temporary blindness sometimes, too. :)

from fswatch.

Hi @grosch,

After thinking about your use case I came to the conclusion that it makes sense to support it from fswatch. I've just released fswatch 1.4.3 which adds, amongst other improvements, support for your request. I renamed it batch marker instead of bundle marker.

You use it using the --batch-marker option. By default, fswatch outputs a record with just an event flag called NoOp. But if you pass an optional argument to --batch-marker you can customize the text of the marker.

I'm closing this issue.

from fswatch.

This doesn't seem to work like I was thinking it would. If I say "--batch-marker --latency=5" then it just spits out a batch marker every 5 seconds if something has changed. I was wanting to only spit things out if nothing had changed for 5 seconds. i.e. if they add a file every second for a minute, nothing happens. Then they don't do anything for 5 seconds, so I get that batch marker and the 60 files listed that had been changed.

from fswatch.

Hi @grosch,

Yeah, that's true, that use case introducing a separate thread in fswatch, which is something I refrained from until now. I'll investigate it.

from fswatch.

Hi Enrico,

just wanted to see if you’d had a chance to work on this? My client who originally started asking me for upgrades which required the changes to fswatch just officially asked me to implement them. Hoping I can just ‘use’ fswatch instead of wrapping an alarm type thing in there myself.

On Sep 29, 2014, at 10:21 AM, Enrico Maria Crisostomo [email protected] wrote:

Hi @grosch,

Yeah, that's true, that use case introducing a separate thread in fswatch, which is something I refrained from until now. I'll investigate it.

—
Reply to this email directly or view it on GitHub.

from fswatch.

Hi Enrico,

I’m going to have to implement this timeout myself at this point. Your man page talks about the numeric values via the -n flag, but doesn’t specify how I tell what those flags mean. Can you clarify please?

I want to know when any FILE has been created or modified. I don’t need to know anything else.

On Oct 10, 2014, at 8:55 AM, Scott Grosch [email protected] wrote:

Hi Enrico,

just wanted to see if you’d had a chance to work on this? My client who originally started asking me for upgrades which required the changes to fswatch just officially asked me to implement them. Hoping I can just ‘use’ fswatch instead of wrapping an alarm type thing in there myself.

On Sep 29, 2014, at 10:21 AM, Enrico Maria Crisostomo <[email protected] mailto:[email protected]> wrote:

Hi @grosch https://github.com/grosch,

Yeah, that's true, that use case introducing a separate thread in fswatch, which is something I refrained from until now. I'll investigate it.

—
Reply to this email directly or view it on GitHub #52 (comment).

from fswatch.

Hi @grosch,

Sorry for not answering before but I was having a thought at this.

My plan is not integrating this feature into fswatch, but add an external program implementing this feature and pipe fswatch output into it. I see the following benefits here:

• The functionality is trivial to implement in a standalone program: one thread reads the input and writes to a buffer, another threads wakes up and consumes the buffer.
• You do not have to integrate with fswatch: nowadays fswatch is a single-threaded program, but it may not be in the future. And besides threading, integrating with any program is usually more difficult that separating a concern and implementing it somewhere else: in this case it sounds natural to create another program which operates on standard input.
• You may reuse such program in other scenarios (thinking about tail -f output, for instance).

As far as numeric event flags are concerned, they are defined in c/cevent.h and documented in the manual, p. 15 and following. You can download the manual from the link to the documentation you can see in the repository main page (which points here at the moment).

from fswatch.

If I get on my Mac and run fswatch and do “mv from to” then I get events of “Renamed IsFile” for both from and to.

Why aren’t I getting a Removed event for ‘from’?

On Oct 29, 2014, at 1:44 AM, Enrico Maria Crisostomo [email protected] wrote:

Hi @grosch https://github.com/grosch,

Sorry for not answering before but I was having a thought at this.

My plan is not integrating this feature into fswatch, but add an external program implementing this feature and pipe fswatch output into it. I see the following benefits here:

The functionality is trivial to implement in a standalone program: one thread reads the input and writes to a buffer, another threads wakes up and consumes the buffer.
You do not have to integrate with fswatch: nowadays fswatch is a single-threaded program, but it may not be in the future. And besides threading, integrating with any program is usually more difficult that separating a concern and implementing it somewhere else: in this case it sounds natural to create another program which operates on standard input.
You may reuse such program in other scenarios (thinking about tail -f output, for instance).
As far as numeric event flags are concerned, they are defined in c/cevent.h and documented in the manual, p. 15 and following. You can download the manual from the link to the documentation you can see in the repository main page (which points here https://drive.google.com/folderview?id=0BxZtP9CHH-Q6bHF3bmJGRmlVcVU&usp=sharing at the moment).

—
Reply to this email directly or view it on GitHub #52 (comment).

from fswatch.

Hi @grosch,

Because that's the way the FSEvents API behaves. In fact, from the point of view of a file which is renamed inside the same file system there's no "removal", as there would be when a file is moved across file systems.

from fswatch.