falood / file_system Goto Github PK

Maybe I'm missing something, but I can only see an example of writing a module that has the dirs option set at compile time. Is it possible to start a new watcher with a custom path at runtime? I would need to start and stop watchers when needed.

I'm currently using fs but looking into other more Elixir friendly options.

I am interested in using the library inside an escript. When I call {:ok, watcher_pid} = FileSystem.start_link(args) from iex evrithing is ok. But when I run the same code inside an escript it returns :ignore

Is there a way to deal with it?

Thanks.

Hi @falood,

I am trying to use this package from another package (exsync) and keep getting this error message in my console:

16:54:02.170 [error] Can't find executable `mac_listener`

I am running macOS Mojave 10.14.4 and have tried installing mac_listener by running the pkg file as per the README:

On Macos 10.14, you need run `open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg` to compile `mac_listener`.

Installation finished successfully and I even restarted my machine, but I still get the error.

I suspect something is off with my PATH setting. So maybe it would help if you could just reply with the output of this command: which mac_listener.

Thank you!

Everytime I try to run my monitor (the one from the README) I immediately get the following error:

Monitor.start
** (exit) exited in: GenServer.call(ExFSWatch.Supervisor, {:start_child, [Excellent.Monitor]}, :infinity)
    ** (EXIT) no process
    (elixir) lib/gen_server.ex:356: GenServer.call/3

Unable to start exsync application with 0.2.0 exfswatch tag, though 0.1.1 works ok.
The following error arises:

** (Mix) Could not start application exfswatch: exited in: ExFSWatch.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (UndefinedFunctionError) function :inotifywait.find_executable/0 is undefined (module :inotifywait is not available)
            :inotifywait.find_executable()
            (exfswatch) lib/exfswatch.ex:19: ExFSWatch.start/2
            (kernel) application_master.erl:273: :application_master.start_it_old/4

Arch linux, Elixir 1.3.2, erlang 0.18.3.

~which inotifywait
/usr/bin/inotifywait

The backends have this code:

  def handle_info({:EXIT, port, _reason}, %{port: port}=state) do
    send(state.worker_pid, {:backend_file_event, self(), :stop})
    {:noreply, state}
  end

Shouldn't the return be {:stop, :normal, state}? Otherwise the backend will continue running even if they have no port? Or is this intentional?

Some applications can change a file with many :modified/[:modified, :closed]
But it's a good idea to process events only once (for example, at the end of file changes).

It was a bit better in Erlang inotify: I can collect only close_write event.

Hi everyone and thanks for the great work!
Sorry if this can seem a silly question - and I hope it is - but I'm struggling to understand how to detect file renames (using inotify backend atm).
Example:

• touching a file triggers three events: [:created], [:attribute], [:modified, :closed] referring to the same path
• vim-ming a file produces a couple [:modified] more, but ends always with a [:modified, :closed]
• moving a file triggers instead two events: [:deleted], [:created] the first referring to the original path, the latter referring to the renamed file

I don't have here a clue on how to understand when a single [:created] event refers to a file rename, which is a use case I need to address. Am I missing something? Is there any reason I'm not seeing to collapse the MOVED_FROM and MOVED_TO events to :deleted and :created ?

Edit: Just as a note, I took a look on the original fs project. They allow the renamed attribute, but they also don't support the MOVED_FROM flag. I may be missing the reason for you not to support the renamed attribute, in that case I say sorry, but it may be enough to correctly handle renames (even if not knowing from which file).

Re-edit: I see that the desired behavior has been removed in f35b1e8806399c882306d0c9155bead25a89ef7b but, again, I don't get the ratio behind this choice

Thanks!

如果把erlang的虚拟机直接halt掉，inotifywait进程不会销毁。exfswatch没有这个问题。exfswatch用sh的read来kill 进程，而file_system中直接创建了inotifywait，如果erlang的vm被kill掉，这个inotifywait的进程就没办法kill了。导致inotifywait的进程数过多问题！

Using file_system 0.2.10 on Windows Server 2012 R2. Elixir 1.12.0-rc.1. Erland/OTP 21.

File_System has been working fine, but it started having issues recently since I upgraded Elixir and many other dependencies(including file_system) a week or so ago.

After my elixir app starts, file_system successfully detects file changes to the specified folder, but after a while it seems to stop detecting new changes. Is there a way to debug this issue? In :observer, FileSystem.Worker still shows up in the Processes tab, when it stops detecting.

In case it helps, here is my code.

defmodule FinReporting.Monitor do

  use GenServer
  alias FinReporting.Repo
  alias FinReporting.AppState

  def start_link(args) do
    GenServer.start_link(__MODULE__, args)
  end

  def init(args) do
    {:ok, watcher_pid} = FileSystem.start_link(args)

    FileSystem.subscribe(watcher_pid)
    IO.inspect(watcher_pid, label: "Watcher PID")
    {:ok, %{watcher_pid: watcher_pid}}
  end

  def handle_info(
        {:file_event, watcher_pid, {path, events}},
        %{watcher_pid: watcher_pid} = state
      ) do
    # Logic for path and events

    callback(path, events)
    {:noreply, state}
  end

  def handle_info(
        {:file_event, watcher_pid, :stop},
        %{watcher_pid: watcher_pid} = state
      ) do

    callback(:stop)
    {:noreply, state}
  end

  def callback(:stop) do
    IO.puts("STOP")
  end

  def callback(file_path, events) do
   ...

exfswatch isn't based on fswatch but instead of fs. I think this can be confusing, would you consider a namechange?

This happened after updating for Mojave
Mohameds-MacBook-Pro:umm mohamed$ mix deps.compile file_system

01:31:26.566 [info]  Compiling file system watcher for Mac...
==> file_system
c_src/mac/cli.c:1:10: fatal error: 'getopt.h' file not found
#include <getopt.h>
         ^~~~~~~~~~
1 error generated.

01:31:26.606 [error] Could not compile file system watcher for Mac, try to run "clang -framework CoreFoundation -framework CoreServices -Wno-deprecated-declarations c_src/mac/*.c -o priv/mac_listener" manually inside the dependnecy.

In worker.ex, start_port() for inotifywait has this code:

args = [ listener_extra_args, '-c', 'inotifywait $0 $@ & PID=$!; read a; kill $PID',
             '-m', '-e', 'close_write', '-e', 'moved_to', '-e', 'create', '-e',
             'delete_self', '-e', 'delete', '-r' | path
]

listener_extra_args is fed to /bin/sh, instead of inotifywait. It should be placed after '-r'., like this:

args = [ '-c', 'inotifywait $0 $@ & PID=$!; read a; kill $PID',
         '-m', '-e', 'close_write', '-e', 'moved_to', '-e', 'create', '-e',      
         'delete_self', '-e', 'delete', '-r', listener_extra_args | path    
       ]

The current implementation will cause the following error on display:

/bin/sh: : No such file or directory

While going through the source code, I've noticed the required Logger was put outside the FileSystem.Backends.FSInotify module.

Just curious, is there a reason for doing so instead of putting it within the FileSystem.Backends.FSInotify itself?

require Logger                                                                                          
                                                                                                         
defmodule FileSystem.Backends.FSInotify do                                                              
@moduledoc """
...
"""

And not?

defmodule FileSystem.Backends.FSInotify do                                                              
@moduledoc """
...
"""

require Logger

I am trying to use this dependency to notify when a file is created in linux. When I ran the Exsfwatch.known_events, only [:renamed, :closed, :modified, :isdir, :undefined] gets returned. It doesn't match what the inofitywait event list. The one i care the most is :create and not there too. Am i missing something?

Thank you!

Ada

$ mix test.watch

[warn] Could not start the file system monitor.

$ mix --version
Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]

Mix 1.12.2 (compiled with Erlang/OTP 24)

I could fix it to run by manually adding some executable flags:

$ chmod +x  ./deps/mix_test_watch/priv/zombie_killer
$ chmod +x ./deps/file_system/priv/mac_listener

Unluckily I do not know enough on the internals of the mix deps implementation and how to fix this.

I am running Mac OSX Big Sur 11.5.2, and I think it was working before.

$ cat mix.lock
%{
  "file_system": {:hex, :file_system, "0.2.10", "fb082005a9cd1711c05b5248710f8826b02d7d1784e7c3451f9c1231d4fc162d", [:mix], [], "hexpm", "41195edbfb562a593726eda3b3e8b103a309b733ad25f3d642ba49696bf715dc"},
  "mix_test_watch": {:hex, :mix_test_watch, "1.0.3", "63d5b21e9278abf519f359e6d59aed704ed3c72ec38be6ab22306ae5dc9a2e06", [:mix], [{:file_system, "~> 0.2.1 or ~> 0.3", [hex: :file_system, repo: "hexpm", optional: false]}], "hexpm", "7352e91952d9748fb4f8aebe0a60357cdaf4bd6d6c42b5139c78fbcda6a0d7a2"},
}

I use exfswatch in a Phoenix 1.2 application. Everything works fine until I build a release (using distillery) and deploy to the server. The release fails to start, the error message is like below:

$ ./bin/my_app console

Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]


=INFO REPORT==== 20-Apr-2017::16:12:21 ===
    application: plug
    exited: {bad_return,
                {{'Elixir.Plug',start,[normal,[]]},
                 {'EXIT',
                     {noproc,
                         {gen_event,call,
                             ['Elixir.Logger','Elixir.Logger.Config',
                              {add_translator,
                                  {'Elixir.Plug.Adapters.Translator',
                                      translate}}]}}}}}
    type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,plug,{bad_return,{{'Elixir.Plug',start,[normal,[]]},{'EXIT',{noproc,{gen_event,call,['Elixir.Logger','Elixir.Logger.Config',{add_translator,{'Elixir.Plug.Adapters.Translator',translate}}]}}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,plug,{bad_return,{{'Elixir.Plug',start,[normal,[]]},{'EXIT',{noproc,{gen_event,call,['Elixir.Logger','Elixir.Logger.Config',{

Crash dump is being written to: erl_crash.dump...done

I reduced my code to only include exfswatch in application and deps in mix.exs. The error still happened. I guess it's because logger config but my Phoenix app already has them. Not sure what the real reason is.

Is there a reason why option -r is not optional? I'd like to use exfswatch without it.

Longterm goal: it would be nice to define a consistent API from all the backends.

https://docs.google.com/document/d/1-GQrFdDVrA57-ce0kbzSth4lQqfOMMRKpih3hPJmvoU/edit might be a good starting place for research.

/cc @whitfin

OS: macOS High Sierra 10.13.1
Elixir version: 1.6.2
Erlang/OTP version: 20

When there's a dir/filename with the = character in the name, it throws the following error:

[error] GenServer #PID<0.480.0> terminating
** (MatchError) no match of right hand side value: ["20888710", "0x00020100", "[created]", "/tmp/test", "2"]
    (file_system) lib/file_system/backends/fs_mac.ex:174: FileSystem.Backends.FSMac.parse_line/1
    (file_system) lib/file_system/backends/fs_mac.ex:153: FileSystem.Backends.FSMac.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {#Port<0.23416>, {:data, {:eol, '20888710\t0x00020100=[created]\t/Users/rhnonose/dev/emcasa/hammock/page=2'}}}
State: %{port: #Port<0.23416>, worker_pid: #PID<0.479.0>}

To reproduce: create a directory with = in the name while watching for changes.

Please follow the link

What do you think of?

I processe some directory with cyrillic name "!А. Дворжак - Квинтет для фортепиано, двух скрипок, альта и виолончели соч 81 [A 10-00067-001] [1984]".

file_system callback returns:

15:34:57.358 [debug] inotify: dir .../!ÐвоÑжак - винÑÐµÑ Ð´Ð»Ñ Ñ
                                                                                         оÑÑепиано, двÑÑ
 ÑкÑипок, алÑÑа и виолонÑели ÑÐ¾Ñ 81 [A 10-00067-001] [1984] created

After that my directory is not found with File.ls/1.

String.to_charlist/1 conversion is a bad suggestion, because I have error {:error, :enametoolong} with directory listing.

:file.native_name_encoding

:utf8

:io.printable_range

:latin1

shell locale command output:
locale
LANG=ru_RU.UTF-8
...
LC_ALL=ru_RU.UTF-8

Hi!

I have this implementation of the monitor:

defmodule MebeWeb.FSMonitor do
  use ExFSWatch, dirs: [Path.expand("#{__DIR__}/../../../web/static")]

  require Logger

  def callback(:stop) do
    Logger.debug("Stopped file watcher.")
  end

  def callback(file_path, events) do
    Logger.debug(inspect({file_path, events}))
  end
end

When inspecting the path, I can see that the expanded path is /Users/nicd/svn/mebe/web/static. I get correct debug logs from files edited in that path, but I also get notifications from files in /private randomly. Example:

[debug] {"/Users/nicd/svn/mebe/web/static/js/app.js", [:inodemetamod, :modified]}
[debug] {"/private/var/folders/h9/b8n6qz715cj7ws460yq15v240000gn/T/etilqs_34edb43e617ebf2d", [:created, :removed, :xattrmod]}

Don't know if it's a problem in this project or in fs.

Elixir 1.4, Erlang 19.2, macOS 10.12.2.

:fs needs to be in the applications list when deploying. It would be nice to have this included in the README (currently cannot offer a pull request).

  def application do
    [applications: [:exfswatch, :fs, :logger]]
  end

IIRC WTFPL is not really binding in some jurisdictions, which mean that this software can be considered proprietary in some countries. That may be a problem there. So my proposal is to dual license it as WTFPL and 0BSD/MIT or something like that to avoid that trap.

You could use Swift instead of C and release it as a binary with the project.

Is it possible to get a new release of file_system? It would be great to get #66 and #65 released.

fs has received a bug fix that resolves this issue: 5HT/fs#40

Is there a way to reliably and safely kill a FileSystem worker and its port?

I was surprised to see that it's possible to bind a variable and match against it in the same pattern:

def handle_info({:file_event, ^watcher_pid, {path, events}}, %{watcher_pid: watcher_pid}=state) do

But it doesn't really work. Copying the example to iex raises a compilation error:

** (CompileError) iex:14: unbound variable ^watcher_pid
    (stdlib) lists.erl:1354: :lists.mapfoldl/3
    (stdlib) lists.erl:1355: :lists.mapfoldl/3
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

Elixir: 1.4.5.

Hi,

I wish to chose a backend according to my needs. Right now its hardcoded in "backend_module(nil)" where case depends on ":os.type()".

It would be nice to get capability to set custom backend. I don't have inotify-tools and I don't want to install them, :fs_poll is working way better.

With that approach, I think that from Phoenix I would be able to alter backend.

Hello!

Could you please document why one might want to use (or migrate to) this library over fs?

Thanks,
Louis

Thanks for creating ExFSWatch. It works great.
I could make it work on its own in iex.

Can you provide some guideline on how to add it to an existing Elixir app?
I thought it would be as simple as adding a line supervisor(Monitor, []) to the children list in my MyProject.ex, but it gave me Monitor.start_link/0 is undefined error.

Another question is.. I noticed there are multiple callbacks issued for a single modification of a file in the folder that's being watched. (which looks like the following) What's the reason for this behavior?

iex >> {"/folder/file1.txt", [:created]}
iex >> {"/folder/file1.txt", [:modified]}
iex >> {"/folder/file1.txt", [:modified]}
iex >> {"/folder/file1.txt", [:modified]}

I have code like the following to watch two directories. I only care about getting a notification when the directory is updated. For my use-case, I don't care about receiving notifications for #individual files getting changed.

{:ok, watcher_pid} = FileSystem.start_link(dirs: [dir1, dir2], latency: 2, file_events: false)
FileSystem.subscribe(watcher_pid)

With this code, when I create multiple files inside one of my watched directories e.g. by running mix new test inside one watched directory, after 2s (the latency set above) I get lots of notifications in my handle_info (I'm using a GenServer, as specified in your Readme) for all the files that just got created. (Similarly when I do a rm -rf watched_dir/another_dir — I get lots of notifications too)

I would like to only get one notification that the directory has been updated. How do I do this with file_system?

I just posted issue 23 with fs when trying to monitor a file with inotify-tools-3.14-1.el6.x86_64. Its a backend issue with line_to_event. Cross posting here in case anyone else experiences it. I have a temporary fix on the linux_fix branch on my clone.