solotimes / lsyncd Goto Github PK

Hi, 
I don't know, if my question is right here (sorry for bad english ;) )

I have Archlinux and install lsyncd-1.37.

I plan to bild a backup like windows shadowcopy. 
I find incron, but incron can't watch recusive. 

In find --binary in the lsyncd manpage. Start lsyncd only the binary, or can I 
give the binary parameters (varied filename /path) ?

If I can use parameters, then I wirte a bash script and use cp to copy the file 
like that: 

################################
time=$(date '+%H_%M')
cp -r "$1" "/path/to/the/backups/$(date '+%Y')/$(date '+%m')/$(date 
'+%d')/$time"
################################

$1 is the parameter of the file, witch is, varied. 

Thanks
GerhardSchr


PS.: If I must write these question somewhere else, please tell me this :)

OS : Redhat 5.3 x86_64
svn source lsyncd2 : r369

lua packages :
liblua.x86_64                        5.1.4-3.el5.pp         installed
liblua-devel.x86_64                  5.1.4-3.el5.pp         installed
lua.x86_64                           5.1.4-4.el5            installed
lua-devel.x86_64                     5.1.4-4.el5            installed
lua-static.x86_64                    5.1.4-4.el5            installed

coming from rpmbone

lua -v
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
luac -v
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio

1st : 
./autogen.sh OK
./configure KO
checking for LIBLUA... configure: error: Package requirements (lua5.1) were not 
met:

No package 'lua5.1' found

solved by :
ln -s  /usr/lib64/pkgconfig/lua.pc /usr/lib64/pkgconfig/lua5.1.pc
./configure ---> OK

2nd
make KO :

[... svn]# make
make  all-am
make[1]: Entering directory `/usr/src/svn'
objcopy --input-target=binary --output-target=`gcc -dumpmachine` 
--binary-architecture=`cat architecture` luac.out luac.o
objcopy: luac.o: Invalid bfd target
make[1]: *** [luac.o] Error 1
make[1]: Leaving directory `/usr/src/svn'
make: *** [all] Error 2
[... svn]#

with 
# gcc -dumpmachine
x86_64-redhat-linux
# objcopy --info
BFD header file version 2.17.50.0.6-9.el5 20061020
elf64-x86-64
 (header little endian, data little endian)
  i386
elf32-i386
 (header little endian, data little endian)
  i386
a.out-i386-linux
 (header little endian, data little endian)
  i386
efi-app-ia32
 (header little endian, data little endian)
  i386
elf64-little
 (header little endian, data little endian)
  i386
elf64-big
 (header big endian, data big endian)
  i386
elf32-little
 (header little endian, data little endian)
  i386
elf32-big
 (header big endian, data big endian)
  i386
srec
 (header endianness unknown, data endianness unknown)
  i386
symbolsrec
 (header endianness unknown, data endianness unknown)
  i386
tekhex
 (header endianness unknown, data endianness unknown)
  i386
binary
 (header endianness unknown, data endianness unknown)
  i386
ihex
 (header endianness unknown, data endianness unknown)
  i386

               elf64-x86-64 elf32-i386 a.out-i386-linux efi-app-ia32
          i386 elf64-x86-64 elf32-i386 a.out-i386-linux efi-app-ia32

               elf64-little elf64-big elf32-little elf32-big srec symbolsrec
          i386 elf64-little elf64-big elf32-little elf32-big srec symbolsrec

               tekhex binary ihex
          i386 tekhex binary ihex
[...svn]#

any help will be apreciated

[email protected]

In case the target directory is local or on a network share, it would be really 
useful to use all file moving magic without rsyncssh.

The first lua code that I ever read was in lsyncd.lua and I managed to create a 
config file that does the same thing that rsyncssh does without putting ssh in 
between.
I did this by duplicating all the  default_rsyncssh code plus the exit code 
lists into the config file and hacked around a bit.

But probably there is a much much smarter way to do this...

Tryin' to sync a directory with brackets in filename, there's an error managing 
the filename.

Thu Nov 25 16:32:19 2010: rsyncing /myhome/sync/ --> user@host:/space/sync/
Thu Nov 25 16:32:25 2010: rsyncing /myhome/sync/Music/ --> 
user@host:/space/sync/Music/
Thu Nov 25 16:35:29 2010: rsyncing /myhome/sync/Music/album (year)/ --> 
user@host:/space/sync/Music/album (year)/ 
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `rsync --server -ldte.Lsf --no-r --delete . 
/space/sync/Music/album (year)/'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) 
[sender=3.0.7]
Thu Nov 25 16:35:35 2010: Forked binary process returned non-zero return code: 
12
Thu Nov 25 16:35:35 2010: ERROR: Rsync from /myhome/sync/Music/album (year)/ to 
user@host:/space/sync/Music/album (year)/ failed

Debian package currently on the testing branch, Version: 1.34-1

Imagine a setup of workstation A, lsyncd server B, and lsyncd target C.

What steps will reproduce the problem?
1.  On workstation A, `rsync -avz some-dir B:/path/lsyncd/watches/`
2.  rsync creates a bunch of hidden temporary files on B, which triggers 
inotify.
3.  Crazyness ensues, ending with:
xargs: illegal option -- 0
xargs: Usage: xargs: [-t] [-p] [-e[eofstr]] [-E eofstr] [-I replstr] 
[-i[replstr]] [-L #] [-l[#]] [-n # [-x]] [-s size] [cmd [args ...]]
Mon Jan 17 15:43:44 2011 Error: Critical exitcode.
xargs: illegal option -- 0
xargs: Usage: xargs: [-t] [-p] [-e[eofstr]] [-E eofstr] [-I replstr] 
[-i[replstr]] [-L #] [-l[#]] [-n # [-x]] [-s size] [cmd [args ...]]


What is the expected output? What do you see instead?
Server should handle this gracefully.

What version of the product are you using? On what operating system?
2.0.1

The file keeps being made after it starts and it being excluded in the place 
where daemon of lsyncd is started even if there is an update when describing it 
like this. 

sync{
        default.rsync,
        source="/var/tmp/",
        target="root@datanode1:/data2/mirror/",
        rsyncOps="-avlt",
        exclude = { "define.php" }
}

----------------------

When daemon of lsyncd is started, it is made disregarding the exclusion list 
when the file is made after it starts or there is an update when describing it 
like this though it is excluded. 

sync{
        default.rsync,
        source="/var/tmp/",
        target="root@datanode1:/data2/mirror/",
        rsyncOps="-avlt",
        exclude = { "/define.php" }
}

What version of the product are you using? On what operating system?
2.0.2



Thanks.

This patch allows to use sysconfig directly. This is used so that if you change 
the proxy using YaST, it will continue to work with the new configured proxy 
before the environment is regenerated.

This allows for libzypp to drop it proxy code and use libproxy exclusively.

You can see the module is only activated if running as root.

lsync 1.39 ignores error codes from rsync while syncing dirs, so if remote 
rsyncd is unreachable directory never gets synced.

Problem is very easy to reproduce
- configure lsyncd to watch some dir, check updates are propagated OK
- stop rsyncd on remote side
- make changes to local dir, watch something like this in debug log
  select() timeout, doing delays.
  in queue: 1 expired / 0 delayed dirs
  delay expired: acting for /apps/samba/shares/deps/novgorod/out/.
  calling /usr/bin/rsync -ltd --delete --bwlimit=100 --timeout=60 --password-file=/home/transf-novgorod/.rsyncpass --exclude-from /home/transf-novgorod/lsyncd.exclusions /apps/samba/shares/deps/novgorod/out/ [email protected]::from-spb/
    /usr/bin/rsync /apps/samba/shares/deps/novgorod/out/ --> [email protected]::from-spb/ [7756]
  rsync: failed to connect to hawk-n1.n1.maxidom.ru: Connection refused (111)
  rsync error: error in socket IO (code 10) at clientserver.c(98)
  Forked binary process returned non-zero return code: 10
  Finished waiting for all children
  in queue: 0 expired / 0 delayed dirs
  gone blocking
- so changes are lost

This severely limits reliablity of lsyncd as mirroring solution.

i ve added regex support, it 's similar to exclude file but with regex patterns 
...

Maybe I am mistaken on the use of %r in configuration but lsyncd seems 
extremely inefficient when there are too many directories involved (such as 
svn).

For example, if I have a directory structure similar to this:

var
`-- www
    |-- images
    |  `-- pic.jpg
    |-- css
       `-- main.css

If I modify pic.jpg then within the delay time I modify main.css is it possible 
to make lsyncd just rsync on /var/www instead of once for /var/www/images and 
/var/www/css?

What steps will reproduce the problem?
1. download 1.33
2. compile 1.33
3. execute in no-daemon

What is the expected output? What do you see instead?
segmentation fault


What version of the product are you using? On what operating system?
1.33

Please provide any additional information below.

I don't need lsyncd to do initial rsync. How do I prevent it from doing it?

What steps will reproduce the problem?
======================================

 [host A]$ touch 123
 [host A]$ ll
 -rw-r--r-- 1 root  root     0 Aug 13 12:38 123

 [host B]$ ll
 -rw-r--r-- 1 root  root     0 Aug 13 12:38 123

 [host A]$ chown cf 123
 [host A]$ ll
 -rw-r--r-- 1 cf    root     0 Aug 13 12:38 123

 [host B]$ ll
 -rw-r--r-- 1 root  root     0 Aug 13 12:38 123

 [host A]$ chmod +x 123
 [host A]$ ll
 -rwxr-xr-x 1 cf    root     0 Aug 13 12:38 123

 [host B]$ ll
 -rw-r--r-- 1 root  root     0 Aug 13 12:38 123


What is the expected output? What do you see instead?
=====================================================

I expected preserved permissions and ownerships.

What version of the product are you using? On what operating system?
====================================================================

# cat /etc/redhat-release 
CentOS release 5.3 (Final)

# rsync --version
rsync  version 2.6.8  protocol version 29

# lsyncd --version
Version: 1.25

Is there any option to set the expected behavior?

(enhancement request)

Linux 2.6.31 or later have a new feature, "fsnotify", that is supposed to
have good performance for the "all files" case, and so may make lsyncd
suitable for entire-system backup.

http://www.pubbs.net/kernel/200905/109416/ has some good discussion.

What steps will reproduce the problem?
1.

mkdir /tmp/1 /tmp/2

[alex@alex-c5-64 2]$ cat /etc/lsyncd.conf 

sync { default.rsync,
   name="alex-c5",
   source="/tmp/2",
   target="/tmp/1",
   rsyncOps={"-a", "--numeric-ids", "--bwlimit=10000", "--delay-updates", "--partial-dir=.rsync.tmp" },
   exclude=".rsync.tmp",
   init=false
}

gzip_text_pdf = {
   onCreate = "sleep 5; test -f ^sourcePathname && echo ^sourcePathname | grep -q txt$ && gzip -f ^sourcePathname",
}
sync { gzip_text_pdf,
   name="alex-c5-gzip",
   source="/tmp/2",
}
[alex@alex-c5-64 2]$ lsyncd -nodaemon -log all /etc/lsyncd.conf 


2. in another terminal:

cd /tmp/2
for i in aa bb cc dd; do echo $i > $i.txt; sleep 15 ; done

What is the expected output? What do you see instead?

files in /tmp/2 should be gzipped and synced to /tmp/1 , instead I see

15:27:38 Inotify: got event Create bb.txt(1)
15:27:38 Function: delay(alex-c5, Create, /bb.txt, (nil))
15:27:38 Exclude: '/bb.txt' NOT matches '/.rsync.tmp'
15:27:38 Delay: New Create:/bb.txt
15:27:38 Function: delay(alex-c5-gzip, Create, /bb.txt, (nil))
15:27:38 Delay: New Create:/bb.txt
15:27:38 Call: inotifyEvent()
15:27:38 Inotify: got event Modify bb.txt(1)
15:27:38 Function: delay(alex-c5, Modify, /bb.txt, (nil))
15:27:38 Exclude: '/bb.txt' NOT matches '/.rsync.tmp'
15:27:38 Delay: Modify:/bb.txt absorbed by Create:/bb.txt
15:27:38 Function: delay(alex-c5-gzip, Modify, /bb.txt, (nil))
15:27:38 Delay: Modify:/bb.txt absorbed by Create:/bb.txt
15:27:38 Call: cycle()
15:27:38 Function: invokeActions('alex-c5',(Timestamp: 316531.19))
15:27:38 Function: invokeActions('alex-c5-gzip',(Timestamp: 316531.19))
15:27:38 Normal: Event Create spawns shell 'sleep 5; test -f /tmp/2//bb.txt && 
echo /tmp/2//bb.txt | grep -q txt$ && gzip -f /tmp/2//bb.txt'
15:27:38 Exec: /bin/sh [-c] [sleep 5; test -f "$1" && echo "$1" | grep -q txt$ 
&& gzip -f "$1"] [/bin/sh] [/tmp/2//bb.txt]
15:27:38 Call: getAlarm()
15:27:38 Debug: getAlarm returns: (false)
15:27:38 Masterloop: going into select (no timeout).
rsync error: some files could not be transferred (code 23) at main.c(892) 
[sender=2.6.8]
15:27:38 Call: collectProcess()
15:27:38 Delay: collected a list
15:27:38 Normal: Finished a list = 23
15:27:38 Delay: Finished list = 23
15:27:38 Call: cycle()
15:27:38 Function: invokeActions('alex-c5',(Timestamp: 316531.21))
15:27:38 Function: invokeActions('alex-c5-gzip',(Timestamp: 316531.21))
15:27:38 Call: getAlarm()
15:27:38 Debug: getAlarm returns: (Timestamp: 316546.19)
15:27:38 Masterloop: going into select (timeout 14.98 seconds)
15:27:43 Call: inotifyEvent()
15:27:43 Inotify: got event Create bb.txt.gz(1)
15:27:43 Function: delay(alex-c5, Create, /bb.txt.gz, (nil))
15:27:43 Exclude: '/bb.txt.gz' NOT matches '/.rsync.tmp'
15:27:43 Delay: New Create:/bb.txt.gz
15:27:43 Function: delay(alex-c5-gzip, Create, /bb.txt.gz, (nil))
15:27:43 Delay: New Create:/bb.txt.gz
15:27:43 Call: inotifyEvent()
15:27:43 Inotify: got event Attrib bb.txt.gz(1)
15:27:43 Function: delay(alex-c5, Attrib, /bb.txt.gz, (nil))
15:27:43 Exclude: '/bb.txt.gz' NOT matches '/.rsync.tmp'
15:27:43 Delay: Attrib:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Function: delay(alex-c5-gzip, Attrib, /bb.txt.gz, (nil))
15:27:43 Delay: Attrib:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Call: inotifyEvent()
15:27:43 Inotify: got event Modify bb.txt.gz(1)
15:27:43 Function: delay(alex-c5, Modify, /bb.txt.gz, (nil))
15:27:43 Exclude: '/bb.txt.gz' NOT matches '/.rsync.tmp'
15:27:43 Delay: Modify:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Function: delay(alex-c5-gzip, Modify, /bb.txt.gz, (nil))
15:27:43 Delay: Modify:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Call: inotifyEvent()
15:27:43 Inotify: got event Attrib bb.txt.gz(1)
15:27:43 Function: delay(alex-c5, Attrib, /bb.txt.gz, (nil))
15:27:43 Exclude: '/bb.txt.gz' NOT matches '/.rsync.tmp'
15:27:43 Delay: Attrib:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Function: delay(alex-c5-gzip, Attrib, /bb.txt.gz, (nil))
15:27:43 Delay: Attrib:/bb.txt.gz absorbed by Create:/bb.txt.gz
15:27:43 Call: inotifyEvent()
15:27:43 Inotify: got event Delete bb.txt(1)
15:27:43 Function: delay(alex-c5, Delete, /bb.txt, (nil))
15:27:43 Exclude: '/bb.txt' NOT matches '/.rsync.tmp'
15:27:43 Delay: Delete:/bb.txt replaces Create:/bb.txt
15:27:43 Function: delay(alex-c5-gzip, Delete, /bb.txt, (nil))
15:27:43 Call: collectProcess()
15:27:43 Delay: collected an event
15:27:43 Error: IN LUA: lsyncd.lua:3393: event does not have field 'isList'
15:27:43 Error: Backtrace 1 :[C]:-1
15:27:43 Error: Backtrace 2 :lsyncd.lua:722
15:27:43 Error: Backtrace 3 :lsyncd.lua:3393
15:27:43 Error: Backtrace 4 :lsyncd.lua:1198
15:27:43 Error: Backtrace 5 :lsyncd.lua:2500



What version of the product are you using? On what operating system?

lsyncd 2.0.2 on centos5 x86_64

Please provide any additional information below.

It doesnt sync subdirs at all :-(
Current SVN got it fixed, by using recurse again, but with filter.
performance unknown.

--from has the disadvantage that rsync hicups on delted files.

What steps will reproduce the problem?

I use a lua config file that will spawn programs (that will run for severals 
seconds) and then copy intermediate files to a remote server using sftp.

What is the expected output? What do you see instead?

Everything works correctly but at the end lsyncd crashes saying :
.....
00:55:24 Call: getAlarm()
00:55:24 Debug: getAlarm returns: (false)
00:55:24 Masterloop: going into select (no timeout).
00:55:24 Call: collectProcess()
00:55:24 Delay: collected an event
00:55:24 Delay: Finish of Create on /home/xxxx
00:55:24 Call: cycle()
00:55:24 Function: invokeActions('Sync1',(Timestamp: 42966265.74))
00:55:24 Call: getAlarm()
00:55:24 Debug: getAlarm returns: (false)
00:55:24 Masterloop: going into select (no timeout).
00:55:24 Call: collectProcess()
00:55:24 Delay: collected an event
00:55:24 Error: IN LUA: lsyncd.lua:1150: Queue is broken, delay not a dpos
00:55:24 Error: Backtrace 1 :[C]:-1
00:55:24 Error: Backtrace 2 :lsyncd.lua:1150
00:55:24 Error: Backtrace 3 :lsyncd.lua:1207
00:55:24 Error: Backtrace 4 :lsyncd.lua:2500


What version of the product are you using? On what operating system?

lsyncd 2.0.2 on ubuntu 64 bits 10.04

Please provide any additional information below.

I write my conf file using the image magic example. No clue at the moment.
I do not understand the error, what does mean the error message ? Lsyncd tries 
to unstack a delay event that does not exist ?
Thank you in advance for looking into this issue

I try to install with lsyncd in FreeBSD 8, but faild.
I had installed: xinetd build libxml2.
any ideas?

the configure had ok, and make info is:
# make
make  all-am
if gcc -DHAVE_CONFIG_H -DXML_CONFIG -I. -I. -I.     -Wall  `xml2-config 
--cflags` -g -O2 -MT lsyncd.o -MD -MP -MF ".deps/lsyncd.Tpo"  -c -o lsyncd.o 
`test -f 'lsyncd.c' || echo './'`lsyncd.c;  then mv -f ".deps/lsyncd.Tpo" 
".deps/lsyncd.Po";  else rm -f ".deps/lsyncd.Tpo"; exit 1;  fi
In file included from lsyncd.c:20:
inotify-nosys.h:15:24: error: asm/unistd.h: No such file or directory
In file included from lsyncd.c:20:
inotify-nosys.h:28: error: expected ':', ',', ';', '}' or '__attribute__' 
before '__flexarr'
inotify-nosys.h:155:3: error: #error "Unsupported architecture!"
inotify-nosys.h:158:3: error: #error "Unsupported architecture!"
inotify-nosys.h:161:3: error: #error "Unsupported architecture!"
inotify-nosys.h: In function 'inotify_init':
inotify-nosys.h:167: error: '__NR_inotify_init' undeclared (first use in this 
function)
inotify-nosys.h:167: error: (Each undeclared identifier is reported only once
inotify-nosys.h:167: error: for each function it appears in.)
inotify-nosys.h: In function 'inotify_add_watch':
inotify-nosys.h:172: error: '__NR_inotify_add_watch' undeclared (first use in 
this function)
inotify-nosys.h: In function 'inotify_rm_watch':
inotify-nosys.h:177: error: '__NR_inotify_rm_watch' undeclared (first use in 
this function)
lsyncd.c: At top level:
……

What steps will reproduce the problem?

lsyncd runs for a while. After that it segfaults

syncd[24766] general protection rip:3385e796f0 rsp:7fff75a73448 error:0
lsyncd[24873]: segfault at 00000000ffffffff rip 0000003385e796d0 rsp
00007fff963cd798 error 4
lsyncd[32550] general protection rip:3385e796f0 rsp:7fff96631c18 error:0
lsyncd[996]: segfault at 0000000000000001 rip 0000003385e796d0 rsp
00007fff63767be8 error 4
lsyncd[1018]: segfault at ffffffffffffffd0 rip 0000003385e796f0 rsp
00007fff60f89428 error 4
lsyncd[1177]: segfault at ffffffffffffffd0 rip 0000003385e796f0 rsp
00007fff946a1f98 error 4
lsyncd[1210]: segfault at ffffffffffffffd0 rip 0000003385e796f0 rsp
00007fffa0dfe6d8 error 4
lsyncd[1272]: segfault at ffffffffffffffd0 rip 0000003385e796f0 rsp
00007fffa4442bf8 error 4


What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?

CentOS 5.4

Please provide any additional information below.

running rsync with --timeout 1. It also seems to segfault when a timeout occurs

io timeout after 1 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(171) [sender=2.6.8]
rsync error: timeout in data send/receive (code 30) at io.c(171)
[receiver=2.6.8]
rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: timeout in data send/receive (code 30) at io.c(463) [sender=2.6.8]
rsync error: timeout in data send/receive (code 30) at io.c(171)
[receiver=2.6.8]
rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: timeout in data send/receive (code 30) at io.c(463) [sender=2.6.8]
rsync error: timeout in data send/receive (code 30) at io.c(171)
[receiver=2.6.8]
rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: timeout in data send/receive (code 30) at io.c(463) [sender=2.6.8]


btw --delay really makes my troubles less painful!

thank you!

What steps will reproduce the problem?
1.source compilation generate warnings 
(inotify.c:75: warning /inotify.c:91: warning)

# make
make  all-am
make[1]: Entering directory `/usr/src/svn'
if gcc -DHAVE_CONFIG_H -I. -I. -I.     -Wall -I/usr/local/include    -MT 
lsyncd.o -MD -MP -MF ".deps/lsyncd.Tpo" -c -o lsyncd.o lsync    d.c; \
        then mv -f ".deps/lsyncd.Tpo" ".deps/lsyncd.Po"; else rm -f ".deps/lsyncd.Tpo"; exit 1; fi
if gcc -DHAVE_CONFIG_H -I. -I. -I.     -Wall -I/usr/local/include    -MT 
inotify.o -MD -MP -MF ".deps/inotify.Tpo" -c -o inotify.o in    otify.c; \
        then mv -f ".deps/inotify.Tpo" ".deps/inotify.Po"; else rm -f ".deps/inotify.Tpo"; exit 1; fi
inotify.c: In function ‘l_addwatch’:
inotify.c:75: warning: format ‘%d’ expects type ‘int’, but argument 6 
has type ‘lua_Integer’
inotify.c: In function ‘l_rmwatch’:
inotify.c:91: warning: format ‘%d’ expects type ‘int’, but argument 5 
has type ‘lua_Integer’
luac lsyncd.lua


2.using lbash.lua with nodaemon=false

3.creating mkdir -p /src/a/b/c/d/e/f

on the target :
# tree -L 10 /trg/
/trg/
`-- a
    `-- b
        |-- b
        |   `-- c
        |       `-- d
        |           `-- e
        |               `-- f
        `-- c
            |-- c
            |   `-- d
            |       `-- e
            |           `-- f
            `-- d
                |-- d
                |   `-- e
                |       `-- f
                `-- e
                    |-- e
                    |   `-- f
                    `-- f
                        `-- f

duplicated directories on target ...

--------------------------------------------------------------
lsyncd 2 - r426 - linux redhat 5.3 64
--------------------------------------------------------------
tail -f "/tmp/lsyncd.log"
Fri Nov 26 10:59:47 2010 Normal: Event Blanket spawns shell 'if [ "$(ls -A 
/src/)" ]; then cp -r /src/* /trg/; fi'
Fri Nov 26 10:59:47 2010 Normal: Startup of '/src/' finished.
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r /src//a 
/trg//a'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/ = 0
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r /src//a/b 
/trg//a/b'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/b/ = 0
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r /src//a/b/c 
/trg//a/b/c'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/b/c/ = 0
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r /src//a/b/c/d 
/trg//a/b/c/d'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/b/c/d/ = 0
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r 
/src//a/b/c/d/e /trg//a/b/c/d/e'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/b/c/d/e/ = 0
Fri Nov 26 11:01:35 2010 Normal: Event Create spawns shell 'cp -r 
/src//a/b/c/d/e/f /trg//a/b/c/d/e/f'
Fri Nov 26 11:01:35 2010 Normal: Finished Create on /src//a/b/c/d/e/f/ = 0

-----------------------------------------------------
# tail -f /tmp/lsyncd.stat
Lsyncd status report at Fri Nov 26 10:59:47 2010

Sync1 source=/src/
There are 0 delays
Excluding:
  nothing.

Watching 1 directories
  1: /src/
  2: /src/a/
  3: /src/a/b/
  4: /src/a/b/c/
  5: /src/a/b/c/d/
  6: /src/a/b/c/d/e/
  7: /src/a/b/c/d/e/f/


sincerely yours ...

What steps will reproduce the problem?
1.server A sync files to Server B
2.server B also sync the same files to Server A
3.there is a recursive loop
4.when sync small file it works fine,but big file , will no
5.in the syncing process there will generate a lot of temp file,named 
.filename.XX31z 
..filename.Xua1 
...filename.A6a51 
or so
6.dose v2.0 solved this problem?

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?

lsyncd 1.0

                } else if (!xmlStrcmp(dnode->name, BAD_CAST "binary")) {
                        xc = xmlGetProp(dnode, BAD_CAST "filename");
                        if (xc == NULL) {
                                fprintf(stderr, "error in config file: 
attribute filename missing from <binary>\n");
                                exit(LSYNCD_BADCONFIGFILE);
                        }
                        dc->exclude_file = s_strdup((char *) xc);

bug??

My maildir mirror was found to have a directory ".Mailing" full of files
that belonged in ".Mailing Lists"

I'm using lsyncd to sync a great many directories to 6 servers. It's working 
very well for me - thanks!

Sometimes, completing an update can take several hours. It would be really 
useful if there was someway of discovering how many items in the queue still 
have to be synced. For a bonus, an estimate of the time left.

Maybe some kind of status sent to syslog or a local file?

Steve

Hi,

I just can't make the "exclude-from"-option work.

I created a file /etc/ha-lsyncd.exclude containing all directories I want
to leave unsynchronized.

I tried
(a) <absolut path of directory i don't want to be mirrored>
(b) <relativ path compared to original directory>
(c) RSYNC syntax: "- <path>"

but when I start lsyncd with
 # lsyncd --exclude-from /etc/ha-lsyncd.exclude
the excluded directories are still in sync. What is the correct syntax for
that file?

What steps will reproduce the problem?
======================================

I just wanted to exclude a complete subdirectory as it is written in
"Lsyncd usage examples"



What is the expected output? What do you see instead?
=====================================================

# lsyncd --no-daemon --exclude-from <subdir> <original> <remote
host>:<replique>
Wed Aug 26 17:10:26 2009: command line options: syncing <original> ->
<remote host>:<replique>

Wed Aug 26 17:10:26 2009: ERROR: Reading file '<subdir>' (21:Is a directory)

This implies that lsyncs has a problem excluding a directory. Excluding
files does work.



What version of the product are you using? On what operating system?
====================================================================

# cat /etc/redhat-release 
CentOS release 5.3 (Final)

# rsync --version
rsync  version 2.6.8  protocol version 29

# lsyncd --version
Version: 1.25

What steps will reproduce the problem?
Try to sync with a rsync version that doesn't support -d option (eg. < 2.6.4).

What is the expected output? What do you see instead?
During normal operations, directories should sync, but the following output
is generated:

rsync: on remote machine: -ldtp: unknown option
rsync error: syntax or usage error (code 1) at main.c(1002)
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[sender=3.0.6]

What version of the product are you using? On what operating system?
lsyncd 1.26
rsync 3.0.6 on source machine
rsync 2.6.2 on remote machine

Please provide any additional information below.
Initial sync works as expected (lsyncd uses -r during startup). I'm tempted
to always add -r option in lsyncd configuration file, but I'm not sure if
it can cause bad side effects. After all lsyncd uses -d for a reason, isn't it?

What steps will reproduce the problem?
1. run lsync on a directory
2. copy a bunch of file to that directory (with 'cp')
3. profit!

What is the expected output? What do you see instead?

I expect lsync to run rsync only once. Instead I see this:

Tue Sep 22 17:06:35 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTB:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:36 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTA:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:36 2009: event CLOSE_WRITE:c6.gif triggered.
Tue Sep 22 17:06:36 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTB:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:36 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTA:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:37 2009: event CLOSE_WRITE:c29.gif triggered.
Tue Sep 22 17:06:37 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTB:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:37 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTA:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:37 2009: event CLOSE_WRITE:c40.gif triggered.
Tue Sep 22 17:06:37 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTB:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:38 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTA:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:38 2009: event CLOSE_WRITE:index.htm triggered.
Tue Sep 22 17:06:38 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTB:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:39 2009: rsyncing
/data/sites/jboek/productie/code/include/vogoo/img/ -->
user@HOSTA:/data/sites/jboek/productie/code/include/vogoo/img/
Tue Sep 22 17:06:39 2009: event CLOSE_WRITE:c30.gif triggered.

Repeating of rsync calls that are all the same.

What version of the product are you using? On what operating system?

1.26, Debian 

What steps will reproduce the problem?
1. ./configure
2. make

What is the expected output? What do you see instead?
make  all-am
make[1]: Entering directory `/root/lsyncd-1.26'
gcc -DHAVE_CONFIG_H -DXML_CONFIG -I.    -Wall `xml2-config --cflags` -g -O2
-MT lsyncd.o -MD -MP -MF .deps/lsyncd.Tpo -c -o lsyncd.o lsyncd.c
/bin/sh: xml2-config: command not found
lsyncd.c:36:27: error: libxml/parser.h: No such file or directory
lsyncd.c:37:25: error: libxml/tree.h: No such file or directory
...

What version of the product are you using? On what operating system?
1.26, Debian Linux (Lenny)

Please provide any additional information below.
Configure should check for libxml2-dev.

Nice work btw! :)

I'd like to see lsyncd supporting fanotify, since we have lots of files in many 
different directories and cannot use inotify.

How long do think would it take to implment fanotify into lsyncd? I'm well 
aware of the "move"-problem, but don't care.

Maybe we can speed things up a little bit by helping you to test or pay a 
reward?

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

rsync: failed to connect to 10.9.10.164: Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.7]
Wed Jun  9 17:12:23 2010: Forked binary process returned non-zero return code: 
10


What version of the product are you using? On what operating system?
lsyncd : 1.33
sync   : 3.07
linux  : gentoo 2.6.26-gentoo-r3 64bit

Please provide any additional information below.

5 nodes sync with lsyncd. where on node down,rsyncd will reutrn 10,and lsyncd 
auto exit.

here's my configure file:

<lsyncd version="1">
<settings>
  <logfile filename="/var/log/lsyncd.log"/>
  <pidfile filename="/var/run/lsyncd.pid"/>
  <debug/>
  <stubborn/>
  <!--scarce/-->

  <!--exclude-from filename="/etc/lsyncd-exclude.conf"/-->

  <callopts>
    <option text="-ltp%r"/>
    <option text="--delete"/>
    <option text="--timeout=60"/>
    <source/>
    <destination/>
  </callopts>
</settings>

<directory>
<source path="/var/www/gfs"/>
   <target path="10.9.10.75::gfs"/>
   <target path="10.9.10.231::gfs"/>
   <target path="10.9.10.222::gfs"/>
   <target path="10.9.10.164::gfs"/>
</directory>
</lsyncd>

What steps will reproduce the problem?
1. large homedir ( million files )
2. lsyncd --exclude-from=/home/oernii/.rsync_exclude --no-daemon --verbose 
/home/oernii/ remote::/BACKUP/

What is the expected output? What do you see instead?
 sync should run without errors

What version of the product are you using? On what operating system?
1.39

Please provide any additional information below.
After running I am getting a LOT of this:

watching /home/oernii/.kde
ERROR: Cannot add watch /home/oernii/.kde (28:No space left on device)

First tests with lsyncd on a medium-size directory (about 2G). I get:

ERROR: Cannot add watch /home/... (28:No space left on device)

Both source and target disk have plenty of space. Maybe it's problem with the 
number of directory to watch?

This is not really an issue, but a request. When I rename/move a large file, 
rsync will copy it again and then delete the "old" version. If an event 
MOVED_FROM to MOVED_TO occurs, wouldn't be better if you simply apply the 
same sequence on the other side?

Wed Dec 16 20:54:15 2009: event MOVED_FROM:00234.m2ts triggered.
Wed Dec 16 20:54:15 2009: rsyncing /mnt/sync1/ --> /mnt/sync2/
Wed Dec 16 20:54:28 2009: event MOVED_TO:same_file.m2ts triggered.
Wed Dec 16 20:54:28 2009: rsyncing /mnt/sync1/ --> /mnt/sync2/

Instead, just move 00234.m2ts to same_file.m2ts without transfering it over 
again.

What steps will reproduce the problem?
1. Try to run on FreeBSD
2. Of course it cannot
3. ???
3. ... No profit :(

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

If you guys could please support kqueue we could really, really use this at my 
job. It would be a much, much better solution than our cron'd rsyncs.

Thanks and keep up the good work!

What steps will reproduce the problem?
1. edit /etc/lsyncd.conf.xml
2. add <no-daemon/> fields
3. return to shell,run lsyncd

What is the expected output? What do you see instead?

SH_jjwxc135 log # lsyncd
Starting up
watching
Segmentation fault


What version of the product are you using? On what operating system?
lsyncd: v1.33 

Linux kernel : Linux SH_jjwxc135 2.6.26-gentoo-r3 #1 SMP Fri Dec 5 11:09:04 CST 
2008 x86_64 Quad-Core AMD Opteron(tm) Processor 2346 HE AuthenticAMD GNU/Linux

glibc version: 2.11

Please provide any additional information below.

1.when remove the <no-daemon/> setting ,lsyncd will ok
2.add <debug/> option,and remove <no-daemon/> in lsyncd.conf.xml,run lsyncd 
,will output this information :

.........

found new directory: <c in (null) -- P╜
found new directory: 嗷b in (null) -- P╜
found new directory: /var/www/gfs/abc.net/api.abc/var/Lair/Log/.. in (null) -- 
P╜
found new directory: <c in (null) -- P╜
found new directory: 嗷b in (null) -- P╜
found new directory: 
/var/www/gfs/abc.net/api.abc/www/scripts/dojo/io/xip_server.html in (null) -- 
P╜
found new directory: 
/var/www/gfs/abc.net/api.abc/www/scripts/dojo/cal/textDirectory.js in (null) -- 
P╜
found new directory: 
/var/www/gfs/abc.net/api.abc/www/scripts/dojo/dnd/HtmlDragAndDrop.js in (null) 
-- P╜
found new directory: 
/var/www/gfs/abc.net/api.abc/www/scripts/dojo/dot/ui-template/redball.png in 
(null) -- P╜
found new directory: p<d in (null) -- P╜

.........

strange symbol "P╜" ??

i work with eclipse+svn and i use lsyncd to mirror my project to the
working dev directory server.

lsyncd add all subdirectories so with .svn tree I have nearly 10000
directories being watched and i of course exploded my max_user_watches limit.

ok i defined it to 32000 and it s ok now but it would be very convenient
and more performant if .svn subdirs were not watched at all ;p

In the new release 2.0.0 the option --pidfile writes the wrong pid into the pid 
file: 

As the function write_pidfile() is called before daemonize(), the pid of the 
first process is written, not the pid of the daemon. So start scripts which 
check that number do not work right.

Lsyncd does not allow the user to configure the triggering inotify events.
Instead lsyncd registers the following events ATTRIB, CLOSE_WRITE, CREATE,
DELETE, DELETE_SELF, MOVED_FROM, MOVED_TO, DONT_FOLLOW, ONLYDIR. In some
use cases this will result in unnecessary sync operations which is in
particular painful if the network bandwidth between source and destination
is an issue.

Given an remote application which puts files in the lsyncd source by
creating a temporary file (CREATE), copy the data (CLOSE_WRITE), moving the
temporary file to the final filename (MOVE_FROM, MOVE_TO). This application
behavior is common because the final move command is an atomic operations
which ensures that only completely transmitted files are available within
the data source. In this use case lsyncd will trigger 4 rsync operations
and the data file will be transmitted twice.

The attached patch introduces the new configuration node <inotify> which
may be configured in the global <settings> section or on a <directory>
basis. If no <inotify> node is configured then the previously described
default event list is used.

Sample configuration:

<lsyncd version="1">
    <directory>
        <inotify>
            <event id="MOVED_TO"/>
            <event id="DELETE"/>
        </inotify>
        <source path="/var/data"/>
        <target path="destination_host::dest"/>
    </directory>    
</lsyncd>

The log is not correctly vomited in "Logfile" and "Pidfile" though it describes 
in the "lsyncd.conf" file as stated above. 
"Nodaemon =  true" mode starts correctly. 

The command specified when lsyncd is started is
 # lsyncd /etc/lsyncd.conf

-----
setting = {
        logfile = "/var/tmp/lsyncd.log",
        statusFile = "/var/tmp/lsyncd.status",
        nodaemon = true,
}
-----

By the way,
 lsyncd /etc/lsyncd.conf -logfile /var/tmp/lsyncd.log -pidfile /var/tmp/lsyncd.status

When the command is specified like this, the log file is correctly written. 
Should I specify the command though it describes in "lsyncd.conf"?

The version used is "lsyncd-2.0.2".

Best regards, Tachi.

Hi,

Here is my lsyncd.conf :


settings = {
  logfile = "/home/toto/log/lsyncd.log",
  statusFile = "/home/toto/log/lsyncd.status",
}

sync{
  default.rsyncssh,
  source = "/home/src/example",
  host = "192.168.1.100",
  targetdir = "/home/dest/example",
  exclude = {"//home/src/example/tmp/**"}
}


Launched with :

> lsyncd -nodaemon -log all etc/lsyncd.conf

I get :


Exclude: toLuaPattern '//home/src/example/tmp/**' = '^/home/src/example/tmp/.*"

So the pattern looks ok (although I'd suggest let people use real Lua regex 
patterns, it will make easier for everybody, and allow more possibilities. The 
translation from your pseudo patterns to Lua patterns is unnecessary in my 
opinion).

21:13:23 Function: Inotify.addWatch(/home/src/example/tmp/, (true), (nil), 
(nil))
21:13:23 Inotify: not concerning '/home/src/example/tmp/'

So pattern seems to work. But it is not used in the rsync command :


21:13:23 Exec: /usr/bin/rsync [--delete] [-r] [-lts] [/home/src/example/] 
[192.168.1.100:/home/dest/example/]

And I get many :

rsync: send_files failed to open "/home/src/example/tmp/sessions/sess_1234": 
Permission denied (13)

Although I told the conf to ignore this directory.

Am I doing something wrong or is this a bug ?
Do i have to add something specific for the rsync command ? The -exclude-from 
doesn't seem to exist anymore.

Thanks in advance :)

The Debian developer tool Lintian found some minor spelling issues that are 
fixed by the attached patch.


Regards
Jan Dittberner

What steps will reproduce the problem?
1. unzip 
2. ./configure
3. make

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?

ubuntu 10.04 x64

Please provide any additional information below.

--------------------------

lsyncd.c: In function ‘reset_options’:
lsyncd.c:966: warning: comparison with string literal results in unspecified 
behavior
lsyncd.c:972: warning: comparison with string literal results in unspecified 
behavior
lsyncd.c: In function ‘one_main’:
lsyncd.c:3049: warning: format ‘%d’ expects type ‘int’, but argument 4 
has type ‘size_t’


--------------------------

Is it possible to run lsyncd as a different user?

What steps will reproduce the problem?
My ssh server is not on port 22, how can I instruct lsyncd to use my other
port, I tried the xml config file with no luck.

What is the expected output? What do you see instead?
Fail

What version of the product are you using? On what operating system?
last release

Please provide any additional information below.

Thanks
Thomas

What steps will reproduce the problem?
1. Recursively delete some files
2. Watch the log as lsyncd tries to rsync files that are no longer there
3. Takes forever - the more that was deleted, the longer it takes

What is the expected output? What do you see instead?

I would expect that lsyncd would at least do a stat on the file to see if
it existed before trying to rsync it.  Obviously if it is not there, lsyncd
should discard the sync and move on to the next inotify event - eventually
the event containing the correct sync level will get hit and rsync can then
be triggered.  This would be MUCH faster.


What version of the product are you using? On what operating system?

lsync 1.26 / rsync  version 3.0.6  protocol version 30
2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009 x86_64 x86_64 x86_64
GNU/Linux


Please provide any additional information below.

File creation or updates work great - just deletes where the problem seems
to be limited.

What steps will reproduce the problem?
1. sudo lsyncd [options] [source] [target] [target]
2.
3.

What is the expected output? What do you see instead?
In previous versions I believe you were able to sync one source to multiple 
targets. When I supply the syntax for multiple targets I get an error that says 
I'm only allowed one config file. So Lsyncd is reading one of my targets as the 
location of the config file. 

What version of the product are you using? On what operating system?
lsyncd 2.0 Ubuntu 10.0.4 Server

Please provide any additional information below.
Is syncing one source to multiple targets even possible with 2.0? If it is how 
would I accomplish that without getting the error? Is this a bug? Thanks for 
your help. Love your work by the way!

What steps will reproduce the problem?
1. Add '--relative' option to rsyncOps
2. Run lsyncd command as usual

What is the expected output? What do you see instead?
With a source of /home/me/tmp and destination of /backup/me, lsyncd should 
create and sync with /backup/me/home/me/tmp.  It does the initial rsync to this 
directory, but it syncs /home/me/tmp with /backup/me.  For example, creating 
'test' in /home/me/tmp should create /backup/me/home/me/tmp/test but instead it 
creates /backup/me/test.

What version of the product are you using? On what operating system?
2.0.2 on Ubuntu 10.10

Please provide any additional information below.
The issue appears to be with how the paths are tracked by lsyncd because the 
rsyncOps are definitely still used for the incremental updates.

Is this project abandoned?

Because it's right next to the lsyncd.1 page, to be installed as man1 in
Maikefile.am, the lsyncd.conf.xml.5 gets installed in the wrong place, as
it should go to man5.

The fix is trivial :

-nodist_man1_MANS = doc/lsyncd.1 doc/lsyncd.conf.xml.5
+nodist_man1_MANS = doc/lsyncd.1
+nodist_man5_MANS = doc/lsyncd.conf.xml.5