ku1ik / bitpocket Goto Github PK

When trying to run the script on a ubuntu box I get the following error:

sh: 27084: Syntax error: "fi" unexpected (expecting "then")

I understand it happens on this line:

• rsync --dry-run \

•    -auvzxi --delete $RSYNC_OPTS --exclude "/$DOT_DIR" --exclude-from "$TMP_DIR/fetch-exclude" $USER_RULES $REMOTE/ . \
  

•  | grep "^[ch<>\.\*][f]\|\*deleting" | sed "s:^\S*\s*::" | sed "s:\(.*\):if [ -f \"\1\" ]; then cp --parents \"\1\" $DOT_DIR/backups/$TIMESTAMP; fi:" | sh || die "BACKUP"
  

apparently sh is not accepting nicely what sed feeds him... I tried to change the code to use

sed "s:(.*):test -f "\1" && cp --parents "\1" $DOT_DIR/backups/$TIMESTAMP:"

but that won't work nicely either...

Anyone has had the same problem?

I have a bitpocket repo setup to a remote server with passwordless ssh keys.
While editing files in vim on the master repo vim constantly complains about the .*.swp files existing after I have been editing a file then edit another one and thus vim removes the .swp file for that file but bitpocket resyncs the swp file after deletion

On my PC I would like to set up two slaves. One folder for syncing from server-a and an other folder for syncing from server-b.

Both servers contain different data.

Is this possible with bitpocket?

hi,

I was just looking through the code and was wondering about the following case:
slave A removes a file while slave B updates the file.
In this case I would guess that a convenient way to procede would be to update the file and not delete it.
From what I saw, the script does it the other way around. Is this the desired outcome?

Thanks
Tom

leaving UTF8 for the locale is better to be able to handle any filename. The sort function may differ, but it is still consistent across the two files, meaning that only non duplicates will be found by uniq.

Hi.

I'v made the mistake of using the wrong path for my master folder, and the hard drive became full, and all sorts of operation became impossible. I've changed the path and now I have a bunch of files called \ or ' they seem to be copy of other files in the directory, and they seem to be unimportant and I can delete them (I store the original folder for sanity). I suspect that these files are generated by bitpocket, is this so? Can I delete them?

local data is backed up before deletion when pulling from master, but master data is not backed up when pushing from local.

This problem arises since I use my master/server to also crunch numbers. If a sync is initiated in the middle of a long process that periodically spits out benchmark data, bitpocket gets confused and will delete data on master during a subsequent sync.

The best solution is probably to put master on a server that isn't used to crunch numbers, but just a word of warning to anyone out there that data on master can be lost and not backed up in some scenarios. I'm not sure how hard it will be to implement backups on master. I notice there is no rsync dry run on the push from local...

There are several issues with OSX compaitibility

1. The gnu coreutils not present issue, need details about workaround.
2. mkdir is used incorrectly for backups

# Saving current state and backing up files (if needed)
  | Root dir: /Users/kthirugnanam/BitTorrent Sync/PC Videos
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ..

1. Rsync is also probably not perfectly used, I see 2-3 instances of its usage page being shown before the sync.
2. Even during the sync (push to server) I see a failure
   rsync: link_stat "/Users/kthirugnanam/BitTorrent Sync/PC Videos/admin@oneplus_one.local:/data/data/com.arachnoid.sshelper/home/SDCard/PC" failed: No such file or directory (2)
   I cant understand this, it has both source and destination, without the last word of the destination.

Is it possible to include some form of notifications, such as growl?

I've been testing this on my laptop and should I sleep the machine and make the connection drop, the lockfile seems to persist preventing future updates (and with no apparent notifications).

Is there a way to make the lockfile expire?

May be helpful.

Love this script! Works like a champ. Thanks so much for creating it.

I have a question. How possible/annoying would it be to create some sort of versioning system for files that have changed? Like, keep the last x number of versions? Or something like a trash feature that moves deleted/changed files to a trash folder in case you screw up and need to restore something?

Whenever a sync is interrupted (e.g. manually with Ctrl-C), the lock directory is left behind, with a file containing the PID of the process that created it.

When it tries to sync again, the function acquire_lock tries to kill the process with this PID, and if it fails (i.e. the process does not exist), it exits with this error message:

There's stale lock directory at <LOCK_DIR>. Please remove it and try again.

Why not remove the lock directory directly in the script, instead of asking the user, if it is known to be stale?

Is there a case in which there is a stale directory but removing it could be harmful?

When an exclusion line is removed from .bitpocket/exclude, the expected behavior would be to start syncing that file/directory. However, the actual behavior is that this file is deleted. The following script, as well as script output, illustrates the issue (using a fresh BitPocketMaster directory):

Script

mkdir BitPocket
cd BitPocket
bitpocket init [email protected] /home/user/BitPocketMaster
bitpocket sync
bitpocket sync
echo content>file-1.txt
echo content>file-2.txt
bitpocket sync
echo content>file-excluded.txt
echo "/file-excluded.txt">.bitpocket/exclude
bitpocket sync
ll
rm .bitpocket/exclude
bitpocket sync
ll

Result

file-excluded.txt is gone, which is not the expected behavior, since it has never been deleted.

With output

[tmp]$ mkdir BitPocket
[tmp]$ cd BitPocket
[BitPocket]$ bitpocket init [email protected] /home/user/BitPocketMaster
Initialized bitpocket directory at /home/user/tmp/BitPocket
Please have a look at the config file (.bitpocket/config)
[BitPocket]$ bitpocket sync
bitpocket started at Fri Jan 11 15:34:14 EST 2013.

Fetching changes from server
receiving incremental file list
.d..t...... ./

sent 31 bytes received 59 bytes 60.00 bytes/sec
total size is 0 speedup is 0.00
Pushing changes to server
sending incremental file list

sent 74 bytes received 12 bytes 57.33 bytes/sec
total size is 0 speedup is 0.00
[BitPocket]$ bitpocket sync
bitpocket started at Fri Jan 11 15:34:31 EST 2013.

Fetching changes from server
receiving incremental file list

sent 28 bytes received 56 bytes 56.00 bytes/sec
total size is 0 speedup is 0.00
Pushing changes to server
sending incremental file list

sent 74 bytes received 12 bytes 57.33 bytes/sec
total size is 0 speedup is 0.00
[BitPocket]$ echo content>file-1.txt
[BitPocket]$ echo content>file-2.txt
[BitPocket]$ bitpocket sync
bitpocket started at Fri Jan 11 15:34:49 EST 2013.

Fetching changes from server
receiving incremental file list
.d..t...... ./

sent 65 bytes received 59 bytes 82.67 bytes/sec
total size is 0 speedup is 0.00
Pushing changes to server
sending incremental file list
<f+++++++++ file-1.txt
<f+++++++++ file-2.txt

sent 208 bytes received 50 bytes 172.00 bytes/sec
total size is 16 speedup is 0.06
[BitPocket]$ echo content>file-excluded.txt
[BitPocket]$ echo "/file-excluded.txt">.bitpocket/exclude
[BitPocket]$ bitpocket sync
bitpocket started at Fri Jan 11 15:35:13 EST 2013.

Fetching changes from server
receiving incremental file list
.d..t...... ./

sent 79 bytes received 97 bytes 117.33 bytes/sec
total size is 16 speedup is 0.09
Pushing changes to server
sending incremental file list

sent 136 bytes received 12 bytes 98.67 bytes/sec
total size is 16 speedup is 0.11
[BitPocket]$ ll
total 3
-rw------- 1 mtorfason Domain Users 8 Jan 11 15:34 file-1.txt
-rw------- 1 mtorfason Domain Users 8 Jan 11 15:34 file-2.txt
-rw------- 1 mtorfason Domain Users 8 Jan 11 15:35 file-excluded.txt
[BitPocket]$ rm .bitpocket/exclude
[BitPocket]$
[BitPocket]$ bitpocket sync
bitpocket started at Fri Jan 11 15:35:38 EST 2013.

Fetching changes from server
receiving incremental file list
*deleting file-excluded.txt

sent 28 bytes received 94 bytes 81.33 bytes/sec
total size is 16 speedup is 0.13
Pushing changes to server
sending incremental file list

sent 112 bytes received 12 bytes 82.67 bytes/sec
total size is 16 speedup is 0.13
[BitPocket]$ ll
total 2
-rw------- 1 mtorfason Domain Users 8 Jan 11 15:34 file-1.txt
-rw------- 1 mtorfason Domain Users 8 Jan 11 15:34 file-2.txt
[BitPocket]$ # file-excluded.txt is gone, which is not the expected
[BitPocket]$ # behavior, since it has never been deleted.

What do you think about an notification (like "BitPocket sync in progress...") if the sync fails?
So BitPocket doesn't just echo the notification (e.g. echo "Couldn't acquire remote lock. Another client is syncing with $REMOTE or lock file couldn't be created. Exiting."), but would as well do a notify-send of that messege (if its enabled in options)

There are some feature requests related to versioning synced files. I don't have implementing native versioning in plans because the goal of git-dude was and still is to be simple sync tool built on rsync.

However, I can see one, possibly working, solution based on git:

bitpocket could have internal, local, git repository (ie. hidden in .bitpocket/git) and working tree being your ~/BitPocket dir. All files could be ignored by default and you would run bitpocket track somefile to add it to git and start versioning. Later you could either run bitpocket snapshot at any time to commit all the changes from tracked files or run bitpocket sync --snapshot to commit+sync. This local git repository would be transferred with rsync to other machines like all other files resulting in all of the versions being accessible on all your machines.

This way the repository size can be reasonably small because of tracking only files you want (and you definitely don't want to track mp3 or avi files). Files not tracked by git would still be synced to other machines like they're now.

You could also be able to adjust .git_ignore file to ie. have all *.txt files automatically tracked without need for manual bitpocket track for each file.

Thoughts? Suggestions?

Neither me nor @torfason is be able to provide required amount of maintenance time for bitpocket these days. We're looking for new maintainer of the project.

If you have good understanding of bitpocket's code or you have good level of bash-fu and a will to use bitpocket and keep it healthy then let us know!

since yesterday, bitsync fails to backup my local folders. I get the following error

Saving current state and backing up files (if needed)
| Root dir: /home/yotam/Sync
sh: 5: Syntax error: "fi" unexpected (expecting "then")
fatal: command failed BACKUP

The message first appeared when I was working on my server machine, but I was working the previous day as well and there weren't any issues then.

Is it just me or doesn't bitpocket follow symlinks?
I would kill for that as a feature

After I set up my remote and init'd my local bitpocket, I issued a 'bitpocket sync'

I got these warnings

"# Saving current state and backing up files (if needed)
| Root dir: /Users/whanners/BitPocket
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
ls: .bitpocket/backups/2013-04-16.071137: No such file or directory
ls: .bitpocket/backups/2013-04-16.071137: No such file or directory
rmdir: .bitpocket/backups/2013-04-16.071137: No such file or directory"

apparently, it also deletes anything I put in my bitpocket folder now when it syncs. Am i missing something here?

How does bitpocket distinguish between delete/create?

AFAIK for this to work you need a list of all files of the last sync.

It would be nice if you could explain your solution in the README.

Client A adds folder X to his BitPocket
Client A and B are both set up to sync every minute and are currently both online.
They start both syncing at the same time.

2 Syncs later folder X is gone.

bitpocket log:

First Sync:
Client B:
sending incremental file list
*deleting X/

Client A:
sending incremental file list
cd+++++++++ X/

Second Sync:
Client B:
sending incremental file list

Client A:
sending incremental file list
*deleting X/

Maybe there should be a "lock file" in the BitPocketMaster/.bitpocket while a client is syncing?

If I create a soft link on a slave machine it is deleted on sync. If I create the same soft link on the master it is created on the slave.

OOops, looks like this has been fixed

I'd like to use bitpocket with rsyncd because this would speed up the sync but using "bitpocket rsync://user@host /dir".

I recommend you use null separated list for the exclude files when generating tree paths.

I've been using bitpocket for a while with great success. I can't seem to track down why
soft links created on my slave machine are deleted when I run a sync. I saw a note referencing a missing RSYNC_OPTS bug so I downloaded the most recent version to test if it fixes the problem. It does not.

I assume this issue is caused by the same reason explained in #24.

Changing persmission on a file on the slave, gets reverted when syncing with master.

Thanks for a great neat little script!

If I create some files on the master and then do a sync on the client, the files appear on the client after the first sync, but they do not get added to .bitpocket/state/tree-prev until the second sync. Consequently, if I delete some of the files on the client between the first and second sync, they reappear on the second sync.

Both ends are running OS X. (10.5 on the master and 10.6 on the client.)

|>ouglas

Can this be done during init command?

Hi, I'm getting this error because I'm using fish as shell:

fish: Expected a command name, got token of type 'Run job in background'. Did you mean 'COMMAND; and COMMAND'? See the help section for the 'and' builtin command by typing 'help and'.
Standard input: mkdir -p "/data/autoadd//.bitpocket/tmp"; cd "/data/autoadd/" && mkdir ".bitpocket/tmp/lock" 2>/dev/null
^
Couldn't acquire remote lock. Another client is syncing with [email protected]:/data/autoadd/ or lock file couldn't be created. Exiting.

btw, awesome tool !!

I haven't tested every permutation of possible conflicts. But my first basic test failed:

1. Create Master with single file as contents: "testing".
2. Sync slave1 with Master.
3. Delete file from Master (or delete from slave2 and then sync slave2 with Master)
4. Modify file on slave1.
5. Sync slave1 with Master.

The file that was just modified (and modified after it was deleted on the Master) on slave1 is removed. Deletion takes precedence over any other changes to the file regardless of the timing of those changes.

Proposed solution: keep a timestamp from last successful sync in the .bitpocket directory and use that plus the time modified of the files to prevent deleting modified files, or at least throw up a question.

I've always been a fan of using passphrase SSH keys even for my automated scripts. The keychain application leverages ssh-agent to manage the keys with passphrases, and enables scripts to access the keys from within cron. I use something like this to source my keys:
type -P keychain &>/dev/null || { echo "I require keychain but it's not installed. Aborting." >&2; exit 1; } eval keychain --noask --eval --agents ssh id_rsa || exit 1
I believe something similar can be done with ssh-agent by itself.

I just think it's important to make people aware that there are easy options for using passphrase-encrypted keys from within cron jobs and scripts.

Thanks!
Scott

unfortunately, OSX Lion compatibility is recently broken....

I get the following errors when I run bitpocket:

 find: -printf: unknown option
 uniq: illegal option -- z
 usage: uniq [-c | -d | -u] [-i] [-f fields] [-s chars] [input [output]]

My bash-skills are insufficient to fix it.. (of course, I could remove the options, but I don't know the consequences...)

I use the latest commit (2013922) and OSX 10.7.3:

$ /bin/bash --version
GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)

bitpocket init fails if the remote path has a space in it (regardless of quotes or escapes). To fix
init $2 $3 $4 needs to be init $2 "$3" $4. Also the $4 here seems like a bug, the function fails if more than 2 args are passed.

Hi,
is bitpocket still maintained ? If not, is there any official fork ?
It's a nice tool

thanks

I'm curious why you instruct people to upload their private keys to the host server in this command:

$ ssh-copy-id -i .bitpocket/id_rsa [email protected]

From what I know of public/private key encryption, you would want to upload the public key and send a key challenge response via the private key. This would greatly reduce the risk of someone on the remote server from getting their hands on your private key. Thus, the following commands would suffice:

$ cd ~/BitPocket
$ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa
$ ssh-copy-id -i .bitpocket/id_rsa.pub [email protected]

Thanks!

This is a feature request.

I would like to write the output that is currently being printed to STDOUT to a log file in a specified directory. The logfile's filename could be the date and time (YYYY-MM-DD_HH:ii:ss.log, or something like that.). This way, as cron executes the sync, it is possible to see failures. and a list of what files were synced, etc.

I want to keep directories in sync, which both reside on my system, so I don't need ssh. I don't know if this makes a difference for this scenario, but at least for completeness' sake.

So, when I set up bitpocket with just a path, it works nicely, as long as this path has no spaces in it. If there are spaces, however, it drops the last chunk of the path and can't complete its mission

For instance, I set up

bitpocket init "/media/newhome/sync test/"
bitpocket sync
rsync: [sender] link_stat "/media/newhome/sync" failed: No such file or directory (2)

Is there a workaround?

Nice little script! Wonder if it'd be useful for anyone to be able to sync multiple directories with one command. Perhaps an env variable like BITPOCKET_DIRS = ("/docs" "/music" "~/.config") or something similar.

Thanks!
Scott

The errors are:

  1) bitpocket locking exits with status 2 when stale lock found
     Failure/Error: max_pid = File.read('/proc/sys/kernel/pid_max').to_i
     Errno::ENOENT:
       No such file or directory - /proc/sys/kernel/pid_max
     # /Users/lunks/Code/bitpocket/spec/locking_spec.rb:13:in `read'
     # /Users/lunks/Code/bitpocket/spec/locking_spec.rb:13:in `block (2 levels) in <top (required)>'

  2) bitpocket sync transfers updated file from local to remote
     Failure/Error: File.read(local_path('a')).should == content
       expected: "foo"
            got: "" (using ==)
     # /Users/lunks/Code/bitpocket/spec/sync_spec.rb:24:in `block (2 levels) in <top (required)>'

I'll try to fix them and make a pull request, but they might break on other platforms. I'm creating this issue just to make the author aware, and make sure I'm doing nothing wrong. :)

One of the issues is that the test is using touch with -d instead of -A, which seems the right way of updating the timestamp. Can someone confirm this is correct also for Linux?

Thanks!

I have a machine in the office which I use bitpocket to sync work between it and my laptop. I had a problem with the machine being down for several days and the files on the laptop weren't properly synced. I removed the lock file on the remote machine, ran bitpocket sync and got an error message that a folder which I copied into two different folders and then deleted on the office machine cannot be deleted locally. I moved the folder to a different location and then the sync went through. However, now I see that the folder only has it's subfolders but no files. I ran bitpocket pack on the laptop, and I can see the files in the .bitpocket/pack directory, when I run git log filename I get some output showing there was some action going on last time I was working on these files, but when I run git diff filename I get empty output.

1. I think that the copying the folder as empty is a bug. I'm not sure how to recreate this.
2. Is there any hope that the files are salvageable?

One of the big strengths of Dropbox is that it's speedy and it syncs the current files. The problem with other systems (such as Sparkleshare, also based on git) is that it does a complete sync of the repository, automatically syncing revisions as well. This means that your usable size can be 20MB but the size of the repo can be as large as 1GB due to all revisions.

You would be able to differentiate from the competition by implementing this, becoming a speedy and stable sync app.

When using shellcheck there some warnings. Is there any interest in PRs fixing those?