Comments (10)
Hi! Thanks for the kind words.
Running as a daemon definitely makes sense, but just isn't something I've gotten to needing/building.
Daemonizing the mirror server
command would be pretty easy, b/c there is little config for it. But mirror client ...
has a number of options that means mirror would probably need to learn about config files. E.g. setup a mirror.conf
for the client daemon to read in.
Which would be great, but again just haven't gotten around to it.
I don't have a ton of free/hacking time lately, so personally won't get a chance to work on it. If you want to hack around with something like:
https://commons.apache.org/proper/commons-daemon/
Or maybe:
https://github.com/brianm/gressil
I can provide feedback on PRs/etc.
from mirror.
Thanks for the reply.
Unfortunately I'm not dev, just sys admin guy who is looking for some solution for realtime sync between 2 or more servers. For the moment I'll be looking for another solution, but definitely will keep an eye on the project.
from mirror.
Hey @shinebayar-g ! At our organisation we've basically followed the same path as you've described (Unison) and hit the same downsides (lack of real-time sync). As well as same conclusions regarding lsyncd :D
We simply packed mirror into a custom docker image and we're running this on each server. We've organised it in start topology (1 server and many clients). So far works like a charm, but it's not handling any severe production loads yet. We sync whole /var/www path with it, that stores deployed code (with Unison we've synced particular paths in that root)
from mirror.
@shinebayar-g You can use the following commands:
nohup /path/to/mirror/mirror server --enable-log-file > /dev/null 2>&1 &
nohup /path/to/mirror/mirror client --enable-log-file -h servergoeshere.com -l /testsync/ -r /testsync/ > /dev/null 2>&1 &
from mirror.
Funnily enough, I just finished writing a init.d script before looking here to put forward my own feature request, so I may as well post my (admittedly somewhat amateurish) attempt to daemonise:
Assuming mirror is installed in the path,
$ nano /etc/init.d/mirror-server
and put this in:
#! /bin/sh
#
### BEGIN INIT INFO
# Required-Start: $local_fs $remote_fs $network $java
# Required-Stop: $local_fs $remote_fs $network $java
# Provides: mirror-server
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: File tree sync mirror server
# Description: File tree sync mirror server
### END INIT INFO
NAME="mirror-server"
DESC="File tree mirror service"
SCRIPTNAME="/etc/init.d/$NAME"
SHUTDOWN_HELPER="pkill java"
# Load the variables
. /etc/mirror/mirror-server.conf
OPTIONS=""
if [ "$SKIP_LIMIT_CHECKS" = "TRUE" ] ; then
OPTIONS="--skip-limit-checks"
fi
#additional option logic to be built here
case "$1" in
start)
mirror server $OPTIONS > /var/log/mirror.log 2>&1 &
#change the first > to >> if you want to log everything ever
echo "mirror service started"
;;
stop)
pkill java >null 2>&1
pkill mirror >null 2>&1
# this is obviously overkill and may kill needed processes
# but I don't see a graceful shutdown option available yet
# there is definitely room for improvement here
;;
status)
if ps -Al| grep mirror|grep -v "mirror-server" >null 2>null ; then
echo "mirror server running"
else
echo "mirror server not running"
fi
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|status}" >&2
exit 3
;;
esac
:
exit 0
$ nano /etc/mirror/mirror-server.conf
#! /bin/sh
# uncomment this line to enable it
# other variables can go here, but must also be accounted for in the script
#SKIP_LIMIT_CHECKS=TRUE
$ chmod 755 /etc/init.d/mirror-server
$ chmod 755 /etc/mirror/mirror-server.conf
$ systemctl enable mirror-server
from mirror.
@fezzzza nice, thanks!
from mirror.
Well, that at least works to give me an autostart on the server although someone more experienced with Linux than me pointed out that I'm actually running through systemd, though I guess it's backwards compatible with this init script. The client script I have devised so far does not work on autostart because it's trying to resolve DNS before the internet connectivity is fully up, but it seems to work at least manually. Here it is:
mirror-client can go in /usr/local/bin or your preferred location alongside the mirror executables which need to be in your $PATH:
#! /bin/sh
#
### BEGIN INIT INFO
# Required-Start: $local_fs $remote_fs $network $java $named $all
# Required-Stop: $local_fs $remote_fs $network $java $named $all
# Provides: mirror-client
# Default-Start: 3 5
# Default-Stop: 0 1 2 4 6
# Short-Description: File tree sync mirror client daemon
# Description: File tree sync mirror client daemon thru ssh tunnel
### END INIT INFO
NAME="mirror-client"
DESC="File tree mirror client service"
SCRIPTNAME="/etc/init.d/$NAME"
SHUTDOWN_HELPER="pkill java" #I have no idea whether this is correct
# Load the configuration
#root has them in /etc/mirror/mirror.conf, logs to /var/log/mirror.log
#user has them in ~/.mirror/mirror.conf, logs to ~/.mirror/mirror.log
LOGDIR=~/.mirror/
CONFDIR=~/.mirror/
if [ "$EUID" = 0 ] ; then #ie if we are root
LOGDIR=/var/log/
CONFDIR=/etc/mirror/
fi
echo $CONFDIR
LOGFILE="$LOGDIR"mirror-client.log #set a default that may be overwritten by CONFFILE load
CONFFILE=$2
if [ "$2" = "" ] ; then
CONFFILE="$CONFDIR"mirror-client.conf
fi
. $CONFFILE
case "$1" in
start)
echo "mirror start control received" >> $LOGFILE
echo "ssh -f -L $CLIENTPORT:localhost:$SERVERPORT $USERNAME@$SERVERNAME -i $KEYFILE -N"
ssh -f -L $CLIENTPORT:localhost:$SERVERPORT $USERNAME@$SERVERNAME -i $KEYFILE -N > $LOGFILE 2>&1
SSHERRORLEVEL=$?
if [ $SSHERRORLEVEL != 0 ] ; then
echo "ssh tunnel failed - see $LOGFILE"
fi
if [ $SSHERRORLEVEL = 0 ] || [ "$FORCESTART" = "TRUE" ] ; then
echo "mirror client $OPTIONS $INCLUDEOPTIONS $EXCLUDEOPTIONS -h localhost -l $CLIENTDIR -r $SERVER$
mirror client $OPTIONS $INCLUDEOPTIONS $EXCLUDEOPTIONS -h localhost -l $CLIENTDIR -r $SERVERDIR >>$
if [ $? = 0 ] ; then
echo "mirror client started"
echo "mirror client started" >> $LOGFILE
else
echo "mirror client not started - see $LOGFILE"
echo "mirror client not started" >> $LOGFILE
fi
else
echo "start mirror not attempted"
echo "start mirror not attempted" >> $LOGFILE
fi
;;
stop)
#beware - if you have multiple mirror instances, this will kill them all
echo "mirror stop control received" >> $LOGFILE
PID=1 # until doesn't work quite as expected without this
until [ "$PID" = "" ] ; do
PID=$(ps -Al|grep mirror|grep -v "mirror-client"|awk '{print $4}')
if [ "$PID" != "" ] ; then
echo "killing mirror with PID $PID" >> $LOGFILE
kill $PID >> $LOGFILE 2>&1
fi
done
if ps -Al| grep java > /dev/null ; then
echo "killing java" >> $LOGFILE
pkill java >> $LOGFILE 2>&1
# this is obviously overkill and may kill needed processes
# but I don't see a graceful shutdown option yet
fi
if ps -Al| grep ssh|grep -v "ssh-agent" > /dev/null ; then
echo "killing ssh" >> $LOGFILE
pkill ssh >> $LOGFILE 2>&1
fi
echo "all mirror processes killed" >> $LOGFILE
echo "all mirror processes killed"
;;
status)
if ps -Al| grep ssh|grep -v "ssh-agent" > /dev/null ; then
echo "ssh session detected - it might be our tunnel"
else
echo "ssh tunnel not found"
fi
if ps -Al| grep mirror|grep -v "mirror-client" > /dev/null ; then
echo "mirror client running"
else
echo "mirror client not running"
fi
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|status} alternative-file.conf" >&2
exit 3
;;
esac
:
exit 0
mirror-client.conf either belongs in ~/.mirror or /etc/mirror if you are running it as root to daemonise:
#! /bin/sh
#
USERNAME=root
SERVERNAME=www.myserver.com
CLIENTPORT=49172
SERVERPORT=49172
CLIENTDIR=/var/www/html
SERVERDIR=/var/www/html
INCLUDEOPTIONS="--include file.ext --include example.dir/"
EXCLUDEOPTIONS="--exclude dir/file.ext --exclude log/"
#put as many include/exclude arguments as you need, just remember
#to include an "--include" or "--exclude" prefix for each one
#use of quote marks to delimit the file names seems to cause failure
#these options can easily be switched on by commenting or uncommenting
#Some require arguments. See "mirror help client" for details
#OPTIONS="$OPTIONS --enable-log-file"
#OPTIONS="$OPTIONS --debug-all"
#OPTIONS="$OPTIONS --debug-prefixes LIST_OF_PATHS" #replace LIST_OF_PATHS with your list
#OPTIONS="$OPTIONS --skip-limit-checks"
#OPTIONS="$OPTIONS --use-internal-patterns"
#FORCESTART=TRUE
#include this line if mirror should start regardless of SSH tunnel success
KEYFILE=/root/.ssh/id_rsa
#this assumes you have already set up your ssh authorisation
#by creating an SSH key pair using, eg, ssh-keygen
#and distributed the public key to the server in .ssh/authorized_keys
#LOGFILE=~/.mirror/mirror.log
#LOGFILE=/var/log/mirror.log
#this will override the default
Don't forget to make them both executable as before.
I'm abandoning any hope of getting this to autorun right now because my use case is a roaming laptop that will be ducking in and out of internet connectivity, and the 2 big unanswered questions for me are:
- What happens to the ssh tunnel when internet connectivity drops and reconnects?
- What happens to mirror when internet connectivity drops and reconnects?
...and right now for me I have bigger fish to fry, so I'm happy enough starting and stopping the client manually. That said, it would be nice if this sort of state management could be handled eventually. Perhaps it would make more sense to deal with both problems at once if/when request #10 gets implemented.
Probably the wrong place to put a feature request, but seeing as I'm sharing my progress, just to keep things in one place:
I need to connect one workstation to multiple servers, dependent on which set of code I'm modifying, and don't believe that's possible at the moment. A simple fix would be to be able to have multiple instances of mirror running and specify the port number in the options (both client and server side) and thereby communicate with each server via multiple ssh tunnels, each with its own port number. As I'm handling connection state manually for the time being, I'll just start and stop whichever connection suits by modifying the above to select between specified .conf files. I've just edited the code above to show the variant for those who are interested. Anyone using this please be aware I'm under no delusion that this is "good" code, so use with caution!
Then the icing on the cake would be to implement rsync to minimise network traffic, but I know you've already put that down as a non-goal... just my 2p ;)
Thanks, Stephen, for your work so far - it's made my job as an amateurish webdev much easier.
from mirror.
What happens to the ssh tunnel when internet connectivity drops and reconnects
FWIW http://www.harding.motd.ca/autossh/ might be "good enough" to keep auto-detect/reconnect the tunnels for you.
What happens to mirror when internet connectivity drops and reconnects
The tunneling aside, mirror should notice the disconnect, and repeatedly ping for when the reconnect is back, then reconnect and resync. It should generally just work. The only caveat is that if both sides have drifted way out of sync, mirror does a ~dumb "last-write-wins" union/merge.
Admittedly, I should really get around to implementing this; personally I boot up my server and laptop and then: 1) manually run ./mirror server
and 2) manually run three ./mirror client
commands for the three directories I sync between the two. It would be nice to have that just happen. Have a few other things I'm busy with, but will nudge it up on my list.
from mirror.
Thanks. I have updated my scripts above that you may freely incorporate into your package (you can see the edit history for older versions) and will further update it for autossh if and when I get a chance. Same warning goes though - I'm not really a bash programmer by any stretch of the imagination, so take nothing for granted that this is "good" code! However I've hit a bunch of problems that I've filed issues for, some of which are making it impossible for me to continue with mirror for the time being. Would love to come back to it if you get time to deal with these issues as it's wonderfully lightweight. Best of luck!
from mirror.
@fezzzza thanks for the scrips, any update on these as well ?
/edit
Please update your copy/paste code as it's missing some quotes :)
from mirror.
Related Issues (20)
- "OutOfMemoryError: Java heap space" error while running mirror HOT 4
- Support very large files HOT 1
- Deleting a directory hierarchy leaves empty directory
- ignoring directories with many files HOT 2
- About package name HOT 3
- Can we use the diff port instead of '49172'? HOT 5
- How to keep the file permissions? HOT 1
- CentOS docker image HOT 2
- Question: What causes lag between watchman and mirror sending a file? HOT 4
- java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.facebook.watchman.WatchmanException: unknown command watch-project HOT 1
- logs are not rolling. HOT 2
- exclude file pattern issue HOT 3
- Deadline exceeded after 3 minutes HOT 8
- Issue with bulk file uploads HOT 1
- restricting server to only one interface HOT 7
- Output per-file status info to systemd/syslog HOT 1
- Specify log file path? HOT 3
- Mirroring directories on same server (Docker) HOT 1
- How do I debug mirror? HOT 1
- on debian client fails HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mirror.