Mirror is built to support a two-machine (e.g. desktop+laptop) development workflow where you want to run a command line compile/build process on a powerful/dedicated desktop, but still edit files remotely on a laptop.
This is fairly common (see "Comparison to Existing Options" section below), but what makes Mirror unique is that it is two-way: it simultaneously syncs both laptop-to-desktop as well as desktop-to-laptop, in real time.
For my personal use case, this is to facilitate using an IDE on the laptop (e.g. for code completion, navigation, etc.), and IDEs often need local access to the binary artifacts (or build time-generated source code) from the desktop-hosted build process, e.g.:
- On your laptop, save
projectA/foo.java- Mirror sends
foo.javato the desktop
- Mirror sends
- On your desktop, the build system picks up the
projectA/foo.javachange and createsprojectA-snapshot.jar- Mirror sends
projectA-snapshot.jarback to the laptop
- Mirror sends
- On your laptop, the IDE can now use
projectA-snapshot.jarfor code completion/etc. when editingprojectB/bar.java
Granted, the IDE will also do local compilation of projectA/foo.java, so ideally the IDE could use in-workspace references, where the IDE-compiled projectA/foo.java is already/immediately on the IDE classpath for projectB. If you can setup your projects this way (e.g. with m2e or IvyDE), that is generally preferable.
However, for larger/more complex projects, e.g. those with various pre-/post-compilation code generation steps (that are often only performed in the CLI build), the IDE just can't reproduce the build process closely enough to fully compile projectA on it's own, and so using CLI-provided projectX-snapshot.jar artifacts is the only way to do local cross-project imports.
This scenario (local edits + remotely-built cross-project artifacts) is what Mirror addresses.
- Real-time, two-way sync between a desktop development machine, and an editor-/IDE-only laptop
- Native file system events (e.g. inotify) fired when files are changed (otherwise IDEs require explicit refreshes/polling)
- Unison-style/long-duration disconnected support
mirrorwill automatically re-connect (e.g. if you close your laptop and then go home) and restart syncing when it detects the server is available again (inspired by mosh), but if files have changed on both sides while disconnected, then the last write wins- This hueristic is generally fine, it just means
mirroris not meant for a use case of "make new changes on the desktop for a few days, make new changes on the laptop for a few days, and then runmirroronce per week to intelligently merge your work". Usegitorunisonfor that;mirroris for real-time syncing.
- Maintain Unix permissions/owner/group
- Whatever Unix user runs the
mirrorcommands will be the owner/group/etc. of the files
- Whatever Unix user runs the
- Support for huge files
- The assumption is that most files are source code, and occassional binary artifacts that are generally in the below-100mb range
- Super-efficient diff/transmission logic like rsync
- Instead we assume a generally fast network connection (as in "faster than a modem", i.e. mirror works fine over a VPN)
- Basically, if a file changes,
mirrorretransmits the whole file instead of trying to diff only what changed
I looked at several sync options before starting mirror, but didn't find anything that quite fit:
- X-Forwarding (to just run everything, IDE included, on the desktop) has noticeable lag, even on a LAN
- rsync is not two-way, nor real-time
- unison is two-way, but not real-time
- lsyncd is real-time, but does not officially support two-way (see issue 303)
- sshfs is too slow and doesn't support inotify
- NFS and other network file systems don't support inotify
- SyncThing is a dropbox alternative if you're looking for a backup solution
- doppleganger (an internal tool) is real-time, but not two-way
- It also generally assumes you can run the build tool (e.g. gradle) on both laptop and desktop, and for my setup I want to only run it on the desktop
- Similarly, doppleganger does let you have a git working copy on both laptop/desktop, but so far mirror generally assumes you have a git copy on only one of the desktop or laptop (your choice) and then the other side is just a dumb copy. Which means git commands, like
git log,tig, etc., will only work on whatever machine you have your git check on.
To use the latest release/pre-built jars:
- Install Java 8
- You can try running
java -versionto see if Java is already installed - If not, a you'll need a platform-specific installation step, like
sudo apt install openjdk-8-jreon Ubuntu
- You can try running
- Install watchman
- Download the latest mirror and mirror-all.jar to your home directory (or some other directory on your path, e.g.
~/bin)wget https://github.com/stephenh/mirror/releases/latest/download/mirror-all.jar ~/wget https://github.com/stephenh/mirror/releases/latest/download/mirror ~/
- Make the
mirrorfile executablechmod u+x mirror
- Copy both to your remote home directory (or some other directory on your path, e.g.
~/bin)scp mirror-all.jar your-desktop.com:~/scp mirror your-desktop.com:~/
- Start the server-side from the desktop's home directory
./mirror server
- Start the client-side from the laptop's home directory
./mirror client -h your-desktop.com -l ./code/ -r ./code/- Note: Be careful using the tilda (e.g.
~/code), as your shell will resolve that, e.g. to/Users/you/code, and that resolved path on the client might be not valid on the server
This will sync the $HOME/code directory on your two machines.
(Note: for Arch Linux users, AUR has the mirror-sync and mirror-sync-git packages.)
By default, mirror will not sync any files in your .gitignore files.
However, you can also configure mirror with extra includes or excludes in addition to the .gitignore, e.g. if for some reason you want to not sync files that are not ignored, or sync files that are actually ignored (e.g. certain build artifacts).
Extra includes and excludes patterns can be passed when starting the client, and follow the .gitignore format, e.g.:
./mirror client --include '*-SNAPSHOT.jar` --include '.classpath' --include '.project' --exclude `build/`
If you'd like mirror to completely ignore your .gitignore files, i.e. to sync everything, you can use --include '*'.
(There is also a --use-internal-patterns that has useful defaults if you work at the same place I do.)
Options for both the client and server commands are available by running:
`./mirror help client`
Or:
`./mirror help server`
For example, the client supports --debug-all and --debug-prefixes command line parameters to output debug info for all or subsets of the paths (setting these parameters on the client session will pass them to the client's server session, so that the server will use those options for its own session with that specific client).
(Update February 2020: Lately I have been ignoring this advice and passing -i .git to have both sides sync their .git metadata so that staging/index/etc. status is all shared, and it's been working out really well so far.)
In general, mirror will work best if you have just one machine (either the desktop or laptop) have the git (or svn/other SCM) working copy (or working copies if you're syncing a directory like ~/code with multiple repositories), and have the non-git machine just get all of it's files via mirror from git-using machine, and not by also having it's own git working copy.
The reason is that if you had git working copies on both the laptop and desktop, you could run into:
- Desktop:
git checkout master - Laptop:
git checkout master - Run
mirror, everything syncs fine, because they're on the same branch - On the laptop, run
git checkout feature_a mirrorcopies all the files changes on the laptop (basically allfeature_a's changes) to the server- Now the server git directory thinks it's still on
master, but we've written thefeature_afiles to it
Basically, mirror makes no attempts to keep your two git working copies on the same branch, and instead just naively copies files back/forth.
So, instead, it works better if you only ever run git commands on the desktop, and so any branch changes happen there, and then the laptop just follows/copies where ever the server is at.
The downside of this approach is that you can't use git log, git blame, etc. on the laptop. But the upshot is that's pretty simple, and makes it harder/less likely to accidentally nuke code by either knowingly or unknowingly getting the two machines on different branches, and then having one overwrite the other.
Note that if you have a lot of directories, you might have to increase the native file system limits, e.g.
- For Linux see this readme
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -pecho fs.inotify.max_queued_events=50000 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
- For Mac, see the watchman docs
If you have an extremely large number of files that you don't want to sync, .gitignore may not be enough, because mirror and the underlying Watchman tool will still initially load those files into memory, even to decide "oh right, don't sync them".
To have Mirror and watchman fundamentally ignore things, you can create a .watchmanconfig file with the ignore_dirs property set.
See the watchman config.
Although not an initial design goal, due to it's approach, mirror also supports a hub and spoke model of syncing more than two machines.
E.g. you could have:
- Desktop runs the mirror server process (you don't need to start multiple
mirror serverprocesses) - Laptop 1 connects to the desktop and syncs it's
~/codeto the desktop's~/code - Laptop 2 also connects to the desktop and syncs it's
~/codeto the desktop's~/code
Now all three machines will be kept in sync.
If you're running most build commands on your desktop, but the IDE on your laptop, your .classpath/etc. files will likely have references to downloaded jars, e.g. in the Maven cache or Gradle cache directories.
Since these files are cached and so don't change, and aren't created very often (only when dependencies are updated), I currently sync these jar caches as needed with a shell script:
rsync -azP [email protected]:.ivy2/sbt/ ~/.ivy2/sbt/
rsync -azP [email protected]:.gradle/caches/ ~/.gradle/caches/In theory mirror could keep these in sync as well, either by just running ~2-3 more invocations of the mirror client, but it would also be nice to pass in multiple sync directories in a single mirror command invocation, e.g.:
mirror \
--sync ./.ivy2/sbt:./.ivy2/sbt \
--sync ./.gradle/caches:./.gradle/caches \
--sync ./code:./codeBut currently mirror only supports a single remote/local sync at a time.
mirror currently uses plain text for its communication protocol, as the primary use case is syncing a desktop/laptop that are on an assumed-secure internal network or VPN connection.
The underlying RPC framework, GRPC, supports TLS communication, but mirror currently does not leverage that.
If you need secure communication, e.g. are syncing across the open Internet with sensitive data, you can use SSH tunneling, e.g.:
- On your client, run
ssh -L 49172:localhost:49172 your-remote-host - On your remote host, start the server as usual,
./mirror server - On your client, run mirror with
./mirror client -h localhost ...
This will have the mirror client send traffic to the localhost:49172 port, which SSH will securely tunnel to your remote host.
A docker image is available at quay.io/stephenh/mirror.
Since the container will have its own filesystem separate from the host's filesystem,
usually you'll want to mount some directory into the container to make it available for
synchronisation.
To mount the current working directory as /data into the container, pass -v $(pwd):/data to docker.
By default docker runs processes as root, which results in all written files owned by root on the host system.
To run the process as your current user, pass -u $(id -u):$(id -g) to docker.
To start a mirror server available on port 49172, with the current working directory mounted at /data run:
docker run --rm --init -it -u $(id -u):$(id -g) -v $(pwd):/data -p 49172:49172 \
quay.io/stephenh/mirror serverTo start a mirror client with the current working directory mounted at /data (and syncing the local /data with the remote /data) run:
docker run --rm --init -it -u $(id -u):$(id -g) -v $(pwd):/data \
quay.io/stephenh/mirror client \
--local-root /data \
--remote-root /data \
--host <SERVER-HOST>If you want to hack on mirror locally, you should be able to:
- Clone this repository
- Run
./gradlew shadowJar- This will download mirror's dependencies and produce an all-in-one jar in
build/lib/mirror-all.jar
- This will download mirror's dependencies and produce an all-in-one jar in
- Run
./mirror ...(e.g. eithermirror clientormirror server- The
mirrorscript in the base directory should pick up your locally-builtbuild/lib/mirror-all.jar
- The
If you want to use your locally-built jar on your remote host, you'll need to scp the new mirror-all.jar to your remote host (in the same directory as the mirror script, which will then use your new mirror-all.jar instead of the previously-downloaded version.)
- Configuration via a
.mirrorrcfile (if necessary) - Really easy setup/install process, e.g.:
- Currently you have to run
mirror ...andmirror ...on both sides - Ideally you could just run
mirroron the client, and it would self-start the desktop-side process by SSH'ing the jar over - Maybe the server-side should be an always-running daemon? Same thing with the client-side?
- Currently you have to run
- Really easy upgrade process
- Ping
repo.joist.ws/mirror-version.txt, and if a new version, exit with a code that tells bootscript script to redownload the jar
- Ping
- Include file hashes to avoid sending files with different modification times
- See Digest for an experiment; collecting digests does slow down the initial scan, but not egregiously
- Currently needless syncs don't seem that frequent/expensive (text files are small), so this is low priority
- Use file hashes to detect renames
- Right now renaming
~/code/project-ato~/code/project-bwill treat every file as a delete+create - So far seems like a limited use case, and also is still handled pretty quickly, so this is also low priority
- Right now renaming
- Support
svn:ignore- Would be easiest by converting it in-memory to
.gitignore-format via something like this - Or else just suggest users use git-svn for all svn repos
- Would be easiest by converting it in-memory to
- Support
.git/info/exclude?- Not sure how often this is used
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent