Code Monkey home page Code Monkey logo

instancesync's Introduction

InstanceSync, Resource Cloning

Date: 2013-11-16
tags:migrate, clone, sync, rackspace, aws, hpcloud
category:*nix
Version: 2.0.1

The use cases

  • Migrating from one provider to another
  • Cloning an instance
  • Upgrading Infrastructures
  • Setting up Geographic Redundancies

What I have tested

  • Rackspace Instances from XenClassic To XenServer
  • Rackspace Instances from XenServer to Open Cloud powered by OpenStack
  • Rackspace Instances from XenClassic To Open Cloud powered by OpenStack
  • Amazon EC2 to the Rackspace Cloud
  • KVM to Rackspace Cloud
Caveats :
There are two issues I have found while migrating instances around. However, the most important caveat was related to the instance type. You Must Have a Similar Instance to Migrate To. Additionally to migrate a linux server from one place to another you must also have setup the instance to use a Single partition for installation. If you are using a multi-partition virtual instance, you will have to manually migrate the partitions accordingly. Which can be successfully accomplished by migrating the instance by hand. Other than the one caveat, of having Similar Instance, I have not had this process fail.
Testament that It Works :
I know that this method works for a variety of situations, I have even had this process complete successfully when migrating an instance that uses Amazon AMI Linux.

Available Options

All options can be set as exports in the environment which the script will use with attempting to perform the Sync Operation.

Enable More Verbosity in the script, Default is "False":
VERBOSE="False or True"
Enable Debug Mode in the Script, Default is "False":
DEBUG="False or True"
Add Additional Excludes to the script when Performing the Sync. This is a Space separated list, if you have files with spaces in the name you MUST escape them.
USER_EXCLUDES=""
Change the Retry Max, Default is 5. This is useful on systems with High Load averages, which can cause the Sync Process to die.
MAX_RETRIES="5"
Enable Inflammatory mode, Default is "False":
INFLAMMATORY="False or True"

Estimated time of Completion

The Estimated time to completion based on Gigabytes of Consumed Space Computations have been made using an average transfer rate of 20 Megabytes a second.

The Estimated time
Space Used EST Time
10G 9 Minutes
20G 17 Minutes
40G 34 Minutes
80G 68 Minutes
160G 136 Minutes
320G 272 Minutes
620G 544 Minutes
1200G 1088 Minutes
Do you want to see the script in Action:
I have a screen cast of a migration for a Live Instance to another instance. You can check it out here.

Foot Notes

When Performing a sync you should attempt to minimize load on your "Source" machine. The Sync script works great on a live system though if the load is too high it could cause the sync process to die.

When performing a sync on a system with a database, be aware that if you are writing a lot of data to the database you may want to shutdown the database engine while the sync is in process. By shutting down the database you can be sure that all of the data in the database was cloned without any sharding.

instancesync's People

Contributors

cloudnull avatar danijeljw avatar joeashcraft avatar mystcb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

instancesync's Issues

screen cast link no longer works.

README.rst points to a screen cast of a migration [http://ascii.io/a/1063] - this link is no longer valid. Is there a good link/home for the video (or a newer rev?) Thanks

Syncing large databases (2GB+) causes rsync to hang

When syncing a large MySQL database, rsync hangs at transferring the ibdata1 file.

This happens because the compression option, "z" is enabled in the script on the rsync options.

Perhaps a notification about disabling compression for the rsync transfer may be helpful for some users transferring large databases?

Exclude /run, /proc and /var/run?

I noticed in the exclusion list that /run, /var/run and /proc are not excluded when running rsync. Shouldn't they be, at least on Debian/Ubuntu machines?

System detect does not fail properly.

When running the script if the system detect goes through all supported operating systems and finds nothing that matches in the file /etc/issue the script hangs and does nothing. This should be trapped and fail more appropriately.

Rackspace to Amazon EC2

Hello there, thanks for making this tool. Do you believe this tool will work for moving instances from Rackspace to Amazon EC2 migration? I noticed this direction of migration was not mentioned in the README.

Swap file not being created in Fedora

Reported by Steve:
I've used your script to migrate Ubuntu to Ubuntu successfully several times. Today I tried it for Fedora 19 to Fedora 19 and had problems...

What appears to happen is that the target system runs the script to create /tmp/SwapFile, but the file winds up with group and (I think) world read privilege. When swapon is invoked, it grumbles about wanting 0600 privilege and dies. This brings the entire sync process to a screeching halt.

OS type warning

I am getting the following error: I could not determine your OS Type. This Application has only been tested on :Debian, Ubuntu, Fedora, CentOS, RHEL,
I am running RHEL 6.5 Is this because the latest version is still just being built out? I am hoping to use this script to transfer and appreciate the help.

Selinux enabled on destination causes errors

I have used this script many time with great success, all going to Rackspace, now I am using the destination with a fresh default install. I have twice forgotten to disable Selinux which caused many files not to be sent.

I suggest that a warning is issued or failure of the script is selinux is enabled on the destination

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.