arguslab / ancor Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 4.0 1.13 MB

A Moving Target Defense Platform powered by ANCOR

Home Page: http://arguslab.github.io/ancor

License: GNU General Public License v3.0

Ruby 91.11% HTML 4.31% Shell 1.34% Python 3.24%

ancor's People

Contributors

Stargazers

Watchers

Forkers

weizhou-ccnu lwycool swami2611 cybercim1

ancor's Issues

Replace bootstrap system

Right now we use a single shell script to install Puppet, MCollective, set APT mirrors, etc. The IP addresses and credentials are all hard-coded and the script will only work for a single run. I'd like to replace that with something more flexible.

Implement webhooks for initialize & configuration

Add controller to Rails for webhooks

OsNetworkService should not delete networks with subnets

If a network on Neutron has subnets left over, it should not be deleted.

Implement MCollective registration agent for webhook

Finalize instance deployment task

Split out security group management form DeployInstance task into its own task
Write unit test for DeployInstance
Write integration test for DeployInstance (see https://github.com/ianunruh/ancor/blob/master/spec/tasks/integration_spec.rb)

Provider endpoints should contain domain names

For cases when we need to be able to compute the FQDN of an instance (such as when approving CSRs on Puppet), we will need to somehow figure out the private domain name that the cloud provider assigns to instances.

Examples:

OpenStack uses some-instance.openstacklocal by default
AWS uses ip-10-245-81-136.ec2.internal

Implement configuration tasks

InitializeInstance
PushConfiguration

Model instance verification

During instance deployment, it should be possible to verify that an instance was correctly configured.

Deployment workflow w/ Sensu

Add client definition, check definitions and metric check definitions to Sensu
Wait for configuration to be pushed to instance
Wait for notifications of OK check events

Termination workflow w/ Sensu

Remove client definition, check definitions and metric check definitions from Sensu

Check mechanism

For this process to work, we have to be able to model what service checks we want to define in Sensu. There are two parts to a check:

The check definition on the Sensu monitoring server (this is in JSON)
The check itself on Sensu clients (arbitrary executable, probably Ruby)

We have to distribute the checks via Puppet. The check definitions can be managed via Puppet as well. Results of the check can be distributed in a number of ways, including over RabbitMQ via AMQP. After instance deployment is finished, check failures can be used to notify the adaptation engine of issues.

Metric mechanism

Metrics work in nearly the same way. Metric check definitions are setup on the monitoring server and metric checks are deployed to the clients.

Unlike service checks though, we don't need to wait on metric check results. These checks are just used as a feedback mechanism for the adaptation engine.

Resources

http://docs.sensuapp.org/0.12/index.html
https://github.com/sensu/sensu-community-plugins

Implement Puppet report processor for webhook

http://docs.puppetlabs.com/guides/reporting.html

Implement Hiera HTTP endpoint

Expose the following via /hiera/:fqdn:

Exported channels (channel selections)
Imported channels (instances that export the imported channel)
Data to apply to an instance (from scenario)
Data to apply to an instance (from scenario stage)
Classes to apply to an instance (from scenario stage)

On the Puppet master, configure Hiera with this backend:

https://github.com/crayfishx/hiera-http

Implement a basic CLI

Should support these basic operations:

Import requirement model
Plan deployment
View pending deployment
Launch deployment
View tasks
View instances, roles
Add instance to role
Remove instance from role

Explore the idea of task intents

When a task is executed, could we execute it with an optional "intent"? This would be similar to Android's activity semantics.

Using this, we could know in the task WHY the task was executed. For example:

We could be returning to a task because of a timeout
We could be returning to a task because another task failed (or an external system failed, like a Puppet run)
We could be returning to a task because something good happened (task success)

Right now, the only way we could know is by querying somewhere to find out the reason why the task was executed.

Roles not removed when environment deleted

Mongo DB still has leftover entries from the previous deployment when environment remove is invoked. This is an issue for when a second deployment needs to happen after an environment is removed.

Implement update security group

OsSecurityGroupService should be idempotent

Model floating IP addresses

We'll need to be able to map floating IP addresses (FIPs) to instances of specific roles. We should keep in mind the following points:

FIPs should be allocated as needed from the available pool
FIPs should be able to be static (I want this role to use this FIP/these FIPs)
If FIPs are dynamic, we should be able to update DNS zones (push to external system, maybe with Puppet + BIND or similar)

Instance replacement workflow w/ floating IPs

Create the new instance
Allocate a new FIP, assign it to the new instance
Update DNS zone
Wait for DNS to propagate (??)
Disassociate old instance from FIP, deallocate it
Terminate the old instance

In the future, if we consider client-side resolution, we don't have to wait as long for propagation. We'll still use a push-based model for propagation, however.

This should look pretty similar to how we handle updating dependent instances without interrupting services. Bring up the new instance, wait for all dependents to switch over to the new instance, kill the old one.

Existing floating IPs not being reused

A new floating IP is generated even if an old one is available e.g., remove load balancer and add a new one

Associate tasks with Sidekiq job id

It might be useful to associate tasks with job IDs on Sidekiq. Whenever TaskWorker.perform_async(task_id) is invoked, it will return a unique key that can be used to cancel or check the status of that specific job on Sidekiq.

Implement destructive tasks

The following resources should have tasks for termination:

Instance
Network
Security group

OpenStackSecurityGroupService needs to have #delete implemented.

Deployment hangs on PushConfiguration tasks

After doing a new deployment with the committed changes...ancor ends up having the push config tasks being suspended and the allocate public ips pending. Waited 1 hour for things to complete with no progress. Was replicated a second time today. Here is the task log.

vagrant@ancor-precise64:/vagrant$ ancor task list
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | id                       | type                                    | state     | updated_at           |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000b7 | Ancor::Tasks::AllocatePublicIp          | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000b9 | Ancor::Tasks::AllocatePublicIp          | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000bb | Ancor::Tasks::AssociatePublicIp         | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000bd | Ancor::Tasks::AssociatePublicIp         | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000b6 | Ancor::Tasks::Sink                      | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000c0 | Ancor::Tasks::UnlockEnvironment         | pending   | 2014-04-03T15:15:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000070 | Ancor::Tasks::ProvisionNetwork          | completed | 2014-04-03T15:15:41Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000072 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:15:45Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000074 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:15:48Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000076 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:15:52Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000078 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:15:56Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400007a | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:15:59Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400007c | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:02Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400007e | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:06Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000080 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:09Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000082 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:13Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000084 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:16Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000086 | Ancor::Tasks::CleanPuppetCertificate    | completed | 2014-04-03T15:16:19Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000088 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:20Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400008a | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:20Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400008c | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:21Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400008e | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:21Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000090 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:22Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000092 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:22Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000094 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:23Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000096 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:23Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac4000098 | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:24Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400009a | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:24Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400009c | Ancor::Tasks::SyncSecurityGroup         | completed | 2014-04-03T15:16:24Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400006f | Ancor::Tasks::Sink                      | completed | 2014-04-03T15:16:25Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000001 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:29Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000003 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:33Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000005 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:36Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000007 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:39Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000009 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:42Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b7122804500000b | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:46Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b7122804500000d | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:49Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b7122804500000f | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:53Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000011 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:58Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000013 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:17:01Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b493b71228045000015 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000017 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000019 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b7122804500001b | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b7122804500001d | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b7122804500001f | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000021 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000023 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000025 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000027 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b71228045000029 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b703b7122804500002b | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b7122804500002d | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:14Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b7122804500002f | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:22Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000031 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:30Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000033 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:38Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000035 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:45Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000037 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:17:53Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000039 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:18:02Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b7122804500003b | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:18:10Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b7122804500003d | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:18:18Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b7122804500003f | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:18:26Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b713b71228045000041 | Ancor::Tasks::ProvisionInstance         | completed | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000a3 | Ancor::Tasks::DeployInstance            | suspended | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000a5 | Ancor::Tasks::DeployInstance            | suspended | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000a7 | Ancor::Tasks::DeployInstance            | suspended | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000a9 | Ancor::Tasks::DeployInstance            | suspended | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000ab | Ancor::Tasks::DeployInstance            | suspended | 2014-04-03T15:18:35Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000047 | Ancor::Tasks::PushConfiguration         | suspended | 2014-04-03T15:19:19Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000049 | Ancor::Tasks::PushConfiguration         | suspended | 2014-04-03T15:19:19Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b7122804500004b | Ancor::Tasks::PushConfiguration         | suspended | 2014-04-03T15:19:41Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b7122804500004d | Ancor::Tasks::PushConfiguration         | suspended | 2014-04-03T15:19:41Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b7122804500004f | Ancor::Tasks::PushConfiguration         | suspended | 2014-04-03T15:19:41Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000043 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:20:25Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400009f | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:20:25Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000045 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:20:33Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000a1 | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:20:33Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000051 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:20:47Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000ad | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:20:47Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000057 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:21:44Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000b3 | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:21:44Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000055 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:21:44Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000b1 | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:21:44Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7bcb3b71228045000053 | Ancor::Tasks::PushConfiguration         | completed | 2014-04-03T15:21:45Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac40000af | Ancor::Tasks::DeployInstance            | completed | 2014-04-03T15:21:45Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+
  | 533d7b1b3b71220ac400009e | Ancor::Tasks::Sink                      | suspended | 2014-04-03T15:21:45Z |
  +--------------------------+-----------------------------------------+-----------+----------------------+

Implement a certificate authority

Implement a certificate authority in ANCOR
Replace Puppet CA with our external CA
Use CA for key distribution with RabbitMQ/MCollective/Sensu

http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html

This gets us the following:

A single, unified certificate authority for signing and revoking certificates for instances
Removal of the MCollective initialization process (no longer have to sign keys, can distribute them with Cloud-init)

Model block devices

We should consider modeling block devices that can be attached to managed instances.

IMPORTANT: This will only work with an instance that is clustered (like a MySQL slave). Otherwise, there will be service downtime.

Instance replacement workflow w/ block devices

Provision a new instance, bring instance to setup stage
Inform dependent instances that the old instance is going away
Bring old instance to undeploy stage
Detach volume from old instance, wait until volume is detached
Attach volume to new instance
Bring new instance to the deploy stage
Inform dependent instances that the new instance is ready
Terminate old instance

Models are not locked

The following models should be locked before performing operations:

Instance
Network
Security group

Support application-side failover awareness

Example: MySQL master failover (this would apply to any hot/cold master configuration, like PostgreSQL, Pacemaker/Corosync configurations, etc.)

Create new instance with database_master role
Have it synchronize with the live database_master (determine liveness by looking at the instance that is in deploy state)
Change current database_master to undeploy (don't actually push the undeploy configuration though)
Push deploy configuration to webapp and database_slave instances

At this point, the application and slaves should notice that there is no database_master in the deploy stage. They have the logic in Puppet to put their applications in a read-only or buffered mode.
Once synchronization of the new database_master is finished, push configuration to the old database_master to undeploy.
Push configuration to the new database_master to deploy.
Push configuration to all database_slave and webapp instances to deploy.

I'm probably missing something here. Is this possible with the current model and process? If not, what do we need to do to be able to support it?

You would have to have a special lifecycle stage for the new master so that it will sync with the current master. Otherwise, if you put it in deploy, dependent instances may think that the new instance is the master to use. This will break the application because it assumed the master was ready for use. In my mind, there would be a pre_deploy stage. The scenario for a master would look for existing masters in deploy and setup like a slave until they are put into deploy.
How do we know that synchronization of the new master has completed? Polling? Is there are post-sync hook for MySQL/PostgreSQL?
It could be possible to combine block device migration with this. We take an existing slave, grab its block device, and then we just have to wait for the new data to be synchronized. This would be a lot harder to automate though.

Apache
HAProxy
Broxy
Redis
Dispatchers

In one place, the partitions talk to each other (HAProxy -> Broxy).

They took Redis, which can't be clustered, and effectively clustered it. We could do something like this by grouping roles together in a sort of "partition".

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.