arguslab / ancor Goto Github PK
View Code? Open in Web Editor NEWA Moving Target Defense Platform powered by ANCOR
Home Page: http://arguslab.github.io/ancor
License: GNU General Public License v3.0
A Moving Target Defense Platform powered by ANCOR
Home Page: http://arguslab.github.io/ancor
License: GNU General Public License v3.0
The following models should be locked before performing operations:
We'll need to be able to map floating IP addresses (FIPs) to instances of specific roles. We should keep in mind the following points:
In the future, if we consider client-side resolution, we don't have to wait as long for propagation. We'll still use a push-based model for propagation, however.
This should look pretty similar to how we handle updating dependent instances without interrupting services. Bring up the new instance, wait for all dependents to switch over to the new instance, kill the old one.
When a task is executed, could we execute it with an optional "intent"? This would be similar to Android's activity semantics.
Using this, we could know in the task WHY the task was executed. For example:
Right now, the only way we could know is by querying somewhere to find out the reason why the task was executed.
A new floating IP is generated even if an old one is available e.g., remove load balancer and add a new one
Add controller to Rails for webhooks
After doing a new deployment with the committed changes...ancor ends up having the push config tasks being suspended and the allocate public ips pending. Waited 1 hour for things to complete with no progress. Was replicated a second time today. Here is the task log.
vagrant@ancor-precise64:/vagrant$ ancor task list
+--------------------------+-----------------------------------------+-----------+----------------------+
| id | type | state | updated_at |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000b7 | Ancor::Tasks::AllocatePublicIp | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000b9 | Ancor::Tasks::AllocatePublicIp | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000bb | Ancor::Tasks::AssociatePublicIp | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000bd | Ancor::Tasks::AssociatePublicIp | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000b6 | Ancor::Tasks::Sink | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000c0 | Ancor::Tasks::UnlockEnvironment | pending | 2014-04-03T15:15:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000070 | Ancor::Tasks::ProvisionNetwork | completed | 2014-04-03T15:15:41Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000072 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:15:45Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000074 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:15:48Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000076 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:15:52Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000078 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:15:56Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400007a | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:15:59Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400007c | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:02Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400007e | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:06Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000080 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:09Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000082 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:13Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000084 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:16Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000086 | Ancor::Tasks::CleanPuppetCertificate | completed | 2014-04-03T15:16:19Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000088 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:20Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400008a | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:20Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400008c | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:21Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400008e | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:21Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000090 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:22Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000092 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:22Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000094 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:23Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000096 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:23Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac4000098 | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:24Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400009a | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:24Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400009c | Ancor::Tasks::SyncSecurityGroup | completed | 2014-04-03T15:16:24Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400006f | Ancor::Tasks::Sink | completed | 2014-04-03T15:16:25Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000001 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:29Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000003 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:33Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000005 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:36Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000007 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:39Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000009 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:42Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b7122804500000b | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:46Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b7122804500000d | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:49Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b7122804500000f | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:53Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000011 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:16:58Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000013 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:17:01Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b493b71228045000015 | Ancor::Tasks::GeneratePuppetCertificate | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000017 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000019 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b7122804500001b | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b7122804500001d | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b7122804500001f | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000021 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000023 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000025 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:04Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000027 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b71228045000029 | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b703b7122804500002b | Ancor::Tasks::GenerateInstanceBootstrap | completed | 2014-04-03T15:17:05Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b7122804500002d | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:14Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b7122804500002f | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:22Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000031 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:30Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000033 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:38Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000035 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:45Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000037 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:17:53Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000039 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:18:02Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b7122804500003b | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:18:10Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b7122804500003d | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:18:18Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b7122804500003f | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:18:26Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b713b71228045000041 | Ancor::Tasks::ProvisionInstance | completed | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000a3 | Ancor::Tasks::DeployInstance | suspended | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000a5 | Ancor::Tasks::DeployInstance | suspended | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000a7 | Ancor::Tasks::DeployInstance | suspended | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000a9 | Ancor::Tasks::DeployInstance | suspended | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000ab | Ancor::Tasks::DeployInstance | suspended | 2014-04-03T15:18:35Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000047 | Ancor::Tasks::PushConfiguration | suspended | 2014-04-03T15:19:19Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000049 | Ancor::Tasks::PushConfiguration | suspended | 2014-04-03T15:19:19Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b7122804500004b | Ancor::Tasks::PushConfiguration | suspended | 2014-04-03T15:19:41Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b7122804500004d | Ancor::Tasks::PushConfiguration | suspended | 2014-04-03T15:19:41Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b7122804500004f | Ancor::Tasks::PushConfiguration | suspended | 2014-04-03T15:19:41Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000043 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:20:25Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400009f | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:20:25Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000045 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:20:33Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000a1 | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:20:33Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000051 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:20:47Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000ad | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:20:47Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000057 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:21:44Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000b3 | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:21:44Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000055 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:21:44Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000b1 | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:21:44Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7bcb3b71228045000053 | Ancor::Tasks::PushConfiguration | completed | 2014-04-03T15:21:45Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac40000af | Ancor::Tasks::DeployInstance | completed | 2014-04-03T15:21:45Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
| 533d7b1b3b71220ac400009e | Ancor::Tasks::Sink | suspended | 2014-04-03T15:21:45Z |
+--------------------------+-----------------------------------------+-----------+----------------------+
Example: MySQL master failover (this would apply to any hot/cold master configuration, like PostgreSQL, Pacemaker/Corosync configurations, etc.)
Create new instance with database_master
role
Have it synchronize with the live database_master
(determine liveness by looking at the instance that is in deploy
state)
Change current database_master
to undeploy
(don't actually push the undeploy
configuration though)
Push deploy
configuration to webapp
and database_slave
instances
At this point, the application and slaves should notice that there is no database_master
in the deploy
stage. They have the logic in Puppet to put their applications in a read-only or buffered mode.
Once synchronization of the new database_master
is finished, push configuration to the old database_master
to undeploy
.
Push configuration to the new database_master
to deploy
.
Push configuration to all database_slave
and webapp
instances to deploy
.
I'm probably missing something here. Is this possible with the current model and process? If not, what do we need to do to be able to support it?
deploy
, dependent instances may think that the new instance is the master to use. This will break the application because it assumed the master was ready for use. In my mind, there would be a pre_deploy
stage. The scenario for a master would look for existing masters in deploy
and setup like a slave until they are put into deploy
.Puppet runs should be recorded so that the PushConfiguration
task can analyze runs to determine the success of them. Right now, it's impossible to know whether or not the run successed, only that it finished.
If a parallel task builder is created but no tasks are added to it, there will be an empty sink task. This doesn't affect anything, just confuses people looking at the task list.
Right now we use a single shell script to install Puppet, MCollective, set APT mirrors, etc. The IP addresses and credentials are all hard-coded and the script will only work for a single run. I'd like to replace that with something more flexible.
For cases when we need to be able to compute the FQDN of an instance (such as when approving CSRs on Puppet), we will need to somehow figure out the private domain name that the cloud provider assigns to instances.
Examples:
some-instance.openstacklocal
by defaultip-10-245-81-136.ec2.internal
If you notice in the diagram, there are 2 very distinct partitions of roles. They have load balancers and database on the outside, and then two duplicate partitions containing:
In one place, the partitions talk to each other (HAProxy -> Broxy).
They took Redis, which can't be clustered, and effectively clustered it. We could do something like this by grouping roles together in a sort of "partition".
Expose the following via /hiera/:fqdn
:
On the Puppet master, configure Hiera with this backend:
DeployInstance
task into its own taskShould support these basic operations:
Mongo DB still has leftover entries from the previous deployment when environment remove
is invoked. This is an issue for when a second deployment needs to happen after an environment is removed.
During instance deployment, it should be possible to verify that an instance was correctly configured.
Deployment workflow w/ Sensu
Termination workflow w/ Sensu
For this process to work, we have to be able to model what service checks we want to define in Sensu. There are two parts to a check:
We have to distribute the checks via Puppet. The check definitions can be managed via Puppet as well. Results of the check can be distributed in a number of ways, including over RabbitMQ via AMQP. After instance deployment is finished, check failures can be used to notify the adaptation engine of issues.
Metrics work in nearly the same way. Metric check definitions are setup on the monitoring server and metric checks are deployed to the clients.
Unlike service checks though, we don't need to wait on metric check results. These checks are just used as a feedback mechanism for the adaptation engine.
http://docs.sensuapp.org/0.12/index.html
https://github.com/sensu/sensu-community-plugins
It might be useful to associate tasks with job IDs on Sidekiq. Whenever TaskWorker.perform_async(task_id)
is invoked, it will return a unique key that can be used to cancel or check the status of that specific job on Sidekiq.
If a network on Neutron has subnets left over, it should not be deleted.
We should consider modeling block devices that can be attached to managed instances.
IMPORTANT: This will only work with an instance that is clustered (like a MySQL slave). Otherwise, there will be service downtime.
setup
stageundeploy
stagedetached
deploy
stagehttp://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html
This gets us the following:
The following resources should have tasks for termination:
OpenStackSecurityGroupService
needs to have #delete
implemented.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.