A service to manage and monitor assets in cloud platforms

This cloud management service is part of an MVaP architecture and set of requirements for CrateKube that creates infrastructure VPCs, bootstraps, and configures Kubernetes clusters on AWS EC2 using CloudFormation templates and Terraform. The underlying objective of our product is to provide default secure, ephemeral, cloud-ready instances that will launch on AWS.

• CrateKube website
• System architecture and high-level requirements
• How to contribute
• Contributor architecture, roles and responsibilities, and developer documentation

The cloud-mgmt-service is in charge of provisioning and monitoring all cloud resources and services. Cloud resources are loosely defined as managed virtual infrastructure, or "Infrastructure as code".

In AWS, this is expected to provision any and all resources necessary to prepare for the creation of a Kubernetes cluster, including but not limited to IAM, VPC, EC2 instances, subnets, security groups, and security policies.

When utilized as a component, this service can act as a stand-alone service for creating infrastructure as code, and can be easily extended by forking and developing as a CrateKube contributor.

The DropWizard app.yml can be configured dynamically using environment variables:

Example environment variable file:

$ CONFIG_DIR=/app/config
$ SSH_PUBLIC_KEY_PATH=<public key path>
$ ADMIN_APIKEY=<api key>
$ AWS_ACCESS_KEY_ID=<value>
$ AWS_SECRET_ACCESS_KEY=<value>

CONFIG_DIR specifies the path to configuration directory. This directory is used to store terraform state for environments.

SSH_PUBLIC_KEY_PATH is the path to the public key for the host where this service is deployed.

ADMIN_APIKEY is the bearer token that will be used to authenticate CrateKube platform services.

The AWS keys are your AWS ID and KEY with permissions to configure a full VPC

These environment variables are the preferred method of configuration at runtime.

To run this service, simply execute:

docker run -p 8080:9000 --env-file /path/to/envfile -v /home/user/tfstate:/app/config -d cratekube/cloud-mgmt-service

This application has terraform files and templates that can be found in the resources directory for the project. These terraform files are used to initialize terraform directories when environments are created.

After the required terraform files are created an environment directory is initialized with terraform init. A plan is generated for all the required resources and and then applied via terraform. Accessing state for environments is accomplished using terraform state calls. Destroying environments is accomplished through terraform destroy.

In order for terraform calls to be successful environment variables for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be provided when running the application.

We strive to have our builds repeatable across development environments so we also provide a Docker build to generate the Dropwizard application container. The examples below should be executed from the root of the project.

docker build -t cloud-mgmt-service:local --target build .

Note: This requires docker 19.03.x or above. Docker 18.09 will throw errors for mount points and the --target flag.

docker build -t cloud-mgmt-service:local --target package .

docker run -p 8080:9000 -v /home/user/tfstate:/app/config -d cloud-mgmt-service:local

Note: We are bind mounting the /home/user/tfstate directory inside of the container to preserve the state of the Terraform installation locally on the host. If you don't do that, whenever your container dies you will lose your infrastructure state files. That's bad! Without state, the next time Terraform runs it could remove infrastructure it doesn't know it exists.

If you don't want to preserve state, you can just remove the volume flag.

http://localhost:8080/swagger

This project uses gradle for building and testing. We also use the gradle wrapper to avoid downloading a local distribution. The commands below are helpful for building and testing.

• ./gradlew build compile and build the application
• ./gradlew check run static code analysis and test the application
• ./gradlew shadowJar builds a fat jar that can be used to run the Dropwizard application
• ./gradlew buildClient generates the API client code for the Dropwizard application
• ./gradlew publishToMavenLocal publishes any local artifacts to the local .m2 repository

After you have generated the fat jar you can run your application with java using:

java -jar build/libs/cloud-mgmt-service-1.0.0-SNAPSHOT-all.jar

This microservice can also run on an Amazon EC2 instance:

• Find an Ubuntu Cloud AMI and spin it up in your AWS EC2 console.
• Install Docker
• Push your locally built docker image to a registry
• Configure the necessary environment variables
• Set up your EC2 security groups to protect this app from the outside world
• Pull the docker image from dockerhub i.e.- docker pull yourUser/yourImage:tag
• Run the container

The cloud management service will become available at:

http://<ec2 instance public dns name>:8080/swagger

• Requires SuperUser permission to an AWS account using the IAM keys of your choice.
• Terraform templates will need to have configuration options that support the compliance rules implemented in the policy-mgmt-service.
• The state files generated by Terraform must be persisted. If running this service on EC2, you must persist the data in a volume in order for Docker to attach to it and save the state.

The API has endpoints that allow you to create, read, and delete environments. In order for these operations to be successful, your app.yml file must have been properly configured to utilize your AWS account with the appropriate key. That API key must have access to all the resources necessary to configure a full VPC.

The resulting operations exist as REST endpoints, which you can simply hit in your browser or with a tool such as Postman.

The API docs for this project are powered by the Swagger Specification. After starting up the application the available APIs can be found at http://localhost:<configured port>/swagger

This application generates a client for the Dropwizard application by using the swagger specification. The maven asset is available in JCenter, make sure you include the JCenter repository (https://jcenter.bintray.com/) when pulling this client. To use the client provide the following dependency in your project:

Gradle:

implementation 'io.cratekube:cloud-mgmt-service:1.0.0'

Maven:

<dependency>
  <groupId>io.cratekube</groupId>
  <artifactId>cloud-mgmt-service</artifactId>
  <version>1.0.0</version>
</dependency>

If you are interested in contributing to this project please review the contribution guidelines. Thank you for your interest in CrateKube!