Magic Modules (MM) is a project to auto-generate management libraries (e.g. plugins) for third party tools to operate Google Cloud Platform (GCP) products natively.

Customers -- large and small -- use many different deployment tools to provision and manage their operating environments. Companies like Puppet, Chef, Ansible, Hashicorp produce open source tools to meet these needs and typically provide enterprise versions and support.

Once customers invest in these tools they prefer to use a single management system (and style). For this reason Google wants to provide integration for these tools, so that our customers have an easier path to GCP versus abandoning a tool they're already comfortable with. For example Puppet can provision and manage VMs in Compute Engine, databases in Cloud SQL, Google Kubernetes Engine clusters, and so on.

But given the breadth of products available in Google Cloud Platform, creating modules by hand quickly becomes inefficient and costly. It would involve too much redundant engineering work and create too much code for a team to maintain. There is also the issue of explosion of modules, if you consider for comprehensive coverage: total_libraries = total_products * total_providers . The more products we add to the portfolio, or increase the support to other tools, doing this work by hand will not scale.

Enter Magic Modules.

The philosophy behind Magic Modules is a strongly defined object and object relationship schema. This schema can in turn be used to map the necessary API calls to achieve the object state.

Magic Modules heavily relies on object convergence as its target destination. That means that the user will specify the final state they would like their objects (and/or object relations to be) and the code generated by Magic Modules will do "whatever it takes" to make it so.

Note that there is not a required 1:1 mapping between objects and API calls to GCP. This 1:N mapping is key to the simplicity of the interface exposed to users. There are various wrappers out there that can envelope a Google API into various languages, e.g. Ruby or Java, but the user needs to know all the intricacies of the underlying GCP product API.

Read more about philosophy

• Puppet
• Chef
• Terraform
• Ansible

All the necessary data for a product to compile is under the products/ folder. For simplicity (yet not a requirement) the folder name is the actual product name you will use in the -p parameter later on.

To see the full product list visit the products folder.

Magic Modules requires:

• git (so you can checkout the code)
• Ruby 2.5
• Bundler gem

Since Ruby 2.5 is not yet available in some common distributions, we recommend using rbenv to ensure that you are using the correct Ruby version. If you see an error about unexpected keyword_rescue, that means you're using an older Ruby.

Depending on the product and provider combination the generated code is stored on another Git repository. To make it easier to track them all these foreign Git repositories are tracked as submodules of Magic Modules main repository.

To fetch all products last known good build, run:

git submodule update --init

As you have Ruby installed all you need is Bundler. To install it, run:

gem install bundler

Now that we have Bundler installed, go to the root folder where you checked out the Magic Modules code and run:

bundle install

We are now ready to compile a module!

Each module has its own set of prerequisites for successful compilation. It is advisable to read the documentation within a particular module's project repository before attempting compilation to ensure prerequisites are installed. A list of each module's repository URL can be retrieved via the following command:

git config --file .gitmodules --get-regexp url

Compiling a module is as easy as:

bundle exec compiler -p <product> -e <provider> -o <folder>

For example, to compile Google Compute Engine for Puppet, you invoke:

bundle exec compiler -p products/compute -e puppet -o build/puppet/compute

And the generated code should be written to build/puppet/compute

The Rakefile can be used to compile all of the modules at once. The following rake command can be used to compile all modules for all providers or just a single provider.

bundle exec rake compile:<optional provider>:<optional module>

The following env variables can be set to have certain providers compile into a custom folder. For Terraform and Ansible, only one variable exists. For Chef + Puppet, one variable exists per GCP product. Examples include:

You point Magic Modules to an empty folder and it will output a fully operational module for the selected provider.

Magic modules will compile the schema and generate the following output:

1. The production-grade, user facing code in the target tool language and style
2. Documentation for all objects and interactions between objects
3. All tests that certifies the user facing code
4. All API layer data used during test runs
5. All required libraries for the operation

Use rspec to test Magic Module changes:

bundle exec rspec

Each provider has their own testing mechanism. Run the command in the product's output folder build/<provider>/<product>, e.g. build/puppet/compute.

For Terraform, copy the generated code into the main Terraform repo and run the tests there as per instructions.

And refer to the module documentation on how to execute and debug the code or tests further.

Please refer to Governance for concepts and guidelines on how to develop Magic Modules code.

Please refer to Developer Guide for details on how to create a new product or update an existing one.