Camel is a command line tool that automates processes for developers in our organisation. The have this tool in order to not only automate processes, but it reduces errors. I also acts as a sort of self documentation. Keeping documentation on processes up to date can be tiring. You need to get the process working and then it's all too tempting to move onto the next feature or start using it instead of updating the documentation. This documentation lag can be completely cut with well-structured code that performs our processes. Someone can instantly see the most uptodate process we use by just looking at the code that executes this process. However, before we use it we have to work with profiles.

Profiles is what camel is built on. Each profile houses configs for the processes that we execute. You cannot do anything unless you have a profile and your current profile has to be pointed to a valid profile. This is why every help section has a list of available profiles and the current profile that camel is pointing to. Running the cml command gives us the following printout:

❯ cml

                     ,,__
           ..  ..   / o._)                   .---.
          /--'/--\  \-'||        .----.    .'     '.
         /        \_/ / |      .'      '..'         '-.
       .'\  \__\  __.'.'     .'          “-._
         )\ |  )\ |      _.'
        // \ // \
       ||_  \|_  \_
       '--' '--'' '--' 


                                                                                
      ############      %####%%#%%%%%       /%%%%%%%%%%%%%%%%      %%%%%%%%%%%% 
   ####          #### ###/         .%%%    #%%             %%%   %%%            
  ###              ####/             .%%%   %%%             %%    %%#           
 ###                ###               #%%    /%%%%%%        %%     %%%%%%(      
 /##                ###                #%          %%%%%    %%          ,%%%%%  
  ###              ####%               ##             %%%   %%              %%% 
   .###(        /###. ####         %#  ##%            %%%   %%#             %%% 
      ############       ###########    ########%%%%%%%      *%%%%%%%%%%%%%%%  


current profile: maxwell

available profiles:
maxwell
terraform-dev

available commands:
cml-profile => manages the profile
cml-ssh => manages ssh configurations
cml-terra => manages terraform builds
cml-repo => manages local repo operations

Now that we have looked at the main interface we can access what is supported in profiles by running the cml-profile command giving us the printout below:

❯ cml-profile

current profile: maxwell

available profiles:
maxwell
terraform-dev

available commands:
cml-profile-create => creates a new profile
cml-profile-switch => switches the current profile to a new one
cml-profile-delete => deletes a profile wiping all data associated with the profile

This is the last level of general help commands. Once you are one level off the main help menu all commands listed will execute a process and will sometimes require arguments. We can access help on how to run these commands with the -h argument. For instance, we can see what arguments are required for the cml-profile-create command by running cml-profile-create -h which will give out the following printout:

❯ cml-profile-create -h
usage: cml-profile-create [-h] --name NAME

optional arguments:
  -h, --help   show this help message and exit
  --name NAME  the name of the profile being created

Here we can see that we need to provide a name under the argument --name. With this help approach whenever new processes are added or updated, a user will be able to see what is needed instead of asking around for documentation or seeing if there are changes in release notes, you can just directly look at what commands are available right now. This also enables us to version our processes. For instance, we can keep running old processes in some pipelines or servers by merely having an older version of camel running. A user can instantly get to grips with what is available and how its done by merely routing through the version being used.

In the future we hope to support export and importing of profiles. This will enable us to transfer configs, files, and data under a certain profile to another user onboarding and standardising processes.

Camel supports terraform. The most recent commands available can be seen by running the cml-terra command. However, what is not covered and needs further explanation is the config files for terra builds. Later on we will work on storing these config files in the profile storage just like the SSH configs. For now, we just need to point to terraform configs for running builds. A example of a terraform config file takes the form below:

location: model_runs/aws_generic
build_state:
    bucket: oasislmf-terraform
    key: eu-west-1/model_run/pariswindstorm/terraform.tfstate
    region: eu-west-1
build_variables:
    aws_access_key: =>aws_access_key
    aws_secret_access_key: =>aws_secret_access_key
    subnet_id: =>subnet_id
    server_security_group: =>security_group_id
    instance_type: "t2.large"
    server_tag: "some model camel run"
    key_name: "OasisProject"
    root_block_size: "15"
model_variables:
    repository: "[email protected]:OasisLMF/example.git"
    oasislmf_version: "1.26.3"
    data_bucket: "example-data-bucket"
    data_directory: "/home/ubuntu/some_provider/example/"
local_vars:
  - name: output
    path: /home/ubuntu/
    ip_address: true
steps:
  - name: run_api_test_1
    env_vars:
      OASIS_WORKER_VERSION: "1.26.3"
      OASIS_UI_VERSION: "1.11.3"
      OASIS_PLATFORM_VERSION: "1.26.3"
    script_name: run_api_test_1
    variables:
      git_branch: "develop"
      parent_dir: "/home/ubuntu/some_provider"
      test_dir: "/home/ubuntu/some_provider/example/tests/test_1"
      expected_md5: "/home/ubuntu/some_provider/example/expected/1-26-3/expected.md5"
      worker_name: "example_worker"
      worker_dockerfile: "/home/ubuntu/some_provider/docker/Dockerfile.some_provider_example_worker"
      docker_compose_platform: "/home/ubuntu/some_provider/docker/docker-compose.yml"
      docker_compose_worker: "/home/ubuntu/some_provider/docker/docker-compose_example.yml"
      docker_compose_ui: "/home/ubuntu/some_provider/docker/docker-compose_ui.yml"
  - name: conditional
    operator: ==
    variable: '>>output'
    value: FINISHED
    step_data:
      name: print
      statement: the process is finished

What we first need to note is the location field. This is the location of the build in the camel repo. The base path of the location is here. Your path that the location field has to point to is a path leading to the main.tf file of that build. To demonstrate this, we can revisit our location definition below:

location: model_runs/BGEQ

This will tell camel to look for a main.tf file here. We can now move onto the next field which is variables. This is merely a dictionary of variables passed into the terraform build. Continuing our Paris Windstorm model example, we can look a what variables are needed in the variables.tf file in the build as seen here. It must be noted here that we are working on passing in keys in different ways as they are needed for all builds. Also, they keys are noted as secrets so they will not show up in the terraform logs. With this covered, we can now move onto the final section which is steps. This field is a list. They have a name field which is the name of the step that is suppoorted. If you want to see what steps are available they can be seen here Our example config file runs a python script on a server that has been created. The run_model script is a python script in the terraform build that we are running which can be found here. More steps will be added in time.

We also must state that oasis_version is optional. If you do not supply a version the latest version will be used as default. The oasis_version is the version of oasislmf from pypi.

When it comes to our run_server_command step this is running a command on the recently created server. We can see that right now we are exporting a range of global environment variables into the server. We should work on creating an environment variable step to organise the variables better.

Camel now supports model templates. To create a model template carry out the following command:

cml-model-build --name <NAME_OF_MODEL>

This will produce a template folder will all the files needed to build a model that will run on a server created by terraform. All files are annotated and the example config.yml in the package. Once you have altered the files to enable your model to run, you will have to load it into your camel package with the following command:

cml-model-load --name <NAME_OF_MODEL>

This loads the model package into the camel package so it can be referenced in model builds. You will also be asked if you want to add the config in the model package. If you put in y to this answer it will be loaded into our tconfig.

We are looking into handling profiles and aws authentication using the following guide: https://blog.gruntwork.io/authenticating-to-aws-with-the-credentials-file-d16c0fbcbf9e

We are looking for contributors to help us develop camel. If you are interested in contributing please contact us on the oasis slack channel or email us at [email protected]. We will be looking into supporting other cloud providers in the future such as Azure and GCP as of right now we only support AWS.

The code style we are using is PEP8. We are using flake8 to check the code style. The project is also object orientated where we usually have one class per file. We also have isolated modules where one module focuses on one responsibility. A standard module takes the following form:

├── adapters
│   ├── __init__.py
│   ├── . . .
├── components
│   ├── __init__.py
│   ├── . . .
├── entry_points
│   ├── . . .
│   ├── __init__.py

components are objects that handle the logic of the program. They are usually called by the entry points. As the components are called by the entry points we can isolate the logic. Therefore, when we look at the entry points we can purely focus on the logic of the program. The adapters are used to handle the communication between modules. To concrete the example, if we were to build an object that manages the storage for a profile, the object that handles the saving and getting of information from the storage would be a component. The entry point would be the management of that component based on the user input. An object would take the following form:

"""
This file defines the class around managing data for a profile.
"""
import os
from pathlib import Path

class Profile:
    """
    This class is responsible for managing the data around a user profile.

    Attributes:
        name (str): the name of the profile
    """
    ROOT_PATH: str = str(Path.home()) + "/.camel_storage/"

    def __init__(self, name: str) -> None:
        """
        The constructor for the Profile class.

        Args:
            name: (str) the name of the profile being loaded
        """
        self.name: str = name

    def create_profile(self) -> None:
        """
        Creates the directories for a new profile.

        Returns: None
        """
        os.mkdir(self.profile_base_path)
        os.mkdir(self.keys_path)
        os.mkdir(self.terra_builds_path)
        os.mkdir(self.configs_path)
    . . .

Here we can see that we use Google style docstrings. We also have type hints. We also require that a docstring is at the top of the file describing what the file is for. We also require that every function and class has a docstring.

Each object should have a test file. The test file should be in the same directory as the object but in the tests directory. Below is an example of a test file for a component that creates a bash script:

from unittest import main, TestCase

from camel.terra.components.server_build_bash_generator import ServerBuildBashGenerator


class TestServerBuildBashGenerator(TestCase):

    def setUp(self) -> None:
        self.test = ServerBuildBashGenerator()

    def tearDown(self) -> None:
        pass

    def test___init__(self):
        self.assertEqual([], self.test)

    def test_write_line(self):
        self.test.write_line(line="this is the first line")
        self.test.write_line(line="this is the second line")

        expected_outcome = ['this is the first line\n', 'this is the second line\n']
        self.assertEqual(expected_outcome, self.test)

    def test_generate_script(self):
        self.test.generate_script(repository="test/repo", oasislmf_version="1.26",
                                  aws_key="AWS_KEY", aws_secret_key="AWS_SECRET_KEY")
        self.assertEqual("".join(self.test), str(self.test))

if __name__ == "__main__":
    main()