Code Monkey home page Code Monkey logo

anarchy's Introduction

Anarchy

Anarchy Diagram

Introducing a Little Anarchy

The world of Kubernetes management is experiencing an Operator revolution.

“The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

“People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides.”

Unfortunately, operators are all too often wresting control away from the mere humans who would manage their own resources. Most Operators embed their logic in compiled container images. The typical human systems operator does not program in Go and does not have a container image build pipeline. Human application operators may have experince with builds, but often do not have cluster administrative rights required to deploy operators.

Anarchy democratizes the potential of the Operator pattern by putting the operator logic in the operator configuration. Users write their own logic in Ansible, which requires little more programming experience than using a command line. Through configuration, the Ansible code is directly controlled at runtime to automate anything you can do in the cluster. Anarchy makes your cluster truly self-governing.

Anarchy in Action

TODO - Add example anarchy operator for managing an application with a Helm chart.

Architecture

Anarchy Diagram

Each AnarchySubject is managed by an AnarchyGovernor. Each AnarchyGovernor is configured to run Ansible tasks on a single AnarchyRunner. AnarchySubjects can have AnarchyActions created to schedule task execution. AnarchyRuns are created for specific Ansible task execution, for processing an subject event handler, run an action, or handle an action callback event.

Anarchy Diagram
  1. An action defined in the AnarchyGovernor config is scheduled for an AnarchySubject, creating an AnarchyAction

  2. An AnarchyRun is created to execute the AnarchyAction’s tasks

  3. Ansible in the AnarchyRun calls a Job Template or Survey in Ansible Tower, passing the callback URL and token for the AnarchyAction

  4. Ansible executing in Ansible Tower calls the AnarchyAction callback

  5. A new AnarchyRun is created to process the callback from Tower

  6. The AnarchyRun for the callback updates the status of the AnarchySubject

Installation

Prerequisite

Anarchy is using [Kopf](https://github.com/nolar/kopf) and needs the kopfpeerings.kopf.dev CustomResourceDefinition must be configured before deploying Anarchy.

oc apply -f kopfpeerings.crd.yaml

Install with Helm

Clone this repo to your local workstation and change directory:

git clone https://github.com/redhat-cop/anarchy.git
cd anarchy

Install with Helm

helm template anarchy helm/ --include-crds | oc apply -f -

Check your to make sure the anarchy-operator pod is running and the default runner starts soon after:

$ oc get pods
NAME                                     READY   STATUS    RESTARTS   AGE
anarchy-5f7d7898d7-f6dww                 1/1     Running   0          11m
anarchy-runner-default-9767c79b6-vt6b4   1/1     Running   0          12m

Anarchy can be installed even if you do not have cluster admin privileges. This will require the CRDs and ClusterRoles to already have been installed.

helm template anarchy helm/ | oc apply -f -

Anarchy Design

The Anarchy Operator is configured with custom resource type AnarchyGovernor in order to manage AnarchySubject resources. Each AnarchySubject is managed according to a single AnarchyGovernor. The AnarchyGovernor defines actions to perform against APIs to instantiate and manage the AnarchySubject. Each action performed for an AnarchySubject according to the AnarchyGovernor definition is represented as an AnarchyAction custom resource. An AnarchyAction always begins with an call to an API. The Anarchy operator listens for callbacks to its own API for events relating to actions such as notifications that an action has completed, or encountered an error. The AnarchyGovernor defines event handlers for actions which may include scheduling further AnarchyActions to occur for the AnarchySubject.

This repository includes a test suite that demonstrates these capabilities by calling a test API. The usage of the test suite is explained in the "Testing" section below. The conceptual overview of the test design is described here.

Let’s start with the AnarchySubject definition:

apiVersion: anarchy.gpte.redhat.com/v1
kind: AnarchySubject
metadata:
  generateName: test-
  namespace: anarchy-operator
spec:
  desiredState: started (1)
  governor: test (2)
  parameters: (3)
    openshift_release: "4.1"
    aws_region: us-east-1
    repo_version: "3.11"
    subdomain_base_suffix: .example.opentlc.com
  1. The desired state of the resource, this is an arbitrary string which should be implemented by the AnarchyGovernor.

  2. The test AnarchySubject references the name of the AnarchyGovernor that will manage it.

  3. Each subject may include a list of parameters to pass to the API, though the governor and API get the final say in how and when the parameters are used.

The test AnarchyGovernor definition is shown here:

apiVersion: anarchy.gpte.redhat.com/v1
kind: AnarchyGovernor
metadata:
  name: test
spec:
  # Ansible processing for this governor will occur on the default runner.
  runner: default

  var:
    ansible_tower_hostname: tower.example.com
    cloud_provider: ec2
  varSecrets:
  - name: api-creds
    var: api_creds
  - name: aws-credentials

  # The `subjectEventHandlers` provide configuration for how to respond to
  # AnarchySubjects being added, updated, and deleted.
  subjectEventHandlers:
    # The `create` event is processed only for subjects that are newly created.
    create:
      tasks:
        # The `anarchy_subject_update` module is provided to make it easy to
        # update the AnarchySubject relating to the current action.
        - name: Set state provision-scheduled in subject status
          anarchy_subject_update:
            metadata:
              labels:
                state: provision-scheduled
            status:
              state: provision-scheduled
        # The `anarchy_schedule_action` module is used to create AnarchyActions
        # for the current AnarchySubject. In this case it schedules an
        # AnarchyAction to be processed immediately.
        - name: Start Provision
          anarchy_schedule_action:
            action: provision

    # The `update` event is processed when a resource changes.
    update:
      # The `anarchy_subject` variable stores the state of the AnarchySubject
      # which triggered this update. A useful pattern is to implement state
      # handling using `spec.desiredState` and `status.state`.
      - when: >-
          anarchy_subject.spec.desiredState|default('') == 'started' and
          (anarchy_subject.status|default({})).state|default('') == 'stopped'
        block:
        - name: Set state start-scheduled in subject status
          anarchy_subject_update:
            metadata:
              labels:
                state: start-scheduled
            status:
              state: start-scheduled
        - name: Schedule start
          anarchy_schedule_action:
            action: start
      - when: >-
          anarchy_subject.spec.desiredState|default('stopped') == 'stopped' and
          (anarchy_subject.status|default({})).state|default('') == 'started'
        block:
        - name: Set state stop-scheduled in subject status
          anarchy_subject_update:
            metadata:
              labels:
                state: stop-scheduled
            status:
              state: stop-scheduled
        - name: Schedule stop
          anarchy_schedule_action:
            action: stop

    # The `delete` event is processed when a subject delete is requsted. This
    # is detected by the presence of a `metadata.deletionTimestamp`. This should
    # schedule an action that will result in removing the finalizer from the
    # subject when complete.
    delete:
      tasks:
      - name: Schedule destroy
        anarchy_schedule_action:
          action: destroy

  # Actions represent entry points for doing something related to a resource.
  # Each action here consists of an API request followed by `callbackHandlers`
  # to respond to callbacks from the API endpoint.
  actions:
    provision:
      tasks:
      - name: Call API
        uri:
          url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
          url_username: "{{ api_creds.user }}"
          url_password: "{{ api_creds.password }}"
          validate_certs: false
          method: POST
          return_content: true
          body_format: json
          body:
            extra_vars:
              job_vars: >-
                {{ anarchy_subject.vars.job_vars | default({})
                 | combine(anarchy_governor.vars.job_vars, recursive=True)
                 | combine({
                     'ACTION': 'provision',
                     '__meta__': {
                       'deployer': {'entry_point': 'ansible/main.yml'},
                       'tower': {'action': 'provision'}
                     }
                   }, recursive=True)
                }}
        ignore_errors: true

      callbackHandlers:
        started:
          tasks:
          - name: Set state provisioning in subject status
            anarchy_subject_update:
              metadata:
                labels:
                  state: provisioning
              status:
                state: provisioning
        - event: complete
          tasks:
          - name: Set state started in subject status
            anarchy_subject_update:
              metadata:
                labels:
                  state: started
              status:
                state: started
          # Subsequent actions are scheduled to run later with the `after` parameter.
          - name: Schedule stop
            anarchy_schedule_action:
              action: stop
              after: 8h
          - name: Schedule destroy
            anarchy_schedule_action:
              action: destroy
              after: 6d

    stop:
      tasks:
      - name: Call API for stop
        uri:
          url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
          url_username: "{{ api_creds.user }}"
          url_password: "{{ api_creds.password }}"
          validate_certs: false
          method: POST
          return_content: true
          body_format: json
          body:
            extra_vars:
              job_vars: >-
                {{ anarchy_subject.vars.job_vars | default({})
                 | combine(anarchy_governor.vars.job_vars, recursive=True)
                 | combine({
                     'ACTION': 'stop',
                     '__meta__': {
                       'deployer': {'entry_point': 'ansible/lifecycle.yml'},
                       'tower': {'action': 'stop'}
                     }
                   }, recursive=True)
                }}
        ignore_errors: true

      callbackHandlers:
        started:
          tasks:
          - name: Set state stopping in subject status
            anarchy_subject_update:
              spec:
                desiredState: stopped
              metadata:
                labels:
                  state: stopping
              status:
                state: stopping
        complete:
          tasks:
          - name: Set state stopped in subject status
            anarchy_subject_update:
              metadata:
                labels:
                  state: stopped
              status:
                state: stopped

    start:
      tasks:
      - name: Call API
        uri:
          url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
          url_username: "{{ api_creds.user }}"
          url_password: "{{ api_creds.password }}"
          validate_certs: false
          method: POST
          return_content: true
          body_format: json
          body:
            extra_vars:
              job_vars: >-
                {{ anarchy_subject.vars.job_vars | default({})
                 | combine(anarchy_governor.vars.job_vars, recursive=True)
                 | combine({
                     'ACTION': 'start',
                     '__meta__': {
                       'deployer': {'entry_point': 'ansible/lifecycle.yml'},
                       'tower': {'action': 'start'}
                     }
                   }, recursive=True)
                }}
        ignore_errors: true

      callbackHandlers:
        started:
          tasks:
          - name: Set state starting in subject status
            anarchy_subject_update:
              metadata:
                labels:
                  state: starting
              status:
                state: starting
        complete:
          tasks:
          - name: Set state started in subject status
            anarchy_subject_update:
              metadata:
                labels:
                  state: started
              status:
                state: started
          - name: Schedule stop
            anarchy_schedule_action:
              action: stop
              after: 8h

    destroy:
      tasks:
      - name: Call API for destroy
        uri:
          url: https://{{ babylon_tower_hostname }}/api/v2/job_templates/job-runner/launch/
          url_username: "{{ api_creds.user }}"
          url_password: "{{ api_creds.password }}"
          validate_certs: false
          method: POST
          return_content: true
          body_format: json
          body:
            extra_vars:
              job_vars: >-
                {{ anarchy_subject.vars.job_vars | default({})
                 | combine(anarchy_governor.vars.job_vars, recursive=True)
                 | combine({
                     'ACTION': 'destroy',
                     '__meta__': {
                       'deployer': {'entry_point': 'ansible/destroy.yml'},
                       'tower': {'action': 'destroy'}
                     }
                   }, recursive=True)
                }}
        ignore_errors: true

      callbackHandlers:
        complete:
          tasks:
          - name: Delete anarchy subject
            anarchy_subject_delete:
              remove_finalizers: true

Testing

Examples

Examples are found in the examples folder.

Configuration

Environment valiable to specify how long subjects should remain cached when active: ANARCHY_SUBJECT_CACHE_AGE_LIMIT default 600

anarchy's People

Contributors

9strands avatar dependabot[bot] avatar fridim avatar jkupferer avatar josephassiga avatar mahdtech avatar pabrahamsson avatar tylerauerbeck avatar yvarbanov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

anarchy's Issues

Anarchy Operator appears to hang or get wedged

On occasions, after creating either an individual instance or a resourcepool no activity is seen on the Tower backends.

After

oc delete pod -l name=anarchy-runner -n anarchy-operator

The queue of anarchy events is released and API calls start again to Tower.

README section for deploy-template is out of date

[ec2-user@clientvm 0 ~/anarchy-operator master ⭑|✔]$ 
oc process -f deploy-template.yaml -p APP_DOMAIN=apps.cluster-babyprod.babyprod.babylon.open.redhat.com | oc apply -f -
error: unknown parameter name "APP_DOMAIN"
error: no objects passed to apply

README installation instructions : Error ImagePullBackOff

Step OpenShift) 5) after deploy-template.yml

[root@bastion 0 ~/anarchy-operator master ⭑|✔]# oc get pods
NAME                                      READY     STATUS             RESTARTS   AGE
anarchy-operator-8644944d68-dd9fx         1/1       Running            0          2m
anarchy-runner-default-58597877c6-dz575   0/1       ImagePullBackOff   0          6m
anarchy-runner-default-d96c5855b-r95kz    0/1       ImagePullBackOff   0          2m
  Type     Reason          Age              From                                  Message
  ----     ------          ----             ----                                  -------
  Normal   Scheduled       1m               default-scheduler                     Successfully assigned anarchy-operator/anarchy-runner-default-d96c5855b-r95kz to node1.babyworkshop.internal
  Warning  Failed          1m (x2 over 1m)  kubelet, node1.babyworkshop.internal  Error: ErrImagePull
  Normal   SandboxChanged  1m (x5 over 1m)  kubelet, node1.babyworkshop.internal  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         1m (x6 over 1m)  kubelet, node1.babyworkshop.internal  Back-off pulling image "quay.io/gpte-devops-automation/anarchy-runner:0.3.4"
  Warning  Failed          1m (x6 over 1m)  kubelet, node1.babyworkshop.internal  Error: ImagePullBackOff
  Normal   Pulling         1m (x3 over 1m)  kubelet, node1.babyworkshop.internal  pulling image "quay.io/gpte-devops-automation/anarchy-runner:0.3.4"
  Warning  Failed          1m (x3 over 1m)  kubelet, node1.babyworkshop.internal  Failed to pull image "quay.io/gpte-devops-automation/anarchy-runner:0.3.4": rpc error: code = Unknown desc = Tag 0.3.4 not found in repository quay.io/gpte-devops-automation/anarchy-runner

inconsistent label subject / subject-name

Depending on the resource you want to filter with label, you need to use either subject=... or subject-name=...

Example:

oc get anarchyevents -l   anarchy.gpte.redhat.com/subject-name=gpte.babylon-ocp4-workshop.dev-vzc5n

oc get anarchyactions  -l anarchy.gpte.redhat.com/subject=gpte.babylon-ocp4-workshop.dev-vzc5n

I would expect the label name to be the consistent across Anarchy resources

Streamline delete finalizer design

The original design did not have a way to remove the finalizers directly from the governor. Now it can handle it and would be a better approach.

README: run the test playbook fails

[root@bastion 0 ~/anarchy-operator master ⭑|✚1]# ansible-playbook test/test-playbook.yaml
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/root/anarchy-operator/test/test-playbook.yaml': line 17, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
  - name: Get route for test api server
    ^ here

Add python requirements support to AnarchyRunner

The AnarchyRunner definition should include the ability to pip install packages on the runner pods on start-up to support dependencies. This may be implemented as part of a general AnarchyRunner init structure.

Anarchy Operator Automatic Deployment

A nice to have feature would be to have a higher level operator that manages deployment of anarchy in namespaces. It would watch for the creation of AnarchyGovernors and deploy the anarchy operator into the namespace if it were found to not already be running there.

This higher level operator could also be responsible for automatically bumping versions of the anarchy-operator image.

anarchy route isn't DNS 952 compliant in README installation instruction

Running instructions from the readme:

[root@bastion 1 ~/anarchy-operator master ⭑|✔]# oc process \
>   -f deploy-template.yaml \
>   -p OPERATOR_HOSTNAME=anarchy-operator.$INGRESS_DOMAIN \
> | oc apply -n anarchy-operator -f -
serviceaccount/anarchy-operator created
rolebinding.rbac.authorization.k8s.io/anarchy-operator created
deployment.apps/anarchy-operator created
service/anarchy-operator created
anarchyrunner.anarchy.gpte.redhat.com/default created
The Route "anarchy-operator" is invalid: spec.host: Invalid value: "anarchy-operator.": host must conform to DNS 952 subdomain conventions

Apparently the command:

INGRESS_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')

Doesn't work.

cluster is an OCP 3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.