Code Monkey home page Code Monkey logo

floodgate's Introduction

Floodgate logo

Floodgate

codilime/floodgate SeriesCI

Motivation

This project integrates multiple parts of "as-code" experience in Spinnaker, e.g. API, Sponnet, Pipeline Templates for a complete GitOps workflow.

Features

  • Allows creating configuration of applications, pipeline templates and pipelines as file types:
    • JSONNET
    • JSON
    • YAML
  • Updates only the parts of Spinnaker configuration that have actually changed
  • Reports diffs in managed objects
  • Works with all currently supported versions of Spinnaker
  • Is well suited to run in a CI system (single binary!)

Upcoming features

  • Run as a microservice within Spinnaker installation for seamless integration

Build process

This repository contains a "known-working" version of gate-swagger.json file, which is a definition of Gate's API, used to generate client code in go using swagger-codegen.

CircleCI process uses the swagger-codegen-cli JAR file to generate the API client at build time.

As a developer, you can use generate_swaggerapi.sh script, which will use Docker to generate go client code for Gate API in the gateapi directory. For example:

./generate_swaggerapi.sh gate-swagger.json
* Using swagger-codegen-cli 2.4.12
2.4.12: Pulling from swaggerapi/swagger-codegen-cli
e7c96db7181b: Already exists
f910a506b6cb: Already exists
b6abafe80f63: Already exists
36ebbdce0651: Pull complete
Digest: sha256:fc24e10784390e27fae893a57da7353d695e641920f3eb58f706ee54af92ebed
Status: Downloaded newer image for swaggerapi/swagger-codegen-cli:2.4.12
docker.io/swaggerapi/swagger-codegen-cli:2.4.12
* Cleaning up gateapi directory
* Generating new gateapi code using gate-swagger.json file...
[main] INFO io.swagger.parser.Swagger20Parser - reading from /local/gate-swagger.json
[...]

Note: This will remove current contents of gateapi directory!

You can also obtain the same using raw java file, like in CI:

$ SWAGGER_VERSION=$(cat gateapi/.swagger-codegen/VERSION)
$ wget https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/${SWAGGER_VERSION}/swagger-codegen-cli-${SWAGGER_VERSION}.jar -O /tmp/swagger-codegen-cli.jar
$ rm -r gateapi
$ java -jar /tmp/swagger-codegen-cli.jar generate -l go -i gate-swagger.json -o gateapi

Note: This will remove current contents of gateapi directory!

How to run?

First, make sure you have some version of Spinnaker running.

Download proper binary from releases page.

Note: You should download proper binary for your Spinnaker version (gate part in binary name).

Create a config.yaml file. An example one can be found in examples/config.yaml. Floodgate checks if a config file is present in:

$HOME/.config/floodgate/config.yaml

You can also specify a path to the config file using the --config flag.

Additionally, you can find example resources in the examples/resources directory.

Note: To allow Floodgate to control pipeline templates apply those commands to your environment:

hal config features edit --pipeline-templates true
hal deploy apply

Note: To view pipeline templates in the UI use:

hal config features edit --managed-pipeline-templates-v2-ui true
hal deploy apply

Note: To configure OAuth2 authentication, you need to set callback url in your app to:

http://localhost:8085/callback

How to use?

floodgate has a simple to use CLI:

$ floodgate
Usage:
   [command]

Available Commands:
  compare     Compare local resources' definitions with Spinnaker and show discrepancies
  help        Help about any command
  hydrate     Hydrate pipeline templates with configurations and preview the result
  inspect     Inspect resources' status on Spinnaker
  render      Render Jsonnet files
  synchronize Synchronize resources to Spinnaker

Flags:
      --config string   path to config file (default $HOME/.config/floodgate/config.yaml)
  -h, --help            help for this command
  -q, --quiet           hide non-essential output
  -v, --verbose         show extended output
      --version         version for this command

Use " [command] --help" for more information about a command.

JSON comparison is achieved using an external library. For full output specification please check https://github.com/josephburnett/jd#diff-language

Simple example:

a.json: {"hungry":"true", "pizza":{"eat":"true","like":"true"},"pasta":{"eat":"true","like":"false"}}

b.json: {"hungry":"false", "pizza":{"eat":"true","like":"true"},"pasta":{"eat":"false","like":"true"}}

Difference between a.json and b.json:

@ ["hungry"]
- "true"
+ "false"
@ ["pasta","eat"]
- "true"
+ "false"
@ ["pasta","like"]
- "false"
+ "true"

Using JSONNET

Files that have a .jsonnet file extention will be evaluated as JSONNET files. Output of each file should be either a single object or an array of objects.

License

Floodgate is licensed under Apache 2.0 License, following other Spinnaker's components.

floodgate's People

Contributors

boscard avatar kacpersaw avatar keisukeyamashita avatar mertsaygi avatar mlembke avatar mwierzbicki avatar nvtkaszpir avatar wurbanski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

floodgate's Issues

Create release pipeline.

We need dedicated pipeline for release branch and release tags. We also need documentation for whole release process.

5 Seconds Timeout Does Not Enough

Is your feature request related to a problem? Please describe.
We need to increase 5 seconds timeout in Floodgate config so we need to convert it from static to parametrical. If you have too many applications and pipelines, default timeout duration may cause api errors.

Describe the solution you'd like
Add Timeout parameter in config struct
type Config struct { Endpoint string json:"endpoint"Timeout int64 json:"timeout"Insecure bool json:"insecure"Auth auth.Configjson:"auth"Libraries []string json:"libraries"Resources []string json:"resources" }

Change Timeout from static to parametric in http.Client
var gateHTTPClient = &http.Client{ Timeout: time.Duration(floodgateConfig.Timeout) * time.Second, Transport: &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: floodgateConfig.Insecure}, }, Jar: cookieJar, }

I want to make this change a new branch and create a pull request then.

Render command needs connection to spinnaker

ATM floodgate needs connection to spinnaker to render json files by floodgate render command.

Steps to reproduce the behaviour:

  1. Make sure you don't have configured connection to spinnaker in your floodgate config.
  2. Execute floodgate render command on exaples directory of this repo.
  3. Nothing is generated and no error is given.

Expected behaviour
Json files should be generated or there should be error informing that spinnaker connection is required to generate json files.

Add Pull Request templates

Is your feature request related to a problem? Please describe.
There is no Pull requests template.

Describe the solution you'd like
Add PULL_REQUEST_TEMPLATE.md.

Describe alternatives you've considered
No.

Additional context
Add any other context or screenshots about the feature request here.

Main flow implementation

Basic flow implementation for all resources:

(start) -> read local -> isChanged? -> update remote -> isChanged? -> (stop)

More configuration options in CLI

Sometimes there is situation you are testing changes on separate instance of spinnaker, or you would like to hydrate only one template / diff or sync just one pipeline.

I think it would be useful to be able to set such options like --endpoint or if at the end of command are some dirs/files just hydrate/diff/sync/render those files.
But generally speaking it's about possibility to override config options in cli command.

Add feature for downloading Spinnaker resources

Implement a command, ie. floodgate download --project project_name --output-dir output_dir. Floodgate would download all resources related to provided project and save them in appropriate subdirectories in provided output directory, e.g. project_name/applications, project_name/pipelines etc.

Also, it would be nice if Floodgate could take care of unwanted fields (e.g. update timestamp) while downloading and saving the resources.

Floodgate should return with different code when there is error in compare

https://app.circleci.com/pipelines/github/codilime/floodgate/784/workflows/4f9815e0-20aa-42d8-88fb-425479fbd1dd/jobs/3486

#!/bin/bash -eo pipefail
echo "Print version using version flag"
/floodgate/bin/floodgate --version
echo "Print version using version command"
/floodgate/bin/floodgate version
echo "Comare changes - clean Spinnaker"
/floodgate/bin/floodgate -q --config ~/floodgate.yaml compare && exit 1 || echo "Found changes"
echo "Apply local resources"
/floodgate/bin/floodgate -q --config ~/floodgate.yaml sync
echo "Compare changes - synced resources"
/floodgate/bin/floodgate -q --config ~/floodgate.yaml compare

Print version using version flag
version 0.3.0-pull/126
Print version using version command
Version:		0.3.0-pull/126
Gate API version:	1.21.x
Go version:		1.14.4
Git commit:		9099505e23301e5e55cdf1dfe00f645ea0d0f806
Built:			2020-10-05_20:25:28
Comare changes - clean Spinnaker
FATA[0007] unexpected end of JSON input                 
Found changes
Apply local resources
Compare changes - synced resources

CircleCI received exit code 0

When there is such error Floodgate should exit with different code than 1 because 1 is used when there is difference between local and remote state.

Support for OAuth2 authentication

Floodgate should be able to authenticate using OAuth2 authentication.
This method is required if Spinnaker is configured with Okta or GitHub.

Support for X.509 authentication

Floodgate should be able to authenticate using X.509 authentication as it is one of standard authentication mechanism used in Spinnaker.

Enable pipeline triggering and execution status tracking

It would be nice to have an option to trigger pipeline for example using webhook and be able to track status execution using Floodgate.

Currently I need to use curl to trigger webhook, and then with combination of spin and curl command I'm able to track execution status.
I believe that Floodgate could do this on its own.

It could look like this:

$ floodgate execute deploy-nging-webhook -p parameters.json
triggering 'deploy-nging-webhook'
waiting for execution id................
execution id is 12345463DCAF
execution status: RUNNING.....................
pipeline finished with status: FAILED
$ echo $?
1

Better diff for showing changes

Currently we have implemented diff using jsondiff. It works but is far from perfect. We should find better library or implement it by ourselves.

Implement dependencies recognition

With this feature Floodgate will be able to apply independent parts in the same time.
This should reduce total time required for applying configuration in big deployments.

Improve configuration of logging levels and messages contents.

Currently messages about loading files are set to level INFO, but they should probably be on DEBUG and should contain full path to the loaded file (maybe also the "detected type" - JSON/YAML/JSONNET)

Applications are dumped into logs but they shouldn't be there.

Spinnaker does not provide sufficient information on errors.

INFO[0000] application-demo1.json loaded
INFO[0000] application-demo2.json loaded
INFO[0000] deploy-nginx.json loaded
INFO[0000] deploy-spinnaker-demo-2.json loaded
INFO[0000] deploy-spinnaker-demo-3.json loaded
INFO[0000] deploy-spinnaker-demo.json loaded
INFO[0000] Syncing applications
INFO[0000] Syncing pipelines
INFO[0000] No need to save pipeline &{0xc00019ac30 Deploy nginx demo1 a6e1c6b1-c907-4615-9bfa-534c39410e83}
FATA[0002] failed to sync pipeline: &{0xc000097a40 Deploy spinnaker-demo demo1 deploy-spinnaker-demo-2}

wurbanski@spinnaker:~/config$ spin pipeline save -f deploy-spinnaker-demo-3.json
Status: 400 Bad Request, Body: {"timestamp":1587415075411,"status":400,"error":"Bad Request","message":"A pipeline with name Deploy spinnaker-demo already exists in application demo1"}```

Floodgate makes changes to our existing resources but it never creates our new resources

Describe the bug
When we synth our resources, Floodgate makes changes to our existing resources but it never creates our new resources. When we investigate, we should make change on spinnaker resource directory as like that:

if api != nil { if err := a.LoadRemoteState(api); err != nil { err := a.SaveLocalState(api) return err } }

If loadRemoteState has an error (does not exist error) local state should be saved on remote. This is quick solution. If you want to resolve this issue on me. I can do clean solution.

Create CLI interface

We need to create some kind of CLI interface based on https://github.com/spf13/cobra

Interface should have at least:

  • dry-run option
  • print diff option
  • print raw and hydrated pipeline
  • save to spinnaker option
  • print states from spinnaker

Implement CLI ''render" command

floodgate render should allow to render and store JSON objects rendered from Jsonnet files from directories provided in configuration file

Better GitHub releases

Is your feature request related to a problem? Please describe.
I use macOS but the artifacts in GitHub Releases are not executable.

Describe the solution you'd like
Release a .tar.gz that contains the binary of the release by using goreleaser.

Describe alternatives you've considered
Release the .tar.gz manually.

Additional context
Add any other context or screenshots about the feature request here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.