Code Monkey home page Code Monkey logo

intuit / intuit-spring-cloud-config-validator Goto Github PK

View Code? Open in Web Editor NEW
16.0 4.0 10.0 153 KB

Validation tools for Spring Cloud Config repos: .json, .yam|, .yml and .properties, verified through script or GitHub Pre-receive Hook!

Python 79.11% Dockerfile 2.29% Shell 18.60%
spring-boot github-enterprise spring-cloud-config spring-cloud-config-server spring-cloud-config-client spring-cloud-config-repo spring-cloud-config-git github-hook pre-receive-hooks git-pre-receive-hook

intuit-spring-cloud-config-validator's Issues

Allow Users to specify parsing and filtering capabilities using .validator file

Requirement

  • Yaml lint has too many rules to follow
  • Each team can define its own set... For instance, the code is broken because of spaces
    • Can be adjusted as explained in the URL
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git push test new-branch -f
Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
Total 0 (delta 0), reused 0 (delta 0)
remote: #####################################################
remote: ### Intuit Spring Cloud Config Validator 1.1.1 ####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=5f2acca1293550dc4ecc7aab26e77c52d74f15aa ref=refs/heads/new-branch
remote: BEFORE HEAD...$$$$$$$$
remote: => Validating SHA 5f2acca1293550dc4ecc7aab26e77c52d74f15aa
remote: Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
remote: (v) File config_msaas_test_01-dev.yml is valid!
remote: (v) File config_msaas_test_01-qal.yml is valid!
remote: (x) File idps/idps.yaml is invalid: [30:15: too many spaces inside brackets (brackets), 30:32: too many spaces inside brackets (brackets), 62:15: too many spaces inside brackets (brackets), 62:32: too many spaces inside brackets (brackets)]
remote: (v) File application.yml is valid!
remote: (v) File config_msaas_test_01.yml is valid!
remote: (v) File config_msaas_test_01-e2e.yml is valid!
remote: (v) File config_msaas_test_01-prd.yml is valid!
remote: (v) File config_msaas_test_01-prf.yml is valid!
To 192.168.86.249:test.git
 ! [remote rejected] new-branch -> new-branch (pre-receive hook declined)
error: failed to push some refs to '[email protected]:test.git'
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git push test new-branch -f
Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
Total 0 (delta 0), reused 0 (delta 0)
remote: #####################################################
remote: ### Intuit Spring Cloud Config Validator 1.1.1 ####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=5f2acca1293550dc4ecc7aab26e77c52d74f15aa ref=refs/heads/new-branch
remote: BEFORE HEAD...
remote: => Validating SHA 5f2acca1293550dc4ecc7aab26e77c52d74f15aa
remote: Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
remote: (v) File config_msaas_test_01-dev.yml is valid!
remote: (v) File config_msaas_test_01-qal.yml is valid!
remote: (v) File idps/idps.yaml is valid!
remote: (v) File application.yml is valid!
remote: (v) File config_msaas_test_01.yml is valid!
remote: (v) File config_msaas_test_01-e2e.yml is valid!
remote: (v) File config_msaas_test_01-prd.yml is valid!
remote: (v) File config_msaas_test_01-prf.yml is valid!
To 192.168.86.249:test.git
 * [new branch]      new-branch -> new-branch

Proposal

  • Users can define the rules for "yaml" using .validator
  • Users can define other rules in the root dir

Pushing new directories fail with OError: [Errno 2] No such file or directory

Problem

  • Pushing a new directory fails with a file idps/idps-config.sh

NOTE: sh files must NOT be considered

  • Tested the same with dir/file.json the same way

Logs

# Directory bug

* Created a directory with a file
* Getting the error after creating the dir `idps` with a file `idps-config.sh`

`remote: IOError: [Errno 2] No such file or directory: '/tmp/f30a60ee192a1f22745386cc1f021a5ce718ca3e/idps/idps-config.sh'`

* Here's the output using the local setup

```console
$ BRANCH=develop ./test.sh
Warning: Permanently added '[172.28.110.65]:52311' (ECDSA) to the list of known hosts.
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 301 bytes | 301.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: ##################################################
remote: ###### Spring Cloud Config Validator 1.1.0 #######
remote: ##################################################
remote: Processing commit=f30a60ee192a1f22745386cc1f021a5ce718ca3e ref=refs/heads/develop
remote: Traceback (most recent call last):
remote:   File "hooks/pre-receive", line 334, in <module>
remote:     (executationDirPath, validationIndex) = ShellExecution.run()
remote:   File "hooks/pre-receive", line 306, in run
remote:     currentDirPath = dirPath if dirPath else ExecutionContext.getCurrentDirPath()
remote:   File "hooks/pre-receive", line 83, in getCurrentDirPath
remote:     currentDirPath = Validator.processPreReceivehookFilesInGithub(base, commit)
remote:   File "hooks/pre-receive", line 243, in processPreReceivehookFilesInGithub
remote:     filePath = Validator.saveFileContent(fileName, content, contextDir)
remote:   File "hooks/pre-receive", line 197, in saveFileContent
remote:     with open(filePath, "w") as text_file:
remote: IOError: [Errno 2] No such file or directory: '/tmp/f30a60ee192a1f22745386cc1f021a5ce718ca3e/idps/idps-config.sh'
To 172.28.110.65:test.git
 ! [remote rejected] develop -> develop (pre-receive hook declined)
error: failed to push some refs to '[email protected]:test.git'

Solution

  • Create the directory of files before saving the stream

  • A clean validation of a new branch now shows the information with dirs

  • Created a dir with file idps/idps.yaml

  • Pushed the new dir

+git remote remove test || true
remote: #####################################################
remote: ### Intuit Spring Cloud Config Validator 1.1.1 ####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=9d661175861154e6ebef737fad9d7ddcc4c47288 ref=refs/heads/marcello
remote: => Validating SHA 9d661175861154e6ebef737fad9d7ddcc4c47288
remote: Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
remote: (x) File idps/idps.yaml is invalid: [2:33: syntax error: mapping values are not allowed here]
remote: (v) File config_msaas_test_01.yml is valid!
remote: (v) File application.yml is valid!
remote: (x) File config_msaas_test_01-dev.yml is invalid: [13:4: syntax error: mapping values are not allowed here]
remote: (v) File config_msaas_test_01-prf.yml is valid!
remote: (v) File config_msaas_test_01-prd.yml is valid!
remote: (v) File config_msaas_test_01-e2e.yml is valid!
remote: (v) File config_msaas_test_01-qal.yml is valid!
To 172.28.110.65:test.git
 ! [remote rejected] marcello -> marcello (pre-receive hook declined)
error: failed to push some refs to '[email protected]:test.git'```

.yml files with {cipher} tokens, without quotes, breaks the config service

Problem

If a configuration property starts with {cipher}, but without single/double-quotes, the configuration service breaks. This is specific to .yml files.

Example at marcellodesales/config-repo@3243ff4

Config Server API Output

$ curl config-server:8888/app/default
Error: Failed to load property source from location 'file:/tmp/config-repo-2392906881014410043/application.yml'

Config Server Error Logs

config-server_1           | 	... 87 common frames omitted
config-server_1           |
config-server_1           | 2017-12-07 07:50:05.983 ERROR 1 --- [nio-8888-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: Failed to load property source from location 'file:/tmp/config-repo-2392906881014410043/application.yml'] with root cause
config-server_1           |
config-server_1           | org.yaml.snakeyaml.parser.ParserException: while parsing a block mapping
config-server_1           |  in 'reader', line 5, column 3:
config-server_1           |       password: {cipher}AQBdTe9vjgOqAN ...
config-server_1           |       ^
config-server_1           | expected <block end>, but found Scalar
config-server_1           |  in 'reader', line 5, column 21:
config-server_1           |       password: {cipher}AQBdTe9vjgOqANyfi+Ke5HYmhFJnuvsG ...
config-server_1           |                         ^
config-server_1           |
config-server_1           | 	at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:569) ~[snakeyaml-1.17.jar!/:na]
config-server_1           | 	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:157) ~[snakeyaml-1.17.jar!/:na]
config-server_1           | 	at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:147) ~[snakeyaml-1.17.jar!/:na]
config-server_1           | 	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:224) ~[snakeyaml-1.17.jar!/:na]

Proposed Solution

  • Verify if the value of keys contains the valid tokens for secrets such as {cipher} and {secret} contains quotes (single or double) to make sure secrets are correctly set.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.