kohlstechnology / git2consul-go Goto Github PK
View Code? Open in Web Editor NEWPopulates a Consul key/value store from git
License: Apache License 2.0
Populates a Consul key/value store from git
License: Apache License 2.0
Hello team, congratulations for your work. We are testing git2consul-go as a replacement for Gonsul.
This is the version we are using:
git2consul --version
git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd)
build date: 20200106-19:50:06
go version: go1.13.5
Our case is the following: Gonsul works great for a "flat" infrastructure (mirroring the Git values into Consul KV) but we want to be able to "reuse" some values, creating "default" values that we can override per environment (eg: production)
So, I have a single test repo named consul
with 2 folder: "base" and "prod". All the values are located inside "base". Inside "prod" I just have added the values I want to override for production.
I have created the following config.json file:
{
"local_store": "/tmp/git2consul",
"repos": [
{
"name": "consul-base-development",
"url": "ssh://gitlab.myorganization.com/consul.git",
"source_root": "/base/",
"mount_point": "development/",
"skip_repo_name": true,
"skip_branch_name": true,
"expand_keys": true,
"credentials": {
"private_key": {
"pk_key": "/path/to/.ssh/id_rsa",
"pk_username": "git"
}
}
},
{
"name": "consul-base-production",
"url": "ssh://gitlab.myorganization.com/consul.git",
"source_root": "/base/",
"mount_point": "production/",
"skip_repo_name": true,
"skip_branch_name": true,
"expand_keys": true,
"credentials": {
"private_key": {
"pk_key": "/path/to/.ssh/id_rsa",
"pk_username": "git"
}
}
},
{
"name": "consul-production",
"url": "ssh://gitlab.myorganization.com/consul.git",
"source_root": "/prod/",
"mount_point": "production/",
"skip_repo_name": true,
"skip_branch_name": true,
"expand_keys": true,
"credentials": {
"private_key": {
"pk_key": "/path/to/.ssh/id_rsa",
"pk_username": "git"
}
}
}
]
}
When I execute git2consul -config ./config.json -debug -once
sometimes it works great: the production
KV path is populated with all the values from "base" (the "default" values) and then those specific values in "prod" are used to override the production settings. So, the production
KV path has correctly merged base and prod. So far so good.
But this process seems to be not idempotent. Sometimes, when I execute the same command, the result is that "production" only shows the values from "base".
So, my question is: are these "repositories" (they are actually the same repo) processed sequentially or in parallel? If they are processed in parallel (and the order is not guaranteed), is it possible to force a sequential execution of these repos? (first we apply "base", then we apply "prod" on top)
Thank you very much in advance
Is your feature request related to a problem? Please describe.
I'm running git2consul from a Linux server that does not have a consul agent on it. I need to manually specify the IP/hostname of the remove consul server.
Describe the solution you'd like
I'd like git2consul to dynamically discover consul servers running in cloud environments when a local consul agent is not available. Enabling this feature should be optional.
Describe alternatives you've considered
Manually specifying the IP/hostname of the consul server.
What version of git2consul-go are you using?
$ git2consul -version v0.1.1
Additional context
N/A
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Darwin Charlie-Jones 18.7.0 Darwin Kernel Version 18.7.0: Sat Oct 12 00:02:19 PDT 2019; root:xnu-4903.278.12~1/RELEASE_X86_64 x86_64
What did you do?
Read the readme
What did you expect to see?
The correct protocol for ssh auth in the example - ssh://
What did you see instead?
The incorrect protocol for ssh auth - https://
When developing git2consul, dep
is used to manage dependencies.
Go Modules should be used instead of dep.
N/A
I created a new GitHub issue and pull request for this project, and I struggled to fill in the template without going crazy.
The issue and pull request templates are easy to use.
Open a GitHub issue or pull request.
No.
We should steal the issue and pull request templates from the Kohl's Technology eunomia project. The templates used in that repo seem to work pretty good.
Has anyone managed to get Git2Consul working with Azure Devops repos? I have tried all kinds of authentication approaches, but am having no luck. Just getting ERROR[0001] Runner error error=reference not found
every time, which I'm having trouble deciphering.
If anyone has got this working, any advice would be welcome, especially with some example config.
I'll report back if I get it working in the interim.
Is your feature request related to a problem? Please describe.
Yes. I am trying to load yaml files that contain lists of strings.
git2consul supports only lists of maps and crashes in my case.
Describe the solution you'd like
You already create keys that are list indices, but assume that value of list is a map.
Please create keys that as value have strings.
Describe alternatives you've considered
Having this yaml:
---
main:
name: test
tags:
- lab
- feature-x
I can only consider mutating this into:
---
main:
name: test
tags:
0: lab
1: feature-x
While consul KV contents will be the same, yaml representation is a bit awkward.
What version of git2consul-go are you using?
Latest, cloned from this repo.
$ git2consul -version 0.1.2-dev
Additional content
The crash is here:
git2consul-go/kv/filehandler.go
Line 169 in 9f404c0
panic: interface conversion: interface {} is string, not map[interface {}]interface {}
goroutine 20 [running]:
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00044f3b0, 0xc00044f380)
/home/dzareba/g2c/git2consul-go/kv/filehandler.go:169 +0xa75
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00044f380, 0xc00044f350)
/home/dzareba/g2c/git2consul-go/kv/filehandler.go:164 +0x634
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00044f350, 0xc00044f290)
/home/dzareba/g2c/git2consul-go/kv/filehandler.go:164 +0x634
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00044f290, 0xc0004353fc)
/home/dzareba/g2c/git2consul-go/kv/filehandler.go:164 +0x634
github.com/KohlsTechnology/git2consul-go/kv.(*YAMLFile).Create(0xc000445910, 0x9d5360, 0xc000130db0, 0x9da6c0, 0xc000130ae0, 0xc000445910, 0xffffffffffffff9c)
/home/dzareba/g2c/git2consul-go/kv/filehandler.go:119 +0x1c2
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).putBranch.func1(0xc0004353c0, 0x40, 0x9d7a40, 0xc000155d40, 0x0, 0x0, 0x4c2d99, 0xc000155d40)
/home/dzareba/g2c/git2consul-go/kv/branch.go:56 +0x10b
path/filepath.walk(0xc0004353c0, 0x40, 0x9d7a40, 0xc000155d40, 0xc0000fdca8, 0x0, 0x0)
/home/dzareba/go/src/path/filepath/path.go:358 +0x425
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes, at least this is the latest release
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Linux xxxxxxx 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
What did you do?
git2consul -config ./config.json -debug
What did you expect to see?
I expected to see that my new commits are updated every 60 seconds because the README file says that the default value for repos:hooks:interval is 60.
What did you see instead?
After 60 seconds (or even several minutes) nothing is updated. It seems like the default value is not 60, it's 60000000000 according to the logs:
INFO[0000] Starting git2consul version: v0.1.1
INFO[0000] Setting configuration with sane defaults caller=config
DEBUG[0000] Using configuration: {"local_store":"/tmp/git2consul","webhook":{"port":9000},"repos":[{"name":"consul-base-development","url":"ssh://10.1.1.11/obfuscated/consul.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root [...]
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec 1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64 i386 MacBookPro11,4 Darwin
What did you do?
I read the README.md file.
What did you expect to see?
An easy to follow user guide with examples that show how to get started with and use git2consul.
What did you see instead?
My opinion is that the README.md files is not easy to follow. I think it needs to be re-written.
Is your feature request related to a problem? Please describe.
I would like to have the ability to expand keys in a YAML file, but I also want to import some .yml files as blobs
Describe the solution you'd like
I would like to have the ability to define a custom file extension to treat as expandable YAML
Describe alternatives you've considered
I have made a small patch that will add this functionality:
config/config.go: MountPoint string `json:"mount_point"` ExpandKeys bool `json:"expand_keys,omitempty"` ExpandExt string `json:"expand_ext,omitempty"` SkipBranchName bool `json:"skip_branch_name,omitempty"` kv/filehandler.go: func Init(path string, repo repository.Repo) File { config := repo.GetConfig() expandKeys := config.ExpandKeys expandExt := config.ExpandExt var f File ext := filepath.Ext(path) if expandKeys { if ext == expandExt { f = &YAMLFile{path: path} } }
Unfortunately I don't really know golang, and I don't really know how to reliably default to ".yml" if no extension was specified.
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Additional context
Is your feature request related to a problem? Please describe.
The current log configuration includes the number of seconds from service start on every log line. This timestamp is not useful because it cannot be easily determined when specific logs occurred.
Describe the solution you'd like
The easy solution is to change from text
to logfmt
handler type.
Describe alternatives you've considered
The code includes a comment to allow different formats, but supporting this requires a lot more work due to many formats having required configuration.
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.2-dev (branch: master, revision: 9f404c06fe43479f3d81f237cee0ae93b3964905) build date: 20200226-18:56:58 go version: go1.13.3
Additional context
Switching to logfmt
would result in more usable logs like:
$ ./git2consul
timestamp=2020-02-26T10:57:33.644652-08:00 level=info message="Starting git2consul version: v0.1.2-dev"
timestamp=2020-02-26T10:57:33.64518-08:00 level=error message="No configuration file provided"
What version of git2consul-go are you using?
$ git2consul -version v0.1.2-dev
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Linux mdl-sres0001a.surescripts-dev.qa 3.10.0-1160.24.1.el7.x86_64 #1 SMP Thu Mar 25 21:21:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
What did you do?
We run git2consul on commit from jenkins
git2consul --config config.json -debug -once
Using a configuration like:
{
"repos": [{
"name": "dev",
"url": "https://bitbucket/scm/ops/consul_kv.git",
"credentials": {
"username": "user",
"password": "password"
},
"source_root": "/dev/dc1/",
"skip_branch_name": true,
"skip_repo_name": true,
"expand_keys": true
}],
"consul": {
"address": "http://localhost:8500",
"token":"123"
}
}
What did you expect to see?
We would expect the consul kv to mirror the most recent source control values. Instead on first run, it matches 1 commit older than the most recent. If we look at the os.tempdir() contents the values match the most recent commit. And if we immediately run the command again, the consul kv values are updated to match the most recent commit. Lastly, if we delete the os.tempdir() before every run the consul kv values match the most recent commit. It feels like there's a race condition in the git checkout and the push to the consul kv where the checkout hasn't finished before the push to the kv is started
What did you see instead?
The first run of git2-consul doesn't push out the most recent commit. The second run pushes out the most recent commit. Below are the debug logs from one such "first" run.
"\u001b[34m INFO\u001b[0m[0000] Starting git2consul version: v0.1.2-dev \u001b[34mcaller\u001b[0m=main",
"\u001b[34m INFO\u001b[0m[0000] Setting configuration with sane defaults \u001b[34mcaller\u001b[0m=config",
"\u001b[37m DEBUG\u001b[0m[0000] Using configuration: {\"local_store\":\"/tmp\",\"webhook\":
"\u001b[34m INFO\u001b[0m[0000] Loaded repository devint \u001b[34mcaller\u001b[0m=repository",
"\u001b[34m INFO\u001b[0m[0000] KV GET ref: devint/master \u001b[34mcaller\u001b[0m=consul",
"\u001b[34m INFO\u001b[0m[0000] Changed: devint/master \u001b[34mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0000] Stopping watcher... \u001b[34mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0000] Received finish \u001b[34mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0001] Watcher received finish \u001b[34mcaller\u001b[0m=runner",
"\u001b[34m INFO\u001b[0m[0001] Terminating git2consul "
The fact its says "Changed: devint/master" is interesting to me. That implies that git2consul knows there's a newer commit but then it didn't update any values?
Second run
"\u001b[34m INFO\u001b[0m[0000] Starting git2consul version: v0.1.2-dev \u001b[34mcaller\u001b[0m=main",
"\u001b[34m INFO\u001b[0m[0000] Setting configuration with sane defaults \u001b[34mcaller\u001b[0m=config",
"\u001b[34m INFO\u001b[0m[0000] Loaded repository devint \u001b[34mcaller\u001b[0m=repository",
"\u001b[34m INFO\u001b[0m[0000] KV GET ref: devint/master \u001b[34mcaller\u001b[0m=consul",
"\u001b[37m DEBUG\u001b[0m[0000] Detected modified file: /tmp/devint/dc1/test.yml
\u001b[37mcaller\u001b[0m=consul",
"\u001b[34m INFO\u001b[0m[0000] KV DEL devint/refs/heads/master/test
\u001b[34mcaller\u001b[0m=consul",
"\u001b[37m DEBUG\u001b[0m[0000] KV PUT: devint/master: test/resilience/config \u001b[37mcaller\u001b[0m=consul",
"\u001b[37m DEBUG\u001b[0m[0000] Transaction with 4 items was sent to the KV store \u001b[37mcaller\u001b[0m=consul",
"\u001b[34m INFO\u001b[0m[0000] KV PUT ref: devint/master \u001b[34mcaller\u001b[0m=consul",
"\u001b[37m DEBUG\u001b[0m[0000] Up to date: devint/master \u001b[37mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0000] Stopping watcher... \u001b[34mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0000] Received finish \u001b[34mcaller\u001b[0m=watcher",
"\u001b[34m INFO\u001b[0m[0001] Watcher received finish \u001b[34mcaller\u001b[0m=runner",
"\u001b[34m INFO\u001b[0m[0001] Terminating git2consul "
Identical config.json and command line parameters were used for both runs and there was no change to source control.
Is your feature request related to a problem? Please describe.
When developing git2consul-go, it would be nice to have an end to end test suite so you can easily ensure git2consul can still do the basic create-update-delete workflow for Consul KV pairs.
Describe the solution you'd like
A make target that would run an e2e test suite. The test should:
The documentation should also be updated explaining how to run the e2e tests and what it does. Probably should be included in developer documentation - Ref #28
Describe alternatives you've considered
Continuing to manually test changes to git2consul.
What version of git2consul-go are you using?
$ git2consul, version v0.1.2-dev (branch: master, revision: 9f404c06fe43479f3d81f237cee0ae93b3964905) build date: 20200201-16:38:31 go version: go1.13.5
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Linux mainws 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
What did you do?
I was working to get the "refs" key generated by git2consul and the actual data from git under the same key. The documentation is quite sparse, so this is a trial an error process. One of the setups of config.json resulted in a nameless top-level key being created in Consul that was not readable. Deleting this key completely wiped the Consul KV store.
for a git repo structured like so:
<root>/services/testservice/<data>
with a config structured like so:
{
"repos": [
{
"name": "/playground/tenant/services/testservice/",
"url": "ssh://[email protected]/playground.git",
"source_root": "/services/testservice/",
"expand_keys": true,
"mount_point": "playground/tenant/",
"skip_branch_name": true
}
]
}
The result is a key with no name as the top level key in the Consul KV store. Removing this key using the Consul GUI results in the complete K/V store being wiped.
What did you expect to see?
Anything but this...
What did you see instead?
A key with no name as the top level key in the Consul KV store
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec 1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64 i386 MacBookPro11,4 Darwin
What did you do?
I tried to follow the contributing/development docs.
What did you expect to see?
An easy to follow guide to getting started developing git2consul. The doc should make it easy for contributors on onboard.
What did you see instead?
I noticed that the docs are incomplete and spread across multiple files. The docs are hard to follow.
Need the ability to get a dynamic Consul ACL token from Hashicorp Vault, so that the token no longer needs to be in the config file. Would like the authenticate to Vault using Kubernetes authentication.
The Consul ACL token is static and in plain text in a config file.
Leverage Hashicorp Vault to dynamically get a Consul ACL token.
N/A
N/A
https://www.vaultproject.io/docs/auth/kubernetes.html
https://www.vaultproject.io/docs/secrets/consul/index.html
The golint command is failing in Travis CI, but the tests are still passing
Travis CI should fail the build if there are any golint errors.
Run golint -set_exit_status $(go list ./... | grep -v /vendor/); echo $?
Output from golint.
github.com/KohlsTechnology/git2consul-go/kv/handler.go:30:6: type name will be used as kv.KVHandler by other packages, and that stutters; consider calling this Handler
github.com/KohlsTechnology/git2consul-go/kv/mocks/kv.go:31:6: exported type KV should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/kv.go:36:1: exported method KV.Get should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/kv.go:44:1: exported method KV.Put should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/kv.go:53:1: exported method KV.Delete should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/kv.go:58:1: exported method KV.Txn should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:30:6: exported type Repo should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:39:1: exported method Repo.Name should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:43:1: exported method Repo.GetConfig should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:47:1: exported method Repo.Add should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:51:1: exported method Repo.CheckRef should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:55:1: exported method Repo.CheckoutBranch should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:60:1: exported method Repo.DiffStatus should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:69:1: exported method Repo.Head should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:77:1: exported method Repo.Pull should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:91:1: exported method Repo.ResolveRevision should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:96:1: exported method Repo.Worktree should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:100:1: exported method Repo.Lock should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:101:1: exported method Repo.Unlock should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:103:1: exported method Repo.GetStorer should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/kv/mocks/repository.go:107:1: exported method Repo.Branch should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/repository/mocks/repository.go:30:1: exported function InitRemote should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/repository/mocks/repository.go:48:1: exported function Add should have comment or be unexported
github.com/KohlsTechnology/git2consul-go/repository/mocks/repository.go:85:1: exported function Commit should have comment or be unexported
Found 24 lint suggestions; failing.
Need to add the -set_exit_status
option for golint in the Travis config file to fix this.
We should add a link to git2consul.go on the website: https://www.consul.io/downloads_tools.html
All it needs is a PR: https://github.com/hashicorp/consul
Everybody can find out tool.
N/A
N/A
N/A
N/A
Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
I would like a Dockerfile to be able to create a docker image of the git2consul-go client
What version of git2consul-go are you using?
Currently using the master branch
Hello
I'm trying to run gi2consul.
This is my config.json:
{
"version": "1.0",
"repos" : [
{
"name" : "sample_configuration",
"url" : "http://local.test.com/git/ccms/configurations.git",
"branches" : ["master"],
"credentials": {
"username": "user",
"password": "password"}
}
],
"consul" :
{
"address" : "192.168.0.244:8500"
}
but when I try to run git2consul, I got this error:
ubuntu@ubuntu:~/go/src/git2consul-go/build/bin$ ./git2consul -config config.json
INFO[0000] Starting git2consul version: 0.1.0
INFO[0000] Setting configuration with sane defaults caller=config
INFO[0000] Loaded repository sample_configuration caller=repository
ERROR[0001] Runner error error=reference not found
The CONTRIBUTING.md is missing information.
People don't know what the expectations are for contributing to this project.
Useful information in CONTRIBUTING.md
Read the CONTRIBUTING.md file.
N/A
N/A
N/A
What version of git2consul-go are you using?
git2consul -version
Output$ git2consul -version git2consul, version v0.1.2-dev (branch: master, revision: ba3bd22916b56c288d5e130a03a5611eb4e9af3e) build date: 20200117-15:05:41 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Darwin localhost 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64 i386 MacBookPro14,1 Darwin
What did you do?
I ran git2consul, manually changed a kv pair that git2consul was managing, and then ran git2consul again.
What did you expect to see?
I expected to see git2consul fix the kv pair I manually changed to match what is in the git repo.
What did you see instead?
The key value pair was not reverted and git2consul said the repo/kv pairs were up to date.
$ ./git2consul -config config.json -once -debug
LogsINFO[0000] Starting git2consul version: v0.1.2-dev INFO[0000] Setting configuration with sane defaults caller=config DEBUG[0000] Using configuration: {"local_store":"/var/folders/tc/pkmb04_n6gb23ghdt6x37n719kt_jc/T/","webhook":{"port":9000},"repos":[{"name":"git2consul-test","url":"https://github.com/vinny-sabatini/git2consul-test.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root":"","mount_point":"","credentials":{"private_key":{"pk_key":""}}}],"consul":{"address":"127.0.0.1:8500","ssl":false}} caller=config INFO[0000] Loaded repository git2consul-test caller=repository INFO[0000] KV GET ref: git2consul-test/master caller=consul DEBUG[0000] Up to date: git2consul-test/master caller=watcher INFO[0000] Stopping watcher... caller=watcher INFO[0000] Received finish caller=watcher INFO[0001] Watcher received finish caller=runner INFO[0001] Terminating git2consul
Based on this, it looks like git2consul compares the branch.ref
kv pair to the commit sha in the repository to determine if the kv pairs are up to date.
Possible Solutions
branch.ref
but don't use that to determine if the kv pairs are up to date (always try to put the values).Is your feature request related to a problem? Please describe.
My Git project contains a hidden file that happens to be a YAML file. In my case, this is .gitlab-ci.yml which defines the Gitlab pipeline for executing git2consul in my environment. This file is picked up by git2consul and fails similar to the behavior outlined in this bug (due to a list in the yaml):
#37
Describe the solution you'd like
I would like to be able to ignore certain files (e.g. by list or pattern, or by default ignore all hidden files.
Describe alternatives you've considered
I'm currently stumped. There seems to be no way to specify which .yml files in a repo get processed, and there's no way to script up a solution that removes the file between pulling the repo and loading the values. I would greatly appreciate any ideas for how to work around this file in my repo!
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
What version of git2consul-go are you using?
v0.1.1
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
5.4.0-1049-gcp
What did you do?
Attempted to load:
test.yml
foo:
- 123.0
- 456.0
- 789.0
What did you expect to see?
I expected to see the array in consul k/v.
What did you see instead?
#033[34m INFO#033[0m[0000] KV GET ref: global/master #033[34mcaller#033[0m=consul
panic: interface conversion: interface {} is float64, not map[interface {}]interface {}
goroutine 19 [running]:
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00016c9f0, 0xc00016c870)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/filehandler.go:169 +0xa75
github.com/KohlsTechnology/git2consul-go/kv.entriesToKV(0xc00016c870, 0xc0004ca7c0)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/filehandler.go:164 +0x634
github.com/KohlsTechnology/git2consul-go/kv.(*YAMLFile).Create(0xc00016f930, 0x9ce2a0, 0xc000134cf0, 0x9d3600, 0xc000134a20, 0xc00016f930, 0x0)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/filehandler.go:119 +0x1c2
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).handleDeltas(0xc000134cf0, 0x9d3600, 0xc000134a20, 0xc00017e040, 0x7, 0x8, 0x0, 0x0)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/init.go:117 +0x583
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).UpdateToHead(0xc000134cf0, 0x9d3600, 0xc000134a20, 0x0, 0x1)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/update.go:100 +0x2f1
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).HandleUpdate(0xc000134cf0, 0x9d3600, 0xc000134a20, 0x0, 0x0)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/update.go:47 +0x26c
github.com/KohlsTechnology/git2consul-go/runner.(*Runner).Start(0xc00007df40)
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/runner/runner.go:97 +0xbf
created by main.main
#011/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/main.go:84 +0x5d8
Is your feature request related to a problem? Please describe.
I currently administer my Consul cluster on my local machine using the CONSUL_HTTP_ADDR
and CONSUL_HTTP_SSL_VERIFY
environmental variables. git2consul
currently does not read from these (or any) environmental variables for configuration purposes, meaning I am essentially 'double-handling' the Consul configuration by adding it to git2consul
's config.json
file. If the configuration of my Consul cluster changes, I now have one extra file I need to update.
It also means if I want to version-control the git2consul
configuration I will have tokens and/or passwords in the configuration file, which is something I'd like to avoid.
Describe the solution you'd like
If not overridden by a configuration file, I would like git2consul
to use environmental variables as a means of configuration as described by The Twelve Factor App Methodology. I believe the way to start would be by adding the ability to configure the following Consul options through their equivalent environmental variables:
consul:address
-> CONSUL_HTTP_ADDR
consul:ssl
-> CONSUL_HTTP_SSL
consul:ssl_verify
-> CONSUL_HTTP_SSL_VERIFY
consul:token
-> CONSUL_HTTP_TOKEN
This could then be expanded to other git2consul
configuration options such as repos:credentials:username
and repos:credentials:password
using an environmental prefix like GIT2CONSUL_
(though I haven't really given this naming scheme much thought).
Describe alternatives you've considered
Hard-coding configuration in a config.json
file, but it is not ideal and I would like to avoid it if at all possible.
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.2-dev (branch: master, revision: 842c5f830f947489014f412da18ed01940dc9553) build date: 20200430-11:24:22 go version: go1.14.2
Additional context
I am working on creating a pull request with the aim of just adding support for reading Consul configuration from the four environmental variables listed above, as I believe that would be a relatively sensible and straightforward addition.
Depending on the outcome and feedback of the pull request, I am happy to continue adding support for configuring other git2consul
options through environmental variables.
I do have some experience with Go and while I have had some code merged into Packer, I am still quite new to the language. I am open to feedback and criticism, and I am willing to work with anyone involved to contribute quality code! ๐
the repo is here: https://github.com/ttys3/git2consul-go
compare: master...ttys3:main
changes include:
vendor
dirwatcher
dir to match the package name watch
Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
Expand the keys and nested keys to consul kv store.
What version of git2consul-go are you using?
v0.1.1
$ git2consul -version git2consul, version v0.1.2-dev (branch: master, revision: f7da486e18c7223f93a542f966b14673dff7f6f6) build date: 20200531-16:41:29 go version: go1.12.5
Additional context
gonsul supports this, but I find the git2consul more friends with multiple repos handling. This feature will definitely be an added plus for git2consul
What version of git2consul-go are you using?
v0.1.1
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
Debian amd64
What did you do?
We have 2 similarly names keys, foo_bar
and foo_bar_v2
. When we remove foo_bar
from the JSON file, both foo_bar
and foo_bar_v2
are removed from Consul KV.
Steps to reproduce:
foo_bar
and foo_bar_v2
foo_bar
and foo_bar_v2
are removed from Consul KVWhat did you expect to see?
Only 1 key foo_bar
should be removed from Consul KV.
What did you see instead?
Both keys, foo_bar
and foo_bar_v2
, were removed.
What version of git2consul-go are you using?
$ master
Does this issue reproduce with the latest release?
In master branch
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Linux kurtpc 5.11.0-38-generic #42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
What did you do?
I have a repo , which branch name is topic/kurt.ding/fix_something, but sync doesn't work
What did you expect to see?
I expect sync work correctly
What did you see instead?
Is your feature request related to a problem? Please describe.
Yes. This is a problem with this project and the original git2consul. The problem is that it works only in case where the KVs are sourced from a git repo, which makes it inconvenient because now you have to check-in KV in a repo and the config for git2consul in another repo, or something like that. What I would ideally like is that I can keep my Kubernetes configuration and consul KV files for my service in the same repo as the application code and I use just different tools to work with different code.
Describe the solution you'd like
Add support to work with JSON files or directories on the filesystem i.e. cloning a repo must not be necessary. That can be one of the ways and the other way could be just providing the path to a directory to load the KV data from.
This should be possible already since the code for all of this is there.
Describe alternatives you've considered
NA
What version of git2consul-go are you using?
MA
$ git2consul -version
Additional context
NA
(While submitting an issue briefly describe the problem you are facing or a new feature you would want to see added)
When I don't specify pk_username in my config.json file, I would expect it to be set to git by default as per the documentation. What happens is that when I run the git2consul binary , I got the following error:
ERROR[0001] (runner): Cannot load repositories from configuration: Error loading : ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
...
I would expect the username to be set to git by default.
...
...
...
git2consul-go seems to be correctly use the Consul TX api in order to batch the writes operations and reduce the write load on the cluster.
The max number of operation of the TX api are 64 but there is an additional limit to check: the batch req size.
I tried the following config:
{
"repos": [
{
"name": "example",
"url": "https://github.com/DummyOrg/DummyRep.git"
}
],
"consul": {
"address": "127.0.0.1:8501",
"ssl": true,
"ssl_verify": false,
"token": "<redacted>"
}
}
and received the following error:
./git2consul --once --config config --debug
INFO[0000] Starting git2consul version: v0.1.1
INFO[0000] Setting configuration with sane defaults caller=config
DEBUG[0000] Using configuration: {"local_store":"/tmp","webhook":{"port":9000},"repos":[{"name":"example","url":"https://github.com/DummyOrg/DummyRep.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root":"","mount_point":"","credentials":{"private_key":{"pk_key":""}}}],"consul":{"address":"127.0.0.1:8501","token":"<redacted>","ssl":true}} caller=config
INFO[0000] Loaded repository example caller=repository
INFO[0000] KV GET ref: example/master caller=consul
INFO[0000] KV PUT changes: example/master
DEBUG[0000] KV PUT: example/master: example/master/Dummyfile caller=consul
DEBUG[0000] KV PUT: example/master: example/master/ESDL-database.accdb caller=consul
DEBUG[0000] KV PUT: example/master: example/master/README.md caller=consul
DEBUG[0000] Up to date: example/master caller=watcher
INFO[0000] Stopping watcher... caller=watcher
INFO[0000] Received finish caller=watcher
INFO[0001] Watcher received finish caller=runner
ERROR[0001] Runner error error=Failed request: Request body(830718 bytes) too large, max size: 524288 bytes. See https://www.consul.io/docs/agent/options.html#txn_max_req_len.
as per documentation at this link it is not recommended to change the size of this request, so when looping the size of the request should be taken into account as well.
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (uname -a
)?
2Gb RAM
uname -a
Output$ uname -a Linux myvm 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Oct 21 13:44:38 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
What did you do?
/git2consul -debug -config aservice.json -once
{
"local_store": "/var/checkout/",
"repos": [
{
"name": "aservice",
"url": "ssh://[email protected]:2222/project/aservice.git",
"branches": [
"master"
],
"source_root": "/config/service/",
"skip_branch_name": true,
"mount_point": "config/"
}
]
}
What did you expect to see?
All repo's sync
What did you see instead?
Some repo's run out of memory, other sync fine.
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x92b3a7, 0x16)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc0c8000000, 0x38000000, 0xd01c58)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0xce93a0, 0x36b78000, 0xc00004fe00, 0x435577)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0xce93a0, 0x1b5bc, 0xffffffff)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0xce93a0, 0x1b5bc, 0xd01c68, 0x41d675)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0xce93a0, 0x1b5bc, 0x7f81303e0101, 0x7f81303e71f0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mheap.go:1093 +0x4c
runtime.(*mheap).alloc(0xce93a0, 0x1b5bc, 0xc000000101, 0x7f81303e71f0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/mheap.go:1092 +0x8a
runtime.largeAlloc(0x36b78000, 0x450100, 0x7f81303e71f0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/malloc.go:1138 +0x97
runtime.mallocgc.func1()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/proc.go:1146
goroutine 431 [running]:
runtime.systemstack_switch()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/asm_amd64.s:330 fp=0xc0001878c8 sp=0xc0001878c0 pc=0x458840
runtime.mallocgc(0x36b78000, 0x0, 0xc002549200, 0xc002b24000)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/malloc.go:1032 +0x895 fp=0xc000187968 sp=0xc0001878c8 pc=0x40c1b5
runtime.growslice(0x8638e0, 0xc079e9e000, 0x2bc60000, 0x2bc60000, 0x2bc68000, 0xc079e9e000, 0x23048000, 0x8000)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/slice.go:175 +0x14e fp=0xc0001879d0 sp=0xc000187968 pc=0x44336e
gopkg.in/src-d/go-git.v4/plumbing.(*MemoryObject).Write(0xc0029c0600, 0xc002b24000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/memory.go:53 +0xfd fp=0xc000187a38 sp=0xc0001879d0 pc=0x6e34 8d
io.copyBuffer(0x7f8130410340, 0xc0029c0600, 0x7f81304103f8, 0xc000156050, 0xc002b24000, 0x8000, 0x8000, 0x894980, 0x0, 0x7f81304103f8)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/io/io.go:404 +0x1fb fp=0xc000187ab0 sp=0xc000187a38 pc=0x4977db
io.CopyBuffer(0x7f8130410340, 0xc0029c0600, 0x7f81304103f8, 0xc000156050, 0xc002b24000, 0x8000, 0x8000, 0x0, 0xc000187bd8, 0x7b67ef)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/io/io.go:375 +0x82 fp=0xc000187b10 sp=0xc000187ab0 pc=0x497572
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).copyObject(0xc0001561e0, 0x7f8130410340, 0xc0029c0600, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/scanner.go:297 +0x2c6 fp=0xc000187bf0 sp=0xc 000187b10 pc=0x7a5d16
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).NextObject(0xc0001561e0, 0x7f8130410340, 0xc0029c0600, 0x0, 0xc000000000, 0x 0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/scanner.go:272 +0xbb fp=0xc000187c68 sp=0xc0 00187bf0 pc=0x7a598b
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).fillRegularObjectContent(0xc0003c8cc0, 0x9d1280, 0xc0029c0600, 0x7a6324, 0xc 0000699c0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:364 +0xa0 fp=0xc000187cb8 sp=0xc0 00187c68 pc=0x79d760
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).decodeByHeader(0xc0003c8cc0, 0xc0029f4400, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:304 +0x4b4 fp=0xc000187db0 sp=0xc 000187cb8 pc=0x79d4a4
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).doDecodeObject(0xc0003c8cc0, 0xc000074e81, 0x9d1280, 0xc0029c0580, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:233 +0x5d fp=0xc000187df0 sp=0xc0 00187db0 pc=0x79cb5d
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).DecodeObject(...)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:223
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).decodeObjects(0xc0003c8cc0, 0x19981, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:170 +0x4f fp=0xc000187e38 sp=0xc0 00187df0 pc=0x79c68f
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).doDecode(0xc0003c8cc0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:160 +0x189 fp=0xc000187ec8 sp=0xc 000187e38 pc=0x79c4f9
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Decoder).Decode(0xc0003c8cc0, 0x0, 0x0, 0xc000000000, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/decoder.go:139 +0xa0 fp=0xc000187f58 sp=0xc0 00187ec8 pc=0x79c230
gopkg.in/src-d/go-git.v4/storage/filesystem/internal/dotgit.(*PackWriter).buildIndex(0xc0000c4690)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/storage/filesystem/internal/dotgit/writers.go:64 +0x95 fp=0xc000187fd 8 sp=0xc000187f58 pc=0x7b5cc5
runtime.goexit()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000187fe0 sp=0xc000187fd8 pc=0x45a791
created by gopkg.in/src-d/go-git.v4/storage/filesystem/internal/dotgit.newPackWrite
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/storage/filesystem/internal/dotgit/writers.go:52 +0x324
goroutine 1 [select]:
main.main()
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/main.go:95 +0x76d
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/os/signal/signal_unix.go:29 +0x41
goroutine 7 [chan receive]:
gopkg.in/src-d/go-git.v4/storage/filesystem/internal/dotgit.(*PackWriter).waitBuildIndex(...)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/storage/filesystem/internal/dotgit/writers.go:79
gopkg.in/src-d/go-git.v4/storage/filesystem/internal/dotgit.(*PackWriter).Close(0xc0000c4690, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/storage/filesystem/internal/dotgit/writers.go:106 +0x9f
gopkg.in/src-d/go-git.v4/utils/ioutil.CheckClose(0x7f813041a758, 0xc0000c4690, 0xc000181948)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/utils/ioutil/common.go:89 +0x31
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.writePackfileToObjectStorage(0x7f813041a718, 0xc00005e780, 0x9c3d00, 0xc000654cd0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/common.go:51 +0x181
gopkg.in/src-d/go-git.v4/plumbing/format/packfile.UpdateObjectStorage(0x7f813040c2b8, 0xc00005e780, 0x9c3d00, 0xc000654cd0, 0xc00005e780, 0x9c3d00)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/format/packfile/common.go:30 +0x105
gopkg.in/src-d/go-git%2ev4.(*Remote).fetchPack(0xc0005b9b20, 0x9ce0e0, 0xc0000240d0, 0xc000347bd0, 0x9ccba0, 0xc0006548c0, 0xc0005774a0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/remote.go:361 +0x1fb
gopkg.in/src-d/go-git%2ev4.(*Remote).fetch(0xc0005b9b20, 0x9ce0e0, 0xc0000240d0, 0xc000181bd0, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/remote.go:298 +0x62f
gopkg.in/src-d/go-git%2ev4.(*Remote).FetchContext(...)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/remote.go:233
gopkg.in/src-d/go-git%2ev4.(*Repository).FetchContext(0xc000102ae0, 0x9ce0e0, 0xc0000240d0, 0xc000347bd0, 0x915b20, 0xc000576720)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/repository.go:786 +0xb5
gopkg.in/src-d/go-git%2ev4.(*Repository).Fetch(...)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/repository.go:764
github.com/KohlsTechnology/git2consul-go/repository.(*Repository).CheckoutBranch(0xc000102900, 0xc0005bbd80, 0x6, 0xc000347cb8, 0x1)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/repository/checkout.go:62 +0x11d
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).putBranch(0xc000102bd0, 0x9d3600, 0xc000102900, 0xc0005bbd80, 0x6, 0x2, 0x0)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/branch.go:32 +0x63
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).UpdateToHead(0xc000102bd0, 0x9d3600, 0xc000102900, 0x0, 0x1)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/update.go:78 +0x627
github.com/KohlsTechnology/git2consul-go/kv.(*KVHandler).HandleUpdate(0xc000102bd0, 0x9d3600, 0xc000102900, 0x0, 0x0)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/kv/update.go:47 +0x26c
github.com/KohlsTechnology/git2consul-go/runner.(*Runner).Start(0xc000069f40)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/runner/runner.go:97 +0xbf
created by main.main
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/main.go:84 +0x5d8
goroutine 17 [select]:
github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).Watch(0xc00005e960)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/watcher.go:90 +0x32a
created by github.com/KohlsTechnology/git2consul-go/runner.(*Runner).Start
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/runner/runner.go:88 +0x99
goroutine 18 [semacquire]:
sync.runtime_SemacquireMutex(0xc000102904, 0xc00006fb00, 0x1)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc000102900)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/sync/mutex.go:81
github.com/KohlsTechnology/git2consul-go/repository.(*Repository).Pull(0xc000102900, 0xc00013a1a8, 0x6, 0x0, 0x0)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/repository/pull.go:28 +0x45f
github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).pollBranches.func1(0xc000160040, 0xc000160040, 0x0)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/interval.go:86 +0x124
gopkg.in/src-d/go-git.v4/plumbing/storer.(*referenceFilteredIter).ForEach(0xc0001446a0, 0xc00013c360, 0x0, 0x0)
/home/travis/gopath/pkg/mod/gopkg.in/src-d/[email protected]/plumbing/storer/reference.go:82 +0xb8
github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).pollBranches(0xc00005e960, 0x9d3600, 0xc000102900, 0x0, 0x0)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/interval.go:99 +0x178
github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).pollByInterval(0xc00005e960, 0x9d3600, 0xc000102900, 0xc00013a000)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/interval.go:56 +0x14e
created by github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).Watch
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/watcher.go:75 +0x1af
goroutine 20 [semacquire]:
sync.runtime_Semacquire(0xc00013a008)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00013a000)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/sync/waitgroup.go:130 +0x64
github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).Watch.func1(0xc00013a000, 0xc00005e960)
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/watcher.go:81 +0x2f
created by github.com/KohlsTechnology/git2consul-go/watcher.(*Watcher).Watch
/home/travis/gopath/src/github.com/KohlsTechnology/git2consul-go/watcher/watcher.go:80 +0x225
goroutine 29 [chan receive]:
golang.org/x/crypto/ssh.(*Client).handleChannelOpens(0xc000156190, 0xc0001386c0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:147 +0x52
created by golang.org/x/crypto/ssh.NewClient
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:60 +0xc3
goroutine 23 [IO wait]:
internal/poll.runtime_pollWait(0x7f813040c0b8, 0x72, 0xffffffffffffffff)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00011a998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00011a980, 0xc0000a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00011a980, 0xc0000a4000, 0x1000, 0x1000, 0xc000138060, 0xc000071dd0, 0xc000071be8)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e598, 0xc0000a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/net.go:184 +0x68
net/http.(*persistConn).Read(0xc0000905a0, 0xc0000a4000, 0x1000, 0x1000, 0xc000071ed8, 0x457d40, 0xc000071ed8)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/http/transport.go:1752 +0x75
bufio.(*Reader).fill(0xc000138240)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc000138240, 0x1, 0x2, 0x0, 0x0, 0xc00006a000, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0000905a0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/http/transport.go:1905 +0x1d6
created by net/http.(*Transport).dialConn
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/http/transport.go:1574 +0xafe
goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc0000905a0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/http/transport.go:2204 +0x123
created by net/http.(*Transport).dialConn
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/http/transport.go:1575 +0xb23
goroutine 28 [chan receive]:
golang.org/x/crypto/ssh.(*Client).handleGlobalRequests(0xc000156190, 0xc000138780)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:138 +0x6d
created by golang.org/x/crypto/ssh.NewClient
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:59 +0x97
goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7f813040be48, 0x72, 0xffffffffffffffff)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0001ee218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001ee200, 0xc00018c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0001ee200, 0xc00018c000, 0x1000, 0x1000, 0xb18e5b016f1db66e, 0x7434c23a7838c282, 0x17af8281b9ea38d)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0005100c0, 0xc00018c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc000138360, 0xc0001ee5a1, 0x5, 0x5, 0x0, 0x20, 0xc000020a80)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x9c3660, 0xc000138360, 0xc0001ee5a1, 0x5, 0x5, 0x5, 0x7f8132678008, 0x0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/io/io.go:310 +0x87
io.ReadFull(...)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/io/io.go:329
golang.org/x/crypto/ssh.(*streamPacketCipher).readPacket(0xc0001ee580, 0xc000000eca, 0x9c3660, 0xc000138360, 0xc0003e2000, 0x5, 0x803f, 0 x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/cipher.go:154 +0x83
golang.org/x/crypto/ssh.(*connectionState).readPacket(0xc000110000, 0xc000138360, 0xc003290698, 0xc000075f30, 0x4051d9, 0xc0001384d8, 0xc 000118ee8)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/transport.go:130 +0x68
golang.org/x/crypto/ssh.(*transport).readPacket(0xc000110000, 0x0, 0x5, 0x5, 0x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/transport.go:114 +0x3b
golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket(0xc0005c4420, 0xc000075f00, 0xc003290698, 0x5, 0x5, 0x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:374 +0x50
golang.org/x/crypto/ssh.(*handshakeTransport).readLoop(0xc0005c4420)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:197 +0x4d
created by golang.org/x/crypto/ssh.newClientTransport
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:134 +0x181
goroutine 26 [select]:
golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc0005c4420)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:268 +0xfa
created by golang.org/x/crypto/ssh.newClientTransport
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:135 +0x1a3
goroutine 27 [chan receive]:
golang.org/x/crypto/ssh.(*handshakeTransport).readPacket(0xc0005c4420, 0xc003290698, 0x5, 0x5, 0x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/handshake.go:187 +0x4e
golang.org/x/crypto/ssh.(*mux).onePacket(0xc0000fc000, 0x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/mux.go:215 +0x37
golang.org/x/crypto/ssh.(*mux).loop(0xc0000fc000)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/mux.go:190 +0x35
created by golang.org/x/crypto/ssh.newMux
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/mux.go:128 +0x153
goroutine 30 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0000696d0, 0x0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0000696c0)
/home/travis/.gimme/versions/go1.13.5.linux.amd64/src/sync/cond.go:56 +0x9d
golang.org/x/crypto/ssh.(*mux).Wait(0xc0000fc000, 0x0, 0x0)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/mux.go:110 +0x8c
golang.org/x/crypto/ssh.NewClient.func1(0xc000156190)
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:62 +0x33
created by golang.org/x/crypto/ssh.NewClient
/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:61 +0xe5
Releases are not automated. It is not fun for project maintainers to manually create releases. Also, we don't provide compiled binaries for releases which is just silly.
This project should use https://goreleaser.com for release automation. TravisCI or GitHub Actions can be used to automagically run the goreleaser
command.
n/a
n/a
n/a
n/a
README.md
for repos:hooks:type
says that available values are polling, github, stash, bitbucket, gitlab. According to load.go only polling
and webhook
are supported. Putting anything else results in an Invalid hook type
error.
bitbucket
.Invalid hook type: bitbucket
What version of git2consul-go are you using?
$ git2consul -version git2consul, version v0.1.1 (branch: HEAD, revision: abdf3a1d746539f08ce5122f9d7db43ae4e79afd) build date: 20200106-19:50:06 go version: go1.13.5
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (uname -a
)?
uname -a
Output$ uname -a Darwin Charlie-Jones 18.7.0 Darwin Kernel Version 18.7.0: Sat Oct 12 00:02:19 PDT 2019; root:xnu-4903.278.12~1/RELEASE_X86_64 x86_64
What did you do?
1 - Create a config with an invalid url (note https in url):
{
"repos": [
{
"name": "test",
"url": "https://github.com/ryanbreen/git2consul_data.git",
"credentials": {
"private_key": {
"pk_key": "/Users/charlie/tmp/git2consul/private_key.rsa",
"pk_username": "git"
}
}
}
]
}
2 - Start git2consul:
git2consul -config ./config.json -debug
INFO[0000] Starting git2consul version: v0.1.1
INFO[0000] Setting configuration with sane defaults caller=config
DEBUG[0000] Using configuration: {"local_store":"/var/folders/3p/rr4q161s7936_28d_9njv5m80000gp/T/","webhook":{"port":9000},"repos":[{"name":"test","url":"https://github.com/ryanbreen/git2consul_data.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root":"","mount_point":"","credentials":{"private_key":{"pk_key":"/Users/charlie/tmp/git2consul/private_key.rsa","pk_username":"git"}}}],"consul":{"address":"127.0.0.1:8500","ssl":false}} caller=config
ERROR[0000] (runner): Cannot load repositories from configuration: Error loading test: invalid auth method
3 - Look in local_store (note the existance of .git directory):
ls -la /var/folders/3p/rr4q161s7936_28d_9njv5m80000gp/T/test
total 0
drwxr-xr-x 3 charlie staff 96B Jan 30 06:55 .
drwx------@ 11 charlie staff 352B Jan 30 06:55 ..
drwxr-xr-x 6 charlie staff 192B Jan 30 06:55 .git
3 - Fix invalid url (note ssh in url now):
{
"repos": [
{
"name": "test",
"url": "ssh://github.com/ryanbreen/git2consul_data.git",
"credentials": {
"private_key": {
"pk_key": "/Users/charlie/tmp/git2consul/private_key.rsa",
"pk_username": "git"
}
}
}
]
}
4 - Try to start git2consul with correct configuration (note "reference not found"):
git2consul -config ./config.json -debug
INFO[0000] Starting git2consul version: v0.1.1
INFO[0000] Setting configuration with sane defaults caller=config
DEBUG[0000] Using configuration: {"local_store":"/var/folders/3p/rr4q161s7936_28d_9njv5m80000gp/T/","webhook":{"port":9000},"repos":[{"name":"test","url":"ssh://github.com/ryanbreen/git2consul_data.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root":"","mount_point":"","credentials":{"private_key":{"pk_key":"/Users/charlie/tmp/git2consul/private_key.rsa","pk_username":"git"}}}],"consul":{"address":"127.0.0.1:8500","ssl":false}} caller=config
INFO[0000] Loaded repository test caller=repository
ERROR[0001] Runner error error=reference not found
5 - Clear local_store, and try again with the same config, and it starts up fine:
rm -rf /var/folders/3p/rr4q161s7936_28d_9njv5m80000gp/T/test
INFO[0000] Starting git2consul version: v0.1.1
INFO[0000] Setting configuration with sane defaults caller=config
DEBUG[0000] Using configuration: {"local_store":"/var/folders/3p/rr4q161s7936_28d_9njv5m80000gp/T/","webhook":{"port":9000},"repos":[{"name":"test","url":"ssh://github.com/ryanbreen/git2consul_data.git","branches":["master"],"hooks":[{"type":"polling","interval":60000000000}],"source_root":"","mount_point":"","credentials":{"private_key":{"pk_key":"/Users/charlie/tmp/git2consul/private_key.rsa","pk_username":"git"}}}],"consul":{"address":"127.0.0.1:8500","ssl":false}} caller=config
INFO[0001] Cloned repository test caller=repository
INFO[0001] KV GET ref: test/master caller=consul
DEBUG[0001] Up to date: test/master caller=watcher
...
What did you expect to see?
I expected git2consul to start up successfully after I corrected my configuration error.
What did you see instead?
git2consul errored with reference not found
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.