xorpaul / g10k Goto Github PK

View Code? Open in Web Editor NEW

124.0 124.0 50.0 4.3 MB

my r10k fork in Go

License: Apache License 2.0

Go 98.13% Shell 1.02% Dockerfile 0.30% Makefile 0.54%

forge-modules golang puppet puppet-control puppet-git puppet-modules puppetfile puppetlabs-forge

g10k's People

Contributors

Stargazers

Watchers

Forkers

mattriesterer bkuebler mikey179 tuxmea ptierno themeier ntnn slm0n87 stuxboulot gdubicki devopsbox alexharv074 mooreandrew jovrum chutzimir rogerlz r0b1ndot mcanevet weaselshit nrqmartinez nrq-martinez nabertrand juliantaylor innogames preisbeck cbowman0 pmuller maxserv giedriuss pillarsdotnet maxenced iguanesolutions elserhumano maxadamo sedoj33 ptrtonull pasonsystems dreamblack86 davidgamba sanjaymsh tmack0 nudgegoonies s-johansson roock andni233 mmolnar krinkle dhollinger eternalerrors cmd-ntrf j0sh3rs

g10k's Issues

variable interpolation in Puppetfile not working

Hi.
I am using variables in my Puppetfile which work with r10k, but they don't seem to work with g10k.

Example from my Puppetfile:

PUPPETENVIRONMENT='test'
DEPOT="ssh://[email protected]/puppet"

mod 'company_firewall',
  :git => "#{DEPOT}/company_firewall",
  :ref => "#{PUPPETENVIRONMENT}"

by doing taht, my puppetfile was almost the same across branches. Only this line was changing:
PUPPETENVIRONMENT='test'

Question: Concurrent Cache Access?

Hello,

This is not an issue but more of a general question regarding the cache.

Can I share the cache between concurrent runs? Say I have my control-repo, a bunch of private modules and a bunch of forge modules. Can I run multiple instances of g10k at the same time while using the same cachedir?

I looked quickly at the code and it doesn't seem to be any kind of lock except for mutexes?

Thanks again for all your hard work!

Regards,
Mathieu

Optionally specifying filename to -puppetfile

librarian-puppet allows the name Modulefile with the same syntax as Puppetfile, and g10k could be used as a drop-in replacement. Also, such an option could be used to specify a Puppetfile in a different directory than the working directory.

Support for deploying just a single module

r10k supports deploying a single module to a specific environment like so:

r10k deploy module foobar -e production

This deploys module foobar to environment production but leaves other modules untouched.

This is useful when pinning a lot of modules to the latest version or head of a git branch and only doing full environment deploys on a regular basis. It would be really nice if g10k supported the same workflow.

Problems with puppet runs during g10kr

Hi,

I have ~50 environments/branches in my production repository (yeah, I know) and I'm using g10k to deploy the Puppetfiles which is 10x faster than r10k. Thanks for that.

Due to the number of environments and IO of my EC2 puppet master, g10k is taking ~20s to run. Having ~400 puppet agents behind the master, almost always there is puppet run during the g10k run and, of course, an incomplete repository.

Provisioning more IO to the EC2 seems to reduce the issue since the g10k will be able to finish even faster, but the cost math becomes a problem, especially with 15 puppet masters.

I thought in a feature to symlink/move the environment dir after the g10k successful run.

Any better solution?

Thanks

git checkouts wich ask for auth input must be serialized

if you have multiple references to git modules in your Puppetfile wich will trigger git to prompt you for a password input g10k will give you multiple password inputs all at once.
i would expect g10k to serialize the git interactive inputs.

Please add possibility to disable ":latest" in environment yaml

Can you please add a configparameter in the environment yamls to disable :latest or no version pinning only for forgemodules? The g10k run should fail if the parameter is set to true and no version or :latest is given.

limiting concurrent sessions

I was wondering whether it could be convenient to have an option to limit the number of sessions.
In my case I have around 85 repository, and I see that gitlab is not very happy. Soon I am gonna add a second puppet server and therefore, I'll need to use flock to span the trigger once at time .

I could also considering to go back to Go(-lang), try to add this option and make a pull request....

p.s.: congrats for the nice job that you have done, which I like very much.
With r10k I was able to fetch the repositories in almost 5 minutes. Thanks to g10k (also adding ssh control master, and one CPU to Gitlab) I am able to fetch in 8 seconds!!

Please make g10k inform about missing branches in the sources

I am using v0.2.9b.

This is my config file:

---
cachedir: '/var/cache/g10k'
sources:
  hieradata:
    remote: [email protected]:puppet/hiera.git
    basedir: /etc/puppet/hieradata
  envs:
    remote: [email protected]:puppet/envs.git
    basedir: /etc/puppet/environments

...and this is the Puppetfile in g10k_test branch of envs repo:

mod 'puppetboard',
  :git => 'https://github.com/nibalizer/puppet-module-puppetboard.git',
  :ref => '2.7.1'

When I run g10k with:

g10k -config /etc/puppet/g10k.yaml -branch g10k_test

..while having g10k_test branch in envs repo but NOT in the hiera repo I get the following output:

# g10k -config /etc/puppet/g10k.yaml -branch g10k_test
Resolving Git modules (1/1)    0s [====================================================================] 100%
Synced /etc/puppet/g10k.yaml with branch g10k_test with 2 git repositories and 0 Forge modules in 2.0s with git (3.4s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s)

It would be nice if g10k could inform (warn?) me about missing branch in hieradata source.

(r10k 2.5.1 also doesn't inform about missing branches, so this is definitely a feature request, not a bug report.)

PS I was also a bit surprised to see 2 git repos synced reported above. Which repos are counted here, the 2 sources repos?

Allow overriding moduledir set in puppetfile with `-moduledir $dir` flag

This would primarily be handy to use g10k as a fixture tool to place modules in a temporary directory to refer to while testing.

It would be great for both the puppet- and configfile mode.

Doesn't like comments

I have a Puppetfile dec like this:

mod 'sensu', # Puppet 4 supported
     :git => 'github.com/sensu/sensu-puppet.git',
     :commit => '8f4fc5780071c4895dec559eafc6030511b0caaa'

g10k doesn't like this, and bombs :(

2016/08/21 13:39:57 Error: trailing comma found in/etc/puppet/environments/env/Puppetfilesomewhere here:      :git => 'github.com/sensu/sensu-puppet.git',     :commit => '8f4fc5780071c4895dec559eafc6030511b0caaa'

Terminal text stays red after help output

After calling g10k without a config file, the console text stays red:

~/sw/go/bin$ ./g10k 
2017/06/16 09:51:35 Error: you need to specify at least a config file or use the Puppetfile mode
Example call: ./g10k -config test.yaml or ./g10k -puppetfile
kheymann@dodo:~/sw/go/bin$  # console text is now red

Add r10k's purge_level, revert unmanaged/change files

It would be great if the r10k feature purge_level could be added. Currently $someone could manually edit the deployed module/environment without g10k noticing.

That said, we're looking for a feature which verifies on each g10k run that all files are in the state the Puppetfile/control repository describes them.

Support entries like "mod 'pcfens-filebeat', '0.10.3'"

Currently g10k fails to deploy Puppetfile with such entry with this error:

2017/03/15 21:43:12 Error: Missing :git url in /etc/puppet/environments/_myenv/Puppetfile for module pcfens-filebeat line: mod 'pcfens-filebeat', '0.10.3'

I suppose that it would accept it if it was mod 'pcfens/filebeat', '0.10.3' (slash instead of dash).

Different behavior between r10k and g10k when referencing Puppetforge module with git address

I just stumbled about a difference between r10k and g10k. Given the following definition in my Puppetfile:

mod 'mayflower/php',
  :git => 'https://github.com/mayflower/puppet-php.git',
  :tag => '3.4.2'

With g10k the following error appears:

2015/11/10 15:56:56 downloadForgeModule(): Unexpected response code while GETing https://forgeapi.puppetlabs.com/v3/files/mayflower-php-:git => .tar.gz400 Bad Request

With r10k instead, it will create simply drop the mayflower/ part and create the folder php where it stores the module in.

mayflower/php is the official name of the Puppetforge module which I'd like to use for reference purpose, but as Puppetforge modules are sometimes out of date I like to reference the Github repository with the tag directly. Or, people use the forge module first and switch to the Github repo later but don't rename it in the Puppetfile and might run into this problem then.

Puppetfile option with a config file

Hello,

This issue is somewhat related to issue #70 where with r10k, I can have both a simple config file and still use: puppetfile install.

My yaml config (r10k.yaml) file contains:

---
cachedir: /tmp/somerandomdirectory
forge: {baseurl: 'http://myforge:42080'}
git:
    private_key: /home/user/.ssh/id_rsa
    provider: rugged

And I run r10k with the following options (which works properly):

r10k -c r10k.yaml puppetfile install -v info --puppetfile Puppetfile --moduledir modules

If I try the same thing with g10k, I get this message "-puppetfile parameter is not allowed with -config parameter" which is different from what r10k supports: is there a reason for that?

So either:

I figure out how to write a full file based configuration using my Puppetfile (so far I haven't haha)
or like with #70, provide a way to set more than just cachedir
or support a config file with -puppetfile parameter

Cheers,
Mathieu

Some valid Ruby Puppetfiles are not interpreted

When executing g10k -puppetfile I get:

$ g10k -puppetfile 
Error: Missing :git url in ./Puppetfile for module ad line: mod 'ad',git: '[email protected]:FOO/puppet-ad',ref: 'master'

My Puppetfile has modules declared as:

mod 'ad',
  git: '[email protected]:FOO/puppet-ad',
  ref: 'master'
...

Workaround for me was to rewrite the Puppetfiles using a regexp:

:%s/^  \(...\): /  :\1 =>

Enhancement: Puppetfile - module link support

If you create environments like "production/development/staging/testing" and so on it would be a nice feature to link these environment branches again your puppet module branches. For example:

Environment "production" <=> Environment branch "production" <=> Module branch "production"

The default solution to do this is to have different "Puppetfile" configuration in your environments:

Puppetfile (production)

mod 'awesomemodule',
  :git => 'http://github.com/foo/bar.git',
  :branch => 'production'

Puppetfile (development)

mod 'awesomemodule',
  :git => 'http://github.com/foo/bar.git',
  :branch => 'development'

But in this scenario you will loose the nice git workflow for merge between the environments. Because each Puppetfile is different and if you would like to use your puppet environments as real Changemanagement stages for example:

Do puppet changes in development environment => merge these changes to testing/staging environment => merge testing/staging environment into production.

So a solution to get this working is support i called it "module link support". The Puppetfile could be looks like this:

mod 'awesomemodule',
  :git => 'http://github.com/foo/bar.git',
  :link => 'true'

So each Puppetfile in environments is the same, but the used module in each environment is on different branch with the same name of the environment branch.

I'll send you a merge request for this feature in the next time.

regards Benjamin

-version doesn't show correct version

I've just discovered that (at least for the last 3 releases) the -version parameter outputs the wrong version number:

mbaur@mbaur-g10k:~/test$ wget https://github.com/xorpaul/g10k/releases/download/v0.3.12/g10k-linux-amd64.zip
--2017-11-07 14:58:06--  https://github.com/xorpaul/g10k/releases/download/v0.3.12/g10k-linux-amd64.zip
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/39644433/607f6d42-8f39-11e7-8c70-f842e8cb830a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171107%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171107T135806Z&X-Amz-Expires=300&X-Amz-Signature=d2de63fd80cf53bb26227c238916a14f49efcfeba97749696905bb92263b771d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dg10k-linux-amd64.zip&response-content-type=application%2Foctet-stream [following]
--2017-11-07 14:58:06--  https://github-production-release-asset-2e65be.s3.amazonaws.com/39644433/607f6d42-8f39-11e7-8c70-f842e8cb830a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171107%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171107T135806Z&X-Amz-Expires=300&X-Amz-Signature=d2de63fd80cf53bb26227c238916a14f49efcfeba97749696905bb92263b771d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dg10k-linux-amd64.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.131.187
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.131.187|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2407015 (2.3M) [application/octet-stream]
Saving to: ‘g10k-linux-amd64.zip’

g10k-linux-amd64.zip                          100%[==============================================================================================>]   2.29M  1.62MB/s    in 1.4s

2017-11-07 14:58:08 (1.62 MB/s) - ‘g10k-linux-amd64.zip’ saved [2407015/2407015]

mbaur@mbaur-g10k:~/test$ unzip g10k-linux-amd64.zip
Archive:  g10k-linux-amd64.zip
  inflating: g10k
mbaur@mbaur-g10k:~/test$ ./g10k -version
g10k Version 1.0 Build time: 2017-09-01_15:16:52 UTC

Can this please the synced to the tag version? Thanks! :)

debug switch not working in puppetfile mode

the debug switch seems not to be working in puppetfile mode

10:39 $ ./g10k -debug -config ./r10k.yaml 2016/08/11 10:39:24 DEBUG Using as config file: ./r10k.yaml 2016/08/11 10:39:24 DEBUG Using as cachedir: ./tmp/g10k/ 2016/08/11 10:39:24 DEBUG Using as cachedir/forge: ./tmp/g10k/forge/ 2016/08/11 10:39:24 DEBUG Using as cachedir/modules: ./tmp/g10k/modules/ 2016/08/11 10:39:24 DEBUG Using as cachedir/environments: ./tmp/g10k/environments/ Synced ./r10k.yaml with 0 git repositories and 0 Forge modules in 0.0s with git (0.0s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s)

10:39 $ ./g10k -puppetfile install -debug -config ./r10k.yaml Synced ./Puppetfile with 14 git repositories and 35 Forge modules in 3.1s with git (23.5s sync, I/O 0.8s) and Forge (0.0s query+download, I/O 1.9s)

g10k fails on some yaml configs that are valid and work with r10k

g10k fails to parse the following config correctly:

---
:cachedir: '/var/cache/r10k'
:sources:
  :hieradata:
    remote: [email protected]:puppet/hiera.git
    basedir: /etc/puppet/hieradata
  :envs:
    remote: [email protected]:puppet/envs.git
    basedir: /etc/puppet/environments

...with the following output:

Synced /etc/puppetlabs/r10k/r10k.yaml with 0 git repositories and 0 Forge modules in 0.0s with git (0.0s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s)

That config with removed some of :s, like this:

---
cachedir: '/var/cache/r10k'
sources:
  hieradata:
    remote: [email protected]:puppet/hiera.git
    basedir: /etc/puppet/hieradata
  envs:
    remote: [email protected]:puppet/envs.git
    basedir: /etc/puppet/environments

...works correctly in g10k.

Note that the first one is valid YAML, according to http://www.yamllint.com/ AND is correctly parsed by r10k.

Of course it was easy to resolve, but it was not so easy to come up with the reason g10k is not working, so perhaps it's worth to do something about it.

subcommands missing

in order to be a r10k replacement all subcommands should be implemented.

COMMANDS
deploy Puppet dynamic environment deployment
help show help
puppetfile Perform operations on a Puppetfile
version Print the version of r10k

Puppetfile mode: allow configuration of cache directories

When using g10k -puppetfile inside a Docker container the final copy from /tmp to module dir causes problems when both are on different devices, e.g. because the module dir is mounted into the Docker container. This could be solved when the local forge directory could be influenced via configuration or command line argument.

For reference, here is the output of a g10k -puppetfile run in such a container:

root@c748758fac38:~/environments# g10k -puppetfile -verbose
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_dalenys_puppet-supervisor.git remote update --prune took 0.71367s
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_xp-framework_puppet-xp-runners.git remote update --prune took 0.72577s
2016/01/26 10:25:47 Executing git --git-dir /tmp/https-__github.com_mayflower_puppet-php.git remote update --prune took 0.81712s
2016/01/26 10:25:47 Executing git --git-dir /tmp/[email protected]_modules_core_sshd.git remote update --prune took 0.93008s
2016/01/26 10:25:47 Executing git --git-dir /tmp/[email protected]_modules_core_facts.git remote update --prune took 1.14196s
2016/01/26 10:25:47 Executing git --git-dir /tmp/[email protected]_modules_core_osum.git remote update --prune took 1.29663s
2016/01/26 10:25:47 Executing git --git-dir /tmp/[email protected]_modules_core_syslog.git remote update --prune took 1.32702s
2016/01/26 10:25:47 Executing git --git-dir /tmp/[email protected]_installapp.git remote update --prune took 1.34041s
Need to sync .//external_modules/ntp/
Need to sync .//external_modules/postfix/
Need to sync .//external_modules/apt/
Need to sync .//external_modules/inifile/
Need to sync .//external_modules/stdlib/
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_mayflower_puppet-php.git log -n1 --pretty=format:%H 3.4.2 took 0.04670s
2016/01/26 10:25:48 Executing git --git-dir /tmp/[email protected]_modules_core_facts.git log -n1 --pretty=format:%H master took 0.04681s
Need to sync .//external_modules//php
Need to sync .//external_modules/concat/
2016/01/26 10:25:48 Executing git --git-dir /tmp/[email protected]_modules_core_syslog.git log -n1 --pretty=format:%H master took 0.04797s
2016/01/26 10:25:48 Executing git --git-dir /tmp/[email protected]_modules_core_osum.git log -n1 --pretty=format:%H master took 0.04800s
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_dalenys_puppet-supervisor.git log -n1 --pretty=format:%H master took 0.05688s
2016/01/26 10:25:48 Executing git --git-dir /tmp/[email protected]_installapp.git log -n1 --pretty=format:%H master took 0.05132s
2016/01/26 10:25:48 Executing git --git-dir /tmp/https-__github.com_xp-framework_puppet-xp-runners.git log -n1 --pretty=format:%H master took 0.05062s
2016/01/26 10:25:48 Executing git --git-dir /tmp/[email protected]_modules_core_sshd.git log -n1 --pretty=format:%H master took 0.06124s
2016/01/26 10:25:48 Executing cp --link --archive /tmp/puppetlabs-ntp-4.1.0/* .//external_modules/ntp/ took 0.70185s
2016/01/26 10:25:48 Failed to execute command:  cp --link --archive /tmp/puppetlabs-ntp-4.1.0/* .//external_modules/ntp/  Output:  cp: cannot create hard link './/external_modules/ntp/CHANGELOG.md' to '/tmp/puppetlabs-ntp-4.1.0/CHANGELOG.md': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/CONTRIBUTING.md' to '/tmp/puppetlabs-ntp-4.1.0/CONTRIBUTING.md': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/Gemfile' to '/tmp/puppetlabs-ntp-4.1.0/Gemfile': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/LICENSE' to '/tmp/puppetlabs-ntp-4.1.0/LICENSE': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/Rakefile' to '/tmp/puppetlabs-ntp-4.1.0/Rakefile': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/checksums.json' to '/tmp/puppetlabs-ntp-4.1.0/checksums.json': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/lib/puppet/parser/functions/ntp_dirname.rb' to '/tmp/puppetlabs-ntp-4.1.0/lib/puppet/parser/functions/ntp_dirname.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/config.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/config.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/init.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/init.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/install.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/install.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/params.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/params.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/manifests/service.pp' to '/tmp/puppetlabs-ntp-4.1.0/manifests/service.pp': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/metadata.json' to '/tmp/puppetlabs-ntp-4.1.0/metadata.json': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/class_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/class_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/disable_monitoring_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/disable_monitoring_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-59-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-59-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-64-x64-pe.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-64-x64-pe.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-64-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-64-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/centos-65-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/centos-65-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/default.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/default.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/fedora-18-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/fedora-18-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/fedora-21-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/fedora-21-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/sles-11-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/sles-11-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/sles-12-64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/sles-12-64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-10044-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-10044-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-12042-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-12042-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-1404-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-1404-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/nodesets/ubuntu-server-14042-x64.yml' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/nodesets/ubuntu-server-14042-x64.yml': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_config_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_config_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_install_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_install_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_parameters_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_parameters_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/ntp_service_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/ntp_service_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/preferred_servers_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/preferred_servers_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/restrict_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/restrict_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/acceptance/unsupported_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/acceptance/unsupported_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/classes/ntp_spec.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/classes/ntp_spec.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec.opts' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec.opts': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec_helper.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec_helper.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/spec_helper_acceptance.rb' to '/tmp/puppetlabs-ntp-4.1.0/spec/spec_helper_acceptance.rb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/unit/puppet/provider/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/spec/unit/puppet/provider/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/spec/unit/puppet/type/README.markdown' to '/tmp/puppetlabs-ntp-4.1.0/spec/unit/puppet/type/README.markdown': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/templates/ntp.conf.erb' to '/tmp/puppetlabs-ntp-4.1.0/templates/ntp.conf.erb': Invalid cross-device link
cp: cannot create hard link './/external_modules/ntp/tests/init.pp' to '/tmp/puppetlabs-ntp-4.1.0/tests/init.pp': Invalid cross-device link

2016/01/26 10:25:48 syncForgeToModuleDir(): Error while trying to hardlink /tmp/puppetlabs-ntp-4.1.0/ to .//external_modules/ntp/ :exit status 1

Autocorrecting environment naming

r10k has this super useful feature to auto correcting environment names.

This is especially helpful if the branches naming follows the bug tracker (e.g. Jira/Youtrack/...) as they tend to be something like [A-Za-z]+-[0-9]+

Could this feature be please added to g10k? :)

g10k output is not log-friendly

The animated progress bars look cool when run interactively but look really bad in case you want to log the output to a file:

Resolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (6/51)  11ms [======>-------------------------------------------------------------]  12%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (12/51)  23ms [==============>-----------------------------------------------------]  24%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (13/51)  34ms [==============>-----------------------------------------------------]  24%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (19/51)  46ms [======================>---------------------------------------------]  35%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (20/51)  50ms [========================>-------------------------------------------]  39%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (24/51)  58ms [==============================>-------------------------------------]  47%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (27/51)  76ms [===================================>--------------------------------]  53%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (32/51)  81ms [=========================================>--------------------------]  63%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (34/51)  94ms [============================================>-----------------------]  67%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (35/51) 101ms [==============================================>---------------------]  69%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (38/51) 110ms [==================================================>-----------------]  75%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (43/51) 124ms [=======================================================>------------]  84%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (44/51) 140ms [==========================================================>---------]  86%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (46/51) 152ms [=============================================================>------]  90%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
Resolving Forge modules (47/51) 164ms [==============================================================>-----]  92%
ESC[0AESC[2K^MESC[0AESC[2K^MResolving Git modules (0/79)   --- [--------------------------------------------------------------------]   0%
(...)

It would be nice if you could disable those bars with a parameter.

error with symlinks

Hey,
I've just installed the newer version, and I have seen this error with symlinks:

2017/03/22 13:25:39 unTar(): Unable to untar type: 2 in file hostgroups/host-a.yaml

I rolled back to the older g10k and it started working again.

External modules installed to wrong location

External modules are installed to wrong locations when not using a prefix.

Example:

Config:

---
cachedir: /tmp/test/cache
sources:
  envs:
    remote: '[email protected]'
    basedir: /tmp/test/envs
    prefix: false

Puppetfile:

moduledir 'external_modules'

mod 'puppetlabs/stdlib', '4.13.1'

Executing g10k -config /tmp/test/test.yaml produces the following
structure:

% tree -d /tmp/test/envs
/tmp/test/envs
├── envs_master
│   └── external_modules
│       └── stdlib
│           ├── examples
│           ├── lib
│           │   ├── facter
│           │   │   └── util
│           │   └── puppet
│           │       ├── functions
│           │       ├── parser
│           │       │   └── functions
│           │       ├── provider
│           │       │   └── file_line
│           │       └── type
│           ├── manifests
│           ├── spec
│           │   ├── acceptance
│           │   │   └── nodesets
│           │   │       └── docker
│           │   ├── aliases
│           │   ├── fixtures
│           │   │   ├── dscacheutil
│           │   │   ├── lsuser
│           │   │   └── test
│           │   │       └── manifests
│           │   ├── functions
│           │   ├── monkey_patches
│           │   └── unit
│           │       ├── facter
│           │       │   └── util
│           │       └── puppet
│           │           ├── parser
│           │           │   └── functions
│           │           ├── provider
│           │           │   └── file_line
│           │           └── type
│           └── types
│               └── compat
└── master

39 directories

Here's the log file:

2016/11/15 00:54:32 DEBUG Using as config file: test.yaml
2016/11/15 00:54:32 DEBUG Using as cachedir: /tmp/test/cache/
2016/11/15 00:54:32 DEBUG Using as cachedir/forge: /tmp/test/cache/forge/
2016/11/15 00:54:32 DEBUG Using as cachedir/modules: /tmp/test/cache/modules/
2016/11/15 00:54:32 DEBUG Using as cachedir/environments: /tmp/test/cache/environments/
2016/11/15 00:54:32 DEBUG Using as basedir for source envs: /tmp/test/envs/
2016/11/15 00:54:32 DEBUG Puppet environment: envs (remote=/tmp/test/git, basedir=/tmp/test/envs/, private_key=, prefix=false)
2016/11/15 00:54:32 DEBUG Using as basedir: /tmp/test/envs/
2016/11/15 00:54:32 DEBUG Executing git clone --mirror /tmp/test/git /tmp/test/cache/environments/envs.git
2016/11/15 00:54:32 Executing git clone --mirror /tmp/test/git /tmp/test/cache/environments/envs.git took 0.04717s
2016/11/15 00:54:32 DEBUG Executing git --git-dir /tmp/test/cache/environments/envs.git branch
2016/11/15 00:54:32 Executing git --git-dir /tmp/test/cache/environments/envs.git branch took 0.00134s
2016/11/15 00:54:32 DEBUG Resolving branch: master
2016/11/15 00:54:32 DEBUG Executing git --git-dir /tmp/test/cache/environments/envs.git log -n1 --pretty=format:%H master
2016/11/15 00:54:32 Executing git --git-dir /tmp/test/cache/environments/envs.git log -n1 --pretty=format:%H master took 0.00144s
Need to sync /tmp/test/envs/master/
2016/11/15 00:54:32 DEBUG createOrPurgeDir(): Trying to create dir: /tmp/test/envs/master/ called from syncToModuleDir()
2016/11/15 00:54:32 syncToModuleDir(): Executing git --git-dir /tmp/test/cache/environments/envs.git archive master | tar -x -C /tmp/test/envs/master/ took 0.00317s
2016/11/15 00:54:32 DEBUG Executing git --git-dir /tmp/test/cache/environments/envs.git log -n1 --pretty=format:%H master
2016/11/15 00:54:32 Executing git --git-dir /tmp/test/cache/environments/envs.git log -n1 --pretty=format:%H master took 0.00145s
2016/11/15 00:54:32 DEBUG Writing hash 4ad8bc6049829a065c6273a79786b1c252ca5e0a from command git --git-dir /tmp/test/cache/environments/envs.git log -n1 --pretty=format:%H master to /tmp/test/envs/master//.latest_commit
2016/11/15 00:54:32 DEBUG readPuppetfile(): Trying to parse: /tmp/test/envs/master/Puppetfile
2016/11/15 00:54:32 DEBUG adding line:moduledir 'external_modules'
2016/11/15 00:54:32 DEBUG adding line:mod 'puppetlabs/stdlib', '4.13.1'
2016/11/15 00:54:32 DEBUG Resolving envs_master
2016/11/15 00:54:32 DEBUG Trying to get forge module puppetlabs-stdlib-4.13.1 with Forge base url and CacheTtl set to 0s
2016/11/15 00:54:32 DEBUG doModuleInstallOrNothing(): Trying to remove: /tmp/test/cache/forge/puppetlabs-stdlib-4.13.1
2016/11/15 00:54:32 DEBUG resolveGitRepositories(): uniqueGitModules[] is empty, skipping...
2016/11/15 00:54:32 DEBUG GETing https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.13.1.tar.gz
2016/11/15 00:54:33 GETing https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.13.1.tar.gz took 1.46960s
2016/11/15 00:54:33 DEBUG downloadForgeModule(): Trying to create /tmp/test/cache/forge/puppetlabs-stdlib-4.13.1.tar.gz
2016/11/15 00:54:35 Extracting https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.13.1.tar.gz took 1.46427s
2016/11/15 00:54:35 DEBUG Syncing envs_master
2016/11/15 00:54:35 DEBUG Using as basedir 2 for source envs: /tmp/test/envs/
2016/11/15 00:54:35 DEBUG Using as moduleDir for envs: /tmp/test/envs/envs_master/external_modules/
2016/11/15 00:54:35 DEBUG Using as as targetDir for module puppetlabs/stdlib: /tmp/test/envs/envs_master/external_modules/stdlib/
Need to sync /tmp/test/envs/envs_master/external_modules/stdlib/
2016/11/15 00:54:35 Executing cp --link --archive /tmp/test/cache/forge/puppetlabs-stdlib-4.13.1/* /tmp/test/envs/envs_master/external_modules/stdlib/ took 0.01201s
2016/11/15 00:54:35 DEBUG Forge response JSON parsing took 0.0000 seconds
2016/11/15 00:54:35 DEBUG Forge modules metadata.json parsing took 0.0000 seconds
Synced test.yaml with 1 git repositories and 1 Forge modules in 3.0s with git (0.1s sync, I/O 0.0s) and Forge (1.5s query+download, I/O 1.5s)

I believe the error stems from L143-L149 in puppetfile.go, where the moduleDir is fixed for puppetfile mode but not for config file mode.
I'm not sure how this could be fixed easily since the resolvePuppetfile function doesn't have direct access to the source struct containing the prefix information. One option would be to extend signature of the function with a boolean flag prefix bool which defines if the prefix should be used or not.

L145-147 could be changed to something akin to

if pfMode || !prefix {
	moduleDir = basedir + "/" + pf.moduleDir
}

Whole g10k run stops on a single broken branch

I have ~40 environments with Puppetfiles that I want to deploy with g10k.

But among those environments there are a few with Puppetfiles, which reference branches of some modules, which do not exist anymore.

In this case whole g10k run fails on this env.

I have two problems here:

I would prefer g10k to just skip broken modules and just notify me at the end of the run about problems per each environments.

But I guess that some people may prefer this to be configurable - skip or fail.

If I don't enable verbose mode I can't tell in which environment there is the reference to nonexistent branch as the output looks like this:

Resolving Git modules (117/117) 8.204s [====================================================================] 100%
Resolving Forge modules (103/103) 744ms [====================================================================] 100%
2017/05/11 20:31:42 executeCommand(): git command failed: git --git-dir /var/cache/g10k/modules/[email protected]_roles.git rev-parse --verify 'OPS_13195' exit status 128
Output: fatal: Needed a single revision

If you are using GitLab be sure that you added your deploy key to your repository

(I am using GitLab but of course the deploy key is in that repo - OPS_13195 branch isn't)

If I do enable verbose it gets better if the problem was in non-first module in the environment, but if it would be in the first one then I would be out of luck.

g10k always tries to download mod 'foo/bar' and mod 'foo-bar' from the forge

I am using v0.2.9b.

Entries in Puppetfile like this:

mod 'nibalizer/puppetboard',
  :git => 'https://github.com/nibalizer/puppet-module-puppetboard.git',
  :ref => '2.7.1'

..or like this:

mod 'nibalizer-puppetboard',
  (...)

..are incorrectly (as r10k 2.5.1 does it differently) treated as puppetforge modules even if they point to a git repo.

The result is:

2017/03/18 10:23:22 DEBUG readPuppetfile(): Trying to parse: /etc/puppet/environments/_g10k_test/Puppetfile
2017/03/18 10:23:22 DEBUG preparePuppetfile(): adding line:mod 'nibalizer/puppetboard',
2017/03/18 10:23:22 DEBUG preparePuppetfile(): adding line::git => 'https://github.com/nibalizer/puppet-module-puppetboard.git',
2017/03/18 10:23:22 DEBUG preparePuppetfile(): adding line::ref => '2.7.1'
2017/03/18 10:23:22 DEBUG readPuppetfile(): setting forge module nibalizer/puppetboard to version git
2017/03/18 10:23:22 DEBUG readPuppetfile(): found forge attribute ---> :git with value ---> https://github.com/nibalizer/puppet-module-puppetboard.git
2017/03/18 10:23:22 DEBUG readPuppetfile(): setting forge module nibalizer/puppetboard to version ref
2017/03/18 10:23:22 DEBUG readPuppetfile(): found forge attribute ---> :ref with value ---> 2.7.1
2017/03/18 10:23:22 DEBUG readPuppetfile(): Setting moduledir for Puppetfile /etc/puppet/environments/_g10k_test/Puppetfile to modules
2017/03/18 10:23:22 DEBUG resolvePuppetfile(): Resolving envs_g10k_test
2017/03/18 10:23:22 DEBUG resolveGitRepositories(): uniqueGitModules[] is empty, skipping...
2017/03/18 10:23:22 DEBUG resolveForgeModules(): Trying to get forge module nibalizer-puppetboard-ref with Forge base url  and CacheTtl set to 0s
2017/03/18 10:23:22 DEBUG doModuleInstallOrNothing(): Trying to remove: /var/cache/g10k/forge/nibalizer-puppetboard-ref
2017/03/18 10:23:22 DEBUG downloadForgeModule(): GETing https://forgeapi.puppetlabs.com/v3/files/nibalizer-puppetboard-ref.tar.gz
2017/03/18 10:23:22 GETing https://forgeapi.puppetlabs.com/v3/files/nibalizer-puppetboard-ref.tar.gz took 0.35643s
2017/03/18 10:23:22 downloadForgeModule(): Unexpected response code while GETing https://forgeapi.puppetlabs.com/v3/files/nibalizer-puppetboard-ref.tar.gz 400 Bad Request

Changing the entry to:

mod 'puppetboard',
  :git => 'https://github.com/nibalizer/puppet-module-puppetboard.git',
  :ref => '2.7.1'

...makes g10k work, but breaks the "drop-in replacement" promise. ;)

:default_branch and ':branch => :control_branch' are not supported

Hello,

It's me again! :-)

Looking at the code it appears g10k doesn't support a default_branch keyword nor using a fallback to the control_branch as per:
https://docs.puppet.com/pe/latest/cmgmt_puppetfile.html#set-a-default-branch-for-content-deployment

Set a default branch for content deployment
You can set a default_branch option, which specifies what branch to deploy content from if the given ref, tag, commit, or branch option cannot be resolved and deployed. This is mostly useful when you set branch to the :control_branch value.

If the desired content cannot be resolved and no default branch is given, or if the default branch cannot be resolved, an error is logged and the content is not deployed or updated.

As such, they give the following example (which is pretty much how I use it):

mod 'hieradata',
  :git    => '[email protected]:organization/hieradata.git',
  :branch => :control_branch,
  :default_branch => 'master'

Let me know if this is something you plan to support.

Cheers,
Mathieu

g10k -puppetfile deletes other stuff in moduledir

I would like to include roles and profiles in my control repo modules directory, as is common. Then I would like to use g10k to install the other modules from Puppetfile into the same directory.

It looks like r10k made this possible here. However, I see that g10k doesn't support this.

Is that something that's easy to fix?

g10k cache sometimes gets corrupted

From time to time I find that the g10k cache becomes corrupted and I am forced to delete it. This is a big problem in production and ultimately may mean I can't use g10k in production. A recent example was a failure like this:

executeCommand(): git command failed: git --git-dir /tmp/g10k/[email protected]_puppet-fstab.git remote update --prune exit status 128
Output: fatal: Not a git repository: '/tmp/g10k/[email protected]_puppet-fstab.git'

feature: check forge modules for updates in puppetfile mode

Add a switch to puppetfile mode to check forge modules in Puppetfile againts the forge api if newer versions are available. Output the update candidates on stdout. When activated only the check should be performed and no content must be changed.
For use in scripts a dedicated exit code which indicates if updates are available would be nice.

Support a config file

It would be great if this tool supported a config file at somewhere like ~/.g10k/config.yml. My immediate use-case is that I want to move the cachedir on my development laptop only, perhaps to /var/tmp/g10k because /tmp is cleaned every time the machine is booted. It looks like while I could specify an alternate cache dir on the command line, that's not ideal because I would need to either change it in the code which is also used in the CI system, or remember to type it each time I run g10k.

feature: only clone if branch/tag/commit exists

It would be cool additional to feature #6 bringing linking support to puppet environments to have the possibility to specify in the puppetfile to ignore checkout errors.
That could mean something like

mod 'site',
  :git => '[email protected]:pomailcore/site.git',
  :branch => "mybranch",
  :ignore-unreachable => 'True'

Or combined with the new feature referenced:

mod 'site',
  :git => '[email protected]:pomailcore/site.git',
  :link => 'True',
  :ignore-unreachable => 'True'

That causes all you puppetfiles in all environments to look the very same.
Now if you're working on a module and want to test these changes e.g. in your "QA" stage, you could branch the specific git repository to the branch "QA" and after triggering g10k it's available within the environment, without the need to manually specify any new versions through your Puppetfile.

Now it's not necessary anymore to keep all branches in all GIT Repositories "available" but just create them, once you need them.

Please update documentation about different behavior between r10k and g10k

First of all, thanks !
I was planning to implement something like r10k in GoLang and then I found your project. Big kudos to you !

Then I've tried to use g10k on the Puppetfile we're using and I hit bug #2 .
I had to dig into the code to really understand why it was mis-interpreting (differently as r10k) our git based modules as forge modules. And then I found the bug.

Can you please update the documentation and mention that this behavior is different than r10k one ? This could save next person some time :-)

Note : maybe that #2 can be closed if you do not plan to support it at all ?

Thanks !

Per item install_path is not supported

Hello,

Looking at the code, it appears "install_path" isn't supported. It is described as the following by puppetlabs:
https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd

Is it something you plan to support?

Cheers,
Mathieu

Per-Item Install Path

Git and SVN content types support installing into an alternate path without changing the value of moduledir by specifying an 'install_path' option:

mod 'apache',
  :git => '[email protected]:puppetlabs/puppetlabs-apache.git',
  :install_path => 'external'

The given 'install_path' can be an absolute path or a path relative to the base of the environment. Note that r10k will exit with an error if you attempt to set the 'path' option to a directory outside of the environment.

dependencies failing to install

I am trying to install this but getting the following:

Alexs-MacBook-Pro:g10k alexharvey$ export GOPATH=/Users/alexharvey/git/g10k
Alexs-MacBook-Pro:g10k alexharvey$ go get
# cd .; git clone https://gopkg.in/yaml.v2 /Users/alexharvey/git/g10k/src/gopkg.in/yaml.v2
Cloning into '/Users/alexharvey/git/g10k/src/gopkg.in/yaml.v2'...
fatal: unable to access 'https://gopkg.in/yaml.v2/': Unknown SSL protocol error in connection to gopkg.in:-9838
package gopkg.in/yaml.v2: exit status 128

I am completely new to Go. Is the issue here that g10k is depending on gopkg.in/yaml.v2 which no longer exists?

:ignore-unreachable alternative :ignore_unreachable

Ruby symbols cannot have a hyphen in their name, which causes other tools, which evaluate the Puppetfile as ruby code, to fail.

A viable alternative would be :ignore_unreachable, in addition to :ignore-unreachable.

Fallback-Option for linked=>true

The 'link => true' feature from bkuebler in #6 for syncing 'linked' environments is very cool and useful if you are working in predefined environments (dev->prelive->live).
But with this feature you are currently forced to create 'own' branches for every linked module to have a working puppet-environment (working g10k-sync).

So it is not easy to create own branches anymore for testing.

For that a kind of 'link-fallback' would be great.
So that g10k still tries to sync a branch with the same name as the environments branch, but if this does not exist using the fallback-branch.
That fallback-branch could for example be set 'hard-coded' to the default-branch of the git - or could be configured by an additional 'link-fallback'-attribute in Puppetfile.

Thx and Greets
Simon

Make URL for forgeAPI configurable

Please add configuration option similar to r10k to allow accessing forgeAPI via a private forge server.

g10k assumes cache directory is populated if it exists

If the cache directory exists, but is empty, g10k tries to update the non-existent repo instead of cloning it. Not a show stopping bug, but it took a bit to figure out why it kept telling me my control repo was unreachable.

Thanks for this project btw, our r10k runs were approaching 3 minutes and g10k does it in 3 seconds with only 25 workers!

---
:cachedir: '/var/cache/g10k'
sources:
  example:
    remote: 'https://github.localdomain/myorg/r10k.git'
    basedir: '/etc/puppetlabs/code/environments'

[root@puppetmaster ~]# mkdir /var/cache/g10k
[root@puppetmaster ~]# chown puppet:puppet /var/cache/g10k
[root@puppetmaster ~]# su - puppet
[puppet@puppetmaster ~]$ /opt/puppetlabs/puppet/bin/g10k -config /etc/puppetlabs/g10k/g10k_config.yaml
WARN: git repository https://github.localdomain/myorg/r10k.git does not exist or is unreachable at this moment!

Building on Windows

It appears these lines will fail when attempting to build on windows.

https://github.com/xorpaul/g10k/blob/master/forge.go#L689
https://github.com/xorpaul/g10k/blob/master/forge.go#L696

Problem with specific module version -beta1

Specific module:

https://forge.puppet.com/mayflower/php

Puppetfile

mod 'mayflower-php', '4.0.0-beta1'

Error

2017/03/23 08:28:57 downloadForgeModule(): Unexpected response code while GETing https://forgeapi.puppetlabs.com/v3/files/mayflower-php-4.0.0.tar.gz 404 Not Found

g10k seems to be ignoring/stripping the -beta1 from the version and therefore the package name?

GET'ng the mayflower-php-4.0.0-beta1.tar.gz package manually works correctly.

g10k unTar() sets 0644 for external facts

Expected behaviour: Files with executable bits in git have the executable bit set after synchronization
Actual behaviour: All files have mode of 0644

Error: https://github.com/xorpaul/g10k/blob/master/forge.go#L329

The permission bits will have to be recovered through the file header.

Occasional issues with file already closed errors

I sometimes see this issue:

unTar(): error while tar reader.Next() for io.Reader read |0: file already closed

I have my script that runs g10k automatically clear the cache when the last run exited in error, so the next run is usually ok. I only see this happen after changing the Puppetfile.

Add option to fail on control repo unreachable

I didn't see a way to configure this, but if the control repo is unreachable for whatever reason, invalid auth, network, etc., g10k exits 0 with:

WARN: git repository https://myserver/org/r10k.git does not exist or is unreachable at this moment!

I'd like to fail in that case and exit 1.

hardlink creation fails when /tmp is on a different partition

hardlink cannot cross partitions, and /tmp/g10k is often on a different partition
It would be handy to have a chance to set cachedir when using puppetfile

Use `-branch` in Puppetfile mode

In Puppetfile mode g10k refurses to synchronize modules with the :link
parameter, suggesting to set g10k_branch as an environment variable.

Since the flag -branch already exists and is documented for
config-file mode I'd suggest to use this flag as well for Puppetfile
mode.

Introduce vendoring

I'd like to see vendoring.

This would help with automated builds as well as initial setups. Go 1.8 will likely also change the default behaviour with requiring to be inside the gopath to build vendored projects as well, which would remove the hacky-wacky temporary gopaths as well.

xorpaul / g10k Goto Github PK

g10k's People

Contributors

Stargazers

Watchers

Forkers

g10k's Issues

Recommend Projects

Recommend Topics

Recommend Org