puppetlabs-toy-chest / puppetlabs-puppetserver_gem Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 26.0 39 KB

This module provides management of Ruby gems for Puppet Server.

Ruby 99.29% Puppet 0.71%

module

puppetlabs-puppetserver_gem's People

Contributors

Stargazers

Watchers

puppetlabs-puppetserver_gem's Issues

module is not compatible with puppet 6.5.0

get the following error:

Jun 19 19:19:45 master puppet-agent[7559]: Could not prefetch package provider 'puppetserver_gem': No command gemcmd defined for provider puppetserver_gem
Jun 19 19:19:45 master puppet-agent[7559]: Failed to apply catalog: No command gemcmd defined for provider puppetserver_gem
Jun 19 19:19:46 master puppet: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:98:in `exit': no implicit conversion from nil to integer (TypeError)
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:98:in `block in run_in_fork'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:94:in `fork'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:94:in `run_in_fork'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:48:in `block in run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:179:in `controlled_run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:46:in `run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/daemon.rb:170:in `block in run_event_loop'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/scheduler/job.rb:49:in `run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:39:in `block in run_ready'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `each'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `run_ready'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:11:in `run_loop'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/daemon.rb:193:in `run_event_loop'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/daemon.rb:149:in `start'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:389:in `main'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:354:in `block in run_command'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:266:in `override'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:350:in `run_command'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:684:in `exit_on_fail'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
Jun 19 19:19:46 master puppet: from /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

Receiving error when attempting to use module. can't use --version. Use name:ver instead

ERROR: Can't use --version w/ multiple gems. Use name:ver instead.
Error: /Stage[main]/Profile::Packages/Package[net-ldap]/ensure: change from 'absent' to '0.16.1' failed: Could not update: Execution of '/opt/puppetlabs/bin/puppetserver gem install -v 0.16.1 --no-document net-ldap' returned 1

git tags are not in sync with the module version

Could you please tag the git repository to be in sync with the module version?

Can't list installed gems if memory is tight

I'm trying to install hiera-eyaml with this gem, and getting this error, followed by trying to reinstall the gem every time:

Feb 26 07:28:46 ssipuppet10 puppet-agent[46223]: Could not match #
Feb 26 07:28:46 ssipuppet10 puppet-agent[46223]: Could not match # There is insufficient memory for the Java Runtime Environment to continue.
Feb 26 07:28:46 ssipuppet10 puppet-agent[46223]: Could not match # Native memory allocation (mmap) failed to map 80530636800 bytes for committing reserved memory.
Feb 26 07:28:46 ssipuppet10 puppet-agent[46223]: Could not match # An error report file with more information is saved as:
Feb 26 07:28:46 ssipuppet10 puppet-agent[46223]: Could not match # /root/hs_err_pid46911.log
Feb 26 07:28:47 ssipuppet10 puppet-agent[46223]: Could not match #
Feb 26 07:28:47 ssipuppet10 puppet-agent[46223]: Could not match # There is insufficient memory for the Java Runtime Environment to continue.
Feb 26 07:28:47 ssipuppet10 puppet-agent[46223]: Could not match # Native memory allocation (mmap) failed to map 80530636800 bytes for committing reserved memory.
Feb 26 07:28:47 ssipuppet10 puppet-agent[46223]: Could not match # An error report file with more information is saved as:
Feb 26 07:28:47 ssipuppet10 puppet-agent[46223]: Could not match # /root/hs_err_pid46994.log

The problem can be reproduced here:

ssipuppet10 ~# /opt/puppetlabs/server/bin/puppetserver gem list
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f97ac000000, 80530636800, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 80530636800 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid21873.log```

The core issue seems to be that loading up the 'gem list' command requires loading up a whole puppet server, which seems problematic.

When using versioning, this doesn't uninstall other versions of the same gem

Sample input:

package { 'my-gem':
  ensure   => '0.1.2',
  provider => puppetserver_gem,
}

If I had previously installed a different version of the gem, that different version also remains after I do the above. I couldn't figure out the pattern of which version puppetserver will use, because it didn't always seem to be the latest, and certainly wasn't always the version I wanted.

Therefore, I think this provider should uninstall other versions of the same gem that are present when a specific version has been requested.

Source option does not work as documented for install

There are a two problems with the source option used with install as documented here

The documentation states that "If a URL is passed via source, then that URL is used as the remote gem repository". This is not quite true, as it adds a source to the end of the gem source list. The effect of this is that an install will always try rubygems.org first.
You should be able to use the gem install option --clear-sources to workaround the above by adding that to the package install_options. However, this provider puts the install_options on the command line at line 74 but has previously put the source option onto the command line at line 68. The order of options on the gem install command line is important in this case as you need to clear the sources before adding the source you want. Illustrated here from the puppetserver gem command line:

myhost:~ # puppetserver gem install jmx --version 1.0 --no-document --clear-sources --source https://my-internal-gem-repo/default.gems.global/
Successfully installed jmx-1.0
1 gem installed
myhost:~ # puppetserver gem uninstall jmx
Successfully uninstalled jmx-1.0
myhost:~ # puppetserver gem install jmx --version 1.0 --no-document --source https://my-internal-gem-repo/default.gems.global/ --clear-sources 
ERROR:  Could not find a valid gem 'jmx' (= 1.0), here is why:
          Unable to download data from https://rubygems.org/ - Errno::ECONNREFUSED: Connection refused - Connection refused (https://api.rubygems.org/specs.4.8.gz)

I'll prepare a PR that addresses these.

Provider pupppetserver_gem is not functional on this host

Puppet runs are failing with:

Debug: Puppet::Type::Package::ProviderPuppetserver_gem: file  does not exist
Debug: Puppet::Type::Package::ProviderPuppetserver_gem: file  does not exist
Error: /Stage[main]/Site_puppet::Server/Package[puppetserver-hiera-eyaml-gpg]: Provider puppetserver_gem is not functional on this host
Error: /Stage[main]/Site_puppet::Server/Package[puppetserver-ruby_gpg]: Provider puppetserver_gem is not functional on this host

puppet resource fails when attempting to ensure an already installed gem is installed with:

# puppet resource package hiera-eyaml-gpg provider=puppetserver_gem ensure=present
Error: Could not prefetch package provider 'puppetserver_gem': no implicit conversion of Array into String
Error: Could not run: no implicit conversion of Array into String

With some quick printf debugging of the puppet core, I have the following stracktrace:

# puppet resource --codedir /data/puppet package hiera-eyaml-gpg provider=puppetserver_gem ensure=present
File being checked is: ["/opt/puppetlabs/bin/puppetserver", "gem"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:57:in `file?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:57:in `validate_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/gem.rb:52:in `execute_gem_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/gem.rb:90:in `gemlist'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/gem.rb:67:in `instances'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:31:in `block in prefetch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:30:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package_targetable.rb:30:in `prefetch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:364:in `prefetch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:256:in `prefetch_if_necessary'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:111:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:119:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:165:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/resource/ral.rb:40:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/resource.rb:221:in `find_or_save_resources'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/resource.rb:142:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:266:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/resource.rb:137:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:390:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:684:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Error: Could not prefetch package provider 'puppetserver_gem': no implicit conversion of Array into String
Error: Could not run: no implicit conversion of Array into String

# ls -ld "/opt/puppetlabs/bin/puppetserver"
lrwxrwxrwx 1 root root 44 Jul 21 16:22 /opt/puppetlabs/bin/puppetserver -> ../server/apps/puppetserver/bin/puppetserver

Path being passed to File.file? is an array of arguments to be executed, not just the first argument which is the executable file on disk: ["/opt/puppetlabs/bin/puppetserver", "gem"] instead of /opt/puppetlabs/bin/puppetserver.

Gems installed:

# /opt/puppetlabs/bin/puppetserver gem list

*** LOCAL GEMS ***

cmath (default: 1.0.0)
csv (default: 1.0.0)
deep_merge (1.0.1)
fast_gettext (1.1.2)
fileutils (default: 1.1.0)
gettext (3.2.2)
hiera-eyaml (3.0.0)
hiera-eyaml-gpg (0.7.3)
highline (1.6.21)
hocon (1.2.5)
ipaddr (default: 1.2.0)
jar-dependencies (default: 0.3.12)
jruby-openssl (default: 0.10.0 java)
jruby-readline (default: 1.2.2 java)
json (default: 2.1.0 java)
locale (2.1.2)
multi_json (1.13.1)
optimist (3.0.0)
psych (default: 3.0.2 java)
puppet-resource_api (1.8.5)
puppetserver-ca (1.3.1)
rake-ant (default: 1.0.4)
rdoc (default: 6.0.1)
ruby_gpg (0.3.2)
scanf (default: 1.0.0)
semantic_puppet (1.0.2)
text (1.3.1)
webrick (default: 1.4.2)

OS version: CentOS 7.5.1804
Puppet agent 6.6.0
Puppetserver version 6.4.0
Puppetserver_gem version 1.1.1

Module compatibility with puppet 5

Hi,

I actually think, but correct me if I'm wrong, that this module is actually compatible with puppet 5. The metadata suggests otherwise. If it is indeed incompatible, or not recommended, what would you recommend we use to use gems on the puppet server side? Otherwise, can the metadata be updated?

Thanks!

Release 1.0.0

This module is vital to use with puppetserver and should not be a pre-release version. Is there anything that needs finalized to make it 1.0.0 or is it just missing a release?

$HOME environment variable is unset

I'm using this module with a proxy server, and we're adding proxy credentials to the file $HOME/.gemrc.

However, when this module is used, the agent looks for gem configuration at /.gemrc because the $HOME environment variable is unset.

I'm not sure what the best fix is for this. We've created a symlink from /.gemrc -> /root/.gemrc

Install fails on PE 2015.2

Tested on 2015.2.1 and 2015.2.2

Puppetfile:
mod 'puppetlabs-puppetserver_gem', '0.1.0'

This appears to load fine: /opt/puppetlabs/puppet/cache/lib/puppet/provider/package/puppetserver_gem.rb

Then we try and load the gem:
package { 'hiera-eyaml':
ensure => '2.0.8',
provider => 'puppetserver_gem',
}
Notice: /Stage[main]/Puppet_master/Package[hiera-eyaml]/ensure: created

/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-2.0.8/lib/hiera/backend/eyaml_backend.rb

:backends:

eyaml
yaml

I put a variable in common.yaml and in the secure.eyaml, and it never finds the secure one:
Notice: secret_pass: not_secured_file

I run the puppetserver in the foreground and then run puppet agent:
/opt/puppetlabs/bin/puppetserver foreground --debug |grep -i hiera
INFO [qtp1702585616-20] [puppet-server] Puppet hiera(): Cannot load backend eyaml: no such file to load -- hiera/backend/eyaml_bac
kend

I also tried using puppet_gem to install into the non-jruby environment, but it had the same effect.
First I removed the original install:

/opt/puppetlabs/bin/puppetserver gem uninstall highline hiera-eyaml

package { 'hiera-eyaml':
ensure => '2.0.8',
provider => 'puppet_gem',
}

/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.0.8/lib/hiera/backend/eyaml_backend.rb

Still fails the same way.

I then created the following links and then it's happy and works:
file { '/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml':
ensure => link,
target => '/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.0.8/lib/hiera/backend/eyaml',
require => Package['hiera-eyaml'],
}

file { '/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml_backend.rb':
ensure => link,
target => '/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.0.8/lib/hiera/backend/eyaml_backend.rb',
require => Package['hiera-eyaml'],
}

file { '/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml.rb':
ensure => link,
target => '/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.0.8/lib/hiera/backend/eyaml.rb',
require => Package['hiera-eyaml'],
}

file { '/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/highline/':
ensure => link,
target => '/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/highline-1.6.21/lib/highline',
require => Package['hiera-eyaml'],
}

file { '/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/highline.rb':
ensure => link,
target => '/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/highline-1.6.21/lib/highline.rb',
require => Package['hiera-eyaml'],
}

DEBUG [qtp1515916820-20] [puppet-server] Puppet hiera(): [eyaml_backend]: Hiera eYAML backend starting

Notice: secret_pass: secured_file_success

So it seems that puppetserver_gem or puppet_gem isn't installing the gems where hiera needs to be able to use them.

Could not match * LOCAL GEMS * when /tmp set to noexec

Overview

When /tmp is mounted noexec, users will running puppet agent -t will begin to receive a disconcerting error message from facter on every puppet run.

Expected result

Mounting /tmp as noexec is recommended by the OS vendor and many independent hardening guides. Puppet should work without error or warning in such environments.

Actual result

Users receive a warning message on every puppet run:

[root@localhost tmp]# puppet facts > /dev/null
Warning: Could not match *** LOCAL GEMS ***

Analysis

The error is caused by the puppetserver gem list command outputting *** LOCAL GEMS *** when executed, in spite of the notes in the gem provider's gemsplit function:

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/package/gem.rb

  def self.gemsplit(desc)
    # `gem list` when output console has a line like:
    # *** LOCAL GEMS ***
    # but when it's not to the console that line
    # and all blank lines are stripped
    # so we don't need to check for them

Invoking puppetserver gem list on the commandline also gives another message about stty that doesn't seem to impact things:

[root@localhost puppetserver_gem]# puppetserver gem list
io/console on JRuby shells out to stty for most operations

*** LOCAL GEMS ***

To fix the error, we just need to reject the *** LOCAL GEMS *** message and lines consisting of only whitespace in the module's provider.

User impact

Annoying message

Workaround

Hotfix

A bit ..... slow?

So....uhm.. puppetserver gem is around 10x slower than other gem providers. Howcome?

[root@puppet ~]# time puppet resource package hiera-eyaml-gpg provider=puppetserver_gem
package { 'hiera-eyaml-gpg':
  ensure => ['0.6'],
}

real    0m41.926s
user    0m40.222s
sys 0m1.183s

[root@puppet ~]# time puppet resource package hiera-eyaml-gpg provider=puppet_gem
package { 'hiera-eyaml-gpg':
  ensure => ['0.6'],
}

real    0m4.374s
user    0m3.862s
sys 0m0.389s
[root@puppet03 ~]#

v. 0.1.0 doesn't work with puppetserver 1.1.3 on Centos 7

On Centos 7 puppetserver 1.1.3 binary is installed to /bin/puppetserver which results in errors like:

Provider puppetserver_gem is not functional on this host

I fixed it for me by forking this repo, using 0.1.0 tag and changing the path and switching to my fork.

I don't really expect you to fix this - I assume that people who will need it will find this issue and help themselves. :)

Provider puppetserver_gem is not functional on this host

Describe the Bug

I get

Provider puppetserver_gem is not functional on this host

on a Ubuntu 18.04.5 LTS client with a puppet server set up using AWS Ops works (v2018.1.7).

I expected it to install the gem specified in the manifest.

Environment

Version puppet client 5.5.10. Puppet server from AWS Ops works, v2018.1.7.
Platform Client Server Ubuntu 18.04.5 LTS.

Additional Context

Add any other context about the problem here.

Wrong path is hardcoded for FOSS version

The hardcoded path isn't true for puppetserver module available in puppetlabs repo.
https://github.com/puppetlabs/puppetlabs-puppetserver_gem/blob/master/lib/puppet/provider/package/puppetserver_gem.rb#L14

Its /usr/bin/puppetserver.