camptocamp / puppet-varnish Goto Github PK

View Code? Open in Web Editor NEW

36.0 15.0 25.0 262 KB

License: Apache License 2.0

Shell 18.60% Ruby 50.78% Perl 4.39% Augeas 2.08% Puppet 22.22% HTML 1.92%

puppet-varnish's Introduction

Varnish

Overview

This puppet module installs and configures varnish.

Usage

class { 'varnish':
  multi_instances => false,
}

To manage varnishlog:

class { 'varnish::log': }

To manage varnishncsa:

class { 'varnish::ncsa': }

Reference

Classes:

varnish
varnish::log
varnish::ncsa

Resources:

varnish_param

###Class: varnish

####enable Should the service be enabled during boot time?

####multi_instances Wether or not use the multi-instance configuration (see notes).

####start Should the service be started by Puppet?

####admin_listen_address Admin interface listen address.

####admin_listen_port Admin interface listen port.

####group Group for the varnishd worker processes

####listen_address Default address to bind to.

####listen_port Default port to bind to.

####secret_file Shared secret file for admin interface.

####storage Backend storage specification.

####ttl Default TTL used when the backend does not specify one.

####user User for the varnishd worker processes.

####vcl_conf Main configuration file.

###Class: varnish::log

###Class: varnish::ncsa

###Resource: varnish_param

This resource is a native type and a series of Augeas-based providers.

It allows to modify varnish parameters where OSes put them by default:

in /etc/default/varnish for Debian wheezy (as a list of parameters)
in /etc/systemd/system/varnish.service for Debian jessie (as a list of parameters)
in /etc/sysconfig/varnish for RedHat 6 (as standalone variables)
in /etc/varnish/varnish.params for RedHat 7 (as standalone variables)

Example:

varnish_param { 'listen_address':
  ensure => present,
  value   => 'localhost',
}

Notes

Version 1.x supported only multi-instances configuration. As we never use this use case, we decided to switch to a single instance configuration.
Version 1.99.x adds support for single instance configuration but still defaults to multi-instances configuration.
Version 2.x will default to single instance configuration and deprecate multi-instances configuration.
Version 3.x will remove multi-instances configuration support.

puppet-varnish's People

Contributors

Stargazers

Watchers

puppet-varnish's Issues

This module is missing a license

This module is missing a license. Could you please add one? I think all of the past contributors need to agree.

varnish_param does not work on Debian Jessie

Examples of non-working stuff:

  class {'::varnish':
    listen_port     => 80,
    multi_instances => false,
    secret_file     => '/etc/varnish/secret',
    storage         => 'file,/var/cache/varnish/varnish_storage.bin,98%',
    vcl_content     => file("module/foo"),
  }

Expected result: should set Varnish port to 80, set Varnish storage engine to "file" with the other parameter.

Actual result:

puppet fails to set the port:

Notice: /Stage[main]/Varnish::Config/Varnish_param[listen_port]/ensure: created
Error: /Stage[main]/Varnish::Config/Varnish_param[listen_port]: Could not evaluate: Failed to save Augeas tree to file. See debug logs for details.
Notice: /Stage[main]/Systemd/Exec[systemctl-daemon-reload]: Dependency Varnish_param[listen_port] has failures: true
Warning: /Stage[main]/Systemd/Exec[systemctl-daemon-reload]: Skipping because of failed dependencies
Notice: /Stage[main]/Varnish::Service/Service[varnish]: Dependency Varnish_param[listen_port] has failures: true
Warning: /Stage[main]/Varnish::Service/Service[varnish]: Skipping because of failed dependencies
Notice: /Stage[main]/Profiles_chtopo::App::Varnish_proxy/Service[varnishncsa]: Dependency Varnish_param[listen_port] has failures: true
Warning: /Stage[main]/Profiles_chtopo::App::Varnish_proxy/Service[varnishncsa]: Skipping because of failed dependencies
Notice: Finished catalog run in 13.10 seconds

Puppet fails to set storage engine (nothing in both /etc/default/varnish and /lib/systemd/system/varnish.service)

Issue when puppet client runs as dæmon

Seems puppet doesn't detect properly varnish service status when launched as a dæmon instead of cron or manually.

This leads to a weird state: varnish continues to serve content, but varnishncsa/varnishlog aren't able to access the VSM, hence we're unable to retrieve logs.

In daemon.log, we only have this information:
Sep 8 07:10:51 [HOSTNAME] puppet-agent[21511]: (/Stage[main]/Varnish::Service/Service[varnish]/ensure) ensure changed 'stopped' to 'running'

This happens on a Debian Jessie with varnish 4.0.3. The following commands returns the correct status:

# service varnish status
# systemctl status varnish
# /etc/init.d/varnish status

● varnish.service - Varnish HTTP accelerator
   Loaded: loaded (/etc/systemd/system/varnish.service; disabled)
   Active: active (running) since Wed 2015-08-26 10:25:06 CEST; 1 weeks 5 days ago
  Process: 22499 ExecReload=/usr/share/varnish/reload-vcl (code=exited, status=0/SUCCESS)
 Main PID: 22964 (varnishd)
   CGroup: /system.slice/varnish.service
           ├─22964 /usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s file,/var/cache/varnish/varnish_storage.bin,18G -u varnish -g varnish
           └─22965 /usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s file,/var/cache/varnish/varnish_storage.bin,18G -u varnish -g varnish

According to puppet resource service varnish --debug puppet is using systemd provider for the service:

[…]
Debug: /Service[varnish.service]: Provider systemd does not support features flaggable; not managing attribute flags
Debug: /Service[varnishlog.service]: Provider systemd does not support features flaggable; not managing attribute flags
Debug: /Service[varnishncsa.service]: Provider systemd does not support features flaggable; not managing attribute flags
Debug: Executing '/etc/init.d/varnish status'
service { 'varnish':
  ensure => 'running',
  enable => 'true',
}

Weird thing, when we do service puppet status we got this log:

Sep 08 06:39:26 ip-10-35-132-218 crontab[19770]: (root) LIST (root)
Sep 08 06:40:52 ip-10-35-132-218 varnishd[20124]: Platform: Linux,3.16.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit
Sep 08 06:40:52 ip-10-35-132-218 varnishd[20124]: -T localhost:6082 could not be listened on.
Sep 08 06:40:52 ip-10-35-132-218 puppet-agent[19521]: (/Stage[main]/Varnish::Service/Service[varnish]/ensure) ensure changed 'stopped' to 'running'
Sep 08 06:40:54 ip-10-35-132-218 puppet-agent[19521]: Finished catalog run in 92.49 seconds
Sep 08 07:09:25 ip-10-35-132-218 crontab[21760]: (root) LIST (root)
Sep 08 07:10:51 ip-10-35-132-218 varnishd[22114]: Platform: Linux,3.16.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit
Sep 08 07:10:51 ip-10-35-132-218 varnishd[22114]: -T localhost:6082 could not be listened on.
Sep 08 07:10:51 ip-10-35-132-218 puppet-agent[21511]: (/Stage[main]/Varnish::Service/Service[varnish]/ensure) ensure changed 'stopped' to 'running'
Sep 08 07:10:53 ip-10-35-132-218 puppet-agent[21511]: Finished catalog run in 92.24 seconds

Apparently, varnish should be listening on localhost:6082, option -T is for the "management interface listener" and shall be passed at runtime.
Weird part: "varnishadm" can access varnish and manage it, and the -T flag is indeed passed to varnishd dæmon.

At this point, I don't have any more information, just this one: running puppet client as a dæmon breaks varnish logs on (at least) Jessie.

varnish_param should require File['/etc/systemd/system/varnish.service']

The following puppet receipt will fail in most cases because there's a missing requirement in varnish_param type:

      Service<| title == 'varnish' |> {
        provider => 'systemd',
      }
      class {'::varnish':
        listen_port     => '80',
        group           => 'varnish',
        multi_instances => false,
        secret_file     => '/etc/varnish/secret',
        user            => 'varnish',
      }

      varnish_param {'shm_reclen':
        value  => 4084,
        notify => [
          Exec['systemctl-daemon-reload'],
          Service['varnish'],
        ],
      }

Here's an acceptance test in order to show this behavior.

If we put a dependence like this, it fails due to a dependency cycle:

      Service<| title == 'varnish' |> {
        provider => 'systemd',
      }
      class {'::varnish':
        listen_port     => '80',
        group           => 'varnish',
        multi_instances => false,
        secret_file     => '/etc/varnish/secret',
        user            => 'varnish',
      }->
      varnish_param {'shm_reclen':
        value  => 4084,
        notify => [
          Exec['systemctl-daemon-reload'],
          Service['varnish'],
        ],
      }

Error: Could not apply complete catalog: Found 1 dependency cycle:
(Service[varnish] => Class[Varnish::Service] => Class[Varnish] => Class[Varnish] => Varnish_param[shm_reclen] => Service[varnish])

Compatibility with Ubuntu

On 16.04 default Varnish is installed with 4.1.1 and parameter "user" is no longer supported

see https://www.varnish-cache.org/docs/4.1/whats-new/upgrading.html#changes-to-parameters

Also systemd is not yet used in 14.04 but your varnish::config use systemd for Debian version >= 8. Probably better to switch to $::systemd fact provider by the systemd module

Augeas issues getting module to work

I am using Puppet v3.8.3 on EL6 with augeasproviders_core 2.1.2 and augeasproviders_shellvar 2.2.0 and am unable to get this module to work. I'm setting multi_instances to false.

Any pointers would be greatly appreciated.

# rpm -qa |grep aug
augeas-1.0.0-10.el6.x86_64
ruby-augeas-0.4.1-3.el6.x86_64
augeas-libs-1.0.0-10.el6.x86_64

Error: /Stage[main]/Varnish::Config/Varnish_param[storage]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[listen_address]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[secret_file]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[ttl]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[user]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[vcl_conf]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[admin_listen_address]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[listen_port]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[admin_listen_port]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)
Error: /Stage[main]/Varnish::Config/Varnish_param[group]: Could not evaluate: Augeas didn't load /etc/sysconfig/varnish with Shellvars.lns from /var/lib/puppet/lib/augeas/lenses: Syntax error (line:96, character:11)

Expose an Exec['varnish restart']

This is needed when we play with varnish_param (needs to restart daemon, not reload)

Use quoteit when modifying values in RH varnish_param providers

Avoid changing quoting when values already exist.

Move varnish::nagiosplugin to its own module

It seems that varnish::nagiosplugin is self contained and not dependent on the rest of the module. Could we move this to its own module? It is a bit fragile and site specific by including packages such as gcc directly and building things on the host instead of packaging them. It being separate gives me more confidence to introduce this module to my users without fear of them using this bit.

Missing runtime parameter support for varnish_param

varnish_param should support "Runtime Parameters", for example:
-p param=value -p param=value …

An idea would be to add the support like this:

varnish_param {'runtime':
 value => ['param=value', 'param=value', …]
}

It should then output in the configuration file the previously mentioned line

This would prevent deep modification in the current type while allowing to set multiple parameters.

[systemd] varnishncsa service management

We should be able to manage ncsa service in order to, for example, set output path, or options.

For example:

[Unit]
Description=Varnish HTTP accelerator log daemon
After=varnish.service

[Service]
User=root
ExecStart=/usr/bin/varnishncsa -a -w /mnt/varnish/varnishncsa.log
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Using varnish_param might do the trick as long as we can edit another file than the varnish.service.

Allows to disable varnishlog

varnish::log should allow to pass argument in order to disable the service.

It seems some distro enable it by default…

[RHL] varnish_param, DAEMON_OPTS, and runtime arguments

Hello,

It seems we're currently unable to pass varnish_param correctly on RHL system.

Example:

  varnish_param {'shm_reclen':
        value  => '4084',
        notify => Exec['systemctl-daemon-reload'],
   }

What's expected:
update DAEMON_OPTS in /etc/varnish/varnish.params and add a -p shm_reclen=4084 in the existing string.
For example:
DAEMON_OPTS="-p shm_reclen=4084 -p vsl_reclen=4084"'

What it actually does:
creates a VARNISH_SHM_RECLEN=4084 line in /etc/varnish/varnish.params

Cheers,