basho-labs / puppet-riak Goto Github PK

As you can see from the moving around of repos and updates, I'm making a big effort to get our Puppet strategy to parity with other tools. My honest question for those who care most about this is straightforward: which of the following is the best repo to start from?

• The current one @haf has been kind enough to maintain here
• The one @Cornellio built over here
• This one from @jbussdieker over here
• Or something else?

@basho-labs/puppet-committers - I'll need your input here!

Let's all discuss this with a positive view on what we need to accomplish to make the most useful code for our communities. If you'd like more guidance, the goals are all outlined here: https://github.com/basho-labs/the-riak-community/blob/master/config-mgmt-strategy.md

I did a little looking, and apparently it's pretty common for packages to end up being accidentally redeclared.

From my Puppet console output

←[1;31mError: Duplicate declaration: Package[curl] is already declared in file /tmp/vagrant-puppet/modules-0/redis/manifests/init.pp at line 9; cannot redeclare on node ubuntu-12.04.2-server-amd64-mono-nginx-node←[0m

←[1;31mError: Duplicate declaration: Package[curl] is already declared in file /tmp/vagrant-puppet/modules-0/redis/manifests/init.pp at line 9; cannot redeclare on node ubuntu-12.04.2-server-amd64-mono-nginx-node←[0m

I'm using https://github.com/logicalparadox/puppet-redis to install Redis.

In my Puppet file, I'm installing Redis first, then Riak.

Now, if I reorder how my packages are called in my Puppet script to install Riak first and then Redis, it works just fine.

If you look at the init.pp for Redis
https://github.com/logicalparadox/puppet-redis/blob/master/manifests/init.pp#L6

It has the code to prevent the re-declaration.

if defined(Package['curl']) == false {
    package { "curl":
      ensure => "installed"
    }
  }

So I think that this is all that needs to be added to this package as well...but I'm not a Puppet guy ;0

BTW - This is a demo / Vagrant box and not a real production box -- definitely wouldn't run servers with Riak and Redis on them ;0

I realize this is OS specific, but wondering if you've given any thought to including this?

See basho-labs/riak-chef-cookbook@41661c2 for the chef cookbook

So in the Puppet script that Vagrant runs, I have

Notice that I already have 1.3.0 installed.

class { 'riak':
  version => '1.3.0',
  cfg => {
    riak_api => {
       pb_backlog => 64
    },
    riak_kv => {
      storage_backend => '__atom_riak_kv_eleveldb_backend',
      map_js_vm_count => 24,
      reduce_js_vm_count => 24
    },
    riak_control => {
      enabled => true,
      auth => '__atom_none',
      admin => true
    }
  }
}

Now I want to upgrade to 1.3.1 -- and presumably I just need to change the version

class { 'riak':
  version => '1.3.1',
  cfg => {
    riak_api => {
       pb_backlog => 64
    },
    riak_kv => {
      storage_backend => '__atom_riak_kv_eleveldb_backend',
      map_js_vm_count => 24,
      reduce_js_vm_count => 24
    },
    riak_control => {
      enabled => true,
      auth => '__atom_none',
      admin => true
    }
  }
}

But nothing happens... just the normal config runs.

Note also that I get this warning from Hiera, as this is a Vagrant box, etc. -- with no centralized Hiera store. Not sure if that's a problem?

Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults

We have an account at https://forge.puppetlabs.com/users/basholabs. I'm trying to get it going with puppet module build and running into some errors:

 An error occurred while installing nokogiri (1.6.6.2), and Bundler cannot continue.
 Make sure that `gem install nokogiri -v '1.6.6.2'` succeeds before bundling.

It's installing just fine. Getting stumped and could use a hand.

On line 118 its both a case statement and a ? selector.
If I dump the selector it works.

Easy to fix but as is it does not work for me at al on Ubuntu 12.04 so not sure how this was tested, since I saw that this was pretty much the platform it was tested on.

in the appconfig.pp we have this one:

http_url_encoding     => 'on' 

but shouldn't it be

http_url_encoding     => '__atom_on' 

?

It was at least necessary for anti_entropy. Otherwise it wasn't enabled (so "on" didn't work).

Hi,

I've taken a quick look at your code and have found a supposed to be a bug:
https://github.com/haf/puppet-riak/blob/master/manifests/init.pp#L139
https://github.com/haf/puppet-riak/blob/master/manifests/init.pp#L220

The idea of anchors is to have anchor start before EVERY resource of the class and anchor end after EVERY resource of the class. Otherwise they do not make much sense.

So you are likely to need to add require => Anchor[riak::start], before => Anchor[riak::end] to every resource of the class

Cheers!

For some reason I can't get this to start Riak on CentOS, not even from a fresh GitHub clone. I've uploaded the output of rake vagrant:up on my system.

@mjbrender can you enable travis-ci tests on this repo at https://travis-ci.org/profile/basho-labs? I don't seem to have access rights.

I've been trying to figure out which Riak resources would make sense to model using types and providers rather than managing config files. I've mocked up what some resources might look like. Do these look useful? Missing obvious things? Doing it wrong? These will basically wrap a bunch of riak-admin commands.

My goal would be to allow use of the puppet resource command line tool to use these to describe a running riak system and capture the current state in puppet, to facilitate puppetizing an existing manual install.

riak_node { $::fqdn:
  ensure             => 'join', # [ join | leave ]
  replace            => '[email protected]',
  autocommit         => false, # commit plan change immediately?
  plan_change_quorum => 3, # wait for 3 nodes to have staged plan changes before committing
  max_plan_wait      => 300, # how long to wait (in seconds) for the plan to be ready
  join_target        => '[email protected]',
}

riak_user { 'example':
  username => 'daniel', # defaults to resource name, this is just to show it can be different
  groups   => ['admins', 'developers'],
  password => 'hunter2',
  options  => {
    'name' => 'lucius',
    'fav_color' => 'red',
  },
  permissions => {
    bucket_type   => '*',
    bucket => '*',
    grants => [ 'riak_kv.get', 'riak_kv.delete', 'riak_kv.put']
  },
}

riak_group { 'admins':
  name => 'admins',
  permissions => {
    type   => '*',
    bucket => '*',
    grants => [ 'riak_kv.get', 'riak_kv.delete', 'riak_kv.put']
  },
  groups => 'everybody',
}

riak_security_source { 'localhost':
  users   => all,
  type    => 'trust',
  cidr    => '127.0.0.1/32',
  options => []
}

riak_security_source { 'private network':
  users   => all,
  type    => 'password',
  cidr    => '10.0.0.0/24',
  options => []
}

riak_security_source { 'WAN':
  users   => ['riakuser', 'daniel']
  type    => 'certificate',
  cidr    => '10.1.0.0/24',
  options => []
}

riak_bucket_type {'n_equals_1':
  ensure => 'activated',
  props  => {
    'n_val'      => '1',
    'allow_mult' => true,
    'precommit'  => ["syntax_check"],
    'postcommit' => ["welcome_email", "update_registry"]
  },
  max_propagation_wait => 30,
}

riak_cluster_settings { 'arbitrary cluster name':
  ciphers => 'DHE-RSA-AES256-SHA:AES128-GCM-SHA256',
  purge_users => true, # delete riak security users puppet didn't create? (false by default)
  purge_groups => true, # delete riak security groups puppet didn't create? (false by default)
  # probably more purge settings
  # what other settings?
}

Only problem is that 1.3.0 does not exist on the apt riak site for Ubuntu 12.04.
Expectation would be to have it bumped up to latest existing stable version on regular basis.

Its not a big deal since I can do trial and error and set my own version until it clicks.
It worked with ...

class { 'riak':
version => '1.4.2-1',

Rather than refactoring the httpfile resource out, can we simply remove httpfile entirely? I'm guessing somebody out there is using it, but wanted to check before we simply grandfather than installation mode in. It seems like any packages would be in a repo anyway; is this there to support the EE install?

The apt package installation fails on Ubuntu because the module is setting package source to Debian instead of Ubuntu.

The source list at /etc/apt/sources.list.d/riak.list needs to be set as:

deb https://packagecloud.io/basho/riak/ubuntu/ precise main
deb-src https://packagecloud.io/basho/riak/ubuntu/ precise main

Instead of this:

deb https://packagecloud.io/basho/riak/debian/ precise main
deb-src https://packagecloud.io/basho/riak/debian/ precise main

It seems that when I modify appconfig.pp in order to launch the Riak Control it sets by default on the app.config file

    riak_control => {
      enabled  => true,
      auth     => '__atom_userlist',
      userlist => ['__tuple', 'user', 'pass'],
      admin    => true
    },

but this makes riak control fail with a error 500

Internal Server Error

The server encountered an error while processing this request:
{error,function_clause,
       [{proplists,get_value,
                   ["user",{"user","pass"},undefined],
                   [{file,"proplists.erl"},{line,222}]},
        {riak_control_security,valid_userpass,3,
                               [{file,"src/riak_control_security.erl"},
                                {line,127}]},
        {riak_control_security,enforce_user_pass,5,
                               [{file,"src/riak_control_security.erl"},
                                {line,115}]},
        {webmachine_resource,resource_call,3,
                             [{file,"src/webmachine_resource.erl"},
                              {line,183}]},
        {webmachine_resource,do,3,
                             [{file,"src/webmachine_resource.erl"},
                              {line,141}]},
        {webmachine_decision_core,resource_call,1,
                                  [{file,"src/webmachine_decision_core.erl"},
                                   {line,48}]},
        {webmachine_decision_core,decision,1,
                                  [{file,"src/webmachine_decision_core.erl"},
                                   {line,209}]},
        {webmachine_decision_core,handle_request,2,
                                  [{file,"src/webmachine_decision_core.erl"},
                                   {line,33}]}]}

The correct form of the riak_control section is

 {riak_control, [
                {enabled, true},
                {auth, userlist},
                {userlist, [{"user", "pass"}
                           ]},
                {admin, true}
        ]},

I'm new to puppet, but I've tried using the version parameter and it doesn't appear to work - no matter what value you enter, it always installs the latest version of Riak

I'm trying to install the last major release - 1.2.1

When running version 0.2.2 on Puppet 3.7, I get:

[riak@dev1 ~]$ sudo puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Syntax error at 'String'; expected ')' at /etc/puppet/modules/riak/manifests/init.pp:11 on node dev1.home
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

On my puppet master:

[root@marx manifests]# puppet module list
Warning: Missing dependency 'puppetlabs-apt':
'basholabs-riak' (v0.2.2) requires 'puppetlabs-apt' (>= 1.0.0 <2.0.0)
Warning: Missing dependency 'puppetlabs-stdlib':
'basholabs-riak' (v0.2.2) requires 'puppetlabs-stdlib' (>= 4.0.0 <5.0.0)
/etc/puppet/modules
├── base (???)
├── basholabs-riak (v0.2.2)
├── puppetlabs-apt (v1.8.0)
├── puppetlabs-firewall (v1.5.0)
├── puppetlabs-ntp (v3.3.0)
└── puppetlabs-stdlib (v4.6.0)
/usr/share/puppet/modules (no modules installed)

How recent of puppet versions people using, and on what platforms? Specifically:

• How would people feel about using future parser / puppet 4 DSL?
• Puppet version since 3.4.0 have supported contain rather than the anchor pattern
• Can we require structured facts to be enabled?
• Am I the only person who wants FreeBSD support?
• What Linux distros do we need to support, and what major versions? I'm hoping EL 5 isn't on the list.

I'd personally love to just do development in future parser with puppet 3.7, but I realize that option isn't available to everybody, and that documentation on what future parser can do is extremely lacking.

this module is for install riak, do you guys have any plan to add more features? such as forming a cluster by executing cluster joining command.

/cc @jaredmorrow

New config - need specification.

Suggest that we stop supporting puppet 2.7 and its hiera in this next version.

To create a test-matrix for vagrant to test out that this applies to the different operating systems. The Vagrantfile already in the repo was previously used for this testing.

I'm hoping to write types and providers in puppet that wrap riak-admin, for the purpose of automating cluster management operations. For example, we could change ring size in the live cluster, change config files, etc and do that all in the correct order automatically to avoid user error in manual change. Similarly, it would be great to automate the process of joining new nodes to a cluster, safely removing nodes, etc.

I'd really appreciate any input on other automation-friendly options for interacting with the functionality provided by riak-admin. I can't find analogous HTTP API endpoints, can't find a way to get JSON (or other structured output) from riak-admin, etc.

The two options I see are to either wrap riak-admin and parse the text output, or to improve my very basic erlang skills and use whatever libraries riak-admin uses to implement the functionality I need. However, the latter is likely to involve months of me learning a new language before I'm productive, and the former is a bit awkward, though common enough in the puppet world.

Are those the only options?

This project is great and I see it as a definite help to the community. Looking at the open issues, it's not clear whether the repo is run by engineering or is a "best effort" project. If it is best effort, I'd like to make it part of an effort to leverage the Basho Labs account and teams that we're building out on that account to get further contribution.

For more info, see the public conversation on strategy here: https://github.com/basho-labs/the-riak-community/blob/master/config-mgmt-strategy.md

Am I running the specs incorrectly? Please see the Travis output at the link below. I have also run the specs on my laptop and within a VM on my laptop, and the results are nearly identical.

https://travis-ci.org/L2G/puppet-riak/builds/19171821

Riak 2.1 is available and it'd be great to add it to the mix. It also introduces a write_once bucket type that may be of interest to implement.

This may or may not be of concern to you.

The tarball hosted on Puppet Forge uses "SCHILY.*" extensions, which causes GNU tar 1.15.1 (the version on Puppet's "learning" VM) to choke on it:

[root@learn ~]# puppet module install haf-riak
Notice: Preparing to install into /etc/puppetlabs/puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
Error: Could not extract contents of module archive: Execution of 'tar xzf /var/opt/lib/pe-puppet/puppet-module/cache/https_forge_puppetlabs_com-ed3a0e51b3c1d8d395ffb1d12c849e3f50d35a10/haf-riak-0.1.1.tar.gz --no-same-permissions --no-same-owner -C /var/opt/lib/pe-puppet/puppet-module/cache/tmp-unpacker-26c36bcd30dc20caedeb577a8304b7092bc92717' returned 2: tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
...
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Error exit delayed from previous errors

Error: Try 'puppet help module install' for usage

I am still new to Puppet, so I don't know whether there's a way to tell puppet module to use star instead of tar in order to work around this.

We're trying to use within Vagrant, and was hoping you could provide some additional documentation.

As you may know, Puppet is installed / run from the VM, and Vagrant has a mechanism for configuring modules, that may live on the local file system.

Basic mechanism (assuming you're in a working directory of modules that's relative to the Vagrantfile)

• Clone the source - git clone https://github.com/basho/puppet-riak.git
• Tell Vagrant / Puppet how to find the modules, with the following (or similar) in the Vagrantfile

Vagrant::Config.run do |config|
  config.vm.provision :puppet, :module_path => "modules"
end

Then of course it's on to using the riak class.

However, my concern is regarding the use of Hiera. We can clone the Hiera sources in the same way we do for the riak module to the modules directory via git clone -b 1.x https://github.com/puppetlabs/hiera-puppet.git.

This would be necessary for Puppet 2.x. For Puppet 3.x, Hiera is in the box from what I understand.

So I'm not sure if we need the mentioned Gem or not.. ??

In particular, this mention is from the docs

Make sure you are shipping the custom functions in this module to your i nodes, by setting pluginsync=true;

[main]
pluginsync = true

Requires that hiera-puppet gem is present.

Any advice / documentation is helpful here, before we go down a potentially painful road ;0

I've been following the instructions on CONTRIBUTING.md, and I have run into an issue in that the initial tests have failed before I have done anything.

bundle exec rake lint

succeeds, but

bundle exec rake syntax

throws errors:

---> syntax:manifests
Could not parse for environment *root*: Syntax error at 'String'; expected ')' at /etc/puppetlabs/code/environments/production/modules/puppet-riak/manifests/init.pp:6
Could not parse for environment *root*: Syntax error at 'String'; expected ')' at /etc/puppetlabs/code/environments/production/modules/puppet-riak/manifests/tuning.pp:8
rake aborted!

I'm running puppet 4.2.2.

Not sure what I'm doing wrong - this is all from a clean VM.

Cheers,

Joel

I'd like to implement optional OS optimization settings, similar to what @Cornellio did in https://github.com/Cornellio/puppet-riak/blob/master/manifests/baseconfig.pp. I'm still hoping to support non-linux platforms, so a lot of this will need to be wrapped in a case statement.

I haven't found a great single source of recommendations. As source material, what I'm aware of it:

• http://docs.basho.com/riak/1.3.2/cookbooks/Linux-Performance-Tuning/#Linux-Tuning
• http://docs.basho.com/riak/1.3.2/cookbooks/Performance-Tuning-AWS/
• http://docs.basho.com/riak/1.3.2/cookbooks/Open-Files-Limit/
• http://docs.basho.com/riak/latest/ops/tuning/linux/

I'm not clear on how much the riak 1.3 recommendations still apply to riak 2. I'm assuming that IO patterns should be relatively similar, but I don't really have a lot of basis for that assumption.

My current POC looks like https://gist.github.com/danieldreier/a9a20e2dee7d2db955c2 (heavily based on @Cornellio's work) but that doesn't even really work yet, more of a scratch pad for ideas.

Are there other must-read sources? I didn't see anything in the little riak book about it.

We cannot override a single default key in the $cfg in appconfig.pp with a hiera hash. For example, if we wanted to override the ring_creation_size, we would have to supply the entire set of values of the riak_core hash or else the merge would result with the riak_core hash only containing the ring_creation_size k/v pair. I believe that we should be able to specify something like this in a hiera hash:

 cfg:
   riak_core:
     ring_creation_size: 128

If we performed a deep_merge install of a regular merge, the value of ring_creation_size would properly be merged with the default $cfg->riak_core hash.

deep_merge is available with gem install deep_merge

More information can be found here: https://github.com/peritor/deep_merge