Code Monkey home page Code Monkey logo

stackstorm / puppet-st2 Goto Github PK

View Code? Open in Web Editor NEW
21.0 25.0 30.0 7.14 MB

Puppet Module to install and configure StackStorm, IFTTT for Ops

Home Page: https://forge.puppet.com/stackstorm/st2

License: Apache License 2.0

Ruby 30.38% Puppet 46.52% HTML 1.30% Shell 1.43% Dockerfile 2.15% Makefile 1.12% Pascal 0.33% Python 16.76%
configuration-management puppet stackstorm ruby puppet-modules automation event-driven-automation st2 devops deployment

puppet-st2's People

Contributors

arm4b avatar armab avatar asktheaxis avatar bishopbm1 avatar bishopbm2 avatar blag avatar dennybaa avatar enykeev avatar jfryman avatar kami avatar lakshmi-kannan avatar lattwood avatar lstoll avatar m4dcoder avatar manasdk avatar nmaludy avatar ruriky avatar rush-skills avatar tclh123 avatar tobijb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-st2's Issues

Add support for URLs in st2chatops.env

Currently in st2::profile::chatops we can configure most settings in st2chatops.env but the URLs for things like the API, auth and web are still hard coded.

Need to figure out a good way to make the default still point to localhost and then allow them to be overridden when there are complex deployments in place.

update-system shouldn't use docker interface for default urls

Initial set up using the all in one installer uses the correct addresses for urls. If you later run update-system without answers in answers.yaml for those values, it will set them to the docker interface. Pretty sure we don't want that.

Note that I've only run into this when I have not run the webgui setup (because I like to automate my automation).

relevant puppet logs:

...
Notice: /Stage[main]/St2::Helper::Auth_manager/Ini_setting[auth_api_url]/value: value changed 'https://10.0.2.15:9101' to 'https://172.17.42.1:9101'
...
Notice: /Stage[main]/St2::Profile::Client/St2::Client::Settings[root]/Ini_setting[root-st2_cli_api_url]/value: value changed 'https://10.0.2.15/api' to 'https://172.17.42.1/api'
Notice: /Stage[main]/St2::Profile::Client/St2::Client::Settings[root]/Ini_setting[root-st2_cli_auth_url]/value: value changed 'https://10.0.2.15/auth' to 'https://172.17.42.1/auth'
...

Support custom repo URLs for internal mirrors

Some production environments may have internal mirrors of the public PackageCloud repos. Instead of hard-coding the packagecloud as a dependency, we should allow the user to specify custom repo URLs to support this use case.

This needs to work for both debian and redhat.

Allow for configurable RabbitMQ listen address

In the latest release we hard coded the listen address to be 127.0.0.1 to conform to InSpec tests.

We should make this a configuration option so users can configure RabbitMQ as they see fit.

Add support for encrypted key/value pairs

Currently st2::kv only supports plain text key/value pairs.

Should be possible to support encrypted key/value pairs.

We probably need to create a provider implementation to get this working properly.

Drop support for Mistral

Mistral support is being dropped from core StackStorm.

Remove the management and installation of both Mistral and Postgresql

Fix open ended dependencies

Fix open ended dependencies shown by pdk validate

warning: metadata-json-lint: metadata.json: Dependency jamtur01/httpauth has an open ended dependency version requirement >= 0.0.3
warning: metadata-json-lint: metadata.json: Dependency puppetlabs/stdlib has an open ended dependency version requirement >= 4.6.0
warning: metadata-json-lint: metadata.json: Dependency puppetlabs/apt has an open ended dependency version requirement >= 1.7.0
warning: metadata-json-lint: metadata.json: Dependency stahnma/epel has an open ended dependency version requirement >= 1.1.0
warning: metadata-json-lint: metadata.json: Dependency puppet/wget has an open ended dependency version requirement >= 1.5.6
warning: metadata-json-lint: metadata.json: Dependency saz/sudo has an open ended dependency version requirement >= 3.0.9
warning: metadata-json-lint: metadata.json: Dependency puppet/python has an open ended dependency version requirement >= 1.10.0
warning: metadata-json-lint: metadata.json: Dependency puppetlabs/inifile has an open ended dependency version requirement >= 1.2.0
warning: metadata-json-lint: metadata.json: Dependency puppet/mongodb has an open ended dependency version requirement >= 0.14.0
warning: metadata-json-lint: metadata.json: Dependency puppetlabs/postgresql has an open ended dependency version requirement >= 4.4.2
warning: metadata-json-lint: metadata.json: Dependency puppet/rabbitmq has an open ended dependency version requirement >= 4.1.0
warning: metadata-json-lint: metadata.json: Dependency ghoneycutt/facter has an open ended dependency version requirement >= 3.0.0
warning: metadata-json-lint: metadata.json: Dependency computology/packagecloud has an open ended dependency version requirement >= 0.3.1
warning: metadata-json-lint: metadata.json: Dependency puppet/selinux has an open ended dependency version requirement >= 0.5.0
warning: metadata-json-lint: metadata.json: Dependency puppet/nginx has an open ended dependency version requirement >= 0.5.0
warning: metadata-json-lint: metadata.json: Dependency puppet/nodejs has an open ended dependency version requirement >= 1.3.0

Also, why don't we error in travis for this?

Duplicate st2 service logs (disable upstart st2 service logs)

I just noticed on our build servers that we have duplicated st2 service logs.

One of those logs get created by st2 and live in /var/log/st2/st2*.log and other get created by upstart and live in /var/log/upstart/st2*.log.

We need to fix this since default /var/log/upstart logrotate config is very permissive (there is no upper limit in file size) so if the service is running using debug level, disk will fill up really fast.

For one, it seems we should disable upstart logging since we already manage log files ourselves. If not, we should at least add a more restrictive logrotate config for st2 log files in /var/log/upstart/*.

Add support for NodeJS v10

StackStorm/st2chatops#109

Tasks

  • Wait for NodeJS v10 to be supported upstream in StackStorm/st2chatops
  • Edit manifests/profile/nodejs.pp to install v10 if the StackStorm version is >= the version supported upstream
  • Test installation in various platforms

Add support for mongodb 3.6

StackStorm/st2#4098

Tasks

  • Wait for mongodb 3.6 to be support upstream in StackStorm
  • Edit manifests/profile/mongodb.pp to install 3.6 if the StackStorm version is >= the version supported upstream
  • Test installation in various platforms

Workroom integration tests fail for ubuntu14 because ``repo_base`` is not respected.

Execution id in st2build002: 56957ff682fb9b04bf891ddb

Affected host:

lakshmi@st2w-master-u14-c83b08840e:~$ cat  /tmp/answers.yaml
st2::version: 1.3dev
st2::revision: 76
st2::repo_base: http://dl-staging201
lakshmi@st2w-master-u14-c83b08840e:~$
lakshmi@st2w-master-u14-c83b08840e:~$ cat /etc/apt/sources.list.d/stackstorm.list
# This file is managed by Puppet. DO NOT EDIT.
# stackstorm
deb https://downloads.stackstorm.net/deb/ trusty_unstable main
lakshmi@st2w-master-u14-c83b08840e:~$

^^ Note that it is not using staging downloads.

Note that EL6 and EL7 are working fine for exact same params. This means there is a regression for ubuntu.

Executions 56957ff782fb9b04bf891ddeand 56957ff782fb9b04bf891de1

Speculations

How to repro?

  • git clone [email protected]:manasdk/st2-test-ground.git
  • vagrant up u14 --no-provision
  • Edit /tmp/answers.yaml and insert
st2::version: 1.3dev
st2::revision: 76
st2::repo_base: http://dl-staging201
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2common=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2common' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2common]/Package[st2common]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2common=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2common' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2reactor=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2reactor' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2reactor]/Package[st2reactor]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2reactor=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2reactor' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2actions=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2actions' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2actions]/Package[st2actions]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2actions=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2actions' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2api=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2api' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2api]/Package[st2api]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2api=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2api' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2auth=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2auth' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2auth]/Package[st2auth]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2auth=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2auth' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2debug=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2debug' was not found
Error: /Stage[main]/St2::Profile::Server/St2::Package::Install[st2debug]/Package[st2debug]/ensure: change from purged to 1.3dev-76 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install st2debug=1.3dev-76' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '1.3dev-76' for 'st2debug' was not found

Add support for installing EWC enterprise components

Currently this module only installs the Open Source components. We should add support for the EWC enterprise components. We'll most likely need to accept a "license key" variable and use that during the configuration process.

Add support for Ubuntu 18.04

Ubuntu 18.04 is out now. We should look into supporting it in Puppet.

Will need to wait for support from StackStorm/st2 core and StackStorm/st2packaging

Refactor st2::profile::repos

Currently we rely on the packagecloud module. On Ubuntu 14.04 this module has issues and corrupts the apt cache.

We should re-work this and simply rely on the puppetlabs/apt and puppetlabs/yum modules.

Avoid hard-coding git repo URLs

It would be a huge benefit to this module to be able to specify a source and version for StackStorm on the installation that can be cloned and distributed on systems which aren't connected to the outside world. Right now, configuration on that end requires cloning an internal repo, which isn't as desired.

If we could have the ability to install StackStorm via puppet via a packaged-up DEB/RPM, it would be really helpful.

Add the ability to specify a pack version in the pack resource

Right now it looks like the pack resource is only validating the pack exists before the module installs the pack. Allowing pack version specifications would ensure a particular version always exists on the Stackstorm server.

Thinking out loud here...
Add version to the resource definition - something like:

st2::pack { 'backups':
    version => "v2.0.0"
  }

The version is passed along to this function which then executes a modified command

output = exec_st2('pack', 'list', '-a', 'ref', 'version', '-j', '-t', token)

Then if that specific version isn't installed it gets installed here, passing along the version.

This is a massive oversimplification, no doubt.

Add support for RabbitMQ auth

Currently the RabbitMQ configuration in this module relies on the default rabbitmq account.

We should do the following:

  • Add a new vhost (optional)
  • Add a new user + password
  • Remove default user

st2::auth::pam broken looking for st2::helpers::auth_manager

Maybe something didn't get committed?

class { '::st2::helper::auth_manager':

$ sudo puppet agent -t
Fact file /etc/facter/facts.d/facts.txt was parsed but returned an empty data set
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Fact file /etc/facter/facts.d/facts.txt was parsed but returned an empty data set
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::st2::helper::auth_manager at /etc/puppet/modules/st2/manifests/auth/pam.pp:48 on node stage-stackstorm-0a70ba88456eebfb3.aws.logicworks.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Cleanup gems checked into git

I was just poking through the git history using this script https://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/

I found that at some point in the past a bunch of gem binaries where checked directly into the repo.

size  pack  SHA                                       location
2561  2557  be6aa88e765c4a14e70ffeb7f45e20bc28f82a87  vendor/cache/puppet-3.7.3.gem
869   866   0050cbd9d0abf38ff600fea3f3914259d54e2ca9  vendor/cache/term-ansicolor-1.3.2.gem
383   275   b6e2422fca0ced54dece99ee8b756d996ef72c0a  vendor/cache/unf_ext-0.0.7.1.gem
292   289   720d862809234400652619427697ec66fc50642b  vendor/cache/tins-1.5.4.gem
228   225   5fa79db2f805d31212edd78c890df2db2c4fa549  vendor/cache/facter-2.3.0-universal-darwin.gem
185   182   f76f65d8bb56fce308de665923a0876901f26a18  vendor/cache/mime-types-2.6.1.gem
179   175   43921dc55a877f3cc39ddc85f9a99bf71a5c83d1  vendor/cache/simplecov-html-0.10.0.gem
148   145   6474a4fcf3d21038d3adc2ce16e2f22ab68cc663  vendor/cache/json-1.8.3.gem
148   145   f766dc258973eacf0dfbdfdb7add2a30eb2c8a58  vendor/cache/json_pure-1.8.2.gem
127   124   0bea3c015a9245240eada0dfa03a6153fb87fb7b  vendor/cache/rake-10.4.2.gem
122   118   81d44af456c5e9239e58a71d9b4192426d2b8369  vendor/cache/rest-client-1.8.0.gem
117   74    01f1852db84d327dad9e72bd2d80a8f4a2d70074  vendor/cache/unf-0.1.4.gem
109   104   2b20659f2b9762051d8d0cc17c07375ba7dcaee0  vendor/cache/rspec-core-3.1.7.gem
102   98    5c16809eb79f2e1ca8f674d6370f66c4049d240a  vendor/cache/domain_name-0.5.24.gem
89    86    1ca502f3a513ea990d71c14abf341e220995ce06  vendor/cache/thor-0.19.1.gem
85    82    39128ca4c29f26c9f19d816ed73053eb06703fce  vendor/cache/mocha-1.1.0.gem
63    58    1625553b9e32036153499080b1c5feb4ff28fb04  vendor/cache/rspec-mocks-3.1.3.gem
62    57    1b0f2258dcf391db924c9c3726a1cc10ae0eb11a  vendor/cache/rspec-expectations-3.1.2.gem
51    48    372564fe138ab7e3aa85f226fae3d0b07820955f  vendor/cache/simplecov-0.10.0.gem
48    43    e4436ccc54911911961bcc89113b273459102fcf  vendor/cache/diff-lcs-1.2.5.gem
42    40    15c8f4bba1c309843abce49c817f1a620d6f8929  vendor/cache/puppet-lint-1.1.0.gem
37    34    9c90dcd5be1280df2ba177da11c8816f5c02ff7a  vendor/cache/http-cookie-1.0.2.gem
29    26    c7c1a52744283a582420dd3dd77ab7e560ebac84  vendor/cache/hiera-1.3.4.gem
22    16    0a230d618d71654d320d4cbae659f9a287d79392  vendor/cache/rspec-support-3.1.2.gem
18    15    c7f1ed24a85cecc9f64164c9abc8fba879eda755  vendor/cache/puppetlabs_spec_helper-0.8.2.gem
17    6     dc6213f5991c266643fc39d365cf0355389ca9c2  CHANGELOG.md
16    13    37be94b763511aefc813dfcfecfd2b184cec5c86  vendor/cache/coveralls-0.8.2.gem
15    12    122a14b85f69c0fd3d092427ecf21a20dcba1c03  vendor/cache/rspec-puppet-1.0.1.gem
15    12    adf1cf0caf5c8c7ea9f0515b1b43b15f91834b08  vendor/cache/CFPropertyList-2.2.8.gem
14    11    4eb5de66b7af6ac04c5e08713640bf3df0fa77eb  vendor/cache/docile-1.1.5.gem
12    2     d6ea4241821cdafbf50bfad9fdb79ea7e75a1275  manifests/profile/server.pp
11    4     354f387c6b8689bb44308dd9ca0a348def2fd701  README.md
11    2     909da8c4354fec99ef4f7159ff6e7c1d95004722  manifests/init.pp
11    8     10e3ff028d1eba39d5107254eae4e0fda7ee3cc1  vendor/cache/puppet-syntax-1.4.1.gem
10    3     0b9b4ea88a5cab47f4569cca0c6ce2dbeea68bbc  manifests/profile/mistral.pp
10    7     29e5655301e637c20a7ef572977aadd28b7e50c1  vendor/cache/netrc-0.10.3.gem
10    3     a3c475c65f9b58f953e388ae715aff2cd8dcac64  spec/helpers/fact_helper.rb
8     2     574cb0435ec74e6264cc230c5e6998e18cb0dc92  manifests/profile/mongodb.pp

We need to delete these from history so the size of the git repo isn't so big.

Integration Tests

Would be good to have Integration Testing framework in place like KitchenCI and some minimal Integration Tests as a first step to verify if module could be run "for real".

Examples:

We can even grab same Inspec tests from the chef-stackstorm.

Setup Puppetfile with all dependencies

Currently kitchen-puppet uses librarian-puppet to resolve all of this module's dependencies recursively. This means that our current Puppetfiles are only valid when using librarian-puppet. We can potentially create Puppetfiles for use with r10k with all recursive dependencies resolved so that can be used for quick starts.

r10k

  /opt/puppetlabs/puppet/bin/gem install r10k
  /opt/puppetlabs/puppet/bin/r10k puppetfile install --moduledir=./modules --puppetfile=./st2/build/<os_name>/Puppetfile

librarian-puppet

  mkdir modules/
  git clone https://github.com/StackStorm/puppet-st2 modules/st2
  
  # run librarian-puppet to download all of our module dependencies defined in
  # ./modules/st2/build/<os_name>/Puppetfile
  # valid <os_name> values are:
  #  centos6   # Puppet 3.x on RHEL/CentOS 6
  #  centos7   # Puppet 3.x on RHEL/CentOS 7
  #  ubuntu14  # Puppet 3.x on Ubuntu 14
  #  ubuntu16  # Puppet 3.x on Ubuntu 16
  #  puppet4   # OS agnostic for Puppet 4
  #  puppet5   # OS agnostic for Puppet 5
  /opt/puppetlabs/puppet/bin/gem install librarian-puppet
  cp modules/st2/build/<os_name>/Puppetfile .
  /opt/puppetlabs/puppet/bin/librarian-puppet install --path=./modules

  # run StackStorm full install using puppet
  puppet apply --modulepath=./modules -e "include ::st2::profile::fullinstall"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.