ffrank / puppet-mgmtgraph Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 4.0 190 KB

The mgmt translator for Puppet manifests

License: Other

Ruby 97.16% Shell 2.84%

puppet-mgmtgraph's People

Contributors

Stargazers

Watchers

Forkers

purpleidea mpeter aequitas electrofelix

puppet-mgmtgraph's Issues

Don't try running on older Puppet versions

Puppet 3 will happily install the module, but fail to run.

Installation tips/docs/guide would be welcome

A short quick start or installation guide would be very welcome in the README. Particularly for dummies like me who are starting to forget how puppet works. :P

Should missing resources be mentioned in stats command?

With version 0.4.1 (AFAICT)

I ran:

puppet mgmtgraph stats --code 'file { "/tmp/test": ensure => present, seltype => "tmp_t" } -> package { "cowsay": ensure => installed } -> schedule { "puppet": period => hourly, repeat => 2 }'

and got the following output:

Warning: GettextSetup is not available, skipping GettextSetup initialization for modules.
   (file & line not available)
connect: No such file or directory
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library
    1x File[...] cannot translate attribute 'seltype', attribute is ignored

I would have expected to see something about the schedule resource, shouldn't I? (since there isn't an equivalent in mgmt)

Thanks!

find a way to translate exports and imports

First, understand the differences between exports as seen by puppet vs. mgmt.

Then, see if and how exports are even represented in the compiled catalog.

Finally, if possible, see how the translation can be implemented.

don't rebase/overwrite git master

I got a sneaking suspicion this is happening. Except for emergencies where someone hacked your repo or you really made a bad push and you fixed in near instantly, don't change git master once you've pushed something. You can git revert a patch instead.

eg: 1e0d15d hit git master, then disappeared.

Cheers!

metadata version doesn't match with face's version

The current HEAD contains the following line

Puppet::Face.define(:mgmtgraph, '0.0.1') do

This code doesn't match up with the current version encoded in the metadata.json: 0.3.0.

Store puppet translation DSL in mgmt

Here's a thought:

In mgmt, resources can have "traits". This means they get an extra function with a specific signature attached to it. What if we added a "transformer" trait, which was a special thing that returned the translation DSL for that resource? That way, when we add/change resources, it will always be in sync? The only difference is that at runtime, the puppet side would have to request that stuff at runtime. Would that work?

Eg:

We'd add:

func (obj *PkgRes) PuppetTransformer() (string, error) {
return ```
PuppetX::CatalogTranslation::Type.new :package do
  emit :pkg

  spawn :name do
    @resource[:name]
  end

  rename :ensure, :state do |value|
    case value
    when :installed, :present
      :installed
    when :purged, :absent
      :uninstalled
    else
      translation_failure "uses an ensure value that currently cannot be translated for mgmt (defaulting to 'installed')", value
      :installed
    end
  end
end

)


Or similar. Perhaps there's even a more logical or better API.

TBD

Running in agent mode does not appear to retreive catalog

Running 'puppet mgmtgraph' doesn't appear to be retrieving the catalog for my environment yet.

Note: still haven't fixed making sure the environment path exists on machines in my vagrant env by default, which kind of shows that puppet module install expects to only install modules on the puppetmaster as opposed to agents, or at least that's what I'm assuming.

Running in a vagrant environment:

mkdir -p /etc/puppetlabs/code/environments/vagrant
sudo /opt/puppetlabs/bin/puppet module install ffrank-mgmtgraph
sudo /opt/puppetlabs/bin/puppet mgmtgraph

Produces the following output:

---
graph: srv-2.ncs.net
comment: generated from puppet catalog for srv-2.ncs.net
resources:
  noop:
  - name: admissible_Stage[main]
  - name: completed_Stage[main]
  - name: admissible_Class[Settings]
  - name: completed_Class[Settings]
  - name: admissible_Class[Main]
  - name: completed_Class[Main]
edges:
- name: Whit[Completed_class[Settings]] -> Whit[Completed_stage[main]]
  from:
    kind: noop
    name: completed_Class[Settings]
  to:
    kind: noop
    name: completed_Stage[main]
- name: Whit[Completed_class[Main]] -> Whit[Completed_stage[main]]
  from:
    kind: noop
    name: completed_Class[Main]
  to:
    kind: noop
    name: completed_Stage[main]
- name: Whit[Admissible_stage[main]] -> Whit[Admissible_class[Settings]]
  from:
    kind: noop
    name: admissible_Stage[main]
  to:
    kind: noop
    name: admissible_Class[Settings]
- name: Whit[Admissible_class[Settings]] -> Whit[Completed_class[Settings]]
  from:
    kind: noop
    name: admissible_Class[Settings]
  to:
    kind: noop
    name: completed_Class[Settings]
- name: Whit[Admissible_stage[main]] -> Whit[Admissible_class[Main]]
  from:
    kind: noop
    name: admissible_Stage[main]
  to:
    kind: noop
    name: admissible_Class[Main]
- name: Whit[Admissible_class[Main]] -> Whit[Completed_class[Main]]
  from:
    kind: noop
    name: admissible_Class[Main]
  to:
    kind: noop
    name: completed_Class[Main]

Where as running sudo /opt/puppetlabs/bin/puppet agent -t produces the following:

Warning: Setting configtimeout is deprecated. 
   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1163:in `issue_deprecation_warning')
Info: Using configured environment 'vagrant'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for srv-2.ncs.net
Info: Applying configuration version '1552308129'
Notice: Configure srv-2.ncs.net internal interace on eth2
Notice: /Stage[main]/Ncs::Net/Notify[Configure srv-2.ncs.net internal interace on eth2]/message: defined 'message' as 'Configure srv-2.ncs.net internal interace on eth2'
Notice: Configure srv-2.ncs.net external interface on eth1
Notice: /Stage[main]/Ncs::Net/Notify[Configure srv-2.ncs.net external interface on eth1]/message: defined 'message' as 'Configure srv-2.ncs.net external interface on eth1'
Notice: /Stage[main]/Ncs::Ncszing_171_disable_fstrim/Exec[comment_fstrim_from_cron_weekly]/returns: executed successfully
Notice: /Stage[main]/Ncs::Puppet/Exec[trigger download of puppet5-release-xenial.deb]/returns: executed successfully
Notice: Getting image: 192.168.32.1:5000/ncsre/redis:latest
Notice: /Stage[main]/Ncs::Sensu/Ncs::Docker::Run[sensu-redis]/Notify[Getting image: 192.168.32.1:5000/ncsre/redis:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/redis:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/rabbitmq:3.5.7-management
Notice: /Stage[main]/Ncs::Sensu/Ncs::Docker::Run[sensu-rabbitmq]/Notify[Getting image: 192.168.32.1:5000/ncsre/rabbitmq:3.5.7-management]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/rabbitmq:3.5.7-management'
Notice: Getting image: 192.168.32.1:5000/ncsre/sensu-api:latest
Notice: /Stage[main]/Ncs::Sensu/Ncs::Docker::Run[sensu-api]/Notify[Getting image: 192.168.32.1:5000/ncsre/sensu-api:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/sensu-api:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/sensu-server:latest
Notice: /Stage[main]/Ncs::Sensu/Ncs::Docker::Run[sensu-server]/Notify[Getting image: 192.168.32.1:5000/ncsre/sensu-server:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/sensu-server:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/uchiwa:latest
Notice: /Stage[main]/Ncs::Sensu/Ncs::Docker::Run[sensu-uchiwa]/Notify[Getting image: 192.168.32.1:5000/ncsre/uchiwa:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/uchiwa:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/sensu-client:latest
Notice: /Stage[main]/Ncs::Sensu::Services/Ncs::Docker::Run[sensu-client]/Notify[Getting image: 192.168.32.1:5000/ncsre/sensu-client:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/sensu-client:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/cadvisor:latest
Notice: /Stage[main]/Ncs::Cadvisor/Ncs::Docker::Run[cadvisor]/Notify[Getting image: 192.168.32.1:5000/ncsre/cadvisor:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/cadvisor:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/web:latest
Notice: /Stage[main]/Ncs::Nginxweb/Ncs::Docker::Run[web]/Notify[Getting image: 192.168.32.1:5000/ncsre/web:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/web:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/changelogs:latest
Notice: /Stage[main]/Ncs::Changelogs/Ncs::Docker::Run[changelogs]/Notify[Getting image: 192.168.32.1:5000/ncsre/changelogs:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/changelogs:latest'
Notice: /Stage[main]/Ncs::Gerrit/Ncs::Storage[gerrit]/Exec[Check /mnt/gerrit]/returns: executed successfully
Notice: Getting image: 192.168.32.1:5000/ncsre/gerrit:latest
Notice: /Stage[main]/Ncs::Gerrit/Ncs::Docker::Run[gerrit]/Notify[Getting image: 192.168.32.1:5000/ncsre/gerrit:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/gerrit:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/jeepyb:latest
Notice: /Stage[main]/Ncs::Jeepyb/Ncs::Docker::Run[jeepyb]/Notify[Getting image: 192.168.32.1:5000/ncsre/jeepyb:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/jeepyb:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/zuul:latest
Notice: /Stage[main]/Ncs::Zuul::Scheduler/Ncs::Docker::Run[zuul-scheduler]/Notify[Getting image: 192.168.32.1:5000/ncsre/zuul:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/zuul:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/elasticsearch:latest
Notice: /Stage[main]/Ncs::Elasticsearch/Ncs::Docker::Run[elasticsearch]/Notify[Getting image: 192.168.32.1:5000/ncsre/elasticsearch:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/elasticsearch:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/gearmanworker:latest
Notice: /Stage[main]/Ncs::Gearmanworker/Ncs::Docker::Run[gearmanworker]/Notify[Getting image: 192.168.32.1:5000/ncsre/gearmanworker:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/gearmanworker:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/kibana:latest
Notice: /Stage[main]/Ncs::Kibana/Ncs::Docker::Run[kibana]/Notify[Getting image: 192.168.32.1:5000/ncsre/kibana:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/kibana:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/logstash:latest
Notice: /Stage[main]/Ncs::Logstash/Ncs::Docker::Run[logstash]/Notify[Getting image: 192.168.32.1:5000/ncsre/logstash:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/logstash:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/graphite-statsd:latest
Notice: /Stage[main]/Ncs::Graphite_statsd/Ncs::Docker::Run[graphite-statsd]/Notify[Getting image: 192.168.32.1:5000/ncsre/graphite-statsd:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/graphite-statsd:latest'
Notice: Getting image: 192.168.32.1:5000/ncsre/prometheus:latest
Notice: /Stage[main]/Ncs::Prometheus/Ncs::Docker::Run[prometheus]/Notify[Getting image: 192.168.32.1:5000/ncsre/prometheus:latest]/message: defined 'message' as 'Getting image: 192.168.32.1:5000/ncsre/prometheus:latest'
Notice: /Stage[main]/Ncs::Jeepyb/Ncs::Docker::Run[jeepyb]/Docker::Run[jeepyb]/Exec[remove jeepyb container]/returns: executed successfully
Notice: /Stage[main]/Ncs::Jeepyb/Ncs::Docker::Run[jeepyb]/Docker::Run[jeepyb]/Exec[remove jeepyb cidfile]/returns: executed successfully
Info: /Stage[main]/Ncs::Jeepyb/Ncs::Docker::Run[jeepyb]/Docker::Run[jeepyb]/Exec[remove jeepyb cidfile]: Scheduling refresh of Exec[run jeepyb with docker]
Notice: /Stage[main]/Ncs::Jeepyb/Ncs::Docker::Run[jeepyb]/Docker::Run[jeepyb]/Exec[run jeepyb with docker]: Triggered 'refresh' from 1 event
Info: Wait_for[wait for container to exit](provider=wait_for): Query output matched regex.
Notice: Applied catalog in 15.75 seconds

Suggests that the catalog wasn't retrieved in the way expected.

Looking at puppets other commands sudo /opt/puppetlabs/bin/puppet catalog find returns the following output:

Warning: Setting configtimeout is deprecated. 
   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1163:in `issue_deprecation_warning')
Notice: Compiled catalog for srv-2.ncs.net in environment vagrant in 0.02 seconds
{
  "tags": [
    "settings"
  ],
  "name": "srv-2.ncs.net",
  "version": 1552309089,
  "code_id": null,
  "catalog_uuid": "8c2f17b4-35e8-4163-8756-78a98f26e660",
  "catalog_format": 1,
  "environment": "vagrant",
  "resources": [
    {
      "type": "Stage",
      "title": "main",
      "tags": [
        "stage"
      ],
      "exported": false,
      "parameters": {
        "name": "main"
      }
    },
    {
      "type": "Class",
      "title": "Settings",
      "tags": [
        "class",
        "settings"
      ],
      "exported": false
    },
    {
      "type": "Class",
      "title": "main",
      "tags": [
        "class"
      ],
      "exported": false,
      "parameters": {
        "name": "main"
      }
    }
  ],
  "edges": [
    {
      "source": "Stage[main]",
      "target": "Class[Settings]"
    },
    {
      "source": "Stage[main]",
      "target": "Class[main]"
    }
  ],
  "classes": [
    "settings"
  ]
}

Which aligns with what is provided by mgmtgraph, however that does not contain the expected resources.

Performing a sudo /opt/puppetlabs/bin/puppet catalog download however produced the following:

Warning: Setting configtimeout is deprecated. 
   (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1163:in `issue_deprecation_warning')
Notice: Saved catalog for srv-2.ncs.net to /var/lib/puppet/client_yaml/catalog/srv-2.ncs.net.yaml

and the first few lines of this look a bit more promising:

--- !ruby/object:Puppet::Resource::Catalog
tags:
- settings
- ncs::sensu
- ncs
- sensu
- ncs::sensu::services
- services
- ncs::net
- net
- locales::params
- locales
- params
- ncs::ntp
- ntp
- ntp::params
- ca_cert::params
- ca_cert

Stop spawning parameters with nil values

The translation_failure method just leads to malformed resources in non-conservative mode

Update details on supported resources in mgmt

As mentioned in #16, need to refresh the supported resource details so that generation of stats from existing puppet codebases on what resources are needed in mgmt to support translation are more useful.

Edge names should be deterministic

If the edges change constantly because they're numbered from e1, e2, e3... then it's a problem. So the eN generation should be replaced with edge names that are deterministic based on resource names. eg: Edge::(File::$name->File::$name2)

This is mentioned here:
purpleidea/mgmt#8 (comment)

Don't raise exceptions from within the translator DSL

File source is handled wrongly

File resources that manage directories have good handling for the source property, e.g. when it's a puppet:// URL:

puppet mgmtgraph print --code 'file { "/path/to/managed/dir": ensure => "directory", source => "puppet:///modules
/local_files/my_managed_dir" }'
...
Error: File[/path/to/managed/dir] uses a puppet fileserver URL source - this will not be translated
Warning: File[/path/to/managed/dir] emitting a `exec puppet resource` node because of the errors above.
Error: File[/path/to/managed/dir] cannot be translated natively, falling back to pippet
...
resources:
  pippet:
  - name: File[/path/to/managed/dir]
    title: "/path/to/managed/dir"
    type: file
    params: |-
      {"path": "/path/to/managed/dir", "provider": "posix", "ensure": "directory", "source": [
          "puppet:///modules/local_files/my_managed_dir"], "backup": "puppet", "replace": true,
        "links": "manage", "purge": false, "sourceselect": "first", "show_diff": true, "validate_replacement": "%",
        "checksum": "md5", "source_permissions": "ignore", "selinux_ignore_defaults": "false",
        "loglevel": "notice"}
    pollint: 1800

This is not quite the case for flat files.

puppet mgmtgraph print --code 'file { "/path/to/managed/file": source => "puppet:///modules/local_files/my_managed
_file" }'
...
resources:
  file:
  - name: "/path/to/managed/file"
    path: "/path/to/managed/file"

Make sure that source is always handled correctly.

Migrate to PDK

Publishing on the Forge has moved to a PDK-based workflow. Update this module and release an update, will you 🧐

puppet-mgmtgraph should respect classes

When we see classes in puppet like:

Package["foo"] -> Class["bar"]

and

class bar {
    file { 'a':
        ensure => present,
        content => "a",
    }
    file { 'b':
        ensure => present,
        content => "b",
    }
    file { 'c':
        ensure => present,
        content => "c",
    }
}

Is the equivalent raw graph something like:

Package["foo"] -> File["a"]
Package["foo"] -> File["b"]
Package["foo"] -> File["c"]

? I think this is the logical thing to do, but knowing what puppet actually does would be helpful. Maybe @tuxmea can chime in.

If so, we can replace the Class object with a "Noop" resource in mgmt, or just directly connect the edges. My uneducated preference would be to use the Noop resources so we could at least "see" the Class structure a bit.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.