Issue

The add-on settings tab fails to load meaning you can't add in the API token from the Puppet Master.

Config

Splunk Version: 7.0.2
OS: CentOS 7.2.1511

Desc

After removing the passwords.conf file from $SPLUNK_DIR/etc/apps/SplunkTAforPuppetEnterprise/local/passwords.conf and restarting the Splunk server, the add-on settings will not load.

Log

03-13-2018 05:27:53.956 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/puppet_enterprise_extended_details.py" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py\", line 113, in wrapper\n    for name, data, acl in meth(self, *args, **kwargs):\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py\", line 299, in _format_response\n    masked = self.rest_credentials.decrypt_for_get(name, data)\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/credentials.py\", line 196, in decrypt_for_get\n    data[field_name] = clear_password[field_name]\nTypeError: 'NoneType' object has no attribute '__getitem__'\n\".  See splunkd.log for more details."}]}
03-13-2018 05:27:53.989 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/puppet_enterprise_extended_details.py" ERRORHTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py\", line 113, in wrapper\n    for name, data, acl in meth(self, *args, **kwargs):\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py\", line 299, in _format_response\n    masked = self.rest_credentials.decrypt_for_get(name, data)\n  File \"/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/credentials.py\", line 196, in decrypt_for_get\n    data[field_name] = clear_password[field_name]\nTypeError: 'NoneType' object has no attribute '__getitem__'\n\".  See splunkd.log for more details."}]}

Hi - I am in the process of testing the Puppet App and Puppet Add On for a customer. Just installed the TA onto a Splunk 7.2 single server environment.

When I try to open the TA I get the following error - any thoughts or suggestions please?

An error occurred while reading the page template. See web_service.log for more details Click here to return to Splunk homepage.

I am using Splunk 7.2.1 and I am never able to get to the GUI items to enter the api_token or configure the servername \ ports for the inputs.

When I click on the App in the GUI I get a "500 Server Error" with a horse saying "Oops"..

-Archie

I am not seeing the Setup action\page when I attempt to follow the documentation step "open the Setup page from the Manage Apps page to configure your Puppet Enterprise server info"

I do not see the Setup action button to click in the Splunk GUI.

Is there an example of the inputs.conf that can be modified for our Puppet Enterprise server and credentials?

I have installed a client Splunk 7.0 server, and also a 6.3.12, and installed the Add on and the App for Puppet.

On either server version the SplunkTA drops errors in /opt/splunk/var/log/splunk

HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"\n In handler 'SplunkTAforPuppetEnterprise_settings': Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/handler.py", line 299, in _format_response\n masked = self.rest_credentials.decrypt_for_get(name, data)\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/credentials.py", line 184, in decrypt_for_get\n clear_password = self._get(name)\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/splunktaucclib/rest_handler/credentials.py", line 389, in _get\n string = mgr.get_password(user=context.username())\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/solnlib/utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/solnlib/credentials.py", line 118, in get_password\n all_passwords = self._get_all_passwords()\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/solnlib/utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/solnlib/credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n". See splunkd.log for more details."}]}

My steps (RHEL 7.4)

1. Install splunk server (7.0 and/or 6.3.12)
2. Cd /opt/splunk/etc/apps
3. Git clone https://github.com/domeger/SplunkAppforPuppetEnterprise.git SplunkAppforPuppetEnterprise
4. Git clone https://github.com/domeger/SplunkTAforPuppetEnterprise.git SplunkTAforPuppetEnterprise
5. Configure real puppet server and replace default http://x.x.x.x for all inputs
6. Restart splunk

NOTE: PE is listening on 0.0.0.0:8080

Regards,
Rene

Does the TA need to be installed on the Puppet Enterprise server or can it be installed on any HF?
You still have enabled IP's/hostnames in the ../default/inputs.conf??

Here is my input and the log entries for factor collection.

[puppet_enterprise_factors://PuppetEnterpriseFactors_prod]
environment = production
index = unix
interval = 300
port_ = 8081
puppet_enterprise_server_ = xxx.xxx.xxx
server_ = https://xxx.xxx.xxx
token_ = ********
token_generation_date_ = 09/22/18

logs..
2018-09-23 09:39:21,016 INFO pid=14523 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2018-09-23 09:39:22,303 INFO pid=14523 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2018-09-23 09:39:24,035 INFO pid=14523 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2018-09-23 09:39:26,077 INFO pid=14523 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO
2018-09-23 09:39:26,078 INFO pid=14523 tid=MainThread file=setup_util.py:log_info:114 | Customized key can not be found
2018-09-23 09:39:26,078 ERROR pid=14523 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/splunktaforpuppetenterprise/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/puppet_enterprise_factors.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/SplunkTAforPuppetEnterprise/bin/input_module_puppet_enterprise_factors.py", line 63, in collect_events
if pe_link:
NameError: global name 'pe_link' is not defined

Splunk / MIT ▸ Puppet / Apache 2 (I'm guessing)

Hi,

I've sat up the Puppet TA, generated the token, and configured the inputs on a Splunk HF. Most of them works fine, and we're getting data into Splunk, but for one of the inputs we're not getting any data. The faulty input is namely the one for "extended details".

I've looked through the internal logs, but cant find any help there. The only message I'm getting in the internal Puppet TA log is the following.

"timestamp" ERROR pid=12345 tid=MainThread file=base_modinput.py:log_error:307 | []

This error I'm getting from the source /opt/splunk/var/log/splunk/splunkta/forpuppetenterprise_puppet_enterprise_extended_details.log.

Any idea on why it fails just for this one input, and how I could get some more insightful error messages on the subject?

input_mdoule_puppet_enterprise_factor.py throws an exception if a fact isn't returned by the HTTP(S) request. A value (even for a built-in fact) can be absent for legitimate reasons. In my environment, for example, many of our servers disable IPv6, so the ipaddress6 fact is not present. The resulting behavior is that the splunk add-on only collects facts for the nodes that have all the facts which are hard-coded in input_module_puppet_enterprise_factor.py. As soon as a node is encountered for which one of those facts is absent, an exception is thrown (KeyError) and no more are processed.

I think there 2 good solutions to this:

1. If it's not necessary to hard code the key names, you should be able to populate the dictionary dynamically based on the key names that are in the the JSON.

   Note: If you don't need to hard code the key names, but you do need to ensure that the resulting dictionaries all have consistent keys, then you could simply collect all unique key_names in an array while looping over the JSON and then massage the resulting dictionaries so that they all contain the same keys, even if some of those keys will not have a value. This strategy, while not optimal from a performance perspective, does ensure that custom facts are also available in the Splunk fact reports (I think).

2. If it's necessary to hard code the key names, define them in an array somewhere. Then, in the dictionary 'merging' section, you can loop over that array, catching the KeyError exceptions to avoid exiting prematurely.

I have the application working now with some issues. One thing I would recommend is to update the documentation for the API token to include the token "lifetime" longer than the default of 5 minutes.

PE Doc: https://puppet.com/docs/pe/2018.1/rbac_token_auth_intro.html#setting-a-token-specific-lifetime

I ended up with the following:
curl -k -X POST -H 'Content-Type: application/json' -d '{"login": "", "password": "","lifetime": "9y" }' https://$:4433/rbac-api/v1/auth/token

I also was getting some SSL errors and needed to get the puppet cert from my PE server and add it to my Splunk HF that was running the TA.

My issue now that I cannot seem to resolve. The facter collection is only grabbing 1 host for facters collection? I was seeing some messages in the logs about line truncation so I added "TRUNCATE = 0" to the props.conf to troubleshoot. The truncation messages stopped but I am still only getting facters from one certname each collection run.

When I search for (index=* sourcetype="pe:factors" certname="*") I only have facters from 1 server.

Here are some of the messages from the logs..

09-07-2018 18:52:10.065 -0400 INFO TailReader - Will retry path="/opt/splunk/var/log/splunk/splunktaforpuppetenterprise_puppet_enterprise_factors.log.3" after deferring for 10000ms, initCRC changed after being queued (before=0x50d178184950889d, after=0x2b94207408a2eeb0). File growth rate must be higher than indexing or forwarding rate.

09-07-2018 16:34:26.282 -0400 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 56562 - data_source="/opt/splunk/var/log/splunk/splunktaforpuppetenterprise_puppet_enterprise_factors.log", data_host="xxxxxx", data_sourcetype="splunktaforpuppetenterprise:log"

PE Version 2018.1.3
Splunk 7.3.1

-Archie

Update..

I am seeing the error 'base_modinput.py:log_error:307' in many of the splunktaforpuppetenterprise logfiles.

-bash-4.2$ pwd
/opt/splunk/var/log/splunk
-bash-4.2$ grep -c 'base_modinput.py:log_error:307' splunktaforpuppetenterprise_*
splunktaforpuppetenterprise_cloud_connect_engine.log:0
splunktaforpuppetenterprise_puppet_enterprise_aggregate_by_resource.log:8817
splunktaforpuppetenterprise_puppet_enterprise_aggregate_details_by_certname.log:8835
splunktaforpuppetenterprise_puppet_enterprise_aggregate_details_by_classes.log:8810
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log:476
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log.1:3439
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log.2:629
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log.3:682
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log.4:1446
splunktaforpuppetenterprise_puppet_enterprise_extended_details.log.5:1099
splunktaforpuppetenterprise_puppet_enterprise_factors.log:22
splunktaforpuppetenterprise_puppet_enterprise_factors.log.1:65
splunktaforpuppetenterprise_puppet_enterprise_factors.log.2:38
splunktaforpuppetenterprise_puppet_enterprise_factors.log.3:65
splunktaforpuppetenterprise_puppet_enterprise_factors.log.4:67
splunktaforpuppetenterprise_puppet_enterprise_factors.log.5:65
splunktaforpuppetenterprise_puppet_enterprise_node_status.log:247
splunktaforpuppetenterprise_puppet_enterprise_node_status.log.1:409
splunktaforpuppetenterprise_puppet_enterprise_node_status.log.2:409
splunktaforpuppetenterprise_puppet_enterprise_node_status.log.3:410
splunktaforpuppetenterprise_puppet_enterprise_node_status.log.4:409
splunktaforpuppetenterprise_puppet_enterprise_node_status.log.5:409
splunktaforpuppetenterprise_puppet_enterprise_status_0d907c823731123e6d4ec9557a1ca45a99ae8ce6b09ef49fd0d23e502de4ab84.log:0
splunktaforpuppetenterprise_puppet_enterprise_status_ff9a5547463478a06ee6f4ff538967494fcab876970afc436cf74d635af03837.log:0
splunktaforpuppetenterprise_util.log:0
-bash-4.2$

I'm trying to get this SplunkTAforPuppetEnterprise app working, as well as the SplunkAppforPuppetEnterprise. I've been able to get the official 'Puppet Enterprise' app working, but not yet this one.

I have the following test setup:

• 1 Splunk Enterprise server, configured with a receiver on port 9997
• 1 Puppet Master, with a Spunk Universal Forwarder installed. It is forwarding to the Splunk Enterprise server, and it's also configured as a deployment client to the Splunk Enterprise deployment server

To get the 'Puppet Enterprise' app working, I added the app to $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/deployment-apps (both on the Splunk Enterprise server), and then used server classification to push the app to the Universal Forwarder as well. After a few minutes, data started coming in to the Splunk Enterprise server.

I've tried configuring the SplunkTAforPuppetEnterprise app (filling in the API key) via the Splunk Web UI, but the ./local/splunktaforpuppetenterprise_settings.conf file ends up looking like this:

[additional_parameters]
api_token = ********

[logging]
loglevel = INFO

Some questions:

• Is the API key getting mangled somewhere?
• Am I supposed to manually paste the API key into that file?
• Should I preconfigure the splunktaforpuppetenterprise_settings.conf file before copying the app to the deployment-apps folder?
• With the 1.0.7 version, I no longer see any inputs. Is this expected?
• What should be the recommended method of getting this up and running? Which app should I deploy where and how?

Hi,

I've ran the Puppet TA for a while now, and after fixing some timezone issues, it have been working just fine. There is one problem though, I'm getting a lot of error messages in the internal logs from the TA. These error messages seems to be false positives (should be INFO). I'm suspecting that the TA logs an error message in the internal log for every Puppet event the TA collects. The errors looks like the following, with a different stuff in the square brackets.

ERROR pid=12345 tid=MainThread file=base_modinput.py:log_error:307 | [...]

This same issue was also mentioned in my previous thread (https://github.com/mrzarquon/SplunkTAforPuppetEnterprise/issues/6), although the main question was another, so the issue was closed.

Is it something wrong with my setup, og is the TA indeed incorrectly logging INFO logs as ERROR logs?

Hi. I've had some problems with duplicate events from the extended details inputs, from the script input_module_puppet_enterprise_extended_details.py. After doing some tests we've concluded that the checkpoint value is never being used in the script. That is, the following if statement is always true.

    if ckpt_value == None:
        old = now - datetime.timedelta(minutes=5)
        #format the time
        # This is a timestamp in UTC-based ISO-8601 format (YYYY-MM-DDThh:mm:ssZ) 
        start_time = old.strftime("%Y-%m-%dT%H:%M:%SZ") 
    #if it does exist then checkpoint value is start time
    else:
        start_time=ckpt_value

Does anyone got any idea on why this might be the case?

The tests we did were based on setting the old = now - datetime.timedelta(minutes=5) to higher values. If the ckpt_value would have been sat, it wouldn't matter how big the timedelta was, as it's only used in the initial startup of the TA. However when we set the timedelta to e.g. 10 minutes, any extended details event is indexed 10 times, indicating that the lookback with the timedelta is used for every run.