How to use ESS command with ES and shield implemnted about elasticsplunk HOT 6 CLOSED

Hi @praveer19
Unfortunately i don't have a shield setup to test, but i understand it supports basic auth.

Can you try specifying your eaddr like:
|ess eaddr="https://USER:PASS@node1:9200,https://USER:PASS@node2:9200"

And post your results ?

Cheers!

from elasticsplunk.

Hi,
Having a similar experience connecting Splunk 7.1.3 to Elasticsearch 6.6.0 secured with SearchGuard plugin. I am trying to connect a couple different ways:
|ess eaddr=https://node1:9200 produces the following error:
External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e1162ec10>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e1162ec10>: Failed to establish a new connection: [Errno 111] Connection refused) "

And the following log output from splunkd.log:
03-07-2019 15:53:05.336 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,335, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:53:05.336 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,336, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=508, metadata={u'action': u'getinfo', u'preview': True, u'searchinfo': {u'session_key': None, u'app': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'splunkd_uri': None, u'owner': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'earliest_time': None, u'dispatch_dir': None, u'username': None, u'search': u'|ess eaddr=https://192.168.33.50:9200', u'latest_time': None, u'sid': u'', u'splunk_version': u'7.1.3'}}, input_header={u'keywords': u'""', u'preview': u'0', u'search': u'|ess eaddr=https://192.168.33.50:9200', u'sid': u'', u'truncated': u'0', u'allowStream': u'1', u'splunkVersion': u'7.1.3', u'realtime': u'0'} 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://192.168.33.50:9200'] 03-07-2019 15:53:05.337 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,337, Level=DEBUG, Pid=11434, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://192.168.33.50:9200" 03-07-2019 15:53:05.345 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,338, Level=DEBUG, Pid=11434, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1 03-07-2019 15:53:05.918 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,913, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=508, metadata={u'searchinfo': {u'dispatch_dir': None, u'splunkd_uri': None, u'username': None, u'earliest_time': None, u'sid': u'searchparsetmp_820669171', u'app': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://192.168.33.50:9200'], u'splunk_version': u'7.1.3', u'owner': None, u'search': u'|ess eaddr=https://192.168.33.50:9200', u'session_key': None, u'latest_time': None}, u'preview': True, u'action': u'getinfo'}, input_header={u'realtime': u'0', u'keywords': u'""', u'truncated': u'0', u'preview': u'0', u'search': u'|ess eaddr=https://192.168.33.50:9200', u'allowStream': u'1', u'sid': u'searchparsetmp_820669171', u'splunkVersion': u'7.1.3'} 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://192.168.33.50:9200'] 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://192.168.33.50:9200" 03-07-2019 15:53:05.919 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://192.168.33.50:9200': 2019-03-07 15:53:05,914, Level=DEBUG, Pid=11464, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1

When I execute with the 'admin:pass' in the search, I get the same error in the Splunk UI:
|ess eaddr=https://admin:<pass>@192.168.33.50:9200

UI error:
External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6f4234ec50>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6f4234ec50>: Failed to establish a new connection: [Errno 111] Connection refused) "

And the following log set from splunkd.log:
03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=508, metadata={u'action': u'getinfo', u'preview': True, u'searchinfo': {u'session_key': None, u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'owner': None, u'sid': u'', u'earliest_time': None, u'username': None, u'splunk_version': u'7.1.3', u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'splunkd_uri': None, u'dispatch_dir': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'app': None, u'latest_time': None}}, input_header={u'keywords': u'""', u'allowStream': u'1', u'realtime': u'0', u'preview': u'0', u'sid': u'', u'splunkVersion': u'7.1.3', u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'truncated': u'0'} 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,719, Level=DEBUG, Pid=11601, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://admin:<pass>@192.168.33.50:9200'] 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,720, Level=DEBUG, Pid=11601, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://admin:<pass>@192.168.33.50:9200" 03-07-2019 15:54:12.727 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:12,720, Level=DEBUG, Pid=11601, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,282, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=572, ElasticSplunk.process started under protocol_version=1 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,282, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=579, Writing configuration settings 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=508, metadata={u'preview': True, u'searchinfo': {u'sid': u'searchparsetmp_1008607785', u'splunk_version': u'7.1.3', u'latest_time': None, u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'username': None, u'dispatch_dir': None, u'earliest_time': None, u'args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'owner': None, u'raw_args': ['/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py', '__GETINFO__', 'eaddr=https://admin:<pass>@192.168.33.50:9200'], u'session_key': None, u'splunkd_uri': None, u'app': None}, u'action': u'getinfo'}, input_header={u'sid': u'searchparsetmp_1008607785', u'realtime': u'0', u'splunkVersion': u'7.1.3', u'search': u'|ess eaddr=https://admin:<pass>@192.168.33.50:9200', u'allowStream': u'1', u'preview': u'0', u'truncated': u'0', u'keywords': u'""'} 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=515, tempfile.tempdir=None 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=internals.py, Line=119, Parsing ElasticSplunk command line: ['eaddr=https://admin:<pass>@192.168.33.50:9200'] 03-07-2019 15:54:13.283 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=internals.py, Line=155, ElasticSplunk: elasticsplunk eaddr="https://admin:<pass>@192.168.33.50:9200" 03-07-2019 15:54:13.284 -0600 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsplunk.py __GETINFO__ eaddr=https://admin:<pass>@192.168.33.50:9200': 2019-03-07 15:54:13,283, Level=DEBUG, Pid=11632, Logger=splunklib, File=search_command.py, Line=624, ElasticSplunk.process finished under protocol_version=1

I've tried to edit the Urllib3HttpConnection(Connection) class in /opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py to provide different values and those error as well.

Help is appreciated,
Thanks!

from elasticsplunk.

@ecwhipple, the error from the logs is 111 Connection refused. Is https://192.168.33.50:9200 the actual elasticsearch url? do you have ssl enabled for elasticsearch itself (not kibana...)? If so elasticsearch is listening on port 9200?

Can you post your elasticsearch.url from your kibana.yml configuration file or from the kibana command line if specified there ?

from elasticsplunk.

@brunotm I did some netcat tests from the splunk instance (test-centos-7) to the elasticsearch instance (test-single-centos-7) and found I was getting connection refused messages.
[root@test-centos-7 ~]# nc -vz 192.168.33.50 9200 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connection refused.

I updated the elasticsearch.yml, it was listening on 127.0.0.1 and is now updated to host: 0.0.0.0 and can verify that I am able to curl that elasticsearch endpoint from the splunk instance:
[root@test-centos-7 ~]# curl -k https://[email protected]:9200 { "name" : "test-single-centos-7", "cluster_name" : "elk_test_cluster", "cluster_uuid" : "mIFQ8PaxT7W39D4LnyoaNw", "version" : { "number" : "6.6.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "a9861f4", "build_date" : "2019-01-24T11:27:09.439740Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Now, when running the following splunk search, I am getting a new error:
Search:
|ess [email protected]:9200 action=cluster-health
Error:
External search command 'ess' returned error code 1. Script output = "error_message=ConnectionError at "/opt/splunk/etc/apps/elasticsplunk-master/bin/elasticsearch/connection/http_urllib3.py", line 156 : ConnectionError(('Connection aborted.', BadStatusLine("''",))) caused by: ProtocolError(('Connection aborted.', BadStatusLine("''",))) "

Is that due to the ssl not being passed in the search?

from elasticsplunk.

@ecwhipple the eaddr parameter must specify the protocol, otherwise will default to http. So your eaddr would be https://[email protected]:9200. Additionally you can create a cluster configuration in the $SPLUNK_PATH//etc/apps/elasticsplunk/local/elasticsplunk.json:

{
	"cluster1":{
		"hosts": ["https://[email protected]:9200"],
		"tsfield": "timestamp",
		"verify_certs": false
	},

	"cluster2":{
		"hosts": ["https://node1/elastic", "https://node2/elastic", "https://node3/elastic"],
		"tsfield": "time",
		"verify_certs": true,
                 "ca_cert": "/path/to/cert/ca.pem"
	}
}

and reference the named cluster in eaddr, e.g. |ess eaddr=cluster1

from elasticsplunk.

Thanks @brunotm, this is working now. Appreciate the help and fast response.

from elasticsplunk.