At the Splunk startup page I get this error message:

Warning: The time on the server differs significantly from this machine which may cause login problems and other errors.

Is there any way to fix this?

Hello,

I have been trying to enable clustering through compose file with no joy.

Could someone shed some light to the secret sauce?

splunkmaster:
   restart: always
   build: ./splunkmaster
   hostname: splunkmaster
   # image: splunk/splunk:6.5.3
   environment:
     SPLUNK_START_ARGS: --accept-license --answer-yes
     SPLUNK_ENABLE_LISTEN: 9997
     SPLUNK_CMD_1: splunk edit cluster-config -mode master -replication_factor 3 -search_factor 2 -secret newsecret123456 -cluster_label cluster1 auth admin:changeme
     SPLUNK_CMD_2: splunk restart

Hello,

I'm trying to get run a docker container but I'm getting this: Validating databases (splunkd validatedb) failed with code '1'
Facts:
a) command:
docker run
--name splunk --hostname=splunk
-p 8000:8000
-p 8088:8088
-p 9997:9997
-p 1514:1514
-p 1515:1515
-v /opt/splunk/etc:/opt/splunk/etc
-v /opt/splunk/var:/opt/splunk/var
-e "SPLUNK_START_ARGS=--accept-license --answer-yes"
splunk/splunk:latest

host /opt/splunk is owned by root. /opt/splunk/etc and /opt/splunk/var owned by user 999. If I change the permissions on the host for etc and var to root or any other user, it goes little further but I have permission errors on writting the logs.
Of course since the docker container is not running the only way to get inside the container is by 'docker run -ti --entrypoint /bin/bash splunk/splunk' which I can see that inside the var and the etc are owned by root but to be honest I don't really trust this.
So any idea? is it really permissions? Is it possible to change the userid to something different?

PS. the docker command is run as root.

Hi there,

I'm new to Splunk and the OPS team gave me the following files under the /opt/splunkforwarder/etc/system/local/:

$ ls -la /opt/splunkforwarder/etc/system/local/
total 16
drwx------ 2 root root  74 Jan  9 00:29 .
drwx------ 3 root root  19 Jan  8 22:29 ..
-rw------- 1 root root 171 Jan  8 22:29 inputs.conf
-rwx------ 1 root root 195 Jan  8 22:29 outputs.conf
-r-------- 1 root root 265 Jan  8 22:29 README
-rw------- 1 root root 375 Jan  8 22:29 server.conf

According to http://blogs.splunk.com/2015/08/24/collecting-docker-logs-and-stats-with-splunk/, I'd like to configure a single SplunkForwarder container to collect syslog for all the containers... I'd like also to disclose:

• The company below is a BIG Splunk customer, but so far nobody from OPS team supports Docker
• We are still on Splunk 6.2.x and so, we cannot use the Docker Native Driver http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/

This is a host installation with the splunk-forwarder 6.2.x... The content of the files is as follows:

inputs.conf

[root@pe2enpmas300 npmo-server]# cat inputs.conf
[default]
host = pe2enpmas300.corp.company.net

[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
_blacklist = \.(gz)$
index= sp-njsnginx-reference-e2eidx

outputs.conf

[root@pe2enpmas300 npmo-server]# cat outputs.conf
[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = oe2esstlg310.corp.company.net:9997, oe2esstlg311.corp.company.net:9997, oe2esstlg312.corp.company.net:9997
autoLB = true

server.conf

[root@pe2enpmas300 npmo-server]# cat server.conf
[sslConfig]
sslKeysfilePassword = $1$Of8JPJZlRRS2

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$brNdYNMjDka2
serverName = pe2enpmas300.corp.company.net

Questions

• Can I just mount the settings in the Forwarder server?
• How can I debug this?

For instance, I just copied the files from the splunk forward from the host and I'm mounting in the data container... Let's start with the container in a separate docker-compose...

docker-compose-monitoring.yml

Since I'm running docker as root (with SSL enabled), then I opened port 514....

splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  environment:
    - SPLUNK_FORWARD_SERVER="oe2esstlg310.corp.company.net:9997,oe2esstlg311.corp.company.net:9997,oe2esstlg312.corp.company.net:9997"
  ports:
    - 514:514/udp

docker-compose.yml

I just use extensions https://docs.docker.com/compose/extends/ and the ${HOSTNAME} variable substitution https://docs.docker.com/compose/compose-file/#variable-substitution

$ echo $HOSTNAME
pe2enpmas300.corp.company.net

For the configuration, I'm mounting it...

[root@pe2enpmas300 npmo-server]# ls -la monitor/splunk/
total 20
drwx------ 2 polkitd ssh_keys  95 Jan  9 01:00 .
drwx------ 3 root    root      19 Jan  9 01:40 ..
-rw------- 1 polkitd ssh_keys 171 Jan  8 22:29 inputs.conf
-rw------- 1 polkitd ssh_keys  45 Jan  9 01:00 migration.conf
-rwx------ 1 polkitd ssh_keys 195 Jan  8 22:29 outputs.conf
-r-------- 1 polkitd ssh_keys 265 Jan  9 00:51 README
-rw------- 1 polkitd ssh_keys 375 Jan  8 22:29 server.conf

Here's the file Dockerfile

splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local

splunkforwarder:
  extends:
    file: docker-compose-monitoring.yml
    service: splunkforwarder
  volumes_from:
    - "splunkforwarderData"

newww:
  build: roles/newww
  restart: always
  env_file: .env
  expose:
    - "5005"
  ports:
    - "80:8081"
  log_driver: "syslog"
  log_opt:
    syslog-tag: "newww"
    syslog-address: udp://${HOSTNAME}

Docker inspect command shows the mounted settings...

$ docker compose 
    "Mounts": [
        {
            "Source": "/npmo-data/npmo-server/monitor/splunk",
            "Destination": "/opt/splunk/etc/system/local",
            "Mode": "rw",
            "RW": true
        }
    ],

I verified that the file is in the container as well...

root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash

root@bb3bb53aba78:/opt/splunk# ls -la
total 96
drwxr-xr-x  9 splunk splunk  4096 Jan  9 09:53 .
drwxr-xr-x  3 root   root      19 Oct 28 14:28 ..
drwxr-xr-x  3 splunk splunk  4096 Jun 26  2015 bin
-r--r--r--  1 splunk splunk    57 Jun 26  2015 copyright.txt
drwxr-xr-x 13 splunk splunk  4096 Jan  9 08:52 etc
drwxr-xr-x  2 splunk splunk    26 Jun 26  2015 include
drwxr-xr-x  4 splunk splunk  4096 Jun 26  2015 lib
-r--r--r--  1 splunk splunk 52503 Jun 26  2015 license-eula.txt
drwxr-xr-x  3 splunk splunk    55 Jun 26  2015 openssl
-r--r--r--  1 splunk splunk   842 Jun 26  2015 README-splunk.txt
drwxr-xr-x  3 splunk splunk    39 Jun 26  2015 share
-r--r--r--  1 splunk splunk 17634 Jun 26  2015 splunkforwarder-6.2.4-271043-linux-2.6-x86_64-manifest
drwxr-xr-x  6 splunk splunk    48 Jan  9 08:51 var

root@bb3bb53aba78:/opt/splunk# ls -la etc/system/local/
total 20
drwx------ 2 splunk splunk  95 Jan  9 09:00 .
drwxr-xr-x 7 splunk splunk  73 Jan  9 08:51 ..
-rw------- 1 splunk splunk 171 Jan  9 06:29 inputs.conf
-rw------- 1 splunk splunk  45 Jan  9 09:00 migration.conf
-rwx------ 1 splunk splunk 195 Jan  9 06:29 outputs.conf
-r-------- 1 splunk splunk 265 Jan  9 08:51 README
-rw------- 1 splunk splunk 375 Jan  9 06:29 server.conf

Any help is appreciated...

I have been fighting very weird errors trying to get splunk working inside a kubernetes cluster.

Setup:

• Storage: nfs
• Image-version: 6.6.2
• kubernetes: 1.7.1 on Ubuntu 16.04.2 LTS

Mounting /opt/splunk/etc and /opt/splunk/var always gave me errors like https://answers.splunk.com/answers/312247/after-upgrading-a-search-head-cluster-to-splunk-63-1.html. Some of the resources it tried to get was also showing __raw/..../undefined/... where the undefined part was ment to show the username (admin). Lots of small things didn't work.

After a lot of trial and errors, I got it to work with mounting in separate directories under /opt/splunk/var, like spool and run.. It was a lot of trial and errors. But now, splunk gave me errors like ERROR while running renew-certs migration. and Warning: cannot create "/opt/splunk/var/run/splunk" when kubernetes recreated it.

What seams to work is this:

• Run with SPLUNK_USER set to root
• Mount in /opt/splunk/etc, /opt/splunk/var/lib, and /opt/splunk/var/log on their own.
  • /opt/splunk/var/log for good measure...
  • Mounting /opt/splunk/var will give the errors above, even if run as root.

The splunk container process exits immediately when running the steps specified in the README. I'm seeing the issue with 6.3.3 and 6.3.1. Below are the logs

$ docker logs splunk
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 1024 bit long modulus
.......................................++++++
............++++++
e is 65537 (0x10001)
writing RSA key

Generating RSA private key, 1024 bit long modulus
...................++++++
.........++++++
e is 65537 (0x10001)
writing RSA key

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

An unforeseen error occurred:

    Exception: <type 'exceptions.OSError'>, Value: [Errno 39] Directory not empty: '/opt/splunk/share/splunk/search_mrsparkle/modules.new/converters'

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1150, in main
    parseAndRun(argsList)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 997, in parseAndRun
    retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call
    return self.func(args, fromCLI)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFunc
    return func(dictCopy, fromCLI)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/_internal.py", line 160, in firstTimeRun
    comm.moveItem(migration.PATH_UI_MOD_NEW, migration.PATH_UI_MOD_ACTIVE)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 856, in moveItem
    shutil.move(src, dst)
  File "/opt/splunk/lib/python2.7/shutil.py", line 300, in move
    rmtree(src)
  File "/opt/splunk/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/opt/splunk/lib/python2.7/shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/splunk/lib/python2.7/shutil.py", line 254, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/opt/splunk/share/splunk/search_mrsparkle/modules.new/converters'


Please file a case online at http://www.splunk.com/page/submit_issue


This appears to be your first time running this version of Splunk.

I am getting a low disk space warning (Under 5GB free) after a few weeks of use and it has caused Splunk to stop indexing and not allow users to use the web interface. Moving the Splunk setting to 2GB restored functionality, but is obviously a temporary solution.

I am using data volume containers as suggested in the readme.

When I run docker ps -s I get:
14.08 MB (virtual 546.3 MB)

Is there another way I should be checking disk usage?

Any ideas on how to fix this or what may be happening?

Thanks.

Hello world,

I have been trying to load the license file by copying over into the /opt/splunk/etc/licenses/enterprise/ folder and no joy.

I have also set a volume mount to the file which works in other circumstances - just not with this license file.

volumes:
      - ./splunkmaster/volume/etc:/opt/splunk/etc:Z
      - ./splunkmaster/volume/var:/opt/splunk/var:Z
      - ./splunksearch/volume/license/Splunk.License:/opt/splunk/etc/licenses/enterprise/Splunk.License:Z

Anyone care to shed some light?

Thanks..

Is docker-splunk no longer listed on Docker Hub?

The branch 7.0.1 is not exposed to the Docker Repository. Wondering how close we are to have it available ?

From https://github.com/splunk/docker-splunk/blob/master/enterprise/README.md:

docker run --name splunk --hostname splunk -p 8000:8000 -d -e "SPLUNK_START_ARGS=--accept-license"

This command is missing an image name and clearly won't work. If you add an image name, it's still broken. Repro:

$ docker run --name splunk --hostname splunk -p 8000:8000 -d -e "SPLUNK_START_ARGS=--accept-license"
"docker run" requires at least 1 argument(s).
See 'docker run --help'.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container
$ docker run --name splunk --hostname splunk -p 8000:8000 -d -e "SPLUNK_START_ARGS=--accept-license" splunk/splunk
2d45302f44756753383c8a7953ef8b05c69f089dc81d1501939fe2112760e122

working on PR

I've tried to change the default username with the "command" in docker-compose.yml but it doesn't seem to work. Does anyone know how to do this?

Hi,
I have these env variables to suit our environment:

SPLUNK_START_ARGS="--accept-license"
SPLUNK_FORWARD_SERVER=splunk_server:9997
SPLUNK_USER=root
SPLUNK_ADD_1='monitor /var/log/containers -sourcetype docker_json'

After the container starts this is what the /opt/splunk/etc/system/local/outputs.conf file looks like:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk_server:9997

[tcpout-server://splunk_server:9997]

But I need to make changes for our environment like this otherwise the forwarder doesn't connect properly to the indexer:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk_server:9997

[tcpout-server://splunk_server:9997]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslVerifyServerCert = false
sslPassword = some_password

How could I achieve this?

I don't think it's possible to configure sslCertPath, sslRootCAPath etc using the Splunk CLI therefore I cannot use the SPLUNK_CMD env variable.

I thought I could override ENTRYPOINT and CMD something like this:

ENTRYPOINT ["/bin/bash -c"]
CMD ["mkdir -p /opt/splunk/etc/system/local; echo '[tcpout]' > /opt/splunk/etc/system/local/outputs.conf; echo 'sslCertPath = /opt/splunk/etc/auth/server.pem' >> /opt/splunk/etc/system/local/outputs.conf; echo 'sslRootCAPath = /opt/splunk/etc/auth/cacert.pem' >> /opt/splunk/etc/system/local/outputs.conf; echo 'sslVerifyServerCert = false' >> /opt/splunk/etc/system/local/outputs.conf; /sbin/entrypoint.sh start-service"]

But this is really hacky and also I don't know how to find the sslPassword parameter because it's created randomly in /opt/splunk/etc/system/local/server.conf after splunk has started.

Am I missing something? Is there a simple way to do this? I'd rather not have to create my own custom Docker image.

Thanks,
Max

Hi,

After a long time of usage without any problem, I finally restarted the running container (after modifying the host hardware), now the image doesn't want to start any more on my previous volume.

btool.log shows the following

08-29-2016 09:18:07.764 ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
08-29-2016 09:18:07.766 ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
08-29-2016 09:18:07.766 ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
08-29-2016 09:18:07.766 ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini
08-29-2016 09:18:07.773 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
08-29-2016 09:18:07.773 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
08-29-2016 09:18:07.773 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_management_console/metadata/local.meta: Permission denied
08-29-2016 09:18:07.773 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/user-prefs/metadata/local.meta: Permission denied
08-29-2016 09:18:07.776 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
08-29-2016 09:18:07.776 ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied

The user rights of the folder look like the following (in the host, then user are not displayed as splunk:splunk) :

I tried to boot the image on a new volume, it seems the problem doesn't occur.

Do you have any idea how I can fix that ?

Thanks !

Hi there. We're having some challenges using this image since once the data volumes are created and configured, there is no way to modify the configs with future updates.

Say, for example, we want to add a new forward server, app, etc. I could update the ENV variables to accomplish this when first starting a container. But if the data volumes already exist, I have to either completely delete them or log into the container and run commands manually.

It might be nice to have an ENV variable to run splunk commands even if the server is already configured? Or perhaps a flag to re-run configuration regardless of what it finds in the volume (assuming that Splunk properly handles the case of identical commands being re-run)?

Thanks!

As a cloud foundry operator, I want to use Splunk to ingest my rsyslog platform logs. Therefore, I would like the Splunk <version>-monitor docker image to come pre-packed with the RFC5424 Syslog application by default.

The *-monitor image comes packed with the Docker Monitoring Dockerfile goodie, and it would be very helpful for day-to-day operations if the RFC5424 Syslog were packed as well. (

(Unfortunately, the provided syslog source format does not seem to parse logs from cloud foundry's rsyslog properly, whereas the RFC5424 Syslog does.)

Thank you for your consideration.

Is it possible to run ES in a docker container? I've tried and it breaks the container

I bit confused , daemonset was running fine for splunk UF till yesterday but now it logs are showing:
Please enter a new password

How to reproduce this:

• In the main directory, run docker build . -t

It looks like the path to the file is wrong. No big deal, but thought you'd like to know.

-- Doug

Error installing with docker-compose up

git clone https://github.com/splunk/docker-splunk.git
Cloning into 'docker-splunk'...
remote: Counting objects: 817, done.
remote: Total 817 (delta 0), reused 0 (delta 0), pack-reused 817
Receiving objects: 100% (817/817), 156.03 KiB | 0 bytes/s, done.
Resolving deltas: 100% (393/393), done.
[ec2-user@ip-10-193-206-13 SPLUNK]$ ls
docker-splunk
[ec2-user@ip-10-193-206-13 SPLUNK]$ cd docker-splunk/
[ec2-user@ip-10-193-206-13 docker-splunk]$ ls
CONTRIBUTING.md enterprise LICENSE README.md universalforwarder
[ec2-user@ip-10-193-206-13 docker-splunk]$ cd enterprise/
[ec2-user@ip-10-193-206-13 enterprise]$ ls
build.sh docker-compose.yml Dockerfile entrypoint.sh publishImage.sh README.md
[ec2-user@ip-10-193-206-13 enterprise]$ vim docker-compose.yml
[ec2-user@ip-10-193-206-13 enterprise]$ docker-compose up
Creating network "enterprise_default" with the default driver
Creating volume "enterprise_opt-splunk-etc" with default driver
Creating volume "enterprise_opt-splunk-var" with default driver
Pulling splunkenterprise (splunk/splunk:7.0.0)...
7.0.0: Pulling from splunk/splunk
Digest: sha256:216f8511d99b7e79ac147cf49829b92ad49a92ec8de35baf5beaf1bb50d9316c
Status: Downloaded newer image for splunk/splunk:7.0.0
Creating enterprise_splunkenterprise_1 ...
Creating enterprise_splunkenterprise_1 ... done
Attaching to enterprise_splunkenterprise_1
splunkenterprise_1 | Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
splunkenterprise_1 | Generating RSA private key, 2048 bit long modulus
splunkenterprise_1 | .........................+++
splunkenterprise_1 | .......................+++
splunkenterprise_1 | e is 65537 (0x10001)
splunkenterprise_1 | writing RSA key
splunkenterprise_1 |
splunkenterprise_1 | Generating RSA private key, 2048 bit long modulus
splunkenterprise_1 | .....................................................+++
splunkenterprise_1 | .....................+++
splunkenterprise_1 | e is 65537 (0x10001)
splunkenterprise_1 | writing RSA key
splunkenterprise_1 |
splunkenterprise_1 | Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
splunkenterprise_1 |
splunkenterprise_1 | An unforeseen error occurred:
splunkenterprise_1 |
splunkenterprise_1 | Exception: <type 'exceptions.OSError'>, Value: [Errno 39] Directory not empty: '/opt/splunk/share/splunk/search_mrsparkle/modules.new/converters'
splunkenterprise_1 |
splunkenterprise_1 | Traceback (most recent call last):
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1143, in main
splunkenterprise_1 | parseAndRun(argsList)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 998, in parseAndRun
splunkenterprise_1 | retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call
splunkenterprise_1 | return self.func(args, fromCLI)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFunc
splunkenterprise_1 | return func(dictCopy, fromCLI)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/_internal.py", line 176, in firstTimeRun
splunkenterprise_1 | comm.moveItem(migration.PATH_UI_MOD_NEW, migration.PATH_UI_MOD_ACTIVE)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 1017, in moveItem
splunkenterprise_1 | shutil.move(src, dst)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/shutil.py", line 300, in move
splunkenterprise_1 | rmtree(src)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/shutil.py", line 247, in rmtree
splunkenterprise_1 | rmtree(fullname, ignore_errors, onerror)
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/shutil.py", line 256, in rmtree
splunkenterprise_1 | onerror(os.rmdir, path, sys.exc_info())
splunkenterprise_1 | File "/opt/splunk/lib/python2.7/shutil.py", line 254, in rmtree
splunkenterprise_1 | os.rmdir(path)
splunkenterprise_1 | OSError: [Errno 39] Directory not empty: '/opt/splunk/share/splunk/search_mrsparkle/modules.new/converters'
splunkenterprise_1 |
splunkenterprise_1 |
splunkenterprise_1 | Please file a case online at http://www.splunk.com/page/submit_issue
splunkenterprise_1 |
splunkenterprise_1 |
splunkenterprise_1 | This appears to be your first time running this version of Splunk.
enterprise_splunkenterprise_1 exited with code 2
[ec2-user@ip-10-193-206-13 enterprise]$

As an admin, I would like to manage my Splunk deployments on Kubernetes via helm chart(s).

I am currently trying to write a helm chart for deploying a standalone heavy forwarder, but I am very new to k8, helm, and Splunk in general. Therefore, I'm concerned that what I write will be ... well, crap.

Are there any plans for the Splunk team to offer easily consumable helm chart(s) for maintaining Splunk clusters?

I can see charts for different use cases, such as:

• chart for standalone heavy forwarder / local indexer (like I am kind of writing)
• chart for cluster of headless forwarders
• chart for cluster of indexers
• etc.

I realize this repository is mostly for holding the base images / basic config for Splunk on docker images, so forgive me if the scope of this question is too broad

EDIT, I now see there's helm deployments in-progress. Just not my exact use-case: https://github.com/splunk/splunk-connect-for-kubernetes/tree/master/helm-chart/splunk-kubernetes-logging

The entrypoint has a lot of configuration features based on authenticating as admin using -auth admin:changeme

• SPLUNK_ENABLE_DEPLOY_SERVER
• SPLUNK_DEPLOYMENT_SERVER
• SPLUNK_ENABLE_LISTEN
• SPLUNK_FORWARD_SERVER
• SPLUNK_FORWARD_SERVER_*
• SPLUNK_ADD
• SPLUNK_ADD_*

In the 7.1.0 docker image, unlike 7.0.x, the admin password is set at the begining of first time configuration (rather than via the webpage after the first time configuration is done), before these configurations steps run. So the password has already been changed from changeme and none of the splunk configurations work.

Temporary workarround

Set your command to (for example, in a docker-compose.yml file

command: |
  bash -c "
    if [ -e /opt/splunk/etc/str ]; then
      rm -f /opt/splunk/ftr
      exec /sbin/entrypoint.sh start-service
    else
      touch /opt/splunk/etc/str
      exec /sbin/entrypoint.sh start-service --seed-passwd changeme
    fi
  "

At least this way, the password is changeme, and everything works. Only downside to this is the webpage no longer tells you to change the password after logging in, like it did in 7.0.x

Splunk universal forwarder fails to do Deployment server setup step randomly. as it able to find splunk.version file created and it is assuming configuration is already done and is not setting splunk_deployment_server.

this is occurs randomly and does not have certain prediction. probably due to race condition in the entrypoin.sh script.

I think there is a typo in README.md at current master:

8088/tcp is mentioned 2x, but i guess the second 8088/tcp is supposed to be 8089/tcp - correct?

Steps to reproduce:

1. Start a splunk/splunk:7.0.0 container using host-mounted volumes for /opt/splunk/etc as described in the documentation (docker run -p 8000:8000 -d -e "SPLUNK_START_ARGS=--accept-license" v /local/path/optsplunketc:/opt/splunk/etc -v /local/path/optsplunkvar:/opt/splunk/var splunk/splunk:7.0.0)
2. Once the container starts successfully stop it.
3. Start another container from the same splunk/splunk:7.0.0 image ( docker run -p 8000:8000 -d -e "SPLUNK_START_ARGS=--accept-license" v /local/path/optsplunketc:/opt/splunk/etc -v /local/path/optsplunkvar:/opt/splunk/var splunk/splunk:7.0.0) This container will fail with exit code 1 and the logs will read:

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)
Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.
You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:
If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.
tcgetattr: Inappropriate ioctl for device
WARNING: error changing terminal modes - password will echo!
Perform migration and upgrade without previewing configuration changes? [y/n]

If you add --answer-yes to SPLUNK_START_ARGS this container will start but it still goes through the "update" process. I'm not sure if this is intended or unintended behavior but I feel like it should at least be documented because it's not clear from the documentation that starting a new container with those volumes mounted will not work. This is an important use case for persisting settings for a server and one of the big benefits of running splunk in docker.

I have followed all the steps mentioned and it shows that the daemonset is running on each of the nodes, However, when I see the Splunk Dashboard, I see no logs. Any pointers?

How to debug deamonset functionality?

I'm not sure if this version is still supported but I found 3 app files are failing file integrity check and yield different sha256 hashes in the docker image vs. direct tar install. The hashes in the manifest file are consistent with the direct tar installation but not with the files from docker. I can suppress the errors for now or edit the manifest, but both of those solutions are not ideal since I don't want to bake these into my cluster management scripts. I've checked on different machines and environments and arrived at the same result (some local, some staging and some live). Thanks!

Docker version:
docker version 18.03.1-ce

Steps to reproduce:
docker pull splunk/splunk:7.1.0
docker run -d -it splunk/splunk:7.1.0
docker exec -it <container_name> bash
./bin/splunk validate files

results in:

File '/opt/splunk/share/splunk/migration/app_contents_SplunkDeploymentMonitor.tar.gz' changed.
File '/opt/splunk/share/splunk/migration/app_contents_unix.tar.gz' changed.
File '/opt/splunk/share/splunk/migration/app_contents_windows.tar.gz' changed.

Compared with the tar.gz downloaded hashes they are different.

run sha256sum against these files:
sha256sum /opt/splunk/share/splunk/migration/app_contents_*

output:
b3f57820ec6af9c62d6685a6a7a7a2ff7f039be2712c04c1f190785afc34fdc4 /opt/splunk/share/splunk/migration/app_contents_SplunkDeploymentMonitor.tar.gz
75728e3fa3b43e7c9214f36df7cb483079d5d14511b754fd14b439bf0d1ad3bd /opt/splunk/share/splunk/migration/app_contents_unix.tar.gz
b141a423f3b7822673465776596fc8278c12e793b6b1f108045b063c975c130f /opt/splunk/share/splunk/migration/app_contents_windows.tar.gz

To grep the manifest file for the hashes it was expecting for these files (run from $SPLUNK_HOME):
grep "app_contents_" splunk-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest

which results in:

f 444 splunk splunk splunk/share/splunk/migration/app_contents_SplunkDeploymentMonitor.tar.gz 3478cfae2593f6be92fc084f2d195c27be13e11441d4118116e27010a2a041d5
f 444 splunk splunk splunk/share/splunk/migration/app_contents_unix.tar.gz 98cc648a8a0c6901f7d3bb585e8597f410df628ee81e1c65082c63195794e283
f 444 splunk splunk splunk/share/splunk/migration/app_contents_windows.tar.gz 2ae56598076bee59f46823ae3957eb2f422be83976774493423684ab7281dd3e

The readme mentions

Start a Splunk Enterprise container and mount the necessary container volumes

docker run --name vsplunk -v /opt/splunk/etc -v /opt/splunk/var busybox
docker run --hostname splunk --name splunk --volumes-from=vsplunk -p 8000:8000 -d -e "SPLUNK_START_ARGS=--a

Should this be updated to use named volumes instead (as of Docker 1.9.0)?

Using ConfigMap to manage splunk universal forwarder's config has been working well until the release of 1.9.4, which included this security fix.

ConfigMap now gets mounted as read-only always, which breaks this container due to the chown commands in entrypoint.sh.

See: https://answers.splunk.com/answers/626964/kubernetes-194-breaking-changes-universal-forwarde.html

The current URI in both the Forwarder and Instance Dockerfile need to be updated.

Recently the product name has been removed from the URI.

Hello,

I saw that the tag latest is not pulling the 7.1.0 but the 7.0.3 as well that 7.1.0 hasn't pushed to the master branch.
Is there any particular reason for holding the push?

How to stop auto upgrade on splunk docker?

Hi, We would like to add _meta as a command line argument with splunk forwarder cli which is at present not supported with cli, Can you add this as a feature request and also if any substitute is there please do let us know

Following the instructions here under the heading Start a Splunk Enterprise container and mount the necessary container volumes I get the following error:

Docker version: 18.03.0-ce-win59 (16762)
Windows 10 version: 1709

Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.

An unforeseen error occurred:

        Exception: <type 'exceptions.OSError'>, Value: [Errno 1] Operation not permitted: '/opt/splunk/etc/openldap/ldap.conf'

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 1143, in main
    parseAndRun(argsList)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 998, in parseAndRun
    retVal = cList.getCmd(command, subCmd).call(argList, fromCLI = True)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 280, in call
    return self.func(args, fromCLI)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/control_api.py", line 30, in wrapperFunc
    return func(dictCopy, fromCLI)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/_internal.py", line 128, in firstTimeRun
    comm.copyItem(migration.PATH_LDAP_CONF_DEF, migration.PATH_LDAP_CONF)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 1008, in copyItem
    shutil.copy(src, dst)
  File "/opt/splunk/lib/python2.7/shutil.py", line 120, in copy
    copymode(src, dst)
  File "/opt/splunk/lib/python2.7/shutil.py", line 91, in copymode
    os.chmod(dst, mode)
OSError: [Errno 1] Operation not permitted: '/opt/splunk/etc/openldap/ldap.conf'


Please file a case online at http://www.splunk.com/page/submit_issue


This appears to be your first time running this version of Splunk.

Deploying Splunk UF in a Docker SWARM mode will not work if it is trying to resolve the Splunk Enterprise instance by service name.

The reason is that dnsutils must be added to the base image, apt-get install -y dnsutils.

Currently, in entrypoint.sh for Splunk Universal Forwarder, under start-service, it is using "admin:changeme" credential for a few commands, in which is hardcoded. This either needs to be configurable or removed (don't require authentication for such commands)

There doesn't seem to be inputs/options described for that.
Can we do it?
Thanks
--Duncan

Hello, I'm moving from running the Splunk forwarder from an installed debian package to running in a container. We have configured the forwarder to monitor various files on disk, I am planning to mount those directories in my container, and I was wondering about the following discrepancy.

I see the Splunk .deb postinst script (from the Splunk forwarder 6.3.3) has something like

if [ ! -f "$SPLUNK_HOME/etc/splunk-launch.conf" ] ; then
    sed "s%# SPLUNK_HOME=.*%SPLUNK_HOME=$SPLUNK_HOME%g" "$SPLUNK_HOME/etc/splunk-launch.conf.default" > "$SPLUNK_HOME/etc/splunk-launch.conf"
fi

We aren't explicitly configuring the splunk-launch.conf, and the default suits our needs after we drop a few files in /etc/system/local/. This splunk-launch.conf isn't in the Dockerfile, and now I'm worried that I'm missing configuring a few other steps and environment variables.

How would you best approach this? Thanks.

Hi there,

I'm trying to use Docker-compose V2 on Mac Beta and there might be something with the FS...

Docker-compose.yml

version: "2"

services:
  splunk:
    image: outcoldman/splunk:6.4
    hostname: splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
    ports:
      - 8000:8000
    volumes:
      - ./data/etc:/opt/splunk/etc
      - ./data/var:/opt/splunk/var

Error

mdesales@Marcello-New2015 [05/06/201620:39:15] ~/dev/github/intuit/servicesplatform-tools/microservices/logging $ docker-compose up
Starting logging_splunk_1
Attaching to logging_splunk_1
splunk_1  |
splunk_1  | Splunk> Another one.
splunk_1  |
splunk_1  | Checking prerequisites...
splunk_1  |     Checking http port [8000]: open
splunk_1  |     Checking mgmt port [8089]: open
splunk_1  |     Checking appserver port [127.0.0.1:8065]: open
splunk_1  |     Checking kvstore port [8191]:   Checking configuration...  Done.
splunk_1  | homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.
splunk_1  |     Checking critical directories...    Done
splunk_1  |     Checking indexes...
splunk_1  | Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
splunk_1  | open
splunk_1  |         Creating: /opt/splunk/var/lib/splunk
splunk_1  |         Creating: /opt/splunk/var/run/splunk
splunk_1  |         Creating: /opt/splunk/var/run/splunk/appserver/i18n
splunk_1  |         Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
splunk_1  |         Creating: /opt/splunk/var/run/splunk/upload
splunk_1  |         Creating: /opt/splunk/var/spool/splunk
splunk_1  |         Creating: /opt/splunk/var/spool/dirmoncache
splunk_1  |         Creating: /opt/splunk/var/lib/splunk/authDb
splunk_1  |         Creating: /opt/splunk/var/lib/splunk/hashDb
logging_splunk_1 exited with code 10

Running without Data Works

It just worked as expected without the volumes.

Recreating logging_splunk_1
Attaching to logging_splunk_1
splunk_1  | Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
splunk_1  | Generating RSA private key, 1024 bit long modulus
splunk_1  | ................................................................++++++
splunk_1  | ...............................++++++
splunk_1  | e is 65537 (0x10001)
splunk_1  | writing RSA key
splunk_1  |
splunk_1  | Generating RSA private key, 1024 bit long modulus
splunk_1  | ..........................................++++++
splunk_1  | ......++++++
splunk_1  | e is 65537 (0x10001)
splunk_1  | writing RSA key
splunk_1  |
splunk_1  | Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
splunk_1  |
splunk_1  | This appears to be your first time running this version of Splunk.
splunk_1  |
splunk_1  | Splunk> Another one.
splunk_1  |
splunk_1  | Checking prerequisites...
splunk_1  |     Checking http port [8000]: open
splunk_1  |     Checking mgmt port [8089]: open
splunk_1  |     Checking appserver port [127.0.0.1:8065]: open
splunk_1  |     Checking kvstore port [8191]:   Checking configuration...  Done.
splunk_1  |     Checking critical directories...    Done
splunk_1  |     Checking indexes...
splunk_1  |         Validated: _audit _internal _introspection _thefishbucket history main summary
splunk_1  |     Done
splunk_1  | New certs have been generated in '/opt/splunk/etc/auth'.
splunk_1  | open
splunk_1  |         Creating: /opt/splunk/var/lib/splunk
splunk_1  |         Creating: /opt/splunk/var/run/splunk
splunk_1  |         Creating: /opt/splunk/var/run/splunk/appserver/i18n
splunk_1  |         Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
splunk_1  |         Creating: /opt/splunk/var/run/splunk/upload
splunk_1  |         Creating: /opt/splunk/var/spool/splunk
splunk_1  |         Creating: /opt/splunk/var/spool/dirmoncache
splunk_1  |         Creating: /opt/splunk/var/lib/splunk/authDb
splunk_1  |         Creating: /opt/splunk/var/lib/splunk/hashDb
splunk_1  |     Checking filesystem compatibility...  Done
splunk_1  |     Checking conf files for problems...
splunk_1  |     Done
splunk_1  |     Checking default conf files for edits...
splunk_1  |     Validating installed files against hashes from '/opt/splunk/splunk-6.4.0-f2c836328108-linux-2.6-x86_64-manifest'
splunk_1  | Generating a 1024 bit RSA private key
splunk_1  | .......++++++
splunk_1  | ....................................................++++++
splunk_1  | writing new private key to 'privKeySecure.pem'
splunk_1  | -----
splunk_1  | Signature ok
splunk_1  | subject=/CN=splunk/O=SplunkUser
splunk_1  | Getting CA Private Key
splunk_1  | writing RSA key
splunk_1  |     All installed files intact.
splunk_1  |     Done
splunk_1  | All preliminary checks passed.
splunk_1  |
splunk_1  | Starting splunk server daemon (splunkd)...
splunk_1  | Done
splunk_1  |
splunk_1  |
splunk_1  | Waiting for web server at http://127.0.0.1:8000 to be available.. Done
splunk_1  |
splunk_1  |
splunk_1  | If you get stuck, we're here to help.
splunk_1  | Look for answers here: http://docs.splunk.com
splunk_1  |
splunk_1  | The Splunk web interface is at http://splunk:8000
splunk_1  |
splunk_1  | Deployment Server is enabled.
splunk_1  | Stopping splunkd...
splunk_1  | Shutting down.  Please wait, as this may take a few minutes.
splunk_1  | ..
splunk_1  | Stopping splunk helpers...
splunk_1  |
splunk_1  | Done.
splunk_1  |
splunk_1  | Splunk> Another one.
splunk_1  |
splunk_1  | Checking prerequisites...
splunk_1  |     Checking http port [8000]: open
splunk_1  |     Checking mgmt port [8089]: open
splunk_1  |     Checking appserver port [127.0.0.1:8065]: open
splunk_1  |     Checking kvstore port [8191]:   Checking configuration...  Done.
splunk_1  |     Checking critical directories...    Done
splunk_1  |     Checking indexes...
splunk_1  |         Validated: _audit _internal _introspection _thefishbucket history main summary
splunk_1  |     Done
splunk_1  | open
splunk_1  |     Checking filesystem compatibility...  Done
splunk_1  |     Checking conf files for problems...
splunk_1  |     Done
splunk_1  |     Checking default conf files for edits...
splunk_1  |     Validating installed files against hashes from '/opt/splunk/splunk-6.4.0-f2c836328108-linux-2.6-x86_64-manifest'
splunk_1  |     All installed files intact.
splunk_1  |     Done
splunk_1  | All preliminary checks passed.
splunk_1  |
splunk_1  | Starting splunk server daemon (splunkd)...
splunk_1  | Done
splunk_1  |
splunk_1  |
splunk_1  |
splunk_1  | If you get stuck, we're here to help.
splunk_1  | Look for answers here: http://docs.splunk.com
splunk_1  |
splunk_1  | The Splunk web interface is at http://splunk:8000
splunk_1  |