Hello QS Team,

The EC2 resource 'SplunkSHCMember3' fails creation which eventually causes the entire stack to fail on creation. On reviewing system logs for the instance SplunkSHCMember3 it looks like last command in user-data execution, which I'm guessing is responsible to setting captaincy for the search head returns a 'non-zero' exit code.

Specific command:

sudo -u $SPLUNK_USER $SPLUNK_BIN bootstrap shcluster-captain -servers_list "https://10.0.1.38:8089,https://10.0.2.109:8089,https://$LOCALIP:8089" -auth admin:<password>

This causes cfn-signal to send a failure signal to the CloudFormation stack. Can you please replicate this issue and suggest if there something I can do to fix it ?

We would like to have the option of retaining the EBS volume when the indexer dies and when the new indexer comes up it can attach to the existing EBS volume which is available

Hi everyone,

Is there any reason for not deploying indexer and search head instances into a private subnets and only put the LoadBalancer into public ?

Sorry for my dummy question and thx for your answers.

I used the template to setup a POC splunk, but it seems that my licenses were not applied, and the index cluster was not configured. I modified the template so it did not assign public IP addresses to any of the instances. please let me know what information i can provide to help solve this issue.

When I change the AMI used in the template for us-west-2 to latest Splunk AMI (splunk_AMI_8.1.0_2020-10-19_17-02-46-7b65de6c-5006-4ca2-bd75-fdba95ae5d9d-ami-0841021191958485f.4 - ami-00ec39a01d30f499a), the userdata scripts fail since the new AMI installs splunk files and data after the launch while the old AMI had everything installed before userdata scripts executed

It is not currently found in the AMI RegionMap.

Hello,

I have modify the CF template to move all splunk instance private subnet with the exception of the Loadbalancers.
Therefor I modify the template and add 3 private subnet, NAT GW, Route Table.
The modify VPC template created successfully.
The modify Splunkt Template ran in to a timeout when deploying the Cluster Master Instance.

Troubleshooting:
Create a jumphost login to cluster-master and review the cloud-init-output.log.
Last message is
Error: Cannot specify both a WaitConditionHandle URL and a logical resource id
usermod --expiredate 1 splunk

Can somebody give me a hint where I should look and why the CM instance always fails.

regards

i am trying to install AWS app, AWS add on and Python for Scientific Computing (for Linux 64-bit) providing cloudformation template parameter SearchHeadApps as
https://splunkbase.splunk.com/app/2882/,https://splunkbase.splunk.com/app/1274/,https://splunkbase.splunk.com/app/1876/ and when i check the logs i get the error as
Downloading app https://splunkbase.splunk.com/app/1876/
--2018-02-12 19:00:41-- https://splunkbase.splunk.com/app/1876/
Resolving splunkbase.splunk.com (splunkbase.splunk.com)... 54.148.88.225, 35.161.95.55, 52.41.48.242
Connecting to splunkbase.splunk.com (splunkbase.splunk.com)|54.148.88.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/tmp/app2.spl’

/tmp/app2.spl [ <=> ] 87.23K --.-KB/s in 0.002s

2018-02-12 19:00:41 (41.2 MB/s) - ‘/tmp/app2.spl’ saved [89328]

Installing app...

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Extracting tarball failed

When two availability zones are selected, an error/rollback occurs when the third PublicSubnet3ID does not have a value.

Hello,
I'm trying to deploy quickstart template in a new VPC following this steps mentioned here in us-east-1 and i selected c5.4xlarge instance for Splunk launch on EC2.
https://aws-quickstart.s3.amazonaws.com/quickstart-splunk-enterprise/doc/splunk-enterprise-on-the-aws-cloud.pdf#%5B%7B%22num%22%3A25%2C%22gen%22%3A0%7D%2C%7B%22name%22%3A%22XYZ%22%7D%2C69%2C709%2C0%5D

I'm getting the below error in SplunkCM
API: ec2:RunInstances Not authorized for images: [ami-0484972f36720ea7f]

I don't see this ami in my account. In my AWS account, I launched splunk on ami-0a5b63581103ff2ae. Not sure why its picking the wrong ami.
Anyone else facing the same issue? Any pointers would be helpful!

Is there a version of this for a non-clustered environment?

I have attempted to remove the UserData from the SplunkCM resource, but the deployment then times out. Still troubleshooting this one.

What would we comment out if we didn't want to use ELB and didn't want to cluster indexers? (maybe also if just wanted to roll this out into one AZ as well?

When i deploy the Cloudformation template in a new VPC, the deployment fails and rollback complete message.
Please help to resolve this issue ASAP. Thanks.

After much messing around I have come to the conclusion that this QuickStart is broken for 7.1.

There are a number of issues which I started to fix, but my knowledge of Splunk is not good enough to offer pull request.

In summary the userdata scripts do not work with 7.1. If you are suffering the same issue, I would suggest running the March 2018 version of this script, or rolling your own.

Anyone wanting to carry on from where I left off....

/templates/splunk-enterprise.template Line 744 (and all others like it) are missing an escape char on the , should read this: (Although the issue does not seem to break anything)

"printf '%s\t%s\\n' \"$LOCALIP\" 'splunksearch' >> /etc/hosts\n",`

AND in the same file, to avoid the 'there is no admin user' issue, the following lines can be added after line 746 (and all other locations like it): (This will enable you to log in to your servers)

"printf '[user_info]\\nPASSWORD = changeme\\n' > $SPLUNK_HOME/etc/system/local/user-seed.conf\n"
"service splunk restart\n"

However, even after making these changes, while you can no log in, to the best of my knowledge, the clusters fail, and its far from working.

Good luck.

I'm seeing this error on a Splunk restart. I don't believe this option exists with newer versions of Splunk

Invalid key in stanza [tcpout:indexer_cluster_peers] in /opt/splunk/etc/apps/base-autogenerated/local/outputs.conf, line 11: autoLB (value: true).

There is an option if i want the search head clustering, but there is no option for the index clustering and forces me to select minimum 2 as the replication factor, it will be better if we can have similar to search head cluster

specifically from R4 and M4 series, via @amiracle

The stack fails likely due to disk volume issue using the CFN template. The first instance provisioned by the template is Splunk cluster-master, however, if the disk size specified is 14000GB and above the stack creations fails with error: 'Failed to receive 1 resource signal(s) within the specified duration'

This would likely be due to the IOPS allocation taking a bit longer to format and mount the volume which would be causing the cfn-signal to timeout and thus initiating roll-back of the Stack.

We would like to use the new C5 instances for running our splunk which needs Enhanced Network Capability in OS - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

Can you please build the splunk AMI from the base image which supports this

Thanks,
-srini

KV store is not replicating in SHC setup.
Ensure default port 8191 is open.

When I use the template to create a distributed deployment and enable Search Head clustering, I run into a couple of issues that seem related to the security groups setup for port 8000. Neither situation manifests itself when dealing with a single Search Head.

First, while my network CIDR (##.0.0.0/8) should have access to the Search Head ELB on port 8000 via the security group, it does not work - my connection times out. I can see that the ELB can see the SH instances, but I cannot seem to connect. I noticed that the same security group used for the ELB is used for the Search Head instances themselves and wondered if since I am not connecting with them directly, and hence ##.0.0.0/8 is not what the SH is seeing, if I added the VPC CIDR instead (I have used 172.31.0.0/16 or 10.0.0.0/16) if the traffic would pass, and in fact it did resolve the situation but it felt an awkward solution.

Second, when Search Head clustering is employed, you cannot use the Add Data function via the Search Heads. You must connect to any of the Indexers and import the data directly. However, neither of the security groups for the Indexers permitted access to port 8000. Once I added my network CIDR (##.0.0.0/8) and port 8000 to the security group, I was able to access the web interface and had access to the Add Data function.

I can't seem to find direct download links (in .spl format?) for installing add-ons via the CF template. What am I missing?

I am attempting to setup the environment without using (associating Public IP Addresses to the servers) and the cloud formation never completes. The lack of Public IP Addresses is a requirement within the private AWS environment. I am editing the associatepublicIP value from true to false.

Is it possible to use this cf with only private IP addresses/ no public IP addresses?