loggly / install-script Goto Github PK

View Code? Open in Web Editor NEW

28.0 28.0 59.0 693 KB

Loggly Sender Installation Script

Shell 69.83% Python 30.17%

install-script's People

Contributors

Stargazers

Watchers

Forkers

dirk varshneyjayant ericcholis directivegames davidschou subledger welll sirsgriffin indiereign bookmoreio psquickitprageet shweta-jain manav-shaw 3n32002 michaelog kristjanvalur jasiedu paulojblack amauryaops dreznicek glowdb kevinegami loriezn vinaykotla dangtransss jmpesp kiliankoller bbrk364 ppatdev doc27380 tawattantan gangathota akhilsurya55 sitedata webzcc soheylnorozi jenkinsgituser introtucetech michaelyt17 imperfectproduce sandhika kolleyvenkatesh robbied77 fightx sumank792 tirushv shakahl vipinraperia6537 alexis-to-semifir kongkonghao2022

install-script's Issues

LOG-1504: Configure-Linux.sh fails with Azure Linux VMs

The below portion of the script will automatically fail on Azure Linux VMs. Azure does not allow ping to be used by default.

I had to remove this check from the script to properly get the script to complete and have Loggly configured on the VM.

checks if all the various endpoints used for configuring loggly are accessible

checkIfLogglyServersAccessible()
{
echo "INFO: Checking if $LOGS_01_HOST is reachable."
if [ $(ping -c 1 $LOGS_01_HOST | grep "1 packets transmitted, 1 received, 0% packet loss" | wc -l) == 1 ]; then
echo "INFO: $LOGS_01_HOST is reachable."
else
logMsgToConfigSysLog "ERROR" "ERROR: $LOGS_01_HOST is not reachable. Please check your network and firewall settings."
exit 1

https://www.loggly.com/install/configure-file-monitoring.sh

I am trying to use the script available at https://www.loggly.com/install/configure-file-monitoring.sh but getting a message 10 times Did not find the test log message in Loggly's search yet. Waiting for 30 secs and after that, the script is getting terminated.

LOG-1503: Feature Request: Add tag support to configure-syslog.py

We use the configure-syslog.py script in conjunction with AWS VPC NAT instances. The NAT instances use CloudFormation init support to retrieve and execute the configure-syslog.py script.

Since we use this across multiple CloudFormation stacks, it would be very nice if the configure-syslog.py script were enhanced to support tags via a CLI switch, and update the syslog template config file accordingly.

Update Certificate Subdomain Check Fails For Some CURL Versions

The update-loggly-certificate.sh script uses a grep "200 OK" check to see if a subdomain exists. However, some versions of CURL have the following output instead:

HTTP/2 200
date: Fri, 06 Apr 2018 13:46:05 GMT
content-type: text/html; charset=utf-8
server: Apache/2.2.22 (Ubuntu)
content-security-policy: frame-ancestors 'none'
expires: Fri, 06 Apr 2018 13:46:05 GMT
vary: Cookie,Accept-Encoding
last-modified: Fri, 06 Apr 2018 13:46:05 GMT
x-loggly-page: login
cache-control: no-cache, no-store, must-revalidate, max-age=0
x-frame-options: DENY
x-loggly-request-uuid: APP453591de-1cee-4395-8bd9-e2fc03b7eb31
set-cookie: csrftoken=qULhsGspdY7UyVV6CsdQx5atdDoHHB5S; expires=Fri, 05-Apr-2019 13:46:05 GMT; Max-Age=31449600; Path=/
x-varnish: 426339233
age: 0
via: 1.1 varnish-v4
accept-ranges: bytes

Note that it simply states 200 rather than 200 OK.

The grep command could easily be modified to work with this version of CURL as well.

Notification Test #2

test

install script fails

running the install script creates the configuration file , but fails to validate a logged message

here's the output

[ec2-user@ip-172-31-14-149 ~]$ sudo bash configure-file-monitoring.sh -a PRIVATE -u PRIVATE -p SECRET -f /var/www/email-microservice/storage/logs/ -l laravel-log-directory
INFO: Downloading dependencies - configure-linux.sh
Loggly account or subdomain: x
Username is set
Directory to monitor: /var/www/email-microservice/storage/logs
File alias: laravel-log-directory
INFO: Initiating configure Loggly for file monitoring.
INFO: Operating system is Amazon AMI.
INFO: Checking if logs-01.loggly.com is reachable.
INFO: logs-01.loggly.com is reachable.
INFO: Checking if logs-01.loggly.com is reachable via 514 port. This may take some time.
INFO: logs-01.loggly.com is reachable via 514 port.
INFO: Checking if 'x' subdomain is valid.
INFO: https://x.loggly.com is valid and reachable.
INFO: Checking if Gen2 account.
INFO: It is a Gen2 account.
INFO: Checking if provided username and password is correct.
INFO: Username and password authorized successfully.
INFO: Authentication token not provided. Trying to retrieve it from https://x.loggly.com account.
INFO: Retrieved authentication token: 5a3a7753-addc-45bd-a4b2-869364a77228
INFO: Checking if provided auth token is correct.
INFO: Authentication token validated successfully.
INFO: rsyslog is present as service.
INFO: selinux status is not enforced.
INFO: There are 0 files in directory. Configuring each file for monitoring present in this directory.
There are 0 files present in this directory. Would you like to configure all the files (yes/no)?yes
INFO: Initiating Configure Loggly for Linux.
INFO: Loggly rsyslog file /etc/rsyslog.d/22-loggly.conf already exist.
INFO: Restarting the rsyslog service.
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
INFO: /var/spool/rsyslog already exist, so not creating directory.
Would you like install a Cron job to sync the files currently in your Directory every 5 minutes? (yes/no)
Please answer yes or no.
Would you like install a Cron job to sync the files currently in your Directory every 5 minutes? (yes/no)yes
INFO: Creating cron script /root/.loggly/file-monitoring-cron-laravel-log-directory.sh
INFO: Restarting the rsyslog service.
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
INFO: Search URL: https://x.loggly.com/apiv2/search?q=syslog.appName%3Alaravel-log-directory&from=-15m&until=now&size=1
INFO: Getting initial log count.
INFO: Verifying if the logs made it to Loggly.
INFO: Verification # 1 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 2 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 3 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 4 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 5 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 6 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 7 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 8 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 9 of total 10.
INFO: Did not find the test log message in Loggly's search yet. Waiting for 30 secs.
INFO: Done waiting. Verifying again.
INFO: Verification # 10 of total 10.
ERROR: File logs did not make to Loggly in time. Please check network and firewall settings and retry.
Manual instructions to configure a file is available at https://www.loggly.com/docs/file-monitoring/. Rsyslog troubleshooting instructions are available at https://www.loggly.com/docs/troubleshooting-rsyslog/

Error on configure script

The script shows an error when I try to execute any command:

root@web:~# python configure-syslog.py loggly_help

*************************************************************
****** Starting Loggly Syslog Configuration Script ******
*************************************************************

Traceback (most recent call last):
  File "configure-syslog.py", line 1479, in main
    log({"status":"start", "args": vars(options)})
  File "configure-syslog.py", line 1209, in log
    log_msg(json.dumps(msg_dict))
  File "configure-syslog.py", line 1237, in log_msg
    (LOGGLY_SYSLOG_SERVER, LOGGLY_SYSLOG_PORT))
error: [Errno 1] Operation not permitted

Configure script has errors

LOG-1502: Feature Request: Add UDP toggle flag

We use the configure-syslog.py script in conjunction with AWS VPC NAT instances. The NAT instances use CloudFormation init support to retrieve and execute the configure-syslog.py script.

We view the logs for the NAT instances as essentially "best effort" and don't want syslog to hang on the instances in the event our connection to Loggly fails. It would be wonderful if there was a CLI toggle flag to select UDP connections instead of TCP for the generated syslog config template.

LOG-1499: Passwords containing spaces result in fatal error

Issue

If an account password contains spaces, the login process will result in failure, preventing the script from correctly installing.

Expected

Action:
Authentication succeeds and user is able to progress to next step of installation.

Message:

INFO: Checking if provided auth token is correct.
INFO: Authentication token validated successfully.

Result

Action:
Login failure, script will not proceed with installation.

Message:

INFO: Checking if provided username and password is correct.
INFO: Please check your username or reset your password at https://<subdomain>.loggly.com/account/users/
ERROR: Invalid Loggly username or password. Your username is visible at the top right of the Loggly console before the @ symbol. You can reset your password at http://<subdomain>.loggly.com/login.
Manual instructions to configure rsyslog on Linux are available at https://www.loggly.com/docs/rsyslog-manual-configuration/. Rsyslog troubleshooting instructions are available at https://www.loggly.com/docs/troubleshooting-rsyslog/

This is not a recognized subdomain (configure-linux.sh)

Hi guys,
we are using the configure-linux.sh script since years but since yesterday we got the following error while execution:

INFO: Checking if 'xxx' subdomain is valid.
ERROR: This is not a recognized subdomain. Please ask the account owner for the subdomain they signed up with.

We didn't change anything from our side.

Could you please assist us?

LOG-1500: Linux install script requires password

Trying to get everything configured to send logs to loggly from within a docker container. It seems that you have to provide your password to the configure linux script for it to work. Is this correct? If so what is the purpose of the token?

I would personally prefer not to have my password stored in my Dockerfile and hosted on a github repo.

"WARN: The rsyslog-gnutls package could not be download automatically because your package manager could not be found. Please install it and restart the rsyslog service to send logs to Loggly."

The configure-linux.sh is stuck at (see attached screenshot) when trying to install. Any suggestion/idea how to fix this?

Log file monitoring using Loggly

Hi we have been using loggly since 2 months (Paid) and we are ingesting Log files to loggly using configure-filemontoring.sh script.
This script ingests application logs as well as syslog in which we are not looking for and due to this the daily limit hit is reached as we tried to remove syslogs by removing the cronjob where it runs and creates 22-loggly.conf and ingests syslogs, but we are loosing ingestion of application logs after something. So please suggest us any solution so that we can only ingest application/file logs to loggly without syslogs.

LOG-1505: SQS3script.py bug

Some users may have multiple AWS profiles, which causes their credential file to look as follows:

[default]
aws_access_key_id = MY_DEFAULT_ACCOUNT
aws_secret_access_key = MY_DEFAULT_SECRET

[personal]
aws_access_key_id = MY_PERSONAL_ACCOUNT
aws_secret_access_key = MY_PERSONAL_SECRET

the following lines may cause "Unauthorized" errors while running the script:

with open(os.environ['HOME'] + '/.aws/credentials') as f:
    for line in f:
        if "aws_access_key_id" in line:
             access_key = line.split("=",1)[1].strip()
        if "aws_secret_access_key" in line:
             secret_key = line.split("=",1)[1].strip()

Better would be if the python script is migrated to use Boto3 Configuration

However - As an alternative, I suggest the use of the following Terraform configuration: https://github.com/honestbee/loggly-elb-infra

This configuration is more robust and easier customisable for end users.

LOG-1501: script for installing file monitoring prompts for 5 minutes, saves for 10 minutes in cronjob

When you run the file monitoring tool it prompts you for 5 minutes interval cron job sync, it's actually saving 10 minutes.

copy of cron job after running script

*/10 * * * * sudo bash /root/.loggly/file-monitoring-cron-laravel-storage-directory.sh

expected

*/5 * * * * sudo bash /root/.loggly/file-monitoring-cron-laravel-storage-directory.sh

Notification test

Jayant, do you get this notification?