pganalyze / collector Goto Github PK

View Code? Open in Web Editor NEW

320.0 320.0 54.0 30.64 MB

pganalyze statistics collector for gathering PostgreSQL metrics and log data

Home Page: https://pganalyze.com

License: Other

Makefile 2.98% Go 95.32% Shell 1.37% Dockerfile 0.11% Procfile 0.01% Smarty 0.22%

database metrics monitoring postgresql rds

collector's People

Contributors

Stargazers

Watchers

Forkers

askrana deliveroo jonike travis-ci peletiah matthull nrmitchi dombo renesugar evenco amasses mahatmafatalerror tools-env jaa-aa mgalgs sj26 isgasho cybernetics pgiraud timchenkomo diseku jovasquez89 yashbhutwala nikolays joehorsnell caleb15 davidanthonygardner janetechinc philcoggins kandji-chris dullyouth edulib-france pedrommone jalaziz premist antogon zhuomingliang sebfie radiusnetworks jlisthood preovaleo aviau rahulkr7370 josephglanville emilsedgh ccppprogrammer jawnsy togglhire benoittgt enescakir johnbeynonorg snag-solutions hoohawk ianstanton

collector's Issues

Support for AssumeRoleWithWebIdentity?

I have an EKS Fargate cluster using Amazon's recommended practice of supply Role access to an EKS ServiceAccount annotation in order to provision AWS access to a pod. I appear to have everything configured correctly and am seeing the AdmissionController do it's thing. Only problem is, it doesn't appear the PGAnalyze is able to leverage it:

Submitted compact activity snapshot successfully
Could not collect logs for server: could not collect logs: Error finding RDS instance: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Submitted compact activity snapshot successfully

Is there any plan to implement AssumeRoleWithWebIdentity? It appears to be supported in the Go AWS SDK.

Heroku Postgres 10 error and no stats delivered

I [PRODUCTION_URL] pg_stat_statements does not exist, trying to create extension...
E [PRODUCTION_URL] Could not collect query stats for server: error collecting pg_stat_statements: pq: function public.pg_stat_statements(boolean) does not exist

The extension is already enabled, the pg_stat_statements view does exist but the function does not seem to. I'm not sure why.

any help would be appreciated

Missing License

Hi,
This looks like an interesting tool. Do you mind adding a license?

Thanks.

Scan error on column index 0

We get the following error messages from the collector:

2017/03/01 11:32:23 W [server1] Error collecting replication statistics: sql: Scan error on column index 0: unsupported Scan, storing driver.Value type <nil> into type *string 2017/03/01 11:32:23 E [server1] Could not process database: sql: Scan error on column index 0: unsupported Scan, storing driver.Value type <nil> into type *string

Allow filter sensitive data from query logs

Hi. We use a custom collector patch, that removes sensitive content from queries before sending logs to pganalyze server.

diff --git a/input/system/logs/parse.go b/input/system/logs/parse.go
index 01863e7..e71cbd1 100644
--- a/input/system/logs/parse.go
+++ b/input/system/logs/parse.go
@@ -211,6 +211,15 @@ func ParseAndAnalyzeBuffer(buffer string, initialByteStart int64, linesNewerThan
                // Generate unique ID that can be used to reference this line
                logLine.UUID = uuid.NewV4()

+               if strings.Contains(logLine.Content, "_encrypt"){
+                       //logger.PrintError("Censored encrypt line: %s", logLine.Content)
+                       logLine.Content = "--Censored Encryption Content;"
+               }
+               if strings.Contains(logLine.Content, "_decrypt"){
+                       //logger.PrintError("Censored decrypt line: %s", logLine.Content)
+                       logLine.Content = "--Censored Decryption Content;"
+               }
+
                logLines = append(logLines, logLine)
        }

Would be extremely cool if this would be supported by upstream, preferably with configurable filtered function names.

test freezes / times out

In docker on a ec2 instance:

[email protected]:~$ sudo docker run --env-file .env quay.io/pganalyze/collector:stable /home/pganalyze/collector --test -v
2020/12/17 17:08:38 V [default] Identified as api_system_type: amazon_rds, api_system_scope: us-east-1/114141044511, api_system_id: cloud51-master
2020/12/17 17:08:38 V Did not open state file: open /var/lib/pganalyze-collector/state: no such file or directory
2020/12/17 17:08:38 I Running collector test with pganalyze-collector 0.35.0
2020/12/17 17:08:38 I [default] Testing statistics collection...
2020/12/17 17:08:38 V [default] Detected PostgreSQL Version 110008 (PostgreSQL 11.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit)
2020/12/17 17:08:40 V [default] Found pganalyze.get_stat_replication() stats helper
request expired, resigning
2020/12/17 17:16:41 I [default]   Test collection successful for PostgreSQL 11.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
2020/12/17 17:16:41 V [default] Successfully prepared S3 request - size of request body: 0.1212 MB
2020/12/17 17:16:43 I [default]   Test snapshot successful (607 KB)
2020/12/17 17:16:43 I [default] Testing activity snapshots...
2020/12/17 17:16:43 V [default] Detected PostgreSQL Version 110008 (PostgreSQL 11.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit)
2020/12/17 17:16:43 V [default] Successfully prepared S3 request - size of request body: 0.0012 MB
2020/12/17 17:16:56 I [default]   Test snapshot successful (2.42 KB)
2020/12/17 17:16:56 I [default] Testing log collection (Amazon RDS)...
request expired, resigning
2020/12/17 17:24:57 V [default] Successfully prepared S3 request - size of request body: 0.0052 MB
2020/12/17 17:25:03 I [default]   Test snapshot successful (11.1 KB)
2020/12/17 17:25:03 I [default]   Log test successful

Test successful. View servers in pganalyze:
 - [default]: https://app.pganalyze.com/servers/o2sfz3shszhafpjwc6y4zzbgja

Right before request expired, resigning it does not output anything for a very long time and then finally outputs request expired, resigning and proceeds.

@lfittl found the fix for this - I am just documenting the bug in case other people run across it. I will be answering and closing the issue below.

Releaser container image v0.28.0

Hi,
Could you add/release a v0.28.0 image here https://quay.io/repository/pganalyze/collector?tab=tags please ?

Thanks !

Collector docker image for OpenShift ( OKD ) environment

Hi Guys, I tried using existing image of collector ( collector:stable) into okd environment but due to RBAC policy restriction it won't let anything to run with root.. Does anyone know where can i find OKD 3.11 compatible version of collector

When unable to parse result is <truncated query>, even when queries are complete

When the parser is unable to parse a query it returns <truncated query>. But the query isn't necessarily truncated - it is unparsable. We reviewed pg_stat_statements manually and these queries are complete.
In pgAnalyze we see about 300 unparseable queries and we need some way to identify them.
What can we do to help fixing these parser errors?

collector/util/normalize.go

Lines 6 to 9 in 7969c3b

    
           normalizedQuery, err := pg_query.Normalize(query) 
        
           if err != nil { 
        
           	normalizedQuery = "<truncated query>" 
        
           }

`--dry-run-logs` doesn't print anything

When running pganalyze-collector --dry-run-logs I don't see any JSON data printed.
I only get the following messages:

2020/03/31 14:25:28 I [server1] Testing log collection (local)...
2020/03/31 14:25:34 I [server1]   Local log test successful

The --dry-run option does only print the stats and activity snapshots data. Nothing for logs.

Here's however what the --debug-logs option returns:

2020/03/31 14:23:35 I [server1] Would have sent log state:
log lines: 9, query samples: 0
6 x STATEMENT_LOG
1 x PGA_COLLECTOR_IDENTIFY
2020/03/31 14:24:08 I [server1] Would have sent log state:
log lines: 1, query samples: 0
1 x STATEMENT_LOG
2020/03/31 14:24:11 I [server1] Would have sent log state:
log lines: 1, query samples: 0
1 x STATEMENT_LOG

Am I missing something?

[feature request] ability to collect multiple db instances with one docker container

Readme:

We currently require one Docker container per RDS instance monitored.

I was very surprised to learn that a docker container can only collect data from one db instance. This seems pretty odd given that non-docker collector can collect from multiple RDS instances.

Can't use the collector when there is a hash (#) in the password

I guess it treat everything after hash as comment.

Log temp files filling up disk

We recently saw a disk space utilization alert fire due to pganalyze. /tmp filled up with nearly 10GB of RDS logs that have been sitting around for days:

Settings:

PGA_ENABLE_ACTIVITY=1
PGA_ENABLE_LOGS=1
PGA_ENABLE_REPORTS=1

I see in the code that Cleanup exists and should delete log files after use but I'm unable to dig further.

--test-logs ambiguous behavior

$ ./pganalyze-collector --test-logs --config=pganalyze-collector.conf
$ echo $?
0

No output, no error, but unclear what this means.

Collect system information when used with Docker

e5dc775 disabled it.

Perhaps there can be an environment variable to control this?

Collector test returns Killed

Server version: 9.6 - Amazon RDS
Collector version: v0.27.0

When I'm running the following command

sudo pganalyze-collector --test -v

I've got

2020/08/18 13:08:32 V Ignoring state file since the on-disk format has changed
2020/08/18 13:08:32 I [server1] Testing statistics collection...
2020/08/18 13:08:37 V [server1] Detected PostgreSQL Version 90615 (PostgreSQL 9.6.15 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit)
2020/08/18 13:08:37 V [server1] Found pganalyze.get_stat_statements() stats helper
Killed

I'm already tried to, but without sucess:

Check the pganalyze-collector.conf
Upgrade to latest version (v.0.32.0).
Recreate IAM policies.
Recreate extensions and functions over RDS.

I have another server with a similar configuration and works perfectly fine.

I'm happy to provide more information if necessary.

Log: "Error collecting replication statistics"

Server version: 11.4 on RDS
Collector version: v0.21.0

The collector logs the following message:

Error collecting replication statistics: sql: Scan error on column index 11, name "write_lsn": unsupported Scan, storing driver.Value type <nil> into type *string

The only thing which might be a bit extraordinary might be that this server has active logical replication slots.
I'm happy to provide more information if necessary.

No way to enable logs collection when using docker image

Seems like by default logs collection is not enabled when using the docker image without changes, and there's no environment variable or command switch to enable them.

Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout

In our docker container I am seeing frequent errors like below:

2021-03-19T23:22:10.974893936Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-19T23:32:07.913737284Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-19T23:42:10.221978219Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-19T23:52:07.792295692Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:02:11.995701411Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:12:07.517917903Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:22:14.380817556Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:32:09.701303512Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:42:07.446886906Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T00:52:08.508554841Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T01:02:08.764939569Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout
2021-03-20T01:12:12.094003633Z E [default] Could not process server: Error collecting pg_stat_statements: pq: canceling statement due to statement timeout

The database it is connecting to does not have a statement timeout:

[email protected]:~$ sudo docker exec pganalyze-mirror-heavy env
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=secret
DB_URL=postgres://pganalyze:[email protected]/secret
PGA_API_KEY=secret
AWS_INSTANCE_ID=cloud100-mirror0-heavy
GOLANG_VERSION=1.14.4
GOPATH=/go
HOME_DIR=/home/pganalyze
CODE_DIR=/go/src/github.com/pganalyze/collector
HOME=/root

Memory leak on v0.22.0

We are running quay.io/pganalyze/collector:v0.22.0 in Kubernetes and experiencing memory leak. Haven't tried whether an upgrade would fix it but I did not see any mention of memory leak in existing issues on this repo.

SIGSEGV: segmentation violation on 0.38.0

I just installed latest stable (0.38.0) on three VMs (Debian 10.8 with PostgreSQL 13.2) with different cpu/mem sizes and the three of them crashed with a signal SIGSEGV: segmentation violation error. The process restarts and it crashes again every 10 mins.

Please find attached traceback log:

pganalyze_segv.log

Container running as root

Hi,
Is there a reason with the container is running as root ?

It looks like the Dockeerfile (https://github.com/pganalyze/collector/blob/master/Dockerfile#L4) is adding a user but no "USER" directive, that user is not actually used.

Support regex processing of DB role names in collector

Hi @lfittl,

As per our call earlier this week (cc @robharrop), I'm opening an issue for your comment/approval on our proposed collector change, to allow processing/transformation of DB role names in the collector, before sending to the pganalyze server.

Our specific use case is to handle Vault Dynamic Secrets, as the dynamically generated suffixes that Vault adds to the DB roles it creates cause pganalyze to treat statements that we would like to be grouped together as separate, so there is effectively no useful history for each statement. The proposed solution (using regex) supports our requirement, but I believe would also be generic enough to be useful for other contexts, even just literal string replacements.

Detail

The DB role names generated by Vault Dynamic Secrets have two dynamically generated suffixes that we would like to trim and ignore.

For example, for role v-aws-some-service-3tce39Yn1P2lV7RSQnMN-1628677987, we want to ignore the two suffix groups, and just send v-aws-some-service to pganalyze.

The proposed solution is to add some collector config options that accept a Sed-like search/replace pattern, eg.

echo "v-aws-some-service-3tce39Yn1P2lV7RSQnMN-1628677987" | gsed --regexp-extended 's/-[[:alnum:]]+-[[:digit:]]+$//'
v-aws-some-service

It seems the built-in Go regex package supports the required regex syntax.

If you are OK in principle with this idea, we can look at fleshing out a bit more detail on this, eg a simple design (exactly what config options to provide and how), so please let us know. Also could you please confirm whether there's any server-side change that would be required to support this, or if it could be handled completely within the collector?

Thanks,

Joe.

AWS Aurora/RDS log insight limitation

Hi,

Currently, when using Aurora or RDS clusters with multiple instances failovers are a normal part of the ecosystem and therefore any replica of the Aurora or RDS cluster can be the reader or writer.

The collector currently has a limitation where if the cluster name is different from the instance ID you have to specify aws_db_instance_id in order to obtain LogInsights/Explain plans. Considering the above statement, this means the user would have to manually edit the aws_db_instance_id configuration for the cluster every time there is a failover event.

Could potentially using the rds:DescribeDBInstances action to determine the writer be a nice way to make finding the instance ID automated?

No I/O statistics from Docker volume

Under https://github.com/pganalyze/collector/blob/35eadf33238441739488fabe6f3eb79e139ae3ce/README.md#docker-container-non-rds it says:

Note: You can add -v /path/to/database/volume/on/host:/var/lib/postgresql/data in order to collect I/O statistics from your database, this requires that it runs on the same machine.

But I've done exactly that (substituting for the correct path, of course) and it doesn't work.

runtime error: invalid memory address or nil pointer dereference

[email protected]:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[email protected]:~$ sudo docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                                  PORTS               NAMES
a3bd97454bb6        quay.io/pganalyze/collector:stable   "/home/pganalyze/doc…"   24 minutes ago      Up Less than a second                                       pganalyze-mirror
9693f5c44b3c        quay.io/pganalyze/collector:stable   "/home/pganalyze/doc…"   24 minutes ago      Restarting (2) Less than a second ago                       pganalyze
[email protected]:~$ sudo docker logs pganalyze
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005a9b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003daf40, 0xc0002dc1b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003daf80, 0xc00045a150, 0xbff0b37ae4907b8a, 0x65e115f, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000637b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002655c0, 0xc0000a32f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000265600, 0xc0001f95e0, 0xbff0b37b16249c30, 0x3148a71, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000794000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002654c0, 0xc0000a32f0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8000, 0xc0001dc000, 0xbff0b37b5f8be464, 0x708588f, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005bfb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8bc0, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8c00, 0xc0001dc140, 0xbff0b37bc1cf27f0, 0x6eb7a9f, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004d0000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d0b00, 0xc0002da1b0, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d0b40, 0xc0001d4150, 0xbff0b37c28198198, 0x5a1e5e6, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00060bb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003bf780, 0xc0002ba210, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003bf7c0, 0xc000424b60, 0xbff0b37cc3fe74f5, 0x362fbb2, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005abb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003d8e00, 0xc0000a00c0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003d8e40, 0xc000201220, 0xbff0b37db4c92282, 0x3677ef8, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004cc000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076c00, 0xc0002ba1e0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076c40, 0xc00016a150, 0xbff0b37f738f0f87, 0x382eab0, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004ce000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076c00, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076c40, 0xc00016a150, 0xbff0b382cdfcdbc5, 0x3751370, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000627b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002dd680, 0xc00009b2f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0002dd6c0, 0xc000271670, 0xbff0b3895749f265, 0x389a9c7, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005e7b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc00032aa40, 0xc0000a3260, 0x29, 0xc00073fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000264000, 0xc0001f8000, 0xbff0b3964a31aa94, 0x35621d0, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005bfb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc00032cb80, 0xc0000a3260, 0x29, 0xc0006efc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc00032cbc0, 0xc000209500, 0xbff0b3a5701cf61a, 0x35e0bff, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00078c000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002641c0, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000264200, 0xc0001dc150, 0xbff0b3b498657059, 0x4a4b6cc, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000788000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8b00, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8b40, 0xc0001dc150, 0xbff0b3c3c3657188, 0x38aef0e, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004ae000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc00030af00, 0xc00003f2f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242000, 0xc0001d4000, 0xbff0b3d2e66af09c, 0x432f6f1, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00052a000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8bc0, 0xc00003e0c0, 0x29, 0xc00073fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8c00, 0xc0001dc140, 0xbff0b3e20c8d28a8, 0x34ecdb5, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004cc000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000242200, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242240, 0xc00016a150, 0xbff0b3f12fda137e, 0x3497a96, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004c4000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076b40, 0xc0002bc1e0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076b80, 0xc00016a150, 0xbff0b400555b0fc5, 0x381e1f4, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00079c000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc00026d7c0, 0xc0000a92c0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000e0000, 0xc0001e4000, 0xbff0b40f731c7727, 0x22a8985, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00044bb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc00032cb40, 0xc0000a3260, 0x29, 0xc00073fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc00032cb80, 0xc000209510, 0xbff0b41e9867a2fe, 0x30df156, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0000d0000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076c00, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076c40, 0xc0001ea150, 0xbff0b42dbb920bba, 0x3507788, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0004bc000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076b40, 0xc0002be1e0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076b80, 0xc00016a180, 0xbff0b43ce1fc3e7b, 0x3677466, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0000d6d80, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0004be200, 0xc0002e01b0, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0004be240, 0xc0005fe110, 0xbff0b44bfb6691e0, 0x2378bce, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00060fb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000242200, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242240, 0xc0001d6140, 0xbff0b45b1ea44764, 0x35f4296, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005d7b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0004a6500, 0xc00003f2c0, 0x29, 0xc0006dfc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242000, 0xc0001d6000, 0xbff0b46a41a555da, 0x1e28a5b, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00060fb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000242200, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242240, 0xc0001d6140, 0xbff0b479622bab32, 0x34a057a, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000780000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000076b40, 0xc0002bc1e0, 0x29, 0xc00073fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000076b80, 0xc00016a150, 0xbff0b4888983d09c, 0x35d2565, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0000d0000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000136b00, 0xc00033c1e0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000136b40, 0xc0001ea150, 0xbff0b497a440e54f, 0x1c3db68, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00079a000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8b00, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8b40, 0xc0001dc1b0, 0xbff0b4a6c83d21ae, 0x34698fb, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0000b6900, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000243a80, 0xc00003f050, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003bed00, 0xc000424020, 0xbff0b4b5ebbd12fd, 0x3719491, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005f1b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002641c0, 0xc00003e0c0, 0x29, 0xc00073fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000264200, 0xc0001f8140, 0xbff0b4c5130ec88e, 0x32d9a77, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000794000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003e2f00, 0xc0002e41b0, 0x29, 0xc0006dfc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003e2f40, 0xc0003ae190, 0xbff0b4d43752eb5b, 0x2eea665, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005b7b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8b00, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8b80, 0xc0001dc140, 0xbff0b4e35dba9804, 0x352c296, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00060bb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003bff80, 0xc0002bc960, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003bffc0, 0xc0001d7d70, 0xbff0b4efa18b604b, 0x68e39a9, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000794000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002641c0, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000264200, 0xc0001f8140, 0xbff0b4efd6131010, 0x35a1f0f, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0005b1b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0002641c0, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000264200, 0xc0001f8140, 0xbff0b4f01b0c8082, 0x517a8ca, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc00060fb00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000243080, 0xc00003f2f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0002430c0, 0xc0001d6dd0, 0xbff0b4f06b068fbe, 0x52be75e, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc0007b0000, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0000d8b00, 0xc0002e21b0, 0x29, 0xc0006e7c20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0000d8b80, 0xc0001dc140, 0xbff0b4f0d2854d4a, 0x8a7c6bd, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000617b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc0003bef40, 0xc0002bc1e0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc0003097c0, 0xc0001d7310, 0xbff0b4f16179ebd0, 0x1d7df34, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x968dad]

goroutine 1 [running]:
github.com/pganalyze/collector/config.ServerConfig.GetDbHost(0xc00003808c, 0x10, 0x12bdcec, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000036007, ...)
	/go/src/github.com/pganalyze/collector/config/config.go:301 +0x4d
github.com/pganalyze/collector/config.preprocessConfig(0xc000607b00, 0xb, 0xc00003808c, 0x10)
	/go/src/github.com/pganalyze/collector/config/read.go:298 +0x81
github.com/pganalyze/collector/config.Read(0xc000242200, 0xc00003e0f0, 0x29, 0xc00075fc20, 0x8060000020b, 0x80600000201, 0x80800000404, 0xd4)
	/go/src/github.com/pganalyze/collector/config/read.go:447 +0x710
main.run(0x166f660, 0xc000242240, 0xc0001d6140, 0xbff0b4f25398e2e0, 0x34be5f3, 0x1fc7880, 0x101010101010101, 0x1, 0x12b64aa, 0x13, ...)
	/go/src/github.com/pganalyze/collector/main.go:53 +0x90
main.main()
	/go/src/github.com/pganalyze/collector/main.go:453 +0x156b

At first I thought it was because I forgot to initialize the db with the pganalyze user but I initialized the db and I can connect to the db with the pganalyze user and the error still happens :(

error on starting service

Hello,
I am getting this error while starting the service.
"ERROR - 2016-07-22 21:42:55,764 Got an error during query execution
ERROR - 2016-07-22 21:42:55,764 permission denied for schema pganalyze"

Any idea what it is?
I re-ran through all the instructions again. https://pganalyze.com/docs/install/amazon_rds/02_create_monitoring_user

fails to install on Ubuntu 18.04 LTS pganalyze-collector.postinst: start: not found

it's a new EC2 instance created in AWS

uname -a

Linux ip-10-10-11-20 4.15.0-1032-aws #34-Ubuntu SMP Thu Jan 17 15:18:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/lsb*

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

apt-get install -y pganalyze-collector

Reading package lists... Done
Building dependency tree
Reading state information... Done
pganalyze-collector is already the newest version (0.19.1).
0 upgraded, 0 newly installed, 0 to remove and 106 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up pganalyze-collector (0.19.1) ...
/var/lib/dpkg/info/pganalyze-collector.postinst: 46: /var/lib/dpkg/info/pganalyze-collector.postinst: start: not found
dpkg: error processing package pganalyze-collector (--configure):
installed pganalyze-collector package post-installation script subprocess returned error exit status 127
Errors were encountered while processing:
pganalyze-collector
E: Sub-process /usr/bin/dpkg returned an error code (1)

Success callback

Hi,
would it be possible to add something like a success callback feature?
In previous versions of the collector when it was executed with cron I always added monitoring to it eg. https://healthchecks.io/
Thanks to this when there was a problem with collector or it took longer that expected I would get notified that's something wrong.

With the new version where collector is executed as a daemon I'm lacking this monitoring feature. I would propose that you add a config option that would be a shell expression or a shell script that could be run on every successful run?

VACUUM (ANALYZE, VERBOSE) is causing collector to fail collecitng vacuum progress

This error log is shown when running VACUUM manually using:

VACUUM (ANALYZE, VERBOSE) <table name>;

Could not collect activity for server: error collecting pg_stat_vacuum_progress: sql: Scan error on column index 3, name "nspname": converting NULL to string is unsupported

Vacuum stats also seems to be missing during the periods those manual VACUUM + ANALYZE are running.

Could not process database: pq: column "queryid" does not exist

We have pganalyze-collector collecting stats from a few of our production databases in RDS for years now. About a month ago, pganalyze-collector started failing to collect stats for one (and only one) of these databases all of a sudden, like this:

$ /tmp/pganalyze-collector-linux-amd64  --version
0.9.14

$ /tmp/pganalyze-collector-linux-amd64 --config="/home/ubuntu/.pganalyze.proddb.conf" --dry-run
2017/09/27 17:57:23 I [DEFAULT] Warning: You are not connecting as superuser. Please setup the monitoring helper functions (https://github.com/pganalyze/collector#setting-up-a-restricted-monitoring-user) or connect as superuser, to get query statistics for all roles.
2017/09/27 17:57:23 E [DEFAULT] Error collecting pg_stat_statements
2017/09/27 17:57:23 E [DEFAULT] Could not process database: pq: column "queryid" does not exist

(Note, pganalyze complains about not being a superuser, but it is using an rds_superuser account.)

This is with the recent 0.9.14 release of pganalyze-collector, since I wanted to make sure it wasn't a bug fixed in a recent version, but we actually encountered the bug on version 0.8.0, which blows up with a message like this:

$ /usr/bin/pganalyze-collector --dry-run
ERROR - 2017-09-27 18:04:50,762 Got an error during query execution
ERROR - 2017-09-27 18:04:50,762 permission denied for schema pg_toast

I saw ticket #13 complaining about the same error message, and see the suggestion there to try ALTER EXTENSION pg_stat_statements UPDATE if the database has been recently updated. I don't think this database has been updated anytime recently (it is still running 9.4.7), but I tried this anyway, which updated pg_stat_statements from version 1.1 to version 1.2. That didn't make any difference in these errors.

Looking through the RDS logs for this instance, I find the failing query generated by 0.8.0 to be this one:

2017-09-27 18:04:50 UTC:192.168.x.y(57213):username@dbname:[49076]:ERROR:  permission denied for schema pg_toast
2017-09-27 18:04:50 UTC:192.168.x.y(57213):username@dbname:[49076]:STATEMENT:  /* pganalyze-collector */
	        WITH btree_index_atts AS (
	          SELECT nspname, relname, reltuples, relpages, indrelid, relam,
	                 regexp_split_to_table(indkey::text, ' ')::smallint AS attnum,
	                 indexrelid as index_oid
	            FROM pg_index
	            JOIN pg_class ON pg_class.oid=pg_index.indexrelid
	            JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace
	            JOIN pg_am ON pg_class.relam = pg_am.oid
	           WHERE pg_am.amname = 'btree'
	        ),
	        index_item_sizes AS (
	          SELECT i.nspname, i.relname, i.reltuples, i.relpages, i.relam,
	                 (quote_ident(s.schemaname) || '.' || quote_ident(s.tablename))::regclass AS starelid, a.attrelid AS table_oid, index_oid,
	                 current_setting('block_size')::numeric AS bs,
	                 8 AS maxalign,
	                 24 AS pagehdr,
	                 /* per tuple header: add index_attribute_bm if some cols are null-able */
	                 CASE WHEN max(coalesce(s.null_frac,0)) = 0
	                     THEN 2
	                     ELSE 6
	                 END AS index_tuple_hdr,
	                 /* data len: we remove null values save space using it fractionnal part from stats */
	                 sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 2048) ) AS nulldatawidth
	            FROM pg_attribute AS a
	            JOIN pg_stats AS s ON (quote_ident(s.schemaname) || '.' || quote_ident(s.tablename))::regclass = a.attrelid AND s.attname = a.attname
	            JOIN btree_index_atts AS i ON i.indrelid = a.attrelid AND a.attnum = i.attnum
	           WHERE a.attnum > 0
	           GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9
	        ),
	        index_aligned AS (
	          SELECT maxalign, bs, nspname, relname AS index_name, reltuples,
	                 relpages, relam, table_oid, index_oid,
	                 ( 2 +
	                     maxalign - CASE /* Add padding to the index tuple header to align on MAXALIGN */
	                       WHEN index_tuple_hdr % maxalign = 0 THEN maxalign
	                       ELSE index_tuple_hdr % maxalign
	                     END
	                   + nulldatawidth + maxalign - CASE /* Add padding to the data to align on MAXALIGN */
	                       WHEN nulldatawidth::integer % maxalign = 0 THEN maxalign
	                       ELSE nulldatawidth::integer % maxalign
	                     END
	                )::numeric AS nulldatahdrwidth, pagehdr
	           FROM index_item_sizes AS s1
	        ),
	        otta_calc AS (
	          SELECT bs, nspname, table_oid, index_oid, index_name, relpages, coalesce(
	                 ceil((reltuples*(4+nulldatahdrwidth))/(bs-pagehdr::float)) +
	                      CASE WHEN am.amname IN ('hash','btree') THEN 1 ELSE 0 END , 0 -- btree and hash have a metadata reserved block
	                 ) AS otta
	          FROM index_aligned AS s2
	          LEFT JOIN pg_am am ON s2.relam = am.oid
	        )
	        SELECT sub.index_oid,
	          CASE
	          WHEN sub.relpages <= otta THEN 0
	          ELSE bs*(sub.relpages-otta)::bigint END
	          AS wasted_bytes
	        FROM otta_calc AS sub
	          JOIN pg_class AS c ON c.oid = sub.table_oid
	          JOIN pg_stat_user_indexes AS stat ON sub.index_oid = stat.indexrelid

Boiling the failing query down a bit, I can reproduce the same ERROR with this smaller query:

SELECT  (quote_ident(s.schemaname) || '.' || quote_ident(s.tablename))::regclass AS starelid FROM pg_attribute AS a JOIN pg_stats AS s ON (quote_ident(s.schemaname) || '.' || quote_ident(s.tablename))::regclass = a.attrelid AND s.attname = a.attname;

and confirmed that this query ERRORs on only this one database server that I'm having trouble with. It's a little unclear to me why the above query blows up with a complaint about pg_toast.

Any suggestions here on a workaround or a fix?

Helm chart to deploy the collector on k8s

Not sure if this could help here or it is out of the scope of the project. I've available a Helm chart for the collector that we use in prod. If it might be helpful here, I can create a PR for that.

Ping here if you need that

Link to example output broken

The README contains the following link in the "Example output" section: https://gist.github.com/terrorobe/7103234

That link 404's for me. Could be because it's a private gist, or just a bad link.

Configuration File Is Empty message during test run.

Followed the instructions for setting up an AWS RDS postgres instance (using an Amazon Linux AMI EC2 instance).

I get the following message when I try to do a test run:

E Config Error: Configuration file is empty. Please edit /etc/pganalyze-collector.conf and reload the collector.

I tried reloading the collector and still receive the error.

here is a screenshot of the /etc/pganalyze-collector.conf file (with fake text for this screenshot only).

Error collecting replication statistics

I'm getting this warning in the log:

Error collecting replication statistics: sql: Scan error on column index 14: converting driver.Value type ("") to a int64: invalid syntax.

I'm running postgresql 10.3 with replication enabled for backup to barman.

Is the `pganalyze.get_column_stats()` function unsafe ?

Hi,
I had a PG expert examine our PG installation and he mentioned that our function pganalyze.get_column_stats() could be unsafe, linking to this article https://www.cybertec-postgresql.com/en/abusing-security-definer-functions/

Is there a security risk with that function ? He recommended we ran that command to fix the issue: ALTER FUNCTION pganalyze.get_column_stats() SET search_path = pg_catalog; Would that help ? Would pganalyze still be able to function ?

update documentation for configuring docker version

The README describes how to run a dockerized collector and uses environment variables to configure it. But how do we configure a pganalyze-collector.conf as needed by insights? https://app.pganalyze.com/docs/guides/setup_log_insights_on_self_hosted_server

If we create a local conf and volume mount it where? And will that override any default conf file that is needed?

Feature Request: Allow 'db_log_location' to take wildcard expressions

I have a use case in which our postgres servers are logging both csv and stderr logs to the same location. The csv logs are forwarded to our local ELK stack for analysis. It would be tremendously helpful if the 'db_log_location' parameter could take a wildcard path such as:

/path/to/log/*.log

The reason for this request is that pganalyze-collector is ingesting both the csv and stderr logs. When looking at the Log Insights view, I am seeing both stderr and csv logs and suspect that is causing the Query Performance EXPLAIN Plans to not work. I have been running the on-premise version of pganalyze for a year and have yet to see any EXPLAIN Plans.

Thank you!

Mutex error after a few minutes of running

Hey everyone, on Amazon Linux the collector will start up, and run fine, showing that activity and log data are successfully submitted for all log data. But then after around 5 minutes, the service fails with fatal error: sync: unlock of unlocked mutex, spits out a go stack trace, and then closes with pganalyze-collector.service: main process exited, code=exited, status=2/INVALIDARGUMENT.

A few other things to note: our setup ran fine with 2 databases connected in the /etc/pganalyze-collector.conf, and only started erroring when we went up to 4, adding 2 more. There didn't seem to be any memory pressure during all of this. There also seemed to be an error uploading to S3 citing something related to 'EOF' a minute or so before the mutex error - not sure if it's related.

Documentation is out of date

For the docker container -- PG_API_KEY is actually PGA_API_KEY, PG_URL doesn't appear to work, ...

Failed to fetch deb

Followed instructions. After adding:

deb https://packages.pganalyze.com/ubuntu/trusty/ stable main

To apt sources, apt then fails with:

W: Failed to fetch https://packages.pganalyze.com/ubuntu/trusty/dists/stable/InRelease  Unable to find expected entry 'main/binary-i386/Packages' in Release file (Wrong sources.list entry or malformed file)

Looking at that file, seems to be only binary-amd64 and no binary-i386

tmp directory not being cleared

One of our containers has been a naughty naughty boy:

root@influxdb0:/var/lib/docker/overlay2# sudo docker exec pganalyze-mirror-heavy du / -h -d 1
331.6G	/tmp
... other results truncated
root@influxdb0:/var/lib/docker/overlay2# sudo docker exec pganalyze-mirror-heavy du /tmp -h -d 2 -a
913.9M	/tmp/515771707
211.4M	/tmp/206147942
212.5M	/tmp/563747060
211.1M	/tmp/219882107
583.1M	/tmp/675476903
214.5M	/tmp/481994459
132.7M	/tmp/201391050
215.8M	/tmp/994851552
... etc
root@influxdb0:/var/lib/docker/overlay2# sudo docker exec pganalyze-mirror-heavy head /tmp/515771707
2021-03-27 12:54:33 UTC:<ip removed>(<port removed>):<user removed>@<removed>:[<removed>]:LOG:  connection authorized: user=<removed> database=<removed> <removed>
2021-03-27 12:54:33 UTC:<ip removed>(<port removed>):<user removed>@<removed>:[<removed>]:LOG:  connection authorized: user=<removed> database=<removed> <removed>
2021-03-27 12:54:33 UTC:<ip removed>(<port removed>):[<user removed>]<removed>un<removed>]:[5190]:LOG:  connection received: host=<removed> port=<removed>
2021-03-27 12:54:33 UTC:<ip removed>(<port removed>):<user removed>@<removed>:[<removed>]:LOG:  disconnection: session time: 0:00:00.036 user=<removed> database=<removed> host=<removed> port=<removed>
... and so on

The /tmp directory has been growing steadily in size ever since march 18th, when we set a 2 gig memory limit on the container. Oddly it's just pganalyze-mirror-heavy that ran into the issue. The other mirrors are normal:

root@influxdb0:/var/lib/docker/overlay2# sudo docker exec pganalyze-mirror du /tmp -h -d 2 -a
4.0K	/tmp
root@influxdb0:/var/lib/docker/overlay2# sudo docker exec pganalyze-mirror1 du /tmp -h -d 2 -a
4.0K	/tmp

I'll leave the container alone for a bit in case you want to look into it. I'll have to delete the container soon so it doesn't take up all the space on the server.

Could not process database: pq: column min_time does not exist

When running pganalyze-collector --config=/etc/pganalyze-collector.conf --test
I get this for one of my databases
Could not process database: pq: column "min_time" does not exist

I re-created the functions to make sure they were up to date. My postgres instance is on 9.6.1. Any ideas?

auto explain queries include statement secrets

I noticed here:
https://github.com/pganalyze/collector/blob/master/logs/analyze.go#L34

that the remainder, the plan section, which includes Query Text, isn't treated as secret — but the full query text can include sensitive information. Should it be treated as a secret?

Support log downloads for AWS Aurora Serverless v1

I'm trying to integrate the pganalyze collector to my Aurora Serverless Cluster but it can't find the db instance since the command aws rds describre-db-instances doesn't return my instance, but aws rds describe-db-clusters does so.

My collector keeps outputing

E [default] Rds/Logs: Could not find RDS instance: DBInstanceNotFound: DBInstance xxx not found.

Not all config options can be configured via env vars

Why some config options are being read from env vars and some are not?

I really liked the fact I can configure collector via env vars on Docker. The joy ended when I wanted to set db_log_location.

Please make all options configurable via env vars.

Could not process database: pq: column "queryid" does not exist

Trying to install pganalyze on Postgres 9.4.10 on a new CentOS 6 install and getting this when testing:

[root contrib]# pganalyze-collector --test -v
2017/01/05 14:23:27 V Did not open state file: open /var/lib/pganalyze-collector/state: no such file or directory
2017/01/05 14:23:28 V [default] Detected PostgreSQL Version 90410 (PostgreSQL 9.4.10 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17), 64-bit)
2017/01/05 14:23:28 E [default] Error collecting pg_stat_statements
2017/01/05 14:23:28 E [default] Could not process database: pq: column "queryid" does not exist

I have the required settings in postgresql.conf (and am confident they are right because I've been using the same postgresql.conf file successfully with pganalyze on another server).

AWS credentials not being read from ~/.aws/credentials

Hi - we're trying to use pganalyze and have been unsuccessful in configuring it to read AWS credentials from ~/.aws/credentials [0].

This is a well known location that's supported by default by the aws-sdk-go [1]. It looks like instead https://github.com/pganalyze/collector/blob/master/util/awsutil/amazon.go#L18 is being used to read credentials. Why is that?

Would it be possible to use the default credentials chain?

[0] https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
[1] https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Not allowed characters in the password?

The database password was set to 'b8B%6%P!#0' in /etc/pganalyze-collector.conf .
pganalyze-collector --test said authentication was failed.
The given password worked well using pgsql and I was able to login.
Then the password was replaced to a new one without % and !.
Both pgsql and pganalyze-collector --test ran successfully without any issue.
Are there not allowed characters for passwords?

/tmp space used by logs

Hi, I'm using pga 0.22.0, and I have the /tmp partition without disk space free (I know, all about the part problem)

It's posible reduce the log size?

Thanks a lot,
Diego

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	normalizedQuery, err := pg_query.Normalize(query)
	if err != nil {
	normalizedQuery = "<truncated query>"
	}