Comments (4)
lfittl
commented on May 23, 2024
@caleb15 Any chance that container has been crashing and restarting multiple times? (there is a condition when the collector crashes that can leave behind files)
from collector.
caleb15
commented on May 23, 2024
Maybe? Status says the image has only been up for 35 minutes. Is there someplace where I can see how many times / when the image has crashed? 🤔
from collector.
lfittl
commented on May 23, 2024
@caleb15 You could look in "docker ps -a" and then run "docker logs" on any instances that have run recently but have been stopped.
Also, in case it's an out of memory condition, review any memory limit settings you have on the container, or memory limits on the system overall. For some systems with high log volume we've seen issues with high memory usage during log parsing.
from collector.
caleb15
commented on May 23, 2024
nvm, I found it: https://serverfault.com/a/909267/512362
You're exactly right, looks like the container has been repeatedly dying from oom:
[email protected]:~$ sudo docker events --since=120m
2021-04-20T04:06:12.653592243Z container oom 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T04:06:12.886462028Z container die 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (exitCode=137, image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T04:06:12.982581172Z network disconnect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
2021-04-20T04:06:13.049410541Z volume unmount ea1bd04f298155a3275c1da243e766e818779d5150d4fa6ece492a91983ad925 (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, driver=local)
2021-04-20T04:06:13.084732358Z network connect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
2021-04-20T04:06:13.091250793Z volume mount ea1bd04f298155a3275c1da243e766e818779d5150d4fa6ece492a91983ad925 (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, destination=/state, driver=local, propagation=, read/write=true)
2021-04-20T04:06:13.504545174Z container start 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:09:49.300653527Z container oom 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:09:49.606117173Z container die 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (exitCode=137, image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:09:49.734725089Z network disconnect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
2021-04-20T05:09:49.855292233Z volume unmount ea1bd04f298155a3275c1da243e766e818779d5150d4fa6ece492a91983ad925 (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, driver=local)
2021-04-20T05:09:49.900484616Z network connect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
2021-04-20T05:09:49.904166747Z volume mount ea1bd04f298155a3275c1da243e766e818779d5150d4fa6ece492a91983ad925 (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, destination=/state, driver=local, propagation=, read/write=true)
2021-04-20T05:09:50.301050392Z container start 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:16:09.101193715Z container oom 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:16:09.273349383Z container die 490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0 (exitCode=137, image=quay.io/pganalyze/collector:v0.36.0, name=pganalyze-mirror-heavy)
2021-04-20T05:16:09.402131327Z network disconnect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
2021-04-20T05:16:09.419386173Z volume unmount ea1bd04f298155a3275c1da243e766e818779d5150d4fa6ece492a91983ad925 (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, driver=local)
2021-04-20T05:16:09.444734606Z network connect c9bbf642c5c183bff16c2933ecdb988d3b0be6ea6cbc33b856780aacd138617b (container=490ccc38d41dfbe712a80bfa48e033f62173ff6e90167e192702ceaf3812b6c0, name=bridge, type=bridge)
We only have so much memory :( I suppose we will need to buy more memory for our ec2 instance.
I might suggest changing pganalyze to clear /tmp at start to avoid disk space piling up. Not sure if it's worth it to do that considering the root problem but maybe? Up to you. Feel free to close this.
from collector.
Related Issues (20)
- Scheduler: Don't schedule runner intervals on fixed cron expressions HOT 5
- Documentation/support for Google Cloud SQL Auth Proxy or Connector HOT 2
- Update collector README to reference pg_monitor role for setup instructions HOT 3
- Handle read-only filesystem gracefully HOT 1
- Make IdentifierMap cache TTL configurable HOT 3
- Could not collect activity for server HOT 8
- Unable to DB_USE_IAM_AUTH with aws RDS instance HOT 4
- Error calling pg_stat_statements_reset() as requested intermittently HOT 2
- Regular collector oom-kill at midnight UTC on Sundays HOT 1
- Publish Docker image to ECR public registry HOT 1
- Error to send snapshots HOT 2
- Upgrade go.uuid module HOT 3
- Missing image for v0.50.1 HOT 2
- Additional security settings for pganalyze collector HOT 3
- Add rudimentary test suite for Helm chart
- Debian build uses EOL version HOT 1
- Error: Failed to reload collector HOT 1
- OTEL connector - can it support Datadog APM traces for PostgreSQL HOT 1
- OTEL connector - can it support Datadog?
- Feature request: Add DB_PASSWORD_FILE or similar HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from collector.