Comments (2)
I fixed the unhealthy state of the cluster by destroying the instance for now. The operator created a new pod keycloak-db-pgcluster-7
and a temporary join job and after a minute the cluster got healthy again.
However that should not be the solution to this kind of error. The operator should be able to heal the cluster if one worker node goes down for whatever reason. I still hope it is just a misconfiguration but I am quite sure I did nothing wrong here.
from cloudnative-pg.
Before I had destroyed keycloak-db-pgcluster-5
I had a look into its filesystem, especially into the folder /var/lib/postgresql/data/pgdata/pg_wal
:
postgres@keycloak-db-pgcluster-5:~/data/pgdata/pg_wal$ ls -la
total 606256
drwxrws--- 3 postgres tape 4096 May 16 10:41 .
drwx------ 19 postgres tape 4096 May 16 10:41 ..
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000005000000000000001A
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000005000000000000001B
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000005000000000000001C
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000005000000000000001D
-rw-rw---- 1 postgres tape 169 May 15 16:00 00000005.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000006000000000000001E
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000006000000000000001F
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000060000000000000020
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000060000000000000021
-rw-rw---- 1 postgres tape 212 May 15 16:00 00000006.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000022
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000023
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000024
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000025
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000026
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000027
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000028
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 000000070000000000000029
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000007000000000000002A
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000007000000000000002B.partial
-rw-rw---- 1 postgres tape 255 May 15 16:00 00000007.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000008000000000000002B
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000008000000000000002C
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000008000000000000002D
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000008000000000000002E
-rw-rw---- 1 postgres tape 298 May 15 16:00 00000008.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000009000000000000002E
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 00000009000000000000002F
-rw-rw---- 1 postgres tape 341 May 15 16:00 00000009.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000A0000000000000030.partial
-rw-rw---- 1 postgres tape 384 May 15 16:00 0000000A.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000B0000000000000030
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000B0000000000000031
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000B0000000000000032
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000B0000000000000033
-rw-rw---- 1 postgres tape 428 May 15 16:00 0000000B.history
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000C0000000000000033
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000C0000000000000034
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000C0000000000000035
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000C0000000000000036
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000C0000000000000037
-rw-rw---- 1 postgres tape 472 May 15 16:00 0000000C.history
-rw------- 1 postgres tape 16777216 May 16 10:41 0000000D0000000000000038
-rw------- 1 postgres tape 16777216 May 16 10:41 0000000D0000000000000039
-rw-rw---- 1 postgres tape 16777216 May 15 16:00 0000000D000000000000003A
-rw------- 1 postgres tape 516 May 16 10:41 0000000D.history
drwxrws--- 2 postgres tape 4096 May 15 16:00 archive_status
I found it kind of weird that the file 0000000D000000000000003A
exists here but not in the other instances. Can this be a remnant from being the primary instance before the sudden shutdown of the worker node?
I deleted that file manually and restarted the pod but it did not solve the issue. It still was a bit weird to see that file there which does not exist in the S3 backup.
from cloudnative-pg.
Related Issues (20)
- [Bug]: Scaleway S3 Http Errors
- [Bug]: Avoid restore job hang if max_connection is increased after last basebackup
- [Feature]: TTY detection for the plugin
- Backport failure for pull request 4609 HOT 1
- Backport failure for pull request 4650
- [Chore]: Remove SystemUID detection
- [Bug]: Remove the need of clusterWide namespace permissions on OLM
- Backport failure for pull request 4706 HOT 1
- [Feature]: Stop requiring namespace object permissions
- [Feature]: E2E not reporting the command executed when errors out
- [Bug]: Panic while checking backup container ID if pod container statuses are missing
- [Bug]: replica cluster switchover doesn't work when only object store replication is configured HOT 3
- [Feature]: Add new options to ginkgo
- Introduce release candidate process
- Backport failure for pull request 4765
- [E2E]: Add pgaudit tests HOT 1
- [Bug]: xmin on standby cluster replication slots causing heavy io on primary due to permanent autovacuum HOT 6
- [Feature]: Use Ciclops thermometer when sending E2E alerts
- [Bug]: Pooler login failing after migrating to Postgres 16 HOT 5
- [Bug]: PostgreSQL switch over endless loop with WAL errors like servers diverged at WAL location
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudnative-pg.