Comments (11)
leverege-jphelps
commented on May 24, 2024
1
Seeing same running a custom image (timescale+jsquery+pgsql 16 base / gke 1.26 / cnpg 1.22). The system comes up and runs fine but if we start failover testing there seems to be a 50/50 chance that a pod becomes unrecoverable.
from cloudnative-pg.
gbartolini
commented on May 24, 2024
Can you please share the content of:
- custom.conf
- override.conf
- postgresql.auto.conf
(They are inside PGDATA).
It strikes me that you are running with synchronous_commit set to off - which for self-healing is not recommended within a Kubernetes cluster unless you can live with some data loss.
from cloudnative-pg.
gbartolini
commented on May 24, 2024
I think this is related to #3680
from cloudnative-pg.
buroa
commented on May 24, 2024
Can you please share the content of:
- custom.conf
- override.conf
- postgresql.auto.conf
(They are inside PGDATA).
It strikes me that you are running with synchronous_commit set to off - which for self-healing is not recommended within a Kubernetes cluster unless you can live with some data loss.
I'm fully aware of what synchronous_commit does. Here are my files you requested:
custom.conf
override.conf
postgresql.auto.conf
from cloudnative-pg.
gbartolini
commented on May 24, 2024
And what about the former primary?
from cloudnative-pg.
buroa
commented on May 24, 2024
And what about the former primary?
They look quite similar. I should note, nothing is particularly custom about my setup.
custom.conf
override.conf
postgresql.auto.conf
from cloudnative-pg.
clrxbl
commented on May 24, 2024
Have the same issue after deleting the PostgreSQL pods to update a service account. One of two instances can no longer boot back up. Downgrading cloudnativepg does not solve the problem.
from cloudnative-pg.
mnencia
commented on May 24, 2024
This bug is a regression caused by a bad interaction between #3535 plus #3545 and a bug in pg_rewind behavior
The patch in #3728 will fix it.
We will prepare a patch release next week to address the issue.
from cloudnative-pg.
gbartolini
commented on May 24, 2024
As a workaround until their systems are upgraded, can users enable the ALTER SYSTEM command for now @mnencia?
from cloudnative-pg.
mnencia
commented on May 24, 2024
@gbartolini, this is correct, but only if the bug has not yet hit. Once you see the error, the only simple procedure you can do is to delete the node.
from cloudnative-pg.
buroa
commented on May 24, 2024
Thank you! @leonardoce @gbartolini @mnencia
from cloudnative-pg.
Related Issues (20)
- Can cloudnative-pg support longhorn to make data sync instead of pg sync data itself in pg cluster, to support infinity disk capacity
- setup creadentials for preflight check in ci workflow
- [Bug]: Logger PGAudit not working, postgres logger is used instead HOT 2
- [Bug]: Kubectl Plugin displays default Operator Image version, when using ImageCatalog in Cluster CR
- [Bug]: CNPG 1.23 - cannot create Cluster auxiliary objects: expected pointer, but got invalid HOT 2
- [Bug]: Replica starts recovery from backup that is outside of retention period.
- [Bug]: Node restarts may render cluster broken HOT 1
- [Docs]: development env setup instructions don't support Apple Silicon
- [Docs]: Release notes for 1.23.1
- [Feature]: Add Container Lifecycle Hooks (PostStart,PreStop) for containers in the Cluster CRD
- [Bug]: Recovery from WAL fails when tablespace was added HOT 1
- [Bug]: New replica searches for WAL archive that doesn't exist
- [Feature]: ScheduledBackup resource should be able to backup using plugin method
- [Bug]: Operator restarting due to DetectAvailableArchitectures()
- [Bug]: Operator test should rely on Deployments not counting pods
- [Docs]: guide on production readiness for CNPG HOT 2
- [Bug]: Do not start the psql or minio container for tests if not needed
- [Bug]: The instance manager is not able to unfence PG after a previous unfence operation failed
- [Feature]: Specify addtional labels for PodMonitor
- [Bug]: Missing log line if only the private key is changed in a TLS secret
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudnative-pg.