Comments (2)
pgreplay doesn't block on select
, unless it decides there is nothing to do for a while, in which case it it sleeps for a while. But I guess that in your gdb stack trace, do_select
is called from poll_socket
, which uses a timeout of 0 for a non-blocking poll of the socket.
I'd assume that the problem is not that the system call gets stuck or takes too long, but that there is no response from the PostgreSQL server. pgreplay just keeps running select
in a tight loop if it is waiting for a response from the server, so it is normal for the system call to consume a lot of the CPU time.
Perhaps you are suffering from the following problem that is mentioned in the “limitations” section of the README:
While pgreplay makes sure that commands are sent to the server in the
order in which they were originally executed, there is no way to guarantee
that they will be executed in the same order during replay: Network
delay, processor contention and other factors may cause a later command
to "overtake" an earlier one. While this does not matter if the
commands don't affect each other, it can lead to SQL statements hitting
locks unexpectedly, causing replay to deadlock and "hang".
This is particularly likely if many different sessions change the same data
repeatedly in short intervals.
This can easily happen if many concurrent sessions change the same data concurrently, as can often happen in an artificial load test:
Session 1, time x: UPDATE t SET v = 1 WHERE id = 42;
Session 2, time x + 0.002ms: UPDATE t SET v = 1 WHERE id = 42;
Session 1, time x + 1ms: COMMIT
Session 2, time x + 1.5ms: COMMIT
Not pgreplay will faithfully replay the statements in that order and send the second statement 0.002 milliseconds after the first, then it waits for the first statement to return before it can send the third statement (because they use the same database session).
Now it can happen that the second statement gets executed first, since they get sent almost at the same time. Then session 2 has the row locked, and the update of session 1 is blocked. So pgreplay will wait in vain for the first statement to finish, and it won't send the fourth statement (which would remove the lock and allow processing to proceed) unless it has first sent the third statement.
In that situation, pgreplay has deadlocked with the database and is stuck. It has no way to detect that – it cannot tell the difference if the first statement is blocked or just takes a long time to complete. On the other hand, it cannot arbitrarily reorder the statements it sends, because they might depend on each other.
Here is a link to a thread on the (retired) mailing list that had that very problem. You could try the SQL statement mentioned there:
SELECT l.pid, pg_cancel_backend(l.pid)
FROM pg_locks l JOIN
pg_stat_activity a ON (l.pid = a.procpid)
WHERE NOT l.granted
AND current_timestamp - a.query_start > '3s'::interval;
If that gets replay unstuck, you are hitting this problem.
Unfortunately there is not much else that can be done about this, except to use a less artificial work load that does not modify the same table rows over and over in concurrent sessions...
from pgreplay.
Thanks for your (long) explanation! As there is nothing that can be done, I close this issue.
from pgreplay.
Related Issues (20)
- make install fails on Mac OS X HOT 5
- Unable to compile pgreplay on ubuntu 18.04 HOT 2
- pgreplay seems to have stopped suddenly when executing HOT 10
- Generating undo records by using pgreplay HOT 1
- pgreplay not working with RDS csvlog HOT 2
- Queries per second HOT 3
- Replay freezes, all queries stuck in ClientRead state. HOT 3
- replay and uuids HOT 1
- Will the program support pg11 onward? HOT 3
- Transaction Deadlocks on Replay HOT 7
- Invalid free on replaying server-side prepare "ROLLBACK; DEALLOCATE ALL". HOT 4
- Tests are failing after installation HOT 6
- First run always lags behind HOT 3
- connection to database failed for a specific session HOT 1
- Parsing line error HOT 12
- Upgrade from pg 12 to pg 14 HOT 2
- ./configure fails on pg15.0 HOT 2
- Opened connections HOT 2
- Will PQgetResult cause stat_stmt statistics errors? HOT 1
- docker build Fails with Error (from master, commit a2f7c113197ed5623b35f09bfc299aae971d56e8) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgreplay.