Comments (5)
Any chance your blocking function isn't releasing the GIL?
from websockets.
Well. Maybe, how would i know?
Looking at the doc:
some extension modules, either standard or third-party, are designed so as to release the GIL when doing computationally intensive tasks such as compression or hashing. Also, the GIL is always released when doing I/O.
And I'm doing multiple i/o tasks at the time of block (reading from mssql db into pandas dataframe using pandas.read_sql).
Could this be worked around, no?
from websockets.
A relatively straightforward way to tell is:
- Start an asyncio task that prints something every 100ms e.g.
async def count():
for i in range(1_000_000):
print(i)
await asyncio.sleep(0.1)
loop.create_task(count())
- Run the function that may block (i.e. that calls
pandas.read_sql
) viarun_in_executor
async def will_it_block():
await asyncio.sleep(1) # check that count() started
print("will it block?")
await loop.run_in_executor(None, blocking_function, bf_arg)
loop.create_task(will_it_block())
If count()
stops counting, then it's purely a pandas / asyncio problem — nothing that I can fix at the level of websockets.
from websockets.
Ok, good point. I've implemented that test and it seems it runs fine - i can see counter running together with blocking task.
However, I should've also pointed out that i have few background tasks created with asyncio.create_task that are also running while blocking operation is running, but websocket still stops receiving any data (pings and other messages included) while the blocking operation is ongoing..
As for test you proposed, i did a slight variation, like this:
loop.create_task(count()) # count() function is like the one you wrote
await asyncio.sleep(1) # check that count() started
res = await loop.run_in_executor(None, (lambda payload, tunnel: Action(payload,tunnel).call_action()), payload, tunnel)
print("blocking function completed")
return res
So i did not create two tasks.
I have to point out, however, that at some point counter stopped counting. I've then adjusted pandas chunksize and after that, counter did not stop at any point, so it appears that it blocks at the time when pandas is assembling a large dataframe.
Currently, it appears to work, thanks! I did not know about GIL before you mentioned it. Now i can find a way to work around it when needed for some future purpose.
from websockets.
If I understand correctly, the problem was that you were blocking the event loop due to a chunksize too large, blocking the entire Python process (incl. the event loop).
On the side of websockets, you can also adjust ping_timeout
if the default of 20 seconds isn't enough.
from websockets.
Related Issues (20)
- 0 bytes read on a total of 2 expected bytes HOT 2
- Secure websocket using example code unable to handle unsecured requests HOT 4
- No route to host----when access IPv6 address failed,no attempt was made to access an IPv4 address HOT 1
- How to close a client connection using the threading interface while stuck in ClientConnection.recv
- Can we get more attention in the threaded client interface? HOT 1
- Human-readable representation of a frame is too short HOT 2
- No module websockets.imports HOT 2
- Sync client stops process exit when running in a Thread if not explicitly closed HOT 5
- Python 3.11 incompatible with websockets\legacy\protocol.py Error "got an unexpected keyword argument 'loop'" HOT 1
- Strange problem, client cannot receive messages from the server, after sending a certain amount of messages. HOT 2
- Cannot Send message to a single client.
- handling multiple websocket client connections HOT 4
- Enable to connect ! invalid handshake HOT 7
- recv() is slow HOT 5
- Client state is incorrect HOT 1
- Feature batch receive HOT 2
- [QUESTION] Switching from websocket-client to websockets HOT 1
- Sync version of broadcast HOT 2
- Running Scheduled Task in Websokcets Or Scaling with multiple workers HOT 1
- await websocket.recv() block the coroutine loop. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from websockets.