[Question] Merge this POC to CPython about nogil HOT 17 OPEN

Here's the PEP: https://peps.python.org/pep-0703/

from nogil.

@manueldeprada I'm currently working on a PEP (Python enhancement proposal). When it's ready, it will be posted to https://discuss.python.org/c/peps/ and I'll share a link to it here.

from nogil.

@redradist No, it won't be added to CPython 3.11. CPython 3.11 will have a "beta" release soon, which means a feature freeze (bug fixes only).

At the Python language summit this week I suggested integrating some of the "nogil" ideas into CPython 3.12 behind a compile flag. It's not clear if that will happen -- there's still a lot of open questions.

Having additional real use cases may help with adoption upstream. So if you have a program or use case that would benefit from removing the GIL, please try it out and report back both any problems you encounter and the results.

from nogil.

This should get into CPython. I'd love to have it in 3.11 with the fastness and this parallelism. Thanks @colesbury for such a detailed work. Appreciate all the efforts. Hope it's rewarded with a merge soon.

from nogil.

Hi, has your interesting PEP finally been accepted for integration into Python 3.12?

After a lot of drama back and forth, it has been accepted by the SC🎊🎊

https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

Most discussion threads have been closed, the only I see now is: https://discuss.python.org/t/python-abis-and-pep-703/34018

but there is not a central active thread as far as I know. Probably, it will make it into 3.13😁😁

from nogil.

Is there any updated page where we can track what is being incorporated into Python 3.12??

from nogil.

Is there any updated page where we can track what is being incorporated into Python 3.12??

The what's new document keeps track of these things, the current one for 3.12 is available (and continuously updated).

Thank you. I see it's nowhere in the roadmap for 3.12. I was asking more for something like a link where we can follow the status and discussion of the NOGIL integration into Python.

Right now, I could only find some minutes from core-dev meetings and discussions on the mailing list.

It would be great if there was a central thread somewhere that tracked the discussion and status of the NOGIL integration into CPython.

from nogil.

Hi. It is really nice to see such a dedicated work for this. I want to contribute my real use case as follow:

Taken the inspiration from LMAX Disruptor written in Java, we (devs in a stock brokerage company) are developing in-memory queue system similar to Disruptor that heavily utilize shared-memory and concurrent model to achieve low-enough latency.

1. Producer continuously put data into the shared buffer/shared list, but we construct it to be ring-buffer liked
2. Consumers continuously poll for data from the shared buffer and consumer it
3. No lock required if there is single producer
4. Producer increase its offset when producing new data to shared buffer. Consumers observe it by polling to get producer's offset value (while True loop)
5. Producer also check for consumers offset to avoid wrapping the slowest consumers

To implement this, currently we are using multiprocessing module with SharedMemory class, each Consumer/Producer lives on separated process. Our producer and consumers are long-running processes.

Currently, due to the GIL, we are limited at communicating between producer and consumers (most likely sharing offset position). Consumers and producer using multiprocessing.Value to share offset value, and that is not the efficient way to observe those values.

In multithreading model, I believe those operations will be more natural in the code, less work-around, and also we get more performance since threads share the same context, no need to pickle/copy data around.

from nogil.

I really appreciate the dedications to this work and it will be so nice to see it merge back to upstream. We should definitely push on this once things mature.

from nogil.

So say we all, but the timing is right: lot of people and money are working on python 5x speed-up goals.

from nogil.

@colesbury Looks like you are doing lots of work for updating to up-to-stream CPython ...
Will this nogil be added to CPython 3.11 ?

from nogil.

I've been looking forward to it.

from nogil.

and 3.12 is in development. I know that the summit it could add a compile option for nogil, but are there any updates on it?

from nogil.

Is there any updated page where we can track what is being incorporated into Python 3.12??

The what's new document keeps track of these things, the current one for 3.12 is available (and continuously updated).

from nogil.

Here's the PEP: https://peps.python.org/pep-0703/

For python-3.12 ?

from nogil.

from nogil.

Hi, has your interesting PEP finally been accepted for integration into Python 3.12?

from nogil.