Cannot find Quartz library on MacOS Sonoma about vispy HOT 18 OPEN

Ok it is all starting to come back to me. @takacsmark what version of python are you using and where did it come from? I think conda-forge included patches for this issue in the Python that they distribute for a long time, but it has since been released upstream by Python. I'm having trouble tracking down what patch versions of Python 3.9+ include the fix for it, but any modern-ish version should be good.

My guess is that you @takacsmark don't have pyopengl installed and that must be (or so it seems) a requirement for this cocoapy code. If you remove the try/except do you get a dlopen issue or do you get a module/package not found error?

from vispy.

After installing pyopengl the monkey patch does not fire and the error is not thrown, my stable diffusion use case works as expected.

When I look at the libraries in REPL I get this:

❯ python
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import cdll, util
>>> print(util.find_library('objc'))
/usr/lib/libobjc.dylib
>>> print(util.find_library('CoreFoundation'))
/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
>>> print(util.find_library('AppKit'))
/System/Library/Frameworks/AppKit.framework/AppKit
>>> print(util.find_library('quartz'))
None
>>> cdll.LoadLibrary(util.find_library('quartz'))
<CDLL 'None', handle fffffffffffffffe at 0x103187280>
>>> print(util.find_library('CoreText'))
/System/Library/Frameworks/CoreText.framework/CoreText
>>> print(util.find_library('Foundation'))
/System/Library/Frameworks/Foundation.framework/Foundation

The find_library return value for quartz is None, it's loaded as <CDLL 'None', handle fffffffffffffffe at 0x103187280> it does no throw an error. Looks strange, though.

from vispy.

Very weird. What is the return value of util.find_library("quartz") and of util.find_library("Quartz")? I quick google doesn't reveal anything about a name change in the Sonoma changes. And you can see that it is trying to load with dlopen something called Quartz.framework/Quartz so it is finding something.

I don't have a mac, but I've asked some coworkers who do if they know anything about this. If anyone else reading this can help investigate that would be appreciated.

from vispy.

@djhoese these are the return values:

❯ python --version
Python 3.10.13
❯ python
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> print(ctypes.util.find_library('quartz'))
None
>>> print(ctypes.util.find_library('Quartz'))
/System/Library/Frameworks/Quartz.framework/Quartz

from vispy.

@takacsmark Can you make a pull request with your fix? We'll see how CI feels about this. Maybe this has been a suggestion for a long time and we've just been getting away with it and it was finally removed in Sonoma.

from vispy.

@djhoese looked into it and see that there was already a patch at the beginning of the file for Big Sur based on a Stackoverfow solution. I patched that patch to be in line with the Stackoverflow logic. Tested for my usecase, ok.

Created pull request 2549

from vispy.

Do you have a small snippet to reproduce this?
I'm on Ventura 13.6.2 and I have:

Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:41) [Clang 15.0.7 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> print(ctypes.util.find_library('quartz'))
None
>>> print(ctypes.util.find_library('Quartz'))
/System/Library/Frameworks/Quartz.framework/Quartz

So apparently this isn't Sonoma specific?

Edit:
Also, looking at that monkey patch, in my case:

>>> import OpenGL.GL
>>>

So it appears that the except branch of that logic won't even trigger...

from vispy.

Ok so maybe this is an edge case of an edge case. Thanks for checking @psobolewskiPhD.

from vispy.

My understanding is that the monkey patch was created for Big Sur based on the PR and the Stackoverflow question and should trigger also on Ventura. Can you check that?

Just to recap the issue; the monkey patch maps in the current prod release:

quartz --> Quartz.framework/Quartz, thus the code will try cdll.LoadLibrary("Quartz.framework/Quartz") and throws the error on Sonoma:

OSError: dlopen(Quartz.framework/Quartz, 0x0006): tried: 'Quartz.framework/Quartz' (no such file), '/System/Volumes/Preboot/Cryptexes/OSQuartz.framework/Quartz' (no such file), '/opt/homebrew/lib/Quartz.framework/Quartz' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/Quartz.framework/Quartz' (no such file), '/usr/lib/Quartz.framework/Quartz' (no such file, not in dyld cache), 'Quartz.framework/Quartz' (no such file)

So my understanding is this:

• quartz = cdll.LoadLibrary(util.find_library('quartz')) used to work before Big Sur and is left as is for backwards compatibility
• I have no way to test if the monkey patch was actually triggering after Big Sur, but either cdll.LoadLibrary(util.find_library('quartz')) or cdll.LoadLibrary("Quartz.framework/Quartz") worked
• Both cdll.LoadLibrary(util.find_library('quartz')) and cdll.LoadLibrary("Quartz.framework/Quartz") fail on Sonoma, so updated the monkey patch to reflect the Stackoverflow solution exactly.

from vispy.

Kind of related: Should we drop support for Big Sur and older?

https://en.wikipedia.org/wiki/MacOS_version_history#Releases

from vispy.

That's the thing, the monkey patch is gated by a Try:

For me Ventura 13.6.2 arm64, import OpenGL.GL doesn't throw an exception.
Python 3.11.4, 3.10.12 -- both from conda-forge.

I'll test some more this evening.

from vispy.

Is there a better way to test this? a snippet I could run?
Because on my Ventura Macs, I get:

>>> import OpenGL.GL
>>> 
>>> from vispy.ext.cocoapy import quartz
>>> quartz
<CDLL 'None', handle fffffffffffffffe at 0x1053f3050>

So that seems not correct? But I can run napari and e.g. turntable_box example just fine.
Edit: I do have pyopengl
Let me try a vispy only env: mamba create -n vispy python=3.10 vispy
Boom:

╰─ python                                                             (vispy) ─╯
Python 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:09:17) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenGL.GL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'OpenGL'

and

>>> from vispy.ext.cocoapy import quartz
>>> quartz
<CDLL 'Quartz.framework/Quartz', handle 3ad9bf508 at 0x10ebd9c60>

So that makes sense.
Which comes back to needing a proper test snippet that uses quartz

from vispy.

I'm using Python 3.10.13 in a venv created by stable diffusion webui using venv, not conda. pyopengl is not installed, indeed.

from vispy.

I don't even remember what cocoapy is used for. If you install pyopengl, without your fix, does your use case start working?

from vispy.

So...do we spend a ton of time investigating what the CDLL "None" really means or do we just say that this is a missing dependency and close this issue and the PR...or is there a way this can be documented better? I really don't understand what the cocoapy stuff is trying to do.

from vispy.

I do wonder if the line

>>> cdll.LoadLibrary(util.find_library('quartz') or util.find_library('Quartz'))
<CDLL '/System/Library/Frameworks/Quartz.framework/Quartz', handle 3aa411628 at 0x102b97910>

which seems correct versus the current:

>>> cdll.LoadLibrary(util.find_library('quartz'))
<CDLL 'None', handle fffffffffffffffe at 0x102b80410> 

But without a something that specifically tests this it's hard to say?

from vispy.

Your solution cdll.LoadLibrary(util.find_library('quartz') or util.find_library('Quartz')) works on Sonoma, too:

>>> cdll.LoadLibrary(util.find_library('quartz') or util.find_library('Quartz'))
<CDLL '/System/Library/Frameworks/Quartz.framework/Quartz', handle 3cd212088 at 0x102c8f370>

If PyOpenGL is not available though, then the monkey patch will run and it has an override for quartz. So the patch will fire, return `Quartz.framework/Quartz' and throw the error in LoadLibrary. To handle this we can keep the fix that I created in the pull request. My fix changes the logic of the patch, it's in line with the info on stackoverflow that seems legit, but testing on multiple versions of MacOs would be the safe path.

from vispy.

Also experiencing this issue and looking forward to a fix. Using an M1 Max.

from vispy.