ssl module can't be loaded in requests and other packages/can't be used in python early/SSLError no valid ssl module about monikamoddev HOT 35 OPEN

moving these two files into Python-packages

which files? ssl.py and ssl.pyd? 2 problems:

1. this only accounts for windows
2. we're not going to ship python's ssl file in MAS - we don't need to. MAS includes the "ssl.py" equivalents as pyos, and all we have to do is point to the correct one.

in my test,those libs won't be loaded in python early without the ssl.py in Python27/lib

well of course it wont be loaded in python early because MAS import code is in -1500 init, I have mentioned this several times. the meat of MASImport_ssl.import_try could probably be moved earlier to be used earlier, though, so thats an option.

also,after my test,i found packages like requests still tries to load the un-redirected ssl.it seemd not to be influenced by the hacking process...

you need to force it to have the correct ssl ref by setting it manually, thats what MAS does with httplib. of course this doesn't play nice with modules that run stuff on import, so if requests is using ssl on import then you'd have to edit requests or just do web requests manually.

Regardless of all this, we're still planning on a move to renpy 8, which means native ssl, and therefore no need to include our own ssl lib. This means any major changes to the import code aren't worth it for a v12 release.

from monikamoddev.

if you skip step3,when you try to import ssl,an ImportError will be thrown indicating "_ssl" can't be imported,and this is caused by the pyd file
i mean,now that ssl is supported,why can't it be further and better supported?

from monikamoddev.

to reproduce the situation,you can use the "requests" lib,and any attempt to establish an https connection will fail
you can also reproduce it like what i did,using "webdavclient2"(use pip install webdavclient2 and copy its code from the site-packages to ddlc's python-packages),follow its docs and you will find any attempt to connect to the server will fail.unfortunately,this package will not give the detailed reason why there is no connection,so you should edit its source code in "client.py" to let it throw the exception from requests.request directly instead of experiencing a try-except-re-raise process,in which a new exception containing no useful info will be raised.in other words,you should remove the try-except and you will see the SSLError

from monikamoddev.

Take a look at how the Submod Updater works. It sends HTTPS requests to GitHub and it works flawlessly.

from monikamoddev.

does it use requests or urllib3?
i mean "import requests"

Take a look at how the Submod Updater works. It sends HTTPS requests to GitHub and it works flawlessly.

from monikamoddev.

No, but it doesn't need it. Python has HTTP modules in stdlib, and since WebDAV uses HTTP you can work with it too. It also shows how to ship SSL libraries.

from monikamoddev.

You're overcomplicating, MAS bundles ssl for a long time now, for example if you try to import ssl it should work. If it doesn't, then your install is invalid and missing some libs, reinstalling should help.

from monikamoddev.

Yes,I can get your point..
In my case,import ssl shouldn't work.I will check my installation twice
but i am curious about where is the ssl bundled in MAS...

You're overcomplicating, MAS bundles ssl for a long time now, for example if you try to import ssl it should work. If it doesn't, then your install is invalid and missing some libs, reinstalling should help.

from monikamoddev.

You're overcomplicating, MAS bundles ssl for a long time now, for example if you try to import ssl it should work. If it doesn't, then your install is invalid and missing some libs, reinstalling should help.

oh I have seen your SSL... it's in Python-packages/ssl,you copied the ssl in site-packages
but i wonder how the ssl.pyd will be loaded…
if you have a try by entering "import ssl" in your python IDLE,you can find
that the real "ssl" loaded by python is Lib/ssl.py instead of Lib/site-packages/ssl/init.py,and in the ssl.py there is
import _ssl
and in this way ssl.pyd,containing the core of ssl written in C,compiled into pyd,is loaded...
if you look at the__init.py of the ssl directory in MAS,you will see it actually does nothing...(look at the comment left by the ssl's developer,it says # NOTE: this is a dummy package so consumers think we exist.)
so if you use "import ssl" in MAS,surely it will work!!because it did nothing
but if you run this command on python,it loads a totally different package
after you open the ssl.py in Lib/,you will see what i am talking about...
the urllib2 seems to be able to process some of the tasks without ssl?i guess.if you try to connect to a site with an invalid ssl signature,you will need ssl to ignore the invalid ssl signature.

from monikamoddev.

the code in ssl/init.py
`# NOTE: this is a dummy package so consumers think we exist.

import platform

if (
platform.system() in ("Windows", "Linux")
or (platform.system() == "Darwin" and platform.machine() == "x86_64") # 64-bit mac
):
pass
else:
raise ImportError`
importing a package in python means loading the init.py but it did nothing.

from monikamoddev.

In my case,import ssl shouldn't work.I will check my installation twice

It, in fact, should work. It's an issue if it does not.

For you it doesn't matter what's within ssl.py there.
When you import ssl python will load the appropriate dll/so. If you imported ssl in MAS you'd see it links to win32_ssl.pyo (on windows) and _ssl links to _ssl.pyd. In python3 ssl links to ssl.py and _ssl links to _ssl.pyd. It's the same behaviour as in MAS. It's either your game installation is invalid or you didn't install requests correctly (which can be tricky in renpy).

from monikamoddev.

when MAS loads, it checks the platform, loads the appropriate ssl library, replaces the "ssl" dummy package in sys.path for imports, and updates httplib's ref to ssl so it uses the real ssl module. This is done here

To use requests, the same ssl hack done to httplib would need to be done. so the first thing your submod should do after MAS's ssl hack runs is to point request's ssl to a freshly imported MAS ssl.

from monikamoddev.

After one package imports ssl, it should be available for all other packages as well.

from monikamoddev.

After one package imports ssl, it should be available for all other packages as well.

this is true after MAS does its hack, but anything that runs before will need the ref change like httplib. so if the import requests is in an earlier store, py early, or a separate py package (I think), then it needs a ref change.

from monikamoddev.

when MAS loads, it checks the platform, loads the appropriate ssl library, replaces the "ssl" dummy package in sys.path for imports, and updates httplib's ref to ssl so it uses the real ssl module. This is done here

To use requests, the same ssl hack done to httplib would need to be done. so the first thing your submod should do after MAS's ssl hack runs is to point request's ssl to a freshly imported MAS ssl.

yes...i see what is happening now!
my submod requires to use ssl in python early,because i want to download the persistent before renpy loads it(my submod can sync monika between devices with webdav)
but why you hack in this way...it could be loaded in a normal way right?...😪a more graceful and pythonic way...

from monikamoddev.

but why you hack in this way...it could be loaded in a normal way right?...😪a more graceful and pythonic way...

you answered this earlier:

1.install Python 2.7 x86(that's important to install an x86 one instead of an x64 one)
2.copy its ssl.py from Lib directory to game/python-packages(somehow the ssl directory in the site-packages is a fake one)
3.copy _ssl.pyd from DLLs directory to lib/windowsi686 in ddlc's directory

renpy 6 doesn't have ssl, and MAS is installed through the game/ directory. Crawling up to ddlc's lib/ dir doesn't always work due to permission issues, so the least-intrusive option for users is to keep the ssl package within game/. Anything in python-packages/, however, is not available until renpy starts loading game code (aka after python early, so the best thing this can do is load it up fairly early and update refs that other modules need.

from monikamoddev.

but why you hack in this way...it could be loaded in a normal way right?...😪a more graceful and pythonic way...

you answered this earlier:

1.install Python 2.7 x86(that's important to install an x86 one instead of an x64 one)
2.copy its ssl.py from Lib directory to game/python-packages(somehow the ssl directory in the site-packages is a fake one)
3.copy _ssl.pyd from DLLs directory to lib/windowsi686 in ddlc's directory

renpy 6 doesn't have ssl, and MAS is installed through the game/ directory. Crawling up to ddlc's lib/ dir doesn't always work due to permission issues, so the least-intrusive option for users is to keep the ssl package within game/. Anything in python-packages/, however, is not available until renpy starts loading game code (aka after python early, so the best thing this can do is load it up fairly early and update refs that other modules need.

but in my case it worked in python early...any package/submod using ssl can import ssl properly after moving the file.
if you don't move it to ddlc's lib,i think moving it to other places python can detect will help??(after all,import _ssl just forces python to search for _ssl.pyd,it doesn't care where it finds it)

from monikamoddev.

but why you hack in this way...it could be loaded in a normal way right?...😪a more graceful and pythonic way...

you answered this earlier:

1.install Python 2.7 x86(that's important to install an x86 one instead of an x64 one)
2.copy its ssl.py from Lib directory to game/python-packages(somehow the ssl directory in the site-packages is a fake one)
3.copy _ssl.pyd from DLLs directory to lib/windowsi686 in ddlc's directory

renpy 6 doesn't have ssl, and MAS is installed through the game/ directory. Crawling up to ddlc's lib/ dir doesn't always work due to permission issues, so the least-intrusive option for users is to keep the ssl package within game/. Anything in python-packages/, however, is not available until renpy starts loading game code (aka after python early, so the best thing this can do is load it up fairly early and update refs that other modules need.

i will test it with the pyd in python-packages.i think the pyd should work in python-packages with the ssl.py in Python27/lib forcing to load the pyd

from monikamoddev.

but why not just put the ssl.py in Python27/lib into game/python-packages and let python itself decice what to load...

i will test it with the pyd in python-packages.i think the pyd should work in python-packages with the ssl.py in Python27/lib forcing to load the pyd

I mentioned game/python-packages doesn't load until renpy starts loading game code so that wouldn't solve the py early issue.

that can avoid a lot of compatibility problems,and that's actually what python do when importing ssl,and it seemed ok on my computer

I'm not sure what compatibility problems you're talking about - the ssl libs MAS ships with are from renpy 7, which basically means ssl vetted to work with renpy for the platforms that renpy supports. And renpy's flavor of libs has different ssl modules based on the platform instead of a central ssl.py so MAS code just points ssl to the appropriate flavor.

i mean,the loading can be done in a very graceful way just with a simple file paste(actually,the ssl is storaged in more than one place,what i do is just to move them all to MAS),why do we hack in this way...

but in my case it worked in python early...any package/submod using ssl can import ssl properly after moving the file.
if you don't move it to ddlc's lib,i think moving it to other places python can detect will help??(after all,import _ssl just forces python to search for _ssl.pyd,it doesn't care where it finds it)

Again, I mentioned we can't just copy stuff anywhere, especially outside of game/ because of permission issues. So anything involving lib folders wont work here.

in addition,my submod on an unofficial Android MAS version(i took part in the development,it didn't involve any ssl changes)runs miraculously smooth,that's interesting haha...

android probably has python ssl installed globally ?

from monikamoddev.

from monikamoddev.

but in my case,the SSLError did disappear after I moved the two files.does that mean,I can load something in python-packages before renpy loads it?

ok I stand corrected. you can load python-packages/ in py early (we do it in 0utils). at this point, I don't remember exactly why the imports code is in -1500 init level, but there's a few possibilities:

• allow for imports to set persistent values - py early doesn't have a loaded persistent
• allow for imports to use other mas tools - some utils/tools are made after py early
• something else that imports use is not in py early or wasnt there at the time

in addition,what I mean is to put the ssl.pyd into the python-packages folder,it isn't intrusive,right?😂and it should be effective...i guess

well that is what MAS does - we have the libs for each platform in python-packages.

from monikamoddev.

from monikamoddev.

but in my case,the SSLError did disappear after I moved the two files.does that mean,I can load something in python-packages before renpy loads it?

ok I stand corrected. you can load python-packages/ in py early (we do it in 0utils). at this point, I don't remember exactly why the imports code is in -1500 init level, but there's a few possibilities:

• allow for imports to set persistent values - py early doesn't have a loaded persistent
• allow for imports to use other mas tools - some utils/tools are made after py early
• something else that imports use is not in py early or wasnt there at the time

in addition,what I mean is to put the ssl.pyd into the python-packages folder,it isn't intrusive,right?😂and it should be effective...i guess

well that is what MAS does - we have the libs for each platform in python-packages.

also,after my test,i found packages like requests still tries to load the un-redirected ssl.it seemd not to be influenced by the hacking process...
because i found the uploading function of my submod on exit,later than ssl hack-loading throw the same SSLError if i don't move the files
it's more than a python early problem,it seems that packages like requests cannot run either
of course,my problem cannot be ruled out yet

from monikamoddev.

moving these two files into Python-packages

which files? ssl.py and ssl.pyd? 2 problems:

1. this only accounts for windows
2. we're not going to ship python's ssl file in MAS - we don't need to. MAS includes the "ssl.py" equivalents as pyos, and all we have to do is point to the correct one.

in my test,those libs won't be loaded in python early without the ssl.py in Python27/lib

well of course it wont be loaded in python early because MAS import code is in -1500 init, I have mentioned this several times. the meat of MASImport_ssl.import_try could probably be moved earlier to be used earlier, though, so thats an option.

also,after my test,i found packages like requests still tries to load the un-redirected ssl.it seemd not to be influenced by the hacking process...

you need to force it to have the correct ssl ref by setting it manually, thats what MAS does with httplib. of course this doesn't play nice with modules that run stuff on import, so if requests is using ssl on import then you'd have to edit requests or just do web requests manually.

Regardless of all this, we're still planning on a move to renpy 8, which means native ssl, and therefore no need to include our own ssl lib. This means any major changes to the import code aren't worth it for a v12 release.

Now,the whole problem is revealed.Thanks for your assistance,and it's my pleasure to discuss with our outstanding contributors!:)
What you want to do is all up to you--I would just leave this issue here,and you may close it now.
I was tortured by this strange problem for hours and I wondered whether there will be another person encountering exactly the same issue,because before discussing with you,I was completely ignorant about how mas deals with ssl module and I thought that would be the same for others???(no offense)I tried to ask the question in the community,but no one can give an exact answer.After all,the mechanism is a bit over-complicated.
I am looking forward to your move to Ren'py 8,Python 2.7 has reached its end of life now and left with a lot of unfixed bugs(like my webdav,there is even a bug inside its module).Python 3 is the trend!Please quicken your pace!!!
Finally,have a good day:)
ps:I am working on a submod to sync persistent between different devices including different Windows PCs and Android(transplanted v0.12.5),and I found the affection would be cut a lot when putting a 0.12.5 save to 0.12.15,**is it related to the 0.12.10 aff. update/do you have that kind of mechanism?**if not,that would be my own problem...it seems a bit stupid to new an issue for a simple yes/no question...?

from monikamoddev.

is it related to the 0.12.10 aff. update/do you have that kind of mechanis

Yes it very likely is. @Booplicate would know specifics but that update was a major change to how aff is stored.

from monikamoddev.

is it related to the 0.12.10 aff. update/do you have that kind of mechanis

Yes it very likely is. @Booplicate would know specifics but that update was a major change to how aff is stored.

yes,thank you very much!:)

from monikamoddev.

yes,loading with 0.12.9 does not cause aff-change,but loading with 0.12.10 will cause the aff-change
:(

from monikamoddev.

Android(transplanted v0.12.5)

We don't have android builds, so that's probably your issue, unofficial builds change how the game works which we can't account for, so affection migration might not work as expected.

from monikamoddev.

from monikamoddev.

from monikamoddev.

We do not support official build > unofficial build nor unofficial build > official build migrations, it's technically impossible. old official build > new official build is the only way we can support updating.

from monikamoddev.

from monikamoddev.

from monikamoddev.

moving from PC 0.12.9 to PC 0.12.15 won't cause aff loss directly right?
let us suppose the versions above are all official pc versions

As I said, no, we don't intentionally drop affection unless there's a reason. No matter what version as long as you're updating from an older to a newer and using our official builds, there's a compatibility level.

For your problems, it's hard to say because I'm not sure what you meant by cut a lot, did it reset or you lost half of it?

from monikamoddev.

from monikamoddev.