HTTP Error 403: Forbidden after update -os windows -lang en de -skipknown about gogrepo HOT 48 CLOSED

I blame the Recaptcha, it is really hard to see the images, detect cars and such and then tick the box.
Btw. if you just remove the IOError bit from the code, it then runs boths and you can easily import netscape cookies:

def load_cookies():
    # try to load as default lwp format
    try:
        global_cookies.load()
        return
    except:
        pass

    # try to import as mozilla 'cookies.txt' format
    try:
        tmp_jar = cookiejar.MozillaCookieJar(global_cookies.filename)
        tmp_jar.load()
        for c in tmp_jar:
            global_cookies.set_cookie(c)
        global_cookies.save()
        return
    except:
        pass

    error('failed to load cookies, did you login first?')
    raise SystemExit(1)

from gogrepo.

I was able update my library by using find/replace of expires="2019 with expires="2020 on my gog-cookies.dat file.

That won't work for cookies from Firefox, though, because those use the Unix epoch instead of a human readable date. In that case, you would have to calculate a new timestamp.

from gogrepo.

The issue here is (probably) that your session has expired and GOG requires completing a reCAPTCHA, which gogrepo doesn't support. You can get around this by copying your GOG session cookies from your browser as described in #17 (comment)

That solved it for me.

from gogrepo.

I had (to be sure) first deleted all cookies of gog, then logged in. I used the menioned cookies.txt extension to get the cookies for copy'n'paste. Indeed, that format is different from what the python code uses. But it has the same info. E.g., the displayed ones are
.gog.com TRUE / FALSE 1576850035 gog_set some_cookie_value
1576850035 is the unix timestamp code, corresponding to 20.12.2019 13:53:55UT.
I converted those to the format gogrepo.py wants,
Set-Cookie3: gog_set="some_cookie_value"; path="/"; domain=".gog.com"; path_spec; domain_dot; expires="2019-12-20 13:53:55Z"; HttpOnly=None; version=0
Doing that for all gog cookies allows the script to run...

from gogrepo.

OK!!!111elevenoneone
Use Chrome + EditThisCookie Extension
Logon to GOG.com and keep the page open, in EditThisCookie -> Options -> change the format to Netscape
Then export the Cookie to Clipboard with the Extension
Then Open your gog-cookies.dat and replace all with the copied cookie (paste) and save.
Open gogrepo.py, (line 233 fff) comment the complete cookie load procedure and save, so it fails over to the Netscape cookie load attempt:
def load_cookies():
# try to load as default lwp format
#try:
# global_cookies.load()
# return
#except IOError:
# pass

# try to import as mozilla 'cookies.txt' format
try:
    tmp_jar = cookiejar.MozillaCookieJar(global_cookies.filename)
    tmp_jar.load()
    for c in tmp_jar:
        global_cookies.set_cookie(c)
    global_cookies.save()
    return
except IOError:
    pass

error('failed to load cookies, did you login first?')
raise SystemExit(1)

After the first successful run/load of gogrepo.py, it will regenerate a cookie it can handle, so remove the # to un-comment the lines again. Save and be happy

from gogrepo.

At least its a consistent method of making it work. However, to close this issue, we should identify the source of the problem. I am not sure how the google captcha is implemented into the login process (never got one), therefore I am not sure how to find it. Maybe a specific error throwing on login when google captcha is requested would be a good start.

from gogrepo.

I could do it, however, I am not skilled in the Libraries used. For the moment, I will try to reimplement the login with different libraries (requests, BeatifulSoup, eventually robobrowser). I will give feedback when I have reached something, maybe I'll create a fork then.

from gogrepo.

Anything new on this case? Just read, that gog seems to have financial problems.... Would make an "offline backup" of my current (huge!) lib even more usefull but sadly I can't get this script to work again with the proposed workarounds...

from gogrepo.

from gogrepo.

edit: forget it, I realized there is a newer fork by Kalanyr. Saving the cookies as cookies.txt (not gog-cookies.dat) and using his version did the trick :)

Hm? Does that mean that the Kalanyr fork is working? Can you explain what you did exactly?

from gogrepo.

same problem

from gogrepo.

I use a freshly created cookie from the script the same error occurs here unfortunately (See #38). Under this issue there seem to be some people facing this issue.
I'm not sure whether its about the recaptcha, my account never got one before on any system, including the one using the gogrepy.py login request. If there a quick way to test/verify the cookie?

from gogrepo.

Working here for me. Thanks!

from gogrepo.

After copy pasting the values from my cookies.txt file into the gog-cookies.dat, the error 403 remains. Copying was necessary because the script somehow deleted my values from the firefox cookies.txt plugin. Instead, I manually looked up what values were captured in the original gog-cookies.dat, searched them in the downloaded cookies and copied the cookies value (the scrambled number/text thingie) into the gog-cookies.dat, replacing the old one. The following cookie values were present, maybe there is something missing?
gog-al
gog_lc
gog_set
galaxy-login-al
galaxy-login-s

from gogrepo.

I have two additional ones, gog_us and csrf. Not sure if they are important. So I created a fresh gog-cookies.dat from my (Vivaldi) Cookies. Had to convert the expiry date that the cookies.txt extension reports from Unix timestamp to a normal date, but now it indeed does work again.

Probably the easiest solution might be to go to GOG settings and disable the two-step login, so it should not use/send a captcha/confirmation. Have not tried it though, now that it works.
Maybe next time the cookes get invalid (9-12 months or so?)

from gogrepo.

Hmm just tried to do the same as proposed:

• logged out in gog (browser)
• relogged (no captchas here!)
• exported the cookie.txt with the cookies.txt addon (firefox)
• compared both files (cookie.txt and that gog-cookies.dat file)
  Noticing that both files are totally different both in structure and content.

When i replace that files as described here i get a "AttributeError: 'str' object has no attribute 'search'" which was expectd because the file contents are completly different.

from gogrepo.

@WisdomeCode also used Firefox, so I'm guessing there's a difference in the way it represents the cookies between browsers.

from gogrepo.

@DerPit The Idea of converting the timestamp is interesting, how did you do that in a quicker fashion than typing it into some webinterface and back?
Beside of that, I did about what @DerPit did, with the exception that I didn't care for anything other than the "some_cookie_value" section. The other parts should either be identical or represent a sent cookie by gog, either way there are no reasons to assume that there should be any problem there (as long as the expiry date is far enough away). Please correct me if I'm wrong here.
@Shiryou That may be an explanation, however one that I cannot base on any technical basis. A cookie is just a String of letters and numbers, no reason to assume that Firefox somehow gets different cookies than Vivaldi or Chrome, especially if you know that the script basically has no browser typical behavior beside cookies at all.
I will try to add the gog_us and csfr cooky later and report back if that made it work.

from gogrepo.

@WisdomCode: I had a local tool (from some other project). But just as good (if you're in Unix environment) is
date -u --date='@1576850035' +%Y-%m-%d\ %TZ

from gogrepo.

Any hints to solve this bug? I have the same problem...

from gogrepo.

Probably the easiest solution might be to go to GOG settings and disable the two-step login, so it should not use/send a captcha/confirmation. Have not tried it though, now that it works.
Maybe next time the cookes get invalid (9-12 months or so?)
Unfortunately this does not work :(

from gogrepo.

I've always deactivated the two-step thing...
I have deleted the old cookie file and created a new.
But nothing helps...

from gogrepo.

The two factor authentication is not the problem, this is already in the script. It is the google recaptcha, where you click "I am not a Robot".

from gogrepo.

I have never seen a "I am not a Robot" window while login... It seems all is turned off here.

from gogrepo.

Even after setting the other cookie values, error 403 persists. As the cookies captured by the script contain all the important values (I assume gog-al and gog_set to be the important ones, however it doesn't matter), I assume that the login was successful, despite error 403.
Like @Lebostein also said, Google captcha was never a thing for me and should probably end with no cookie sent at all, the login procedure itself would probably fail.
Cokkies ARE send, so it would be best to test them in some way. Is there a way to apply the cookies from the .dat file into the browser (basically the opposite of what cookies.txt does), so that we can see whether the opening of gog will result in being already logged in (assuming that there were no cookies before of course, a cleaned up browser is needed)? Will this even work when this error does not occur with the gog-cookies.dat?

from gogrepo.

@disi: YES! That's ist. It works. Thansk for your investigation!

from gogrepo.

Wow, so it did for me, thanks!
Would you mind to explain what exactly is going on here? Is it something about the cookies format?

from gogrepo.

gloated too soon... it seems it works only for round about 3 hours, then it seems the cookie expired and the error comes back...

from gogrepo.

I also have never seen a captcha when loggin in via browser, so I think it's something different.
Nevertheless, I found two tools. First the browser extension Export Cookies
0.3.0, and then a small perl script from https://stackoverflow.com/questions/27886991/convert-netscape-cookies-to-lwp-perl. I adapted the filenames in the script, so now it's only a three-step thing: Log in in browser, click the 'export cookies' for gog.com, and run the perl script. With that, gogrepo.py works without changes.
Have not yet checked how long it stays valid - there's two cookies (csrf and gog_us) that have a close-by expiry date. And a new one I hadn't seen before, galaxy-login-tsa from login.gog.com.

from gogrepo.

Doesnt seem to work for me (getting "failed to load product data (are you still logged in?)") now when using your workaround (can't execute perl scripts so i used the Chrome Plugin Edit This Cookie which is able to export cookies in LWP format).

Edit:
compared to the .dat file generated by gogrepo.py there are at least 2 gog-galaxy related cookies missing using the "browser method".

from gogrepo.

Some more observations. After some time I had tried connecting to gog again using the cookie file created as described in the previous post. Would get the 403 again :(
So I re-ran the described sequence(*) and with the new generated gog-cookies.dat connection worked again. So I compared the two (old and new) cookie files. 4 cookies were unchanged:
gog_set, galaxy-login-al, galaxy-login-s and galaxy-login-tsa.
gog_lc and csrf have same values, but new expiry dates (but for gog_lc that is 1 year in the future)
gog-al and gog_us have new values (and expiry date).

side note: The cookies come from a different computer. I run browser and conversion script on the desktop machine, and copy the generated gog-cookies.dat to my laptop where I run gogrepo.py

from gogrepo.

Now the proposed method with chrome which was working previously is not working anymore either...
Is this repository still maintained?

from gogrepo.

You mean via the cookie export? For me it does still work (Vivaldi with Export Cookies 0.3.0).

Looks someone with knowledge in both Python and HTML is needed - I suck at both :((

from gogrepo.

from gogrepo.

Well, I rewrote the Login code, I investigated every step and every loaded site. Everything, however, looks absolutely fine. The cookies still are not usable for the script. There were a handful of oddities though:

1. There is already a recaptcha error implemented,there is an error message when it occurs. I made this search even a bit thorougher, risking false positives to absolutely be sure.
2. There is a login_flow in the login process, which is set default, but the login value is empty. Corrected that.
3. The Login request is a post request, not sure if gogrepo is doing that.

None of the code contains a captcha, the last url indicates to a successful URL, and as we already discovered, the cookies look fine too. This makes me sure by now that its not a problem with the login process, but with the usage of the cookies. I may post the Code if anyone wants to investigate further

from gogrepo.

Now I made another interesting discovery that may actually explain it:
If you login under
https://login.gog.com/login
(which is basically the url of the login which is embedded into the actual GOG page). The login is entirely done via the login.gog.com subdomain, this is also true for the script. Logging in there however leaves you not logged in on the original page (of course given your not logged in already). Maybe somebody could confirm this phenomenom? If this is the case, the cookie is probably simply not handed to the main webpage, rendering them useless.

from gogrepo.

After logging in via https://login.gog.com/login , opening another tab at https://www.gog.com/ , results in 'SIGN IN' still visible at the top.
This happened the majority of the login attempts.
So far, only three times has that login page (after inputting credentials) redirected to a logged in gog.com main page.

from gogrepo.

from gogrepo.

Well, I am no real expert to cookies, but I can tell that pythons internal structure on acquiring the cookies is no different than the one the browser uses. No Cookies are directly dropped, and the login process is even quicker than the manual one, so no reason to assume they time out somehow.

I believe (and this is the part where I am not sure) that the embedding of the login page into the main page results in a different kind of cookie. Maybe the same values are given, but the domain of the cookie is different or something like that. Maybe somebody has experience with the netscape format of the cookies and these values in general? Is something like that possible.

from gogrepo.

I have the same problem.

@Ulathar where did you hear about gog having problems? Any link?

from gogrepo.

A good idea to look up how lutris is doing it.
Never actually tried it that way, but now after trying, I can say 2 things:

1. It doesn't work for me either. I enter username and password, enter two factor auth, and... nothing. The window disappears the window previously downloading wine is still stuck there at 100%, no network activity on the device. I fear the same problems are faced there, although I could not find an issue in their tracker. Unfortunately I cannot find the location of the log files to harden this suspicion.

2. The system used differs at least partly. I think it uses electron to embed the website into the local app run in python, which is quite different from what is done here. The result will however probably be the same: Cookies get extracted and used for downloading the application.

Maybe somebody can tell me where to find the log file? Google Fu fails me here...

from gogrepo.

No idea about lutris, I failed to get 0.5 working.
But another possibility to look at might be lgogdownloader.
I just compiled and tested the latest git version, and I'm able to log in and download things(*). But I think it uses the gogdownloader/galaxy api, not sure if that is something different.

(*)Then again, for me also the transfer-cookies-from-browser solution with gogrepo.py does work.

from gogrepo.

I have the same problem.

@Ulathar where did you hear about gog having problems? Any link?

I read an article which said that they had to lay off 10% of their employes alongside with the already mentioned cut in their "fair price" program...

The metioned lgogdownloader is used by one of my colleagues and seems to be working. But it only runs on linux so i can't use it myself.

from gogrepo.

I can confirm, the lgogdownloader works for me (on Linux). Maybe one can create a Dockerfile to make use of it on a different platform?

I only installed it. Then i launched it simply with lgogdownloader and logged in with my data. Afterwards lgogdownloader --directory ./backup/ --download downloads all the files.

from gogrepo.

Since this method works for a while:
#39 (comment)
I think we need to find a way to keep the cookie alive longer. Kalanyrs fork uses session classes and seems to run longer, but also fails to login and generate a usable cookie, also it has some other design decision I don't like.

from gogrepo.

When creating the gog-cookies.dat using the EditThisCookie Extension and editing gogrepo.py as mentioned above, I get the following error:

fatal...
Traceback (most recent call last):
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 1167, in <module>
    main(process_argv(sys.argv))
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 1142, in main
    cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.id)
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 605, in cmd_update
    load_cookies()
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 242, in load_cookies
    tmp_jar.load()
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Anaconda3_64\lib\http\cookiejar.py", line 1784, in load
    self._really_load(f, filename, ignore_discard, ignore_expires)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Anaconda3_64\lib\http\cookiejar.py", line 2004, in _really_load
    if not self.magic_re.search(magic):
AttributeError: 'str' object has no attribute 'search'

Is anybody else getting this error?

edit: forget it, I realized there is a newer fork by Kalanyr. Saving the cookies as cookies.txt (not gog-cookies.dat) and using his version did the trick :)

from gogrepo.

Did some tests with that Kalanyr fork but it doesn't seem to work either.
Although the errors in that fork are different:
The login and cookie handling seems to work just fine (in contrast to this one). But the downloading and updating also fails but with a different reason:

line 262, in getattr
return self[key]
KeyError: 'galaxyDownloads'

Edit: ok this was caused by the gog-manifest.dat file (the one from this gogrepo.py is not compatible with the one from the kalanyr fork).
Looking good so far, fetching stuff again. Hope that the DL will work, too.
Edit2: Downloads are working, too. So switching over to the Kalanyr fork is indeed a working solution! Nice find spitfireCH, thx!

from gogrepo.

I started having this issue today for some games, tried the suggested Kalanyr fork and it gives the same error.

from gogrepo.