Fix Tumblr in countries where GDPR consent is required about sherlock HOT 7 OPEN

It seems that the problem is more fundamental. This involves how Sherlock actually detects that a given user name is not supported.

For example, when I go to a Tumblr page that does not exist (e.g. https://blah.tumblr.com/), I get the following message:

There's nothing here.
Whatever you were looking for doesn't currently exist at this address. Unless you were looking for this error page, in which case: Congrats! You totally found it.

Since there is not an explicit http error message (like a 404 not found), it appears that the algorithm is hardcoded to look for a snippet of the error message text. On your branch, the error message is "Whatever you were looking for doesn't currently exist at this address". Since that does not match for Tumblr, Sherlock incorrectly thinks that there is a user here.

If I change the error message in the JSON to be "Whatever you were looking for doesn't currently exist at this address", then it works. But, if I make the string "whatever you were looking for doesn't currently exist at this address" (just changing the capitalization of the first word), it thinks that there is a user.

So, the sites that do not give an explicit error code are very fragile. Any change that the site owners make to their error page will cause Sherlock to give false positives.

So, you can fix Tumblr just by updating the error text. But, it seems that the check for existence should at least be case insensitive.

from sherlock.

I'm a bit confused as you said the string I used:

Whatever you were looking for doesn't currently exist at this address

doesn't work, but the string that you used:

Whatever you were looking for doesn't currently exist at this address

does work, yet they're exactly the same?

from sherlock.

Hmm... well, I am confused too. I could have sworn that your string was different, but when I look back, they are the same.

So, I am puzzled on what the problem is. When I run Tumblr, it gives the right result. Even though the implementation is fragile as it depends on the exact error text.

The following is the exact entry I have at the end of the JSON file:

"Tumblr": {
"url": "https://{}.tumblr.com",
"errorType": "message",
"errorMsg": "Whatever you were looking for doesn't currently exist at this address"
}

And, this works.

What was making you think that the problem was cookie related?

from sherlock.

If you clear your cookies and browse to https://blah.tumblr.com/ you will get the following page:

This is the page being returned to Sherlock as the request we're making doesn't contain the pfg cookie, and because the error text isn't on this page we get a false positive.

My commit was an attempt to get round this by passing a hardcoded cookie on the request, although this didn't work...

from sherlock.

I am wondering if you are seeing something completely different since you are in the UK. I do not see this GDPR consent page at all. I tried opening a new Incognito page in Chrome, and I just get the text I mentioned. But, perhaps since Tumblr knows that my IP is from the US, it is not putting all of that stuff in front of me.

from sherlock.

Yes, that makes sense.

from sherlock.

I am not coming up with any good options. This is also complicated by the fact that I cannot duplicate the problem here.

How did you get the cookie? It sounds like a valid cookie depends on the user agent string that is presented during the request. I saw this StackOverflow question. There is mention in this answer about the cookie having an expiration date too...it would not be good that users would have to get a new version of Sherlock just to get a later cookie.

There is Python module called PyTumblr which would allow direct access via the API. But, this would result in the secret being boiled into the Sherlock software, which does not seem right. But, it is a way.

I have done browser automation in the past that would allow one to programmatically click the accept button. But, that seems like overkill for this one website.

from sherlock.