Ignore pages that have a 404 status code about suckit HOT 6 OPEN

We could have one 404 error page by website

from suckit.

As long as you're aware that is an opinionated choice :) some sites have custom 404s by section of the site etc, some will keep the original URL like in my screenshot, some will redirect to a dedicated 404 URL, some will show a 404 page with a 200 response.. Web crawling is messy!

Perhaps this could be a configuration thing, but that's up to you :)

from suckit.

A good solution can be to hash a 404 or 200 webpage. This way if the page is specific to this URL it is saved, if not we could make a symbolic link to the generic one.

from suckit.

Yea I think it's tricky. If it's legitimately just a bad link to a page that was never existed or a href that was relative when it shouldn't have been you might hit an infinite loop (i've seen this in practise).

from suckit.

Humm ok. We have more serious issues and very little time currently, we will give this a try latter

from suckit.

Yea no rush :)

from suckit.