404 error with Myanmar Zawgyi about corpuscrawler HOT 2 CLOSED

Seems like the bug is here:

https://github.com/google/corpuscrawler/blob/master/Lib/corpuscrawler/util.py#L756

That function should be more lenient when there is a 404. In this case, it found a link to /page/2 in this category, but that page is actually a 404, and that should not be a fatal error.

I would probably change

assert pgdoc.status == 200, (pgdoc.status, pgurl)

to something more like

if pgdoc.status != 200:
  print("Error %3d:      %s" % (pgdoc.status, pgurl))
  next

Can you open a PR?

from corpuscrawler.

Closing as fixed; let me know if you have any more issues.

from corpuscrawler.