Update robots.txt file to stop legitimate services from unnecessary scanning paths such as the api about orgbook-bc-client HOT 5 CLOSED

In this case the organization behind Bytespider is well known for not respecting the robots.txt file, but we need to update it anyway.

from orgbook-bc-client.

From @cvarjao:
https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#syntax

from orgbook-bc-client.

The Disallow path should start with /

User-agent: *
Disallow: /

Some other comments:
https://www.feitsui.com/en/article/32

It seems Bytespider and Sogou spiders are not fully compatible with robots exclusion standard. These crawlers magically disappeared one week after I created a separate block for each user agent in robots.txt.

from orgbook-bc-client.

Other sources indicate they do not respect the robots.txt file at all.

• https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/
• https://www.reddit.com/r/flask/comments/161fqml/what_to_do_about_bytespider/
• https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent

from orgbook-bc-client.

That said it's worth adding the explicit entries into the robots.txt file to try things out.

from orgbook-bc-client.