A Python script to scrap all links in a given website using requests and Beautiful soup

The detailed article on how to scrap all links in a given website can be found on my blog with an article titled How to extract all website link in Python.

To get started exploring this code you might have to clone or download the repository just as shown below;

-> git clone https://github.com/Kalebu/Link-scraper-in-python
-> cd Link-scraper-in-python

To successfully run this code you're supposed to have requests and BeautifulSoup libary installed on your machine

-> pip install requests
-> pip install beautifulsoup4

Now that we have everything already set up , lets run our code just as shown below;

Link-scraper-in-python -> python link_spider.py
Enter URL of the site : https://kalebujordan.com/
['#content', 'https://kalebujordan.com/', 'https://kalebujordan.com/', 'https://kalebujordan.com/category/projects/...]

Now keep explore it by testing it with various input links to see what links it will scrap

Did you find this information useful, then give it a star

All the credits to kalebu