Web Scraper

Web Scraper is a simple Flask-based API that allows users to retrieve meta data from any website by passing its URL as a request parameter.

How to Use the API

To use the API, simply make a GET request to the following endpoint:

https://web-scrapper-coral.vercel.app/api/read_web_meta_data?url={website_url}

Here, {website_url} refers to the URL of the website whose metadata you want to retrieve. The API returns a JSON object containing the website's title, description, and image etc.

For example, to retrieve the metadata for https://www.theverge.com, you would make the following request:

https://web-scrapper-coral.vercel.app/api/read_web_meta_data?url=https://www.theverge.com

Project Setup To set up the project, follow these steps:

• Clone the repository to your local machine.

git clone https://github.com/TheAlphamerc/web-scrapper

• Change into the project directory.

 cd web-scrapper 

• Create a virtual environment and activate it.

python3 -m venv venv

`source venv/bin/activate

• Install the project dependencies.

pip install -r requirements.txt

• Setup the development environment by running these commands in your terminal.

export FLASK_APP=index.py
export FLASK_ENV=development

• Run the Flask development server.

python app.py

The API will be accessible at http://127.0.0.1:5000/api/read_web_meta_data?url={website_url}.

Contributing Contributions to the project are welcome. If you'd like to contribute, please create a pull request with your changes.

Created & Maintained By

Sonu Sharma (Twitter) (Youtube) (Insta) (Dev.to)

If you found this project helpful or you learned something from the source code and want to thank me, consider buying me a cup of ☕

• PayPal

You can also nominate me for Github Star developer program https://stars.github.com/nominate

Visitors Count