This scrapper/scanner logs into a Upwork account using selenium for the login process, catches all the jobs which are in the main page through a local file created with python and consumed with BeautifoulSoup. Then it navigates to the profile page with selenium, extracts certain information about the profile owner using BeautifoulSoup. Finally, I used pydantic to model the scrapped data and export it into a JSON file.

I also used mypy, flake8 and pytest for checks, linting and testing, besides of typer to facilitate easy and expandable interaction with the CLI.

• Getting Started
  • Prerequisites
  • Installation
• Linting and Checks
• Tests

The next command will execute the process of scanner in the site upwork.com

python scan.py upwork

Execute the next command on your terminal

git clone https://github.com/Alfareiza/scrap-up-work-com.git

Once the repository has been cloned, a folder is created with the name of the project scrap-up-work-com.

Go toward this folder using the terminal :

cd scrap-up-work-com

Make sure you have python 3.11 installed and execute:

python -m venv .venv

Then to activate the isolated environment execute the next command according to your O.S

All the dependencies and sub-dependencies will be installed on the local project.

pip install poetry

poetry install

Type checker.

mypy .

Linting tool for checking pep8 recommendations.

flake8 --exclude .venv --ignore=E501,F401

Execution of tests.

pytest  .