erdos1729 / webscrapping-identify-download-classify-published-pdfs-from-multiple-urls Goto Github PK

• This repository will assist you in scrapping data from multiple websites. It will download the latest pdf files published on a website in a specific folder as per the users requirement. This can be used for automating various operations involved in market research.
• Once the pdfs are downloaded they are classified into oil/no_oil/foreign_language categories based on a string based rule
• You can customize these rules for classification as per your need

• pip install -r requirements
• Run radar_automation.py

I devised the solution from the following pages of the documentation:

• [] package that collects several modules for working with URLs
• [] to scrape information from web pages
• [] is a text extraction tool for PDF documents
• [] for natural language processing
• Keyword based search in extracted text for rule based classification