Code Monkey home page Code Monkey logo

assignment's Introduction

Twitter Trends Scraper

Overview

This script logs into Twitter, fetches the top trending topics, and stores them in a MongoDB database. The script utilizes Selenium for web scraping and MongoDB for data storage.

Features

  • Twitter Login: Automates the login process to Twitter.
  • Trend Fetching: Retrieves the top 5 trending topics.
  • MongoDB Integration: Stores the trending topics in a MongoDB database.
  • User-Agent Rotation: Uses a random user-agent for each run to avoid detection.
  • Proxy Rotation (Commented Out): Includes functionality for rotating proxies using Webshare.io, though it is currently commented out due to running out of free credits.

Requirements

  • Python 3.x
  • MongoDB
  • Chrome WebDriver
  • Selenium
  • Requests
  • Webdriver Manager

Setup

  1. Install Python Dependencies
    pip install -r requirements.txt

Usage

  1. Run the Script

    python script.py
  2. Data Storage The trending topics will be stored in the MongoDB database specified in the MONGO_URI.

Implementation Details

Twitter Login

The script uses Selenium to automate the login process to Twitter. The XPaths for the username and password fields are used to enter credentials and log in.

Trend Fetching

Once logged in, the script navigates to the Twitter trends page and retrieves the top 5 trending topics using their respective XPaths.

MongoDB Integration

The trending topics are stored in a MongoDB collection named trends within the twitter_trends database. Each record includes the trend names, timestamp, and IP address.

User-Agent Rotation

The script rotates through a list of user-agents for each run to avoid detection and reduce the chances of being blocked.

Proxy Rotation

The script includes functionality for rotating proxies using Webshare.io. However, due to running out of free credits, this part of the code is commented out. You can uncomment the relevant lines and provide your Webshare.io API key to enable this feature.

assignment's People

Contributors

rachitkumar205 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.