diastro / zeek Goto Github PK

Not only read back the error code but catch HTTPError exception too

ex :
HTTPError: HTTP Error 404: Not Found

Server first sends a configuration packet to all of its clients so that they share the same crawling settings.

• Server -> Config - > Client
• Client -> Ack -> Server
• Server -> Sends first urls -> ...

Application will be able to parse a configuration file and use the needed values from it.

Server will be able to specify to either crawl a specific list of URL (static), crawl indefinitely (dynamic) or domain specific.

Server will inform node of the crawling type when sending the config info.

Hello
Please i am lost on how to view the output of the scrapped url

Software requirements specification (SRS)

Requirements traceability

Add the size of the queued url to an input in the configuration file

Handle :

• 400
• 500
• etc

• Socket
• Error handling
• Transmission and reception of data (serialized object using pickle)

Coomunication protocol

Mongo DB

Different action will need to be taken depending on the type of packet received

Socket communication wasn't using delimiters which cause (in rare cases since I test Zeek running multiple python process on the same server) the ServerSideClient to block on a read eventhough it had received more then a full packet.

Fix:
Added a delimiter and after receiving data we check to see if there's a delimiter in the buffer. If so, we take the complete packet that's in there.

rule.py and scrapping.py are not being reloaded from client.py but from original imported file.

Enhancement needed to reload them from client.py

Each connection has a different Thread (Server-side).

Creation of the object class that will be passed between the server node and the working nodes

Dynamic reload of imported module from server

Message targetted to specific client

Ouput result to file (csv)

• Data
• Error

Read all the URLs in a webpage

• Handle illegal cases (bad extensions .mp3 .mp4 etc)

Create rules to prevent the scrapper to go visit certain urls:

• .mp3
• .mp4
• .jpg
• etc

Class structure containing all relevant data for a visited site:

• Return code (400, 201)
• Processing time
• The page data
• etc

Protocol needs to be able to send a list of URL and client needs to be able to interpret this list.

User manual - Deployment guide

• Console logging.
• Save logs to an output file.
• Allow multiple level of logging (INFO, DEBUG, CRITICAL, ERROR)

Review of all the documentation :

• SRS
• MORE TO COME

Collect stats

• Create a class which will contain all the stats data (Return code, request time, data, etc)
• Return this object to server
• Server will handle it and store it if needed

URL dispatching and sorting (server)

Add proper error handling throughout the project

Working node needs to be able to scrape URLs from a web page and return the list of scrapped URLs to the server.

In the event that a URLs request isn't successful, the working node needs to return and inform the server.

Deployment of rule.py and scrapping.py to working nodes

Project "retrospective" report

exe_type need to be formated to remove < >

Repro :

• Start server
• Start client
• Kill the server
• Try to restart it

Fix :
Handle CTR-C and force close all socket before exiting

Fix queue size.

Each serverSide client will have a fix number of URL dispatched to them (ie 20). Everytime a client replies after visiting a site a new url will be sent.

Architecture overview

Dispatching of urls :

ServerSide

• Visited URL
• URL to visit

Client:

• URL to visit
• No visited URL queue since the crawling thread will create the response packet to reply to the server