web-crawler's Introduction

serverless web crawler

this application measures the latency and availability of websites.

Desgin Diagram

Technologies used

AWS Lambda
Amazone EvenetBridge
DynamoDB
API Gateway
AWS CloudWatch
CloudWatch Alarms
AWS SNS(Simple Notification Service)
AWS SDK
AWS CDK
AWS CodePipeline
Node js
React
Material UI
MongoDB Atlas

Features

measures latency and availability metrices
generate Alarms on certain threshold
store generated alarm to database.
plot metrices live to aws cloudwatch dashboard
measure metrices every five minutes.
you can create delete get and update list of urls in mongodb using express server.
alarms on metrices will created or deleted according to url operation.
if you add a new url a new alarm will be created automatically.
if you delete a url corresponding alarm will be deleted automatically.
logs will be shown at cloudwatch
CI/CD is implented in this project.
make any changes in your application or stack they will automatically be deployed to your account.
If your web crawler is in alarm state pipeline will automatically rollback applicatin to last successfule build.
you can add your own cloudwatch metrices to applicatin see this guidline: for alarms https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Alarm.html for metric https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html#metricwbrerrorsprops

Run locally

setting up aws cdk.

If you already have aws cdk proceed to next step.

installing aws-cdk

    npm install aws-cdk-lib

set your credentials

    aws configure

Run project

Clone the project

  git clone https://github.com/kashif2022skipq/Sculptor_TS/tree/main/kashif-ghafoor/sprint2

Go to the project directory

  cd sprint4

Install dependencies

  npm install

typescript compilation

  npm run watch

deploy lambda

    cdk deploy

that's it you have deployed your serverless application to aws.

cdk watch onstantly observe the changes in your code and deploy it to aws.

    cdk watch

output

after deployment go to your lambda in aws console. you will have two lambda functions there. you can observer them in their cloudwatch. you can also see the alarms generated in cloudwatch and in dynamodb table.