googlecloudplatform / nodejs-serverless-microservices-demo Goto Github PK

View Code? Open in Web Editor NEW

A set of Node.js microservices to track visual changes of web pages.

License: Apache License 2.0

JavaScript 92.53% CSS 2.91% HTML 4.57%

nodejs-serverless-microservices-demo's Introduction

Web page visual history

A set of microservices to track visual changes of web pages.

This code is provided as example only. It is provided without guarantee or level of support. This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

Dependencies

Google Cloud Platform
App Engine
Cloud Functions
Cloud Datastore
Cloud Storage
Cloud Pub/Sub
App Engine cron

List of micro services

Each micro service is stored in its dedicated folder.

frontend: A frontend to vizualise which websites are tracked and see their screenshots
task-scheduler: Every 5 minutes, looks for screenshots to take and schedule them as tasks.
screenshot: Takes screenshots of the given URL, stores it in Cloud Storage
image-diff: Compares an image with its reference image in Cloud Storage

Architecture

Every 5 minutes, the task-scheduler microservice runs, queries the Cloud Datastore database for webpages ot screenshot. For each webpage, it sends a message to a Pub/Sub topic. This topic is responsible for pushing these messages to the screenshot microservice. This screenshot service receives the message as a regular HTTP POST request, screenshots requested webpage and stores the result in a Cloud Storage bucket. The creation of a new file in the bucket triggers the a Cloud Function (image-diff). The image-diff function compares the new image with a reference image from the references folder. If a difference is found, it stores the image in a keyframe folder and updates the reference image.

The frontend service is a web frontend that allows to browse the data: Its main page lists the currently tracked webpages, clicking on a webpage shows all the saved keyframes for this webpage. Users can start tracking a new webpage by entering its URL, which will add a new entity to the Cloud Datastore database.