Code Monkey home page Code Monkey logo

towwwer's Introduction

Towwwer

CircleCI

Towwwer is a tool for monitoring, collecting and presenting website performance data collected from Google PageSpeed Insights API.

Additionally, Towwwer can run WPScan against the monitored websites and keep track of which security issues are found. WPScan is a separate project, and their license details differ from those of Towwwer.

Lastly, Towwwer can be configured to send messages to a Slack channel via Slack webhooks whenever a significance difference in Lighthouse scores is detected between two consecutive measurements.

Requirements

To run Towwwer, you need:

  • Elixir (or a release built for your system with ERTS bundled in)
  • PostgreSQL (for storing the reports & job queue)
  • Ruby and RubyGems (for installing WPScan)

Configuration

While the PageSpeed Insights API can be used without an API key, you will want to use one for anything more than a single Site + Monitor combination.

Development configuration

  • Configure the PageSpeed Insights API key by setting the PAGESPEED_INSIGHTS_API_KEY environment variable.
  • Configure the Slack webhook URL key by setting the SLACK_WEBHOOK_URL environment variable.

Production configuration

  • Configure the PageSpeed Insights API key by using the config provider of Distillery at rel/config/runtime_config.exs.
  • Configure the Slack webhook URL by using the config provider of Distillery at rel/config/runtime_config.exs.

Deployment

Towwwer uses Distillery to build releases for production. See the Distillery docs for more details.

Steps taken to produce a new deployment:

# To build the Docker image
docker build -t elixir-ubuntu:latest .

# To build the release
docker run -v $(pwd):/opt/build --rm -it elixir-ubuntu:latest /opt/build/bin/build

After which you should see your release tarball in rel/artifacts.

If you add dependencies that require system packages, you will need to update the Dockerfile for the build container, and rerun the docker build command to update it.

Test local deployment: cp rel/artifacts/towwwer-0.1.0.tar.gz /tmp/test/

Start the release with: cd /tmp/test && tar -xf towwwer-0.1.0.tar.gz && ./bin/towwwer start

Production migrations

You can run migrations in production with ./bin/towwwer migrate. This is handled by rel/commands/migrate.sh and Towwwer.ReleaseTasks.

Features

  • Unlimited sites (think domain), monitors (paths for a site), and reports (data for a monitor)
  • Automatic daily (or configurable) updates for every site & monitor reports
  • PostgreSQL-backed job queue
  • WPScan security scans alongside PageSpeed Insights
  • Rate limiting to respect Google API quotas
  • Friendly API for fetching the most relevant information
  • Slack integration

Testing

The following are run in CI, and should always pass:

mix format --check-formatted
mix test
mix dialyzer
mix credo

Local Phoenix

To start your Phoenix server:

  • Install dependencies with mix deps.get
  • Create and migrate your database with mix ecto.setup
  • Install Node.js dependencies with cd assets && npm install
  • Start Phoenix endpoint with mix phx.server

Now you can visit localhost:4000 from your browser.

License

Towwwer is licensed under the MIT License.

towwwer's People

Contributors

juhalehtonen avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

towwwer's Issues

Duplicate jobs cannot be identified or stopped

Currently there is no way to identify or stop duplicate jobs (using same parametes) from running.

To Reproduce
To reproduce, just enqueue jobs more than once.

Expected behavior
We should somehow have a limit to jobs so that only a single job of its kind can be enqueued.

Include low-hanging fruit in Slack messages as points of action

We should display some easy actions to take to improve a given metric. E.g. "You have very large images on your site, consider checking them out to improve performance".

Could check if OptimizeImages -> ruleImpact is high enough, and if it is, output some info such as returning the potential savings in SIZE_IN_BYTES and PERCENTAGE.

Caching saved data

Is your feature request related to a problem? Please describe.
Currently the web UI gets slow fast when data accumulates, because every interaction with the service taxes the database.

Describe the solution you'd like
Use Mnesia or ETS to create a caching layer that is used to serve the content to users. Automatically refresh a given site's cache when the site is updated.

Describe alternatives you've considered
None. We could just not build the cache, but that sucks.

Save mobile data in addition to desktop data

Pass a strategy parameter on request to define desktop and/or mobile.

strategy string The analysis strategy (desktop or mobile) to use, and desktop is the default

Acceptable values are:

“desktop”: Fetch and analyze the URL for desktop browsers
“mobile”: Fetch and analyze the URL for mobile devices

Collect low-hanging fruit to site, not Slack.

Currently it is not very helpful that we have the daily checks, because it often takes longer than one day to fix them, so duplicate reports are quite numbing. Better to have a daily collection in some state (Agent?) and display that on the site.

Update WPScan DB automatically

Theoretically this could be just a cronjob, but I hate the idea of depending on the server :(

I also don't want to use the default update feature on wpscan, because it would trigger way too many times when the scheduled jobs all popped up at the same time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.