alberanid / diffido Goto Github PK

View Code? Open in Web Editor NEW

20.0 2.0 6.0 703 KB

Watch web pages for changes

License: Apache License 2.0

Python 97.97% Shell 0.61% Dockerfile 1.42%

webpages check monitor notifications alert change email vuejs tornado scheduler apscheduler

diffido's Introduction

Diffido

Spot the difference on the web.

Tired of clicking F5 waiting for a change on a web page? Define a list of pages to watch, and receive an email when something has changed.

Install, run, develop and debug

Docker

Just run:

./run-docker.sh

Old-fashioned installation

To install it:

wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
# if you want to install these modules for an unprivileged user, add --user and remove "sudo";
# if you want to upgrade the versions already present in the system, also add --upgrade
sudo pip3 install lxml
sudo pip3 install pytz
sudo pip3 install apscheduler
sudo pip3 install requests
sudo pip3 install sqlalchemy
sudo pip3 install tornado
git clone https://github.com/alberanid/diffido
cd diffido
./diffido.py --debug

Now you can point your browser to http://localhost:3210/

You can also run the server in https, putting in the ssl directory two files named diffido_key.pem and diffido_cert.pem

Settings

You can edit the conf/diffido.conf file (Python syntax) to change the global settings; you almost surely have to configure the SMTP settings, at least.

Each schedule has its own web page GUI; the settings should be pretty self-explanatory, except:

XPath selector: define which portion of a web page to consider
minimum change: float between 0.0 and 1.0, which represent the minimum amount of the page (in percentage of number of lines) that has to be changed to send a notification; if left empty, any change will be notified
crontab: a complete crontab definition, to specify the period of the check

Development

See the docs/DEVELOPMENT.md file for more information about how to contribute.

Technological stack

VueJS 2 for the webApp
Vue Material for the UI components
Tornado web as web server
APScheduler to run the scheduled jobs
Python 3
Git to store the data

License and copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

diffido's People

Contributors

Stargazers

Watchers

Forkers

messyidea nogit karolzlot princebhatia09 arthur-techempower

diffido's Issues

ask for confirmation, deleting an entry

Introduce a modal to ask confirm, deleting an entry.

side-by-side diff view

The html2diff library also supports a side-by-side view of the diff; let's do it!

on Firefox, focused numeric fields look bad

Editing a trigger, numeric fields expand over other elements, when focused.

It happens only on Firefox, they are ok with Chrome. Maybe we can update our version of https://vuematerial.io/

update axios library

Update Axios package due to a security alert.

CVE-2020-28168
high severity
Vulnerable versions: < 0.21.1
Patched version: 0.21.1

Axios NPM package 0.21.0 contains a Server-Side Request Forgery (SSRF) vulnerability where an attacker is able to bypass a proxy by providing a URL that responds with a redirect to a restricted host or IP address.

tables are not paginated

The history page, especially, will become quickly not usable.

Waiting for upstream support: https://vuematerial.io/components/table

show also complete revisions

It should be possible to display also the page at a given revision, in the diff page.

Whole application fails if unable to create a repository

For some still undetected reasons, sometimes a new storage directory is not created.
In this case, the whole daemon is blocked:

Process Process-37:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "./diffido.py", line 199, in _commit
    os.chdir('storage/%s' % id_)
FileNotFoundError: [Errno 2] No such file or directory: 'storage/5'
Process Process-37:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "./diffido.py", line 199, in _commit
    os.chdir('storage/%s' % id_)
FileNotFoundError: [Errno 2] No such file or directory: 'storage/5'

add two command line options (smtp_username and smtp_password) to get username and password (and also add them to the diffido.conf sample)
add s.login before both the s.send_message lines (notice that there are 2 calls, there) in the send_email function, if the above options are set

send emails in case of error

In many cases it's possible to send emails to notify the admin that something horrible happened.

support for authentication and other HTTP requests methods

We should support authentication (Basic/Digest HTTP access authentication and based on sessions), custom cookies and other HTTP methods (POST/PUT)