When fetching XML with defined encoding, XPath transformer raises an exception, since it expects encoded content:

Traceback (most recent call last):
File "/home/drwho/kibitzr/bin/kibitzr", line 11, in <module>
sys.exit(cli())
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(args, kwargs)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, kwargs)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, kwargs)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/cli.py", line 45, in once
sys.exit(main(once=True, log_level=ctx.obj['log_level'], names=name))
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/main.py", line 51, in main
execute_all(checkers)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/main.py", line 133, in execute_all
checker.check()
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/checker.py", line 41, in check
ok, report = self.transform(ok, content)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/transformer/factory.py", line 48, in run_pipeline
ok, content = transform(content)
File "/home/drwho/kibitzr/lib/python2.7/site-packages/kibitzr/transformer/html.py", line 72, in xpath_selector
root = etree.fromstring(html, parser=etree.HTMLParser())
File "src/lxml/lxml.etree.pyx", line 3228, in lxml.etree.fromstring (src/lxml/lxml.etree.c:79609)
File "src/lxml/parser.pxi", line 1843, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:119069)
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

PhantomJS might be a better choice for some cases.
It should be possible to choose it on check level with something like:

checks:
  - name: xxx
    browser: PhantomJS
    ...

Create an extension, like kibitzr-keyring that will add env key to creds dictionary.
Example usage from Python code:

creds['env']['AWS_SECRET_KEY']

Or in Jinja template:

{{ creds.env.AWS_SECRET_KEY }}

sh is only available on POSIX. Also Xvfb is not applicable to Windows.

Using cryptography.fernet and fernet.key from kibitzr-creds.yml:

  transform:
    - encrypt
    - decrypt

This will unlock secure storage on https://dweet.io

Please and a notifier which executes http requests. This could be used to communicate with web hooks or other web based API. For example sending messages via Gotify.

Is there a way to process multiple elements (e.g. an HTML list or JSON Array) and be notified for each item?

using centos 7.8 on a Dell Rsomething

docker version 19.03.08

docker image peterdemin/kibitzer:latest

running the docker command from documentation

• docker run -v $PWD:/root/.config/kibitzr -v $PWD/pages:/pages peterdemin/kibitzr run

Getting error:

2020-04-01 23:33:41,187 [INFO] kibitzr.conf: No credentials file found at /root/.config/kibitzr/kibitzr-creds.yml
Traceback (most recent call last):
  File "/usr/local/bin/kibitzr", line 11, in <module>
    load_entry_point('kibitzr', 'console_scripts', 'kibitzr')()
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/kibitzr/kibitzr/cli.py", line 75, in run
    sys.exit(app.run(once=False, log_level=ctx.obj['log_level'], names=name))
  File "/kibitzr/kibitzr/app.py", line 65, in run
    checks=settings().checks,
  File "/kibitzr/kibitzr/conf.py", line 166, in settings
    return ReloadableSettings.instance()
  File "/kibitzr/kibitzr/conf.py", line 52, in instance
    cls._instance = cls(config_dir)
  File "/kibitzr/kibitzr/conf.py", line 32, in __init__
    self.reread()
  File "/kibitzr/kibitzr/conf.py", line 63, in reread
    checks = self.parser.parse_checks(conf)
  File "/kibitzr/kibitzr/conf.py", line 178, in parse_checks
    checks = conf.get('checks', conf.get('pages', []))
AttributeError: 'NoneType' object has no attribute 'get'

Pbs3 supposedly works on Unix and Windows.
So maybe Kibitzr could use it instead of sh / pbs?

• https://github.com/xapple/pbs3/
• https://pypi.org/project/pbs3/

I think for multiple smtp recipients it would be nice to send the emails as BCC, so recipients cannot see the addresses of other recipients.

The to address could be either the first recipient by convention, or maybe better the sender.

Please consider adding an example for the bash notifier. For bash newbies (like myself), it is not entirely clear how to reference the transformed output from the bash script. Thx! :)

CodeFactor found an issue: Using lxml.etree.fromstring to parse untrusted XML data is known to be vulnerable to XML attacks. Replace lxml.etree.fromstring with its defusedxml equivalent function.

It's currently on:
kibitzr\transformer.py:127
Commit 6e216a8

One use-case is aggregating results of multiple checks in one.
Another is triggering a check on success/failure of another check.

http://stackoverflow.com/a/36856663/135079

Hello,

I'm having an issue running latest docker image published on my system.
I discovered that Geckodriver is broken, manually updating the driver inside the images to latest v0.23 fixes the issue (from https://github.com/mozilla/geckodriver/releases/tag/v0.23.0 )

Log below:

1545203408943 geckodriver INFO geckodriver 0.20.1
1545203408955 geckodriver INFO Listening on 127.0.0.1:53944
1545203409964 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-headless" "-profile" "/tmp/rust_mozprofile.oJMjQRxZqm9Q"
*** You are running in headless mode.
1545203411205 Marionette DEBUG Received observer notification profile-after-change
1545203411678 Marionette DEBUG Received observer notification command-line-startup
1545203411679 Marionette DEBUG Received observer notification nsPref:changed
1545203411679 Marionette DEBUG Init aborted (running=false, enabled=true, finalUIStartup=false)
1545203412747 Marionette DEBUG Received observer notification toplevel-window-ready
1545203414939 Marionette DEBUG Received observer notification sessionstore-windows-restored
1545203414940 Marionette DEBUG Waiting for delayed startup...
1545203415237 Marionette DEBUG Setting recommended pref app.update.disabledForTesting to true
1545203415237 Marionette DEBUG Setting recommended pref security.turn_off_all_security_so_that_viruses_can_take_over_this_computer to true
1545203415241 Marionette DEBUG Setting recommended pref toolkit.cosmeticAnimations.enabled to false
1545203415242 Marionette DEBUG Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
1545203415243 Marionette DEBUG Setting recommended pref dom.disable_beforeunload to true
1545203415379 Marionette INFO Listening on port 59003
1545203415380 Marionette DEBUG Remote service is active
1545203415392 Marionette DEBUG Accepted connection 0 from 127.0.0.1:60704
1545203415404 Marionette TRACE 0 -> [0,1,"newSession",{"acceptInsecureCerts":true,"browserName":"firefox","capabilities":{"desiredCapabilities":{"acceptInsecureCerts":true,"br$
1545203415405 Marionette TRACE 0 <- [1,1,{"error":"unknown command","message":"newSession","stacktrace":"WebDriverError@chrome://marionette/content/error.js:178: ... et@chrome$
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:721:13

The smtp server I'm using has nothing to do with the recipient address.
kibitzr tries to login with an irrelevant value user which makes the server disconnect (501).

The expected behavior is for kibitzr to only try to login if the username and password are present in the config.

Suggested fix:
Default user and password to None.

user = credentials.get('user', None)
password = credentials.get('password', None)

Only attempt a login if the password has been configured.

if password is not None:
    server.login(user, password)

Notifier configuration should look like

smtp: <you email [email protected]>

instead of

 smtp:
            recipients:
              - <you email [email protected]>
            subject: JetPack update

subject should be taken from conf's name.

Because asking to run

rm -rf pages/*

sounds hacky

It must be global - all keys accessible from all checks.
It must persist between restarts (saved on disk).
Writes should be made in notify:

checks:
  - name: Add to stash
    script: echo "Good"
    notify:
      - stash:
          Hubert: {{ content }} news, everyone! 

stash notifier accepts dictionary, where each value is a Jinja template.
All keys are written atomically, e.g. if an error happens in one, none will be written.
Stash should be accessible wherever conf is (fetchers, transforms, notifiers) and should be named stash.

It will unlock checks aggregation.

Would be nice to have support for more notification providers.
Ref: https://github.com/caronc/apprise

• webwatcher version: latest
• Python version: 3.5
• Operating System: raspbian

Description

Hi, I am not well versed with running script. Can you kindly update the readme file on how to run it. I successfully installed it using setup.py, and when I just type command "kibitzr", command fails and says that kibitzr.yml file not found.

Do we need to create this file by ourselves?

Kindly explain the installation and running process a bit more detailed.

TIA
Ankit

Latest version on docker hub is 7.0.5. Seems like job failed:

https://github.com/kibitzr/kibitzr/actions/runs/7427870480/job/20214276680

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Instead of pkill -USR1 kibitzr

Email content to be in human readable form. Like the email I got is like:

Jetpack Updates at 'URL'
+++ b/content
@@ -0,0 +1 @@
+Download Version 4.6

Now here, won't be it great like email content may show:

Jetpack Updates at 'URL'
Previous Value:
Download Version X.X
New Value:
Download Version X.X

CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().

It's currently on:
kibitzr\conf.py:96
Commit e8b498b

It will have regular access to conf and creds, then:

1. content - whole content in one string
2. lines - content splitted by lines
3. xpath - xpath selector function
4. css - css selector function
5. json - lazy JSON loader

Explore https://github.com/zapier/transformer and see if it can be integrated into kibitzr.

Seems like all html transformations are broken in the latest docker image (7.0.10). To reproduce, may use next config:

checks:
  - name: Test
    url: https://travis-ci.org/kibitzr/kibitzr
    transform:
      - text
      - changes
    notify:
      - python: print(content)

And docker-compose config,

services:
  kibitzr:
    container_name: kibitzr
    image: peterdemin/kibitzr:latest
    command: '--log-level=debug run'
    volumes:
      - './kibitzr.yml:/root/kibitzr.yml:ro'
    network_mode: 'host'

kibitzr  | 2024-05-09 10:42:48,956 [DEBUG] kibitzr.conf: Loading credentials from /root/kibitzr-creds.yml

kibitzr  | 2024-05-09 10:42:48,962 [DEBUG] kibitzr.conf: Loading settings from /root/kibitzr.yml

kibitzr  | 2024-05-09 10:42:49,011 [INFO] kibitzr.fetcher.loader: Fetching Test at https://travis-ci.org/kibitzr/kibitzr

kibitzr  | 2024-05-09 10:42:49,012 [DEBUG] cachecontrol.controller: Looking up "https://travis-ci.org/kibitzr/kibitzr" in the cache

kibitzr  | 2024-05-09 10:42:49,013 [DEBUG] cachecontrol.controller: No cache entry available

kibitzr  | 2024-05-09 10:42:49,013 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): travis-ci.org:443

kibitzr  | 2024-05-09 10:42:49,447 [DEBUG] urllib3.connectionpool: https://travis-ci.org:443 "GET /kibitzr/kibitzr HTTP/1.1" 301 167

kibitzr  | 2024-05-09 10:42:49,448 [DEBUG] cachecontrol.controller: Updating cache with response from "https://travis-ci.org/kibitzr/kibitzr"

kibitzr  | 2024-05-09 10:42:49,448 [DEBUG] cachecontrol.controller: Caching permanent redirect

kibitzr  | 2024-05-09 10:42:49,451 [DEBUG] cachecontrol.controller: Looking up "https://app.travis-ci.com/kibitzr/kibitzr" in the cache

kibitzr  | 2024-05-09 10:42:49,451 [DEBUG] cachecontrol.controller: No cache entry available

kibitzr  | 2024-05-09 10:42:49,453 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): app.travis-ci.com:443

kibitzr  | 2024-05-09 10:42:50,266 [DEBUG] urllib3.connectionpool: https://app.travis-ci.com:443 "GET /kibitzr/kibitzr HTTP/1.1" 200 None

kibitzr  | 2024-05-09 10:42:50,404 [DEBUG] cachecontrol.controller: Updating cache with response from "https://app.travis-ci.com/kibitzr/kibitzr"

kibitzr  | 2024-05-09 10:42:50,404 [DEBUG] cachecontrol.controller: Caching b/c of expires header. expires in None seconds

kibitzr exited with code 132

Hi, I'm trying to create dev environment, but current master is broken on *nix systems.
Getting errors around sh usage, for example:

______________________ test_fetch_by_explicit_bash_script ______________________

    def test_fetch_by_explicit_bash_script():
>       ok, content = run_script({'bash': 'echo "hello"'})

tests/unit/fetchers/test_script.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/unit/fetchers/test_script.py:9: in run_script
    return fetch_by_script({'script': script})
kibitzr/fetcher/script.py:28: in fetch_by_script
    return execute_bash(
kibitzr/bash.py:143: in execute_bash
    return executor(code).execute(stdin)
kibitzr/bash.py:159: in execute
    return self.make_report(ok, result)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

ok = True, result = 'hello\n'

    @staticmethod
    def make_report(ok, result):
>       stdout = ensure_text(result.stdout)
E       AttributeError: 'str' object has no attribute 'stdout'

kibitzr/bash.py:191: AttributeError

Seems like sh.Command isn't returning object with stdout and stderr, but just a string. See migration. sh 2.0 was released Feb 10 2023

How to make this app run on Heroku. Is there a $PORT functionality for this app?

Quite easy to send:

PUT /_matrix/client/r0/rooms/{roomid}/send/m.room.message/{txnid}
Authorization: Bearer {accesstoken}
Content-Type: application/json

{"msgtype": "m.notice", "body": "{message}"}

https://matrix.org/docs/spec/client_server/r0.6.0#put-matrix-client-r0-rooms-roomid-send-eventtype-txnid

It would be nice to use the Docker Image on the ARM platform.

Here is an example how that could be done via GitHub Actions:

name: Build Docker Image

on:
  push:
    tags:
      - '*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Check out
        uses: actions/checkout@v2

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm/v7,linux/arm64
          push: true
          tags: peterdemin/kibitzr:latest,peterdemin/kibitzr:${GITHUB_REF/refs\/tags\//}

PS: You mentioned that Travis CI is not working anymore. Maybe you could switch to GitHub Actions altogether.

Currently kibitzr commits in the underying git repo first; and then notifies. So, if a notification fails (e.g. expired api key/network glitch/whatever), you will miss the change.
I guess an easier way to deal with it would be just to undo the latest commit if some of the notification methods failed?

Another issue is: if one of the notifiers fails, the whole program still exits with 0 exit code which makes it a bit harder to track failures. Do you think it makes sense to exit with nonzero code in that case?

Let me know if you think that makes sense; I'll be happy to implement and contribute!

OS: Arch Linux, 20170510 patchlevel
Details: Working local installation of Postfix to send and receive e-mail using SMTP(S) automatically; no credentials required, if it's coming from localhost it gets picked up by the mailer and transmitted. Known to work because it's been running since the server was built a few years ago.

• Successfully installed Kibitzr into a virtualenv
• Configured a JSON feed

  - name: Antarctica Starts Here JSON feed
    url: https://drwho.virtadpt.net/json/feed.json
    transform:
      - jq: ".items[0] | { title: .title, url: .url, body: .content_html }"
      - text
      - changes
    notify:
      - smtp: [email protected]
    period: 5

• Set up SMTP in kibitzr-creds.yml for what should be a "fire and forget" local SMTP server:

smtp:
  host: localhost
  port: 25

• Ran Kibitzr: kibitzr once
• Nothing.

Is there a way to configure Kibitzr such that it can "just use" a local SMTP server if one is present?

It might be useful to have checks running in parallel using a thread pool of predefined size.
Since kibitzr spends most of it's time waiting for network...

Use case from #10 for tutorial in the README.md

Kibitzr uses defusedxml.lxml in the xpath transform.
However, the module is deprecated, so we should look into replacing its usage.

defusedxml.lxml

DEPRECATED The module is deprecated and will be removed in a future release.

https://pypi.org/project/defusedxml/#defusedxml-lxml

• webwatcher version:
• Python version:
• Operating System:

Description

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

It's nice to have more control over times of checks, than just period.
This feature should add alternative cron schedule:

checks:
  - url: https://google.com
    cron: 43 21 * * *

where space-separated fields account for: minutes, hours, a day of the month, month, a day of the week.

Filesystem in cloud deployment is ephemeral. Thus changes history must be stored somewhere outside. In S3, or database.

Hello. I was trying to sent notification by telegram bot and having:
eror ModuleNotFoundError: No module named 'telegram.bot'
Is it possible now to notify by telegram using docker container?

I have an API that takes > 10 seconds on the first execution, but thereafter, API calls are very quick. Is it possible to initialize that API once at startup, and then save the API object somewhere where it can be reused in other checks without the overhead of the first call? I see there's a kibitzr.before_start hook, but it looks like it gets called every time through the run loop, and I want to only run it once before the run loop.

Looks like kibitzr fails under Windows. Need to take a look.

Storing credentials in plain-text is not reliable enough :)
System keyring seems a better option for Desktop environments.