maubot / rss Goto Github PK

View Code? Open in Web Editor NEW

67.0 67.0 21.0 68 KB

A RSS plugin for maubot

License: GNU Affero General Public License v3.0

Python 100.00%

rss's Introduction

maubot

A plugin-based Matrix bot system written in Python.

Documentation

All setup and usage instructions are located on docs.mau.fi. Some quick links:

Discussion

Matrix room: #maubot:maunium.net

Plugins

A list of plugins can be found at plugins.mau.bot.

To add your plugin to the list, send a pull request to https://github.com/maubot/plugins.maubot.xyz.

The plugin wishlist lives at https://github.com/maubot/plugin-wishlist/issues.

rss's People

Contributors

Stargazers

Watchers

Forkers

l0ric0 rda0 a22sc spiritcroc juhovan fl0kie jaywink jchammons noantiq sumpfralle andrewkvalheim edge-coordinates ikkemaniac edwargix j0n0m lukelr north-west-wind

rss's Issues

Support HTTP basic auth via configuration

Right now if I want to subscribe a room to a private feed I can do so via https://user:[email protected]/rss. This is however, unsatisfying because it exposes my credentials to the whole room.

A better solution would be to utilize aiohttp's basic auth capabilities for certain domains if an RSS admin does the subscription.

Some necessary steps:

Check headers['www-authenticate'] in parse_feed for 'Basic realm="Use test/basic"'
This indicates that auth is needed

Add new configuration item, sth. like:

private_feeds:
   - url: "*.example.org"
     user: johndoe
     pw: supersecure

If we have credentials for the url, then self.http.get(url, auth=aiohttp.BasicAuth(user, pass))
If a subscription matches a private feed, only an rss-admin should be allowed to manage it

What do you think about such an enhancement? I would consider providing a pull request but i dont have a good dev setup here to test it.

[Feature] Make the message type used for posting feed updates configuarable

Make it possible for the bot admin to configure the message type used to post feed updates, i.e. m.text instead of m.notice

Relative links in feeds aren't handled correctly

The bot seems to be broken and is posting broken HTML when Github Wiki updates happen: <p>New post in nitter: Recent Wiki Updates: <a href=\"/zedeus/nitter/wiki\"></a></p> instead of the actual link.

(Used for the Nitter room)

Feature request: Add config option for matrix resend backoff

I kind of rely on the RSS bot to send messages of a feed into a room, however some messages are dropped if there are errors (such as the homeserver not responding or similar). It would be great if there was a (configurable) resend backoff which tries to send the matrix messages again if some error occured.

Suggestions:

Exponential backoff with maximum of 4 weeks or so
Hard-coded resend interval of e.g. 4 hours
Configurable by one integer (try again after n hours), maximum 4 weeks or so

This would also cover potential issues if there is any other problem with the bot which is fixed few weeks after, because feed items which weren't processed properly (e.g. ending in a matrix message) would be processed again until it worked or the absolute limit of suggested 4 weeks is reached.

Add mbp to release

New posts are not picked up – how to debug?

Hello everyone, I'm using the hosted version of this bot, provided by t2bot.io.

I've invited the bot to a channel on my homeserver and subscribed to multiple feeds. The subscriptions are picked up correctly, and running !rss subscriptions gives the expected list of my subscribed feed.

However, there are never any messages from the bot about new feed items. Even though the feeds I've subscribed to, are regularly updated with new content, the bot does not post anything. I've waited for weeks now, so this is not about a little delay.

There are no errors anywhere, and since this is the hosted version from t2bot.io, I was wondering if there is any way to debug this at all?

Failed to load feed: generator raised StopIteration

Hello, I am following the instructions at https://t2bot.io/rssbot/

!rss subscribe http://www.openstreetmap.org/api/0.6/notes/feed?bbox=2.224122,48.8547763,2.3311615,48.902157

returns

Failed to load feed: generator raised StopIteration

$feed_title in posts does not match content of title field in feed database

My use case is the following: I have subscribed to a feed but want to customize the title shown when new posts are sent.

To do this I tried to modify the 'title' field in the 'feed' database, but new posts are still sent with the title that was set when subscribing, my notification_template looks like this: New post from $feed_title:<br/>$link

Feed parse error (extra dot)

When subscribing to a feed, the link saved to the database has an extra dot before the TLD.

Example: !rss subscribe https://nitter.privacydev.net/GrapheneOS/rss

The correct link should be http://nitter.privacydev.net.

For comparison, it works correctly when subscribing to a feed from nitter.net. Maybe subdomains are an issue?

Add variable with feed item content

Hi,
it'd be a great enhancement for us to have the ability to post the entire content of the new feed item to the channel we've got the bot in. I tried reading the code but I didn't find this is possible yet. Is it possible to extend the code with this functionality so we can post full-text updates of the RSS feed?

After a fresh install of maubot xyz.maubot.rss no longer works

I just installed maubot from scratch as i upgraded my env from python-3.8 to python-3.9 via pip install --upgrade maubot and got no errors there.

After firing up the fresh maubot it runs, but my one and only plugin xyz.maubot.rss no longer works.

When i try to upload the version 0.3.2 of the maubot plugin (i had 0.3.1 before) i get the following errors:

12:15:33	ERROR	maubot.loader.zip	Failed to load xyz.maubot.rss v0.3.2

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

Error loading updated version of xyz.maubot.rss, rolling back

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/management/api/plugin_upload.py", line 107, in upload_replacement_plugin
    await plugin.reload(new_path=path)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 244, in reload
    return await self.load(reset_cache=True)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

Failed to load xyz.maubot.rss v0.3.1

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/management/api/plugin_upload.py", line 107, in upload_replacement_plugin
    await plugin.reload(new_path=path)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 244, in reload
    return await self.load(reset_cache=True)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

Failed to roll back update of xyz.maubot.rss

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.2.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/management/api/plugin_upload.py", line 107, in upload_replacement_plugin
    await plugin.reload(new_path=path)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 244, in reload
    return await self.load(reset_cache=True)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 216, in _load
    importer.load_module(module)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/lib/zipimport.py", line 280, in load_module
    exec(code, mod.__dict__)
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "<frozen zipimport>", line 259, in load_module
  File "/var/db/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/bot.py", line 27, in <module>
    import feedparser
ModuleNotFoundError: No module named 'feedparser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/management/api/plugin_upload.py", line 111, in upload_replacement_plugin
    await plugin.reload(new_path=old_path)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 244, in reload
    return await self.load(reset_cache=True)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 201, in load
    return self._load(reset_cache)
  File "/home/maubot/maubot-env/lib/python3.9/site-packages/maubot/loader/zip.py", line 220, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss

Any chance i can recover from this?

Thanks
Max

subsribe rss feed, I get sqlite3.OperationalError

Send the following chat command to

!rss subscribe http://www.publickey1.jp/atom.xml

Received the following error

Failed to run handler
Traceback (most recent call last):
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/client/syncer.py", line 229, in _catch_errors
    await handler(data)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/maubot/handlers/command.py", line 141, in __call__
    ok, res = await self.__call_subcommand__(evt, call_args, remaining_val)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/maubot/handlers/command.py", line 162, in __call_subcommand__
    return True, await subcommand(
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/maubot/handlers/command.py", line 153, in __call__
    return await self.__mb_func__(self.__bound_instance__, evt, **call_args)
  File "/home/am/tmp/maubot-py/plugins/xyz.maubot.rss-v0.3.0.mbp/rss/bot.py", line 350, in subscribe
    feed = await self.dbm.create_feed(info)
  File "/home/am/tmp/maubot-py/plugins/xyz.maubot.rss-v0.3.0.mbp/rss/db.py", line 185, in create_feed
    info.id = await self.db.fetchval(
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/util/async_db/database.py", line 150, in fetchval
    return await conn.fetchval(query, *args, column=column, timeout=timeout)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/util/async_db/connection.py", line 37, in wrapper
    ret = await func(self, arg, *args, **kwargs)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/util/async_db/connection.py", line 89, in fetchval
    return await self.wrapped.fetchval(query, *args, column=column, timeout=timeout)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/util/async_db/aiosqlite.py", line 76, in fetchval
    row = await self.fetchrow(query, *args)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/mautrix/util/async_db/aiosqlite.py", line 70, in fetchrow
    async with self.__execute(query, *args) as cursor:
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/aiosqlite/context.py", line 41, in __aenter__
    self._obj = await self._coro
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/aiosqlite/core.py", line 184, in execute
    cursor = await self._execute(self._conn.execute, sql, parameters)
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/aiosqlite/core.py", line 129, in _execute
    return await future
  File "/home/am/tmp/maubot-py/lib/python3.8/site-packages/aiosqlite/core.py", line 102, in run
    result = function()
sqlite3.OperationalError: near "RETURNING": syntax error

How do I resolve this?

failed to load feed: https://sciencebasedmedicine.org/feed

!rss subscribe https://sciencebasedmedicine.org/feed
Failed to load feed: <unknown>:7:2: mismatched tag

However, the feed appears to be well-formed.

>>> import requests
>>> import feedparser
>>> r=requests.get("https://sciencebasedmedicine.org/feed", headers={"User-Agent": "maubot/0.3.1 https://github.com/maubot    /rss"})
>>> p = feedparser.parse(r.text)
>>> p["feed"]["title"]
'Science-Based Medicine'
>>>

(My test was with @rss:t2bot.io. I'm not sure what version of the bot is in use there)

Bot doesn't react.

Heyo,

I'm having an issue with RSS bot. Like, it doesn't react on commands. Logs doesn't shows an error or anything else.
Here's the config:

# Feed update interval in minutes
update_interval: 60
# Maximum backoff in minutes when failing to fetch feeds (defaults to 5 days)
max_backoff: 3000
# The time to sleep between send requests when broadcasting a new feed entry.
# Set to 0 to disable sleep or -1 to run all requests asynchronously at once.
spam_sleep: -1
# The prefix for all commands
# It has to be prefixed with ! in matrix to be recognised
command_prefix: !rss
# Default post notification template for new subscriptions
notification_template: 'New post in $feed_title: [$title]($link)'
# Users who can bypass room permission checks
admins:
- '@gameplayer:matrix.xaviama.dank-me.me'

Startup logs:

GET https://matrix.xaviama.dank-me.me/_matrix/client/v3/account/whoami
req #185 (/v3/account/whoami) completed in 33.2ms with status 200
GET https://matrix.xaviama.dank-me.me/_matrix/client/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me/displayname
req #186 (/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me/displayname) completed in 8.8ms with status 200
GET https://matrix.xaviama.dank-me.me/_matrix/client/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me/avatar_url
req #187 (/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me/avatar_url) completed in 8.0ms with status 200
Enabling end-to-end encryption support
GET https://matrix.xaviama.dank-me.me/_matrix/client/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me
Starting syncing
req #188 (/v3/profile/%40rss%3Amatrix.xaviama.dank-me.me) completed in 27.0ms with status 200
Client started, starting plugin instances...
Connecting to sqlite://data/dbs/rss.db
Database connection init commands: ['PRAGMA foreign_keys = ON', 'PRAGMA journal_mode = WAL', 'PRAGMA busy_timeout = 5000']
Database at v3, not upgrading
Started instance of xyz.maubot.rss v0.3.2 with user @rss:matrix.xaviama.dank-me.me
Polling started

I've used Maubot, latest version as a container.
Maubot database - PostGreSQL.
Bot version: 0.3.2

Ability to filter posted feed items by regex on item title

I'd like to use this bot on a subreddit but only for post titles that match a certain regex.

Cannot import name 'Scheme' from 'mautrix.util.async_db'

Hi,

I just discovered maubot and it seems amazing.

I have some issue. I tried echo, it didn't work. When I type !echo test in a chat with the bot, nothing happens and nothing in logs

I wanted to try rss bot but I face to this issue:

Click to expand and see logs

[2022-06-16 01:31:21,886] [[email protected]] Initializing maubot 0.2.0
[2022-06-16 01:31:21,892] [[email protected]] Preloading plugins...
[2022-06-16 01:31:21,928] [[email protected]] Preloaded plugin xyz.maubot.echo from /home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.echo-v1.4.0.mbp
[2022-06-16 01:31:21,962] [[email protected]] Preloaded plugin xyz.maubot.rss from /home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp
[2022-06-16 01:31:22,211] [[email protected]] Plugin instance dependencies loaded
[2022-06-16 01:31:22,238] [[email protected]] Plugin instance dependencies loaded
[2022-06-16 01:31:22,240] [[email protected]] Initialization complete in 0.77 seconds
[2022-06-16 01:31:22,241] [[email protected]] Running startup actions...
[2022-06-16 01:31:22,318] [[email protected].@buddybot:myserver.zapto.org] Starting syncing
[2022-06-16 01:31:22,342] [[email protected].@buddybot:myserver.zapto.org] Client started, starting plugin instances...
[2022-06-16 01:31:22,391] [[email protected]] Loaded and imported plugin xyz.maubot.echo from /home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.echo-v1.4.0.mbp
[2022-06-16 01:31:22,394] [[email protected]] Started instance of xyz.maubot.echo v1.4.0 with user @buddybot:myserver.zapto.org
[2022-06-16 01:31:22,728] [[email protected]] Failed to load xyz.maubot.rss v0.3.1
Traceback (most recent call last):
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 203, in _load
    importer.load_module(module)
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/lib/zipimport.py", line 282, in load_module
    exec(code, mod.__dict__)
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/bot.py", line 42, in <module>
    from .db import DBManager, Entry, Feed, Subscription
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/db.py", line 26, in <module>
    from mautrix.util.async_db import Database, Scheme
ImportError: cannot import name 'Scheme' from 'mautrix.util.async_db' (/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/mautrix/util/async_db/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 188, in load
    return self._load(reset_cache)
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 207, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss
[2022-06-16 01:31:22,745] [[email protected].@buddybot:myserver.zapto.org] Failed to start
Traceback (most recent call last):
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 203, in _load
    importer.load_module(module)
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/lib/zipimport.py", line 282, in load_module
    exec(code, mod.__dict__)
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/__init__.py", line 1, in <module>
    from .bot import RSSBot
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/bot.py", line 42, in <module>
    from .db import DBManager, Entry, Feed, Subscription
  File "/home/user/projects/matrix/bots/maubot/plugins/xyz.maubot.rss-v0.3.1.mbp/rss/db.py", line 26, in <module>
    from mautrix.util.async_db import Database, Scheme
ImportError: cannot import name 'Scheme' from 'mautrix.util.async_db' (/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/mautrix/util/async_db/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/client.py", line 130, in start
    await self._start(try_n)
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/client.py", line 203, in _start
    await self.start_plugins()
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/client.py", line 206, in start_plugins
    await asyncio.gather(*[plugin.start() for plugin in self.references])
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/instance.py", line 173, in start
    cls = await self.loader.load()
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 188, in load
    return self._load(reset_cache)
  File "/home/user/projects/matrix/bots/maubot/lib/python3.7/site-packages/maubot/loader/zip.py", line 207, in _load
    raise MaubotZipLoadError(f"Failed to load module {module}")
maubot.loader.zip.MaubotZipLoadError: Failed to load module rss
[2022-06-16 01:31:22,755] [[email protected]] Listening on http://0.0.0.0:29316
[2022-06-16 01:31:22,757] [[email protected]] Startup actions complete in 0.52 seconds, now running forever

Click to expand ans see pip list in vhost of maubot

Package             Version
------------------- -------
aiohttp             3.8.1
aiosignal           1.2.0
alembic             1.8.0
async-timeout       4.0.2
asyncpg             0.25.0
asynctest           0.13.0
attrs               21.4.0
bcrypt              3.2.2
cffi                1.15.0
charset-normalizer  2.0.12
click               7.1.2
colorama            0.4.4
commonmark          0.9.1
feedparser          6.0.10
frozenlist          1.3.0
idna                3.3
importlib-metadata  4.11.4
importlib-resources 5.7.1
Jinja2              3.1.2
Mako                1.2.0
MarkupSafe          2.1.1
maubot              0.2.0
mautrix             0.14.10
multidict           6.0.2
packaging           21.3
pip                 22.1.2
prompt-toolkit      3.0.29
psycopg2            2.9.3
pycparser           2.21
pyparsing           3.0.9
questionary         1.10.0
ruamel.yaml         0.17.21
ruamel.yaml.clib    0.2.6
setuptools          57.0.0
sgmllib3k           1.0.0
SQLAlchemy          1.3.24
typing_extensions   4.2.0
wcwidth             0.2.5
wheel               0.37.0
yarl                1.7.2
zipp                3.8.0

What can I do?

Make default template configurable

Failed to load feed: <unknown>:2:0: syntax error for reddit rss feeds

Since a few days I cannot subscribe to reddit feeds anymore. I only get the error:
Failed to load feed: <unknown>:2:0: syntax error

The rss feed validator tells me:
id must be a full and valid URL: /.rss

So it seems that the feeds are indeed invalid, but that seems to be the case since a long time, at least I find results on this from a long time ago. So I don't know what made this happen.

A feed to test it (trying to subscribe should be enough): https://reddit.com/.rss

Q: Is there a way to configure it to send memes from linuxmemes.rss?

Make Twitter available to subscribe

Sometimes there are communities, projects or companies that are just announcing on twitter - it would be good to be able to subscribe to them via this bot.
According to that https://www.reddit.com/r/rss/comments/h0ahkw/how_to_get_a_twitter_rss/?sort=top some RSS-Readers already support this, maybe its possible to get info from them? https://github.com/samuelclay/NewsBlur seems to be able to subscribe to twitter. Maybe its possible via https://nitter.net/ as well?

Year 0 feeds break polling loop

12:28:06	ERROR	instance.rss bot	Error while polling feeds
12:28:06	ERROR	instance.rss bot	Traceback (most recent call last):
  File "/data/plugins/rss.mbp/rss/bot.py", line 112, in _poll_feeds
    await self._poll_once()
  File "/data/plugins/rss.mbp/rss/bot.py", line 101, in _poll_once
    new_entries = {entry.id: entry for entry in self.find_entries(feed.id, entries)}
  File "/data/plugins/rss.mbp/rss/bot.py", line 155, in find_entries
    ) for entry in entries]
  File "/data/plugins/rss.mbp/rss/bot.py", line 155, in <listcomp>
    ) for entry in entries]
  File "/data/plugins/rss.mbp/rss/bot.py", line 137, in get_date
    return datetime.fromtimestamp(mktime(entry["published_parsed"]))
ValueError: year 0 is out of range

Please add install instructions

Hello!

Please add install instructions. :)

Identify Failed Feeds and possibility to delete them

The log does not show which feed fails, so there is no possibility to delete failed feeds from the DB.

Would be cool to add the possibility to delete e.g. feeds with > 5 continuous fails from the feed db in maubot manager.

IntegrityError when subscribing to new feed

Hi all,

I have set up the latest v0.2.3 release on my homeserver. Now, when I try subscribing to a new feed (!rss subscribe URL), I get the following error in my log:

Traceback (most recent call last):
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: NOT NULL constraint failed: entry.feed_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/maubot/lib/python3.8/site-packages/mautrix/client/syncer.py", line 200, in _catch_errors
    await handler(data)
  File "/opt/maubot/lib/python3.8/site-packages/maubot/handlers/command.py", line 106, in __call__
    ok, res = await self.__call_subcommand__(evt, call_args, remaining_val)
  File "/opt/maubot/lib/python3.8/site-packages/maubot/handlers/command.py", line 122, in __call_subcommand__
    return True, await subcommand(evt, _existing_args=call_args,
  File "/opt/maubot/lib/python3.8/site-packages/maubot/handlers/command.py", line 114, in __call__
    return await self.__mb_func__(self.__bound_instance__, evt, **call_args)
  File "/opt/maubot/plugins/xyz.maubot.rss-v0.2.3.mbp/rss/bot.py", line 280, in subscribe
    self.db.add_entries(entries, override_feed_id=feed.id)
  File "/opt/maubot/plugins/xyz.maubot.rss-v0.2.3.mbp/rss/db.py", line 151, in add_entries
    self.db.execute(self.entry.insert(), entries)
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2235, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/opt/maubot/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: entry.feed_id
[SQL: INSERT INTO entry DEFAULT VALUES]
(Background on this error at: http://sqlalche.me/e/13/gkpj)

I can't seem to figure out why this is happening. The feed is displayed in the database's "feed" table, but the "subscription" and "entry" tables both remain empty. Any ideas?

Feature request: `!rss notice * false`

It would be nice to allow setting the !rss notice variable to false as a bot-wide (or room-wide) default. I prefer it to use the m.text for all messages, but it's a hassle to do that for each and every feed individually.

Possible UI:

In a room: !rss notice * false to set all feeds in the room to m.text
In the bot config: default_notice: true, which can be changed to false to set all feeds in all rooms to m.text by default.

Feed with authentification not working

I have a feed which has the form https://username:[email protected].

If I try to subscribe with the valid username and password (I can open it with any browser) I only get
"Failed to load feed: Response payload is not completed"
from the maubot.

Is this possible to fix?

microservices has no field feed.id

There are some rss feeds which do not have an id field. This creates an exception. One example is

https://sdtimes.com/tag/microservices/feed/

But there are way more...

I for myself learned: if you create a rss based program and publish it, the prog gets an exception hell...

[Feature Request] Add the feature of reading the description of RSS

Right now, we can only see the title and URL of the RSS. I wish there was an option for us to see the RSS description.

Automatically Detect Feeds in HTML.

For example

!rss subscribe https://kevincox.ca
!rss subscribe https://www.youtube.com/c/TechnologyConnections

Should work. Instead they give an error because they try to parse the HTML as XML.

Feed discovery is a well understood process but the basics is that if you get an HTML response back you should check for links such as <link rel="alternate" type="application/rss+xml" href="{feed-url}"> in the HTML and HTTP headers. One complication is that if there are multiple feeds available the user should probably be asked which one to select.

A simple UX for selecting feed is just displaying the title and printing !rss subscribe {url} lines for the user to copy-paste. However better UX would be something like listing them with numbers and allowing the user to click a reaction to pick which feed they want to subscribe to.

Failed to load feed: <unknown>:23:70: not well-formed (invalid token)

!rss subscribe https://www.kaggle.com/anistaluqdar/code
!rss subscribe https://github.com/AnisTaluqdar/111_CORA_111

This gives me the error ( Failed to load feed: :23:70: not well-formed (invalid token) ), What's wrong?

RSS feeds broadcasted in matrix room once after maubot start

Something goes wrong. On both self hosted and t2bot.io instances. No errors in log on selfhosted, but rss bot broadcast in matrix room only once after start, then sleep forever. If i stop and start client in webadmin, rss broadcasted again once.

Add no-send option to config, to skip sending updates to matrix

We are currently migrating feeds to a different baseURL, which means that maubot-rss would recognize all items as new, because their url changed. We don't want all users to get spammed with old messages again.

https://github.com/rss2email/rss2email#using-rss2email has a --no-send flag which still logs "new" items into the database as seen, but skips sending a notification for them.

I could imagine such an option beeing useful for maubot-rss as well. If this is wanted, I would offer to implement this somewhat like this:

# bot.py line 143...
skip_broadcast = self.config["skip_broadcast"]
if not skip_broadcast:
    for entry in new_entries.values():
        await self._broadcast(feed, entry, feed.subscriptions)
else:
    self.log.debug("Skipped broadcasting new items because config option skip_broadcasting was set to True")

AttributeError: object has no attribute 'id'

Error while polling feeds
23:17:26 ERROR instance.rss Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/feedparser.py", line 398, in getattr
return self.getitem(key)
File "/usr/lib/python3.8/site-packages/feedparser.py", line 356, in getitem
return dict.getitem(self, key)
KeyError: 'id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data/plugins/xyz.maubot.rss-v0.1.0.mbp/rss/bot.py", line 111, in _poll_feeds
await self._poll_once()
File "/data/plugins/xyz.maubot.rss-v0.1.0.mbp/rss/bot.py", line 100, in _poll_once
new_entries = {entry.id: entry for entry in self.find_entries(feed.id, entries)}
File "/data/plugins/xyz.maubot.rss-v0.1.0.mbp/rss/bot.py", line 147, in find_entries
return [Entry(
File "/data/plugins/xyz.maubot.rss-v0.1.0.mbp/rss/bot.py", line 149, in
id=entry.id,
File "/usr/lib/python3.8/site-packages/feedparser.py", line 400, in getattr
raise AttributeError("object has no attribute '%s'" % key)
AttributeError: object has no attribute 'id'

Modularize parsing to support other feed types

e.g. Wordpress's wp-json API, https://jsonfeed.org/ and others

Pass through HTML when using feed entry body

Currently it escapes the HTML

Feature request: Cleanup DB - Delete older entries so DB does not blow up

It would be great if there would be a way to cleanup the DB from old entries. I run lot of RSS feeds, so DB blows up and e.g. cleaning out every entry older then 1 year, or not anymore existing entries in the feed, would be cool.

Thank you

Failed to load feed: RPiLocator In Stock Feed

Trying to add subscriptions to know when Raspberry PIs come back into stock, however:

!rss subscribe https://rpilocator.com/feed/

Returns:
Failed to load feed: <unknown>:13:2: mismatched tag

document how feed templates work

How can i fetch older Feeds?

Its possible to fetch older feeds e.g 10 days or 10 years agao?

No feedback on invalid RSS URL

When specifying an invalid feed url, the bot says it is subscribed. !rss subscribe https://example.org/nonexistingfeed.xml

add command to see faild feeds

some feed throw an exception when they are polled. it would be nice to have a command that shows the feeds that are failing.

Document feed template

!rss help template command should print template language description.

Not working

Doesn't work when I try to make an instance. It says "internal server error" but shows up when I reload. Then permanently stays off.

Feature Request: add images support in templates, e.g. via posting the image URLs

Mastodon rss feeds can frequently have images, so it might be cool to be able to have them available for templates, or at least have a boolean for the template if there is attached media/files so that one could make an according if-condition in the template to change things slightly if there are