lflare / lemmy-subscriber-bot Goto Github PK

View Code? Open in Web Editor NEW

27.0 2.0 3.0 78 KB

Lemmy Subscriber Bot (LSB) for easy community federation :)

License: GNU Affero General Public License v3.0

Dockerfile 2.02% Python 97.98%

federation lemmy lemmy-bot community seeder bot

lemmy-subscriber-bot's Introduction

Lemmy Subscriber Bot (LSB)

NOTE: THIS TOOL WAS CREATED FOR PERSONAL PURPOSES AND I WILL NOT BE HELD RESPONSIBLE FOR ANY MISUSE OF THIS TOOL.

Tired of having to manually find federated Lemmy communities? Tired of having to rely on centralised Lemmy instances to find the best communities? Tired of having to manually subscribe to every single one?

Look no further, because this tool will do all of that for you!

NOTE: THIS TOOL WAS CREATED FOR PERSONAL PURPOSES AND I WILL NOT BE HELD RESPONSIBLE FOR ANY MISUSE OF THIS TOOL.

P.S. Please only use this tool on your own Lemmy instance servers.

Usage

Docker

# To run it once in the background
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot

# To run it as a daemon
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' --restart always lflare/lemmy-subscriber-bot --daemon

# To run it only on selected instances
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot --instances 'lemmy.ml,beehaw.org'

# To run it except selected instances
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot --instances '!badlemmy.com'

# WARNING: Do not use the following unless you are familiar with the code!
# To reset subscriptions by unsubscribing from everything
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot --reset

# To filter for only non-NSFW communities
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot --no-nsfw

# To filter for only undefined, or english languages
docker run --name lemmy-subscriber-bot -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lflare/lemmy-subscriber-bot --lang 'und,en'

## OR

docker build -t lemmy-subscriber-bot .
docker run --name lemmy-subscriber-bot --restart always -dt --env 'LEMMY_USERNAME=subscriber_bot' --env 'LEMMY_PASSWORD=subscriber_bot' --env 'LEMMY_DOMAIN=lemmy.world' lemmy-subscriber-bot

Manual

$ pip3 install -r requirements.txt

$ python3 bot.py -h
usage: bot.py [-h] [-v] [--reset] [--database DATABASE] [--domain DOMAIN] [--username USERNAME] [--password PASSWORD] [--threshold-add THRESHOLD_ADD] [--threshold-subscribe THRESHOLD_SUBSCRIBE] [--daemon] [--daemon-delay DAEMON_DELAY] [--instances INSTANCES] [--no-nsfw]
              [--lang-codes LANG_CODES]

lemmy-subscriber

options:
  -h, --help            show this help message and exit
  -v, --verbose
  --reset
  --database DATABASE
  --domain DOMAIN
  --username USERNAME
  --password PASSWORD
  --threshold-add THRESHOLD_ADD
  --threshold-subscribe THRESHOLD_SUBSCRIBE
  --daemon
  --daemon-delay DAEMON_DELAY
  --instances INSTANCES
                        comma-separated instances, e.g. 'lemmy.ml,beehaw.org'
  --no-nsfw
  --lang-codes LANG_CODES
                        comma-separated language codes (e.g. und, en, de)

FAQ

What was the motivation behind this?

As of the writing of this tool (Jul 2023), there is no easy way for personal/small Lemmy instance users to discover communities outside of their own. You might suggest using one of the many aggregators out there, like Lemmy Explorer, but the whole act of adding a community, one by one, is a painful, arduous and unintuitive experience altogether. This tool was written to give the users of the instance a view of what communities might be out there without having to jump through hoops, and if popular enough (meeting the requirements of the bot subscribing), an overview of the community without the user having to subscribe to it.

Wouldn't this cause unnecessary load on upstream servers?

Well, yes. It is undeniable that in the wrong hands, this tool, and by extension, the many tools just like this, can cause havoc on the larger Lemmy instances out there. As of this writing, I do not have a solution for that, except to try and alleviate some possible causes for concern by only subscribing to the most active communities that an average Lemmy user might find interesting.

How much disk space is this tool expected to cause?

As of the writing of this tool, and size of the fediverse (Jul 2023), using this tool, may result in disk space usage of around 2GiB/day, according to my own metrics.

lemmy-subscriber-bot's People

Contributors

Stargazers

Watchers

Forkers

gerard780 vgrassia estork09

lemmy-subscriber-bot's Issues

0.19 support

The tool is not working on 0.19 FYI

Error

2023-07-05 18:25:09.244 | ERROR    | __main__:subscribe_community:182 - Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           │           │     │    │       └ {}
           │           │     │    └ <property object at 0x7f6824271bc0>
           │           │     └ <Response [408]>
           │           └ <function loads at 0x7f68250ca0c0>
           └ <module 'json' from '/usr/local/lib/python3.11/json/__init__.py'>
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           │                │      └ ''
           │                └ <function JSONDecoder.decode at 0x7f68250c9a80>
           └ <json.decoder.JSONDecoder object at 0x7f68250cc290>
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               │    │          │      │  └ ''
               │    │          │      └ <built-in method match of re.Pattern object at 0x7f68253b4ee0>
               │    │          └ ''
               │    └ <function JSONDecoder.raw_decode at 0x7f68250c9b20>
               └ <json.decoder.JSONDecoder object at 0x7f68250cc290>
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
          │                                  └ ''
          └ <class 'json.decoder.JSONDecodeError'>

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f68250fea20>
    └ <Thread(Thread-2 (community_subscriber_thread), started daemon 140085250935608)>
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f68250fe700>
    └ <Thread(Thread-2 (community_subscriber_thread), started daemon 140085250935608)>
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(Thread-2 (community_subscriber_thread), started daemon 140085250935608)>
    │    │        │    └ ()
    │    │        └ <Thread(Thread-2 (community_subscriber_thread), started daemon 140085250935608)>
    │    └ <bound method Bot.community_subscriber_thread of <__main__.Bot object at 0x7f6823d2c110>>
    └ <Thread(Thread-2 (community_subscriber_thread), started daemon 140085250935608)>

  File "///bot.py", line 205, in community_subscriber_thread
    self.subscribe_community(community_addr)
    │    │                   └ 'https://lemmy.world/c/wefwef'
    │    └ <function Bot.subscribe_community at 0x7f6823d137e0>
    └ <__main__.Bot object at 0x7f6823d2c110>

> File "///bot.py", line 165, in subscribe_community
    id, resolved = self.resolve_community(community_addr)
                   │    │                 └ 'https://lemmy.world/c/wefwef'
                   │    └ <function Bot.resolve_community at 0x7f6823d13740>
                   └ <__main__.Bot object at 0x7f6823d2c110>

  File "///bot.py", line 148, in resolve_community
    if "error" in r.json():
                  │ └ <function Response.json at 0x7f6823ff4fe0>
                  └ <Response [408]>

  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
          └ <class 'requests.exceptions.JSONDecodeError'>

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

2023-07-05 18:25:09.690 | INFO     | __main__:get_instance_communities:125 - QUEUED RESOLVE: lemmy.studio/atheistmemes
2023-07-05 18:25:10.106 | INFO     | __main__:get_instance_communities:122 - QUEUED SUBSCRIBE: lemmy.click/artificial_intel

Exception in thread Thread-1 (community_resolver_thread):
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "//bot.py", line 194, in community_resolver_thread
    self.resolve_community(community_addr)
  File "//bot.py", line 148, in resolve_community
    if "error" in r.json():
                  ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Stops working without error

I have tried running it on my instance (latest version )but it stops working after checking a few communities.

Here is the log:
2023-07-04 17:17:50.815 | SUCCESS | main:retrieve_jwt:85 - LOGGED IN AS all
2023-07-04 17:17:58.379 | INFO | main:get_instance_communities:125 - QUEUED RESOLVE: lemmygrad.ml/television
2023-07-04 17:17:58.379 | INFO | main:get_instance_communities:122 - QUEUED SUBSCRIBE: lemmygrad.ml/formula1
2023-07-04 17:17:58.422 | INFO | main:resolve_community:151 - RESOLVED: https://lemmy.world/c/television

Introduce automated first-class federation crawling

Bot currently only crawls communities from https://github.com/maltfield/awesome-lemmy-instances as a proof-of-concept code. This is not ideal in the long-term, thus will require better instance crawling support.

Release new Docker image

Could we either get a new release and push to dockerhub or at least get an updated nightly?