eikowagenknecht / lootscraper Goto Github PK

RSS feeds and Telegram bot for free game and loot offers.

License: Other

Dockerfile 0.42% Python 99.24% Mako 0.17% Jinja 0.17%

lootscraper's Introduction

LootScraper

You like getting games for free but you don’t like having to keep track of the various sources (Amazon Prime, Epic Games, Steam, ...) for free offers? Also your F5 key starts to look a bit worn out? Then this is for you!

Public feeds / channels / bots

Let's face it, you're probably not here because you're interested in the technical details of how this works. You just want free games. And that's fine. So without further ado, here's where to find them! You have the following options:

Telegram channels
Telegram bot
RSS feeds
E-Mail
Discord
HTML pages

Telegram channels

Want to get a Telegram notification every time a new offer is discovered? Simply subscribe to the Telegram channels.

For our mobile gamers:

Telegram bot

Want to receive only the offers you want in a single chat? Subscribe directly to the source: The Telegram LootScraperBot will happily send you push notifications for new offers. You can choose which categories you want to subscribe to.

If you want, you can even add the bot to your own groups (including threaded groups) and channels. Just make sure to give it the neccessary permissions (admin rights work best).

This is what it currently looks like in Telegram:

RSS feeds

Prefer the anonymity and manageability of RSS feeds? Sure. You can use the links below. These feeds contain all active offers.

You can also have all sources in one feed.

This is how it looks in Feedly:

E-Mail

If you want to get the offers by email, you can use free services like Blogtrottr or Feedsub to convert from RSS to email.

Discord

If you want to get the offers in a Discord channel, you can use the free MonitoRSS bot to post them there for you. I suggest the following settings:

Feed URL: Use the RSS feed links above
Content:
```
**{{title}}**

{{description}}
```
Button: Label: Claim, URL: {{link}}
Embed: Just select Image URL with the {{extracted::atom:content__#::image1}} variable

HTML pages

Want to check a website for new deals once in a while, but not be bothered by push notifications? Here are the latest offers in a nice and clean HTML page:

You can also have all sources on one page.

This is how it looks like:

HTML archive

There is also an archive version of the HTML pages. These contain all offers that have been discovered so far, including expired ones.

You can also have all sources on one page.

This is how it looks like:

How it works

This Python (3.12+) application uses Playwright to automatically visit websites with free game-related offers (see below for the supported sources) and then puts the collected information neatly into RSS feeds and a Telegram bot.

If you encounter any problems feel free to open an issue here and I'll do my best to help. I'd also love to hear your feature requests! This is also tracked in the GitHub issues.

For power users and developers

You can either run this project locally on your computer or in any environment capable of running a Docker container.

If you want to do this or even contribute, please see the README for developers file.

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

lootscraper's People

Contributors

Stargazers

Watchers

Forkers

lenicyl doiska codewhizz

lootscraper's Issues

Feed name for single feeds does not include the type

Should be "Free Amazon Prime Games", but is "Free Amazon Prime".

No image for games from Steam

Steam now is supported as a new source of free games, but to not scrape it too many times, only the results page is visited.

This leads to a situation where we have no image for the game.

Supposed fix:

Use the gameinfo component to add the steam images to all games that are found there.
In the feed generator, use the image from steam if no own image exists.

Better method for adding announcements

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Currently announcements (see #31) have to be added by manually executing code or with a db edit.

Describe the solution you'd like

Would be nice if that could just be handled with a telegram command like

/announce Title of announcement || Text of announcement

Describe alternatives you've considered

No response

Anything else?

No response

Add support for more sources

Amazon Prime (Games and InGame)
Epic Games
Steam
Web API: https://steamapi.xpaw.me/
SteamKit API (used i.e. by SteamDB): https://github.com/SteamRE/SteamKit
Python library: https://github.com/ValvePython/steam
Maybe https://store.steampowered.com/search/?maxprice=free&specials=1 is a way to get free steam offers (needs to be checked).
GOG.com
https://www.gog.com/partner/free_games
itch.io
Stuff listed on https://www.reddit.com/r/FreeGameFindings/
Xbox Game Pass (https://www.xbox.com/en-US/xbox-game-pass/games#PCgames)
isthereanydeal.com

Create Discord channel

One more way for users to subscribe: Discord. Look into creating a server and design a method that pushes the updates there as well.

Add support for further game information

Add this for game information, it is free and used by Twitch e.g.: https://www.igdb.com/api

Support for filtering (wantlist / blocklist)

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

For the Telegram bot, it would be nice to have a possibility to only get offers matching keywords that you want or block keywords that you don't want.

Probably by category (e.g. a wantlist like "Hearthstone, League of Legends" for Loot) and some blacklist like "Ultima" for games)

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Show "Show details" button in telegram only if there are actually any more details to display

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Add support for Humble Bundle

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Add game info / Steam score

When looking at a free offers there is still some cost in terms of time investment it takes to actually claim the offer. To have a quick way to determine if it's worth your time, it would be nice to have more information about the game, e.g. a short description and a score. Since metacritic seems to be hard to access programatically, the Steam score would be an approachable alternative. Sometimes it even contains the metacritic score!

API: https://partner.steamgames.com/doc/webapi/ISteamApps
All games: http://api.steampowered.com/ISteamApps/GetAppList/v0002/?format=json
Game details: https://store.steampowered.com/api/appdetails?appids=10 > metacritic and total recommendations, not percentage
https://store.steampowered.com/app/10/
Storefront API: https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI

The main problem is matching the scraped game name to an entry in the steam database (=steam app id). A possibility would be to use the search field of the Shop, e.g.

https://store.steampowered.com/search/?term=Rainbow+Six+Siege&category1=998 -> Tom Clancy's Rainbow Six® Siege

Add tests

This project needs some tests. Especially because the scraped web pages are bound to change some time it would be good to have a realiable way of knowing when things break.

Maybe pytest 7 is worth looking at: https://www.heise.de/news/Test-Framework-pytest-7-bietet-mehr-Type-Annotations-und-baut-Altlasten-ab-6349971.html

Add support for announcements

Whenever there is a big new feature (as for exmple a telegram bot), it would be nice to have the possibility to announce that.

Store as a new table "announcements" with fields "date" and "text".

The question is: How to enter new announcements since we have no user interface? A simple way would be to just pase an "announcement.txt" file in the data directory. Whenever this file exists, read the content into the database on the next run and crate an announcement out of it.

Features

Advanced features

Add a preview picture
Add the steam score for the game
API: https://partner.steamgames.com/doc/webapi/ISteamApps
All games: http://api.steampowered.com/ISteamApps/GetAppList/v0002/?format=json
Game details: https://store.steampowered.com/api/appdetails?appids=10 > metacritic
https://store.steampowered.com/app/10/
Storefront API: https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI
Telegram Bot / Notifications
Generate a web page like https://www.indiegamebundles.com/category/free/

Normalize database

Split the table structure in the database into games and offers, so that it is easier to generate useful pages automatically (like "all offers for game xyz).

Automatic creation of config.ini doesn't work.

Reproduction:

Delete / rename the folder "data" containing the config.ini
Run lootscraper.py

Possible root:
The folder data does not exist and is not created by the script.

Problem with docker-compose

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Error when building docker image

Expected Behavior

No response

Steps To Reproduce

docker-compose.yml:

version: "3.9"

services:
    lootscraper:
        image: eikowagenknecht/lootscraper:latest

docker compose up

Log:

rss-server-lootscraper-1  | 2022-09-29 16:21:26,927 alembic.runtime.migration [INFO ] Running upgrade d0cab9037616 -> 8267b60db582, Datetime
rss-server-lootscraper-1  | Traceback (most recent call last):
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
rss-server-lootscraper-1  |     self.dialect.do_execute(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
rss-server-lootscraper-1  |     cursor.execute(statement, parameters)
rss-server-lootscraper-1  | sqlite3.OperationalError: near "DROP": syntax error
rss-server-lootscraper-1  | 
rss-server-lootscraper-1  | The above exception was the direct cause of the following exception:
rss-server-lootscraper-1  | 
rss-server-lootscraper-1  | Traceback (most recent call last):
rss-server-lootscraper-1  |   File "/app/lootscraper.py", line 463, in <module>
rss-server-lootscraper-1  |     main()
rss-server-lootscraper-1  |   File "/app/lootscraper.py", line 51, in main
rss-server-lootscraper-1  |     run_main_loop()
rss-server-lootscraper-1  |   File "/app/lootscraper.py", line 104, in run_main_loop
rss-server-lootscraper-1  |     db = stack.enter_context(LootDatabase(echo=Config.get().db_echo))
rss-server-lootscraper-1  |   File "/app/app/sqlalchemy.py", line 246, in __init__
rss-server-lootscraper-1  |     self.initialize_or_update()
rss-server-lootscraper-1  |   File "/app/app/sqlalchemy.py", line 279, in initialize_or_update
rss-server-lootscraper-1  |     command.upgrade(alembic_cfg, "head")
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
rss-server-lootscraper-1  |     script.run_env()
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
rss-server-lootscraper-1  |     util.load_python_file(self.dir, "env.py")
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
rss-server-lootscraper-1  |     module = load_module_py(module_id, path)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
rss-server-lootscraper-1  |     spec.loader.exec_module(module)  # type: ignore
rss-server-lootscraper-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
rss-server-lootscraper-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
rss-server-lootscraper-1  |   File "/app/alembic/env.py", line 116, in <module>
rss-server-lootscraper-1  |     run_migrations_online()
rss-server-lootscraper-1  |   File "/app/alembic/env.py", line 110, in run_migrations_online
rss-server-lootscraper-1  |     context.run_migrations()
rss-server-lootscraper-1  |   File "<string>", line 8, in run_migrations
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
rss-server-lootscraper-1  |     self.get_context().run_migrations(**kw)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
rss-server-lootscraper-1  |     step.migration_fn(**kw)
rss-server-lootscraper-1  |   File "/app/alembic/versions/20220414_195921_datetime.py", line 70, in upgrade
rss-server-lootscraper-1  |     op.drop_column("loot", "seen_first")
rss-server-lootscraper-1  |   File "<string>", line 8, in drop_column
rss-server-lootscraper-1  |   File "<string>", line 3, in drop_column
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/operations/ops.py", line 2189, in drop_column
rss-server-lootscraper-1  |     return operations.invoke(op)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/operations/base.py", line 399, in invoke
rss-server-lootscraper-1  |     return fn(self, operation)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/operations/toimpl.py", line 89, in drop_column
rss-server-lootscraper-1  |     operations.impl.drop_column(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/ddl/impl.py", line 331, in drop_column
rss-server-lootscraper-1  |     self._exec(base.DropColumn(table_name, column, schema=schema))
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/alembic/ddl/impl.py", line 195, in _exec
rss-server-lootscraper-1  |     return conn.execute(construct, multiparams)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
rss-server-lootscraper-1  |     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
rss-server-lootscraper-1  |     return connection._execute_ddl(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl
rss-server-lootscraper-1  |     ret = self._execute_context(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
rss-server-lootscraper-1  |     self._handle_dbapi_exception(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
rss-server-lootscraper-1  |     util.raise_(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
rss-server-lootscraper-1  |     raise exception
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
rss-server-lootscraper-1  |     self.dialect.do_execute(
rss-server-lootscraper-1  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
rss-server-lootscraper-1  |     cursor.execute(statement, parameters)
rss-server-lootscraper-1  | sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "DROP": syntax error
rss-server-lootscraper-1  | [SQL: ALTER TABLE loot DROP COLUMN seen_first]
rss-server-lootscraper-1  | (Background on this error at: https://sqlalche.me/e/14/e3q8)
rss-server-lootscraper-1 exited with code 1

Anything else?

No response

Notification when something goes wrong

When something goes wrong (e.g. a timeout occurs, the source page format changed, ...) it would be nice to have a notification to reach the administrator (in the case of the phenx.de hosted version me).

That could be managed by:

Sending an email
Sending a message on Telegram

Also errors need to be cought on the top level for this to work because otherwise the script is terminated in a Docker run environment.

Do not search for titles already in the database

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Currently all offers are matched against IGDB with the title (e.g. "The Witcher 3") and then the resulting IGDB id is used to see if the game is already in our database.

Describe the solution you'd like

This could be optimized by looking in our local IGDB and Steam data collection first and only use the web if nothing is found there.

Describe alternatives you've considered

No response

Anything else?

No response

Support for time zone changes.

Is your feature request related to a problem? Please describe.

It would be interesting if there was a way to change the bot's time zone to correspond with the country's time.

Describe the solution you'd like

Create an option to change the time zone in the bot.

Anything else?

No response

Discord Bot

Is your feature request related to a problem? Please describe.

Create a discord bot (open source code) that can send all offers in a channel, similar to what the Telegram bot already does.

Describe the solution you'd like

No response

Anything else?

No response

Add support for Uplay

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Auto serialize rawtext

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Add scraper for free mobile games (iOS, Android)

Sometimes there are offers for iOS and Android as well. We want to catch those as well.

Error in loading the page stops the script

This is bad for automatic runs, skip the run instead and log the error.

2022-03-16 08:18:30 [INFO ] Shutdown complete
2022-03-16 08:18:31 [INFO ] Found 0 new offers
2022-03-16 08:18:31 [INFO ] Waiting until 2022-03-16T08:38:31.313360 for next execution
2022-03-16 08:38:31 [INFO ] Starting Run # 335
2022-03-16 08:38:31 [INFO ] Start scraping of Amazon Prime
2022-03-16 08:38:31 [INFO ] Getting pagedriver options
2022-03-16 08:38:31 [INFO ] Creating driver
2022-03-16 08:38:31 [INFO ] Injecting JS
2022-03-16 08:40:39 [ERROR] Failure starting Chrome WebDriver, aborting: unknown error: net::ERR_CONNECTION_TIMED_OUT
  (Session info: headless chrome=99.0.4844.51)

Use module specific loggers

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Currently quite some code uses the root logger ("logging.info(...)").

Describe the solution you'd like

Use module specific logging instead, as seen here: https://docs.python.org/3.10/howto/logging-cookbook.html

Describe alternatives you've considered

No response

Anything else?

No response

test

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Detection whether a game is always free on GoG does not work

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Gwent is detected as valid offer even tho it's free forever.

Url: https://www.gog.com/en/game/gwent_the_witcher_card_game

Expected Behavior

Gwent is not considered as an offer.

Steps To Reproduce

Run GoG Game Scraper.

Anything else?

As soon as there is a new real offer from GoG compare what differs. I assume the claim button does not say "Play for Free" or there is a price instead of FREE:

Create Telegram bot

Not everyone has an RSS reader, so it would be an interesting alternative to be able to get the offers pushed from a Telegram bot or in a Telegram group instead.

Support for temporary free games

Is your feature request related to a problem? Please describe.

I.e. Steam Weekend: https://steamdb.info/upcoming/free/

Describe the solution you'd like

No response

Anything else?

No response

Publish this project

Check TODO comments in code.
Make URLs in feed customizable

Support multiple languages

Most platforms display the offerings in the language the browser sends. Currently it is hardcoded to "en-US", but it would be nice to (additionally) add more languages. The question is how those languages should be handled. Add all translations in the RSS feeds for the entry? Create seperate feeds for every language?

Filter out games that are always free (and demos)

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Especially Humble Store seems to be spammed with games that are always free anyways, e.g.

Influent, where the base game is always free and you need a DLC to really use it (Base game price = free)
Backbone: Prologue, where the first chapter is always free on steam ("Free to Play" genre on Steam)

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Use more non-static functions

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

A lot of methods look the same in all scrapers. Using non-static functions the base Scraper would call the methods of the child classes.

This get's more important as more functionality is added to the methods.

Describe the solution you'd like

Example scrape:
Current Scraper:

@staticmethod
    def scrape(driver: WebDriver) -> list[Offer]:
        raise NotImplementedError("Please implement this method")

Current GogGamesScraper (and most classes):

@staticmethod
    def scrape(driver: WebDriver) -> list[Offer]:
        return GogGamesScraper.read_offers_from_page(driver)

Proposed:

class Scraper(object):
    def read_offers_from_page(self, driver: WebDriver):
        raise NotImplementedError("Please implement this method")
    def scrape(self, driver):
        return self.read_offers_from_page(driver)

Describe alternatives you've considered

Try to create suitable static methods in the base class.

Anything else?

This could make way for default implementation in the Scraper, but python seems ill suited for that.

Improve scraping performance

Scraping is quite slow for some cases because a new ChromeDriver instance is build for every scraper. Probably a good idea would be to have only one instance for a whole run.

Twitter bot

Is there an existing issue for this?

I have searched the existing issues

Is your feature request related to a problem? Please describe.

Also post content on Twitter

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Anything else?

No response

Add default subscriptions for Telegram Bot

Is your feature request related to a problem? Please describe.

It seems some users are confused by how the bot works and forget to actually subscribe to anything.

Describe the solution you'd like

Try to encourage active use by adding a subscription to the common services that are available without constraints (e.g. Amazon Prime):

Steam games
Epic Games
by default for new users (and tell them about it of course).

Anything else?

No response

Add support for Steam giveaways

Every once in a while, there is a giveaway on Steam itself. But Steam seems not to be too easy to scrape due to the complexity of the API and other sites like SteamDB have anti bot protection. There seems to be a Telegram Bot sending out offers after registration (which itself scrapes SteamDB somehow and then verifies against Steam - https://github.com/etaxi341/FreeSteamGames-TelegramBot). Maybe just use the data from that channel?

Some more relevant links:

Web API: https://steamapi.xpaw.me/
SteamKit API (used i.e. by SteamDB): https://github.com/SteamRE/SteamKit
Python library: https://github.com/ValvePython/steam
Maybe https://store.steampowered.com/search/?maxprice=free&specials=1 is a way to get free steam offers (needs to be checked when there is actually an offer available..).

Telegram bot sometimes looses connection

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

The telegram bot sends telegram related errors to the developer. This however fails when sending a message is not possible, resulting in some ugly log entries:

2022-04-24 12:48:35 root [ERROR] An exception was raised while handling an update:

<pre>update = &quot;None&quot;</pre>

<pre>context.chat_data = None</pre>

<pre>context.user_data = None</pre>

<pre>Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 614, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 402, in _make_request
    six.raise_from(e, None)
  File &quot;&lt;string&gt;&quot;, line 2, in raise_from
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 398, in _make_request
    httplib_response = conn.getresponse()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 1374, in getresponse
    response.begin()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 318, in begin
    version, status, reason = self._read_status()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 287, in _read_status
    raise RemoteDisconnected(&quot;Remote end closed connection without&quot;
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/utils/request.py&quot;, line 259, in _request_wrapper
    resp = self._con_pool.request(*args, **kwargs)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/request.py&quot;, line 68, in request
    return self.request_encode_body(method, url, fields=fields,
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/request.py&quot;, line 148, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/poolmanager.py&quot;, line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 665, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/util/retry.py&quot;, line 347, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/packages/six.py&quot;, line 685, in reraise
    raise value.with_traceback(tb)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 614, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 402, in _make_request
    six.raise_from(e, None)
  File &quot;&lt;string&gt;&quot;, line 2, in raise_from
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connectionpool.py&quot;, line 398, in _make_request
    httplib_response = conn.getresponse()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 1374, in getresponse
    response.begin()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 318, in begin
    version, status, reason = self._read_status()
  File &quot;/usr/local/lib/python3.10/http/client.py&quot;, line 287, in _read_status
    raise RemoteDisconnected(&quot;Remote end closed connection without&quot;
telegram.vendor.ptb_urllib3.urllib3.exceptions.ProtocolError: (&#x27;Connection aborted.&#x27;, RemoteDisconnected(&#x27;Remote end closed connection without response&#x27;))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/ext/updater.py&quot;, line 646, in _network_loop_retry
    if not action_cb():
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/ext/updater.py&quot;, line 597, in polling_action_cb
    updates = self.bot.get_updates(
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/ext/extbot.py&quot;, line 224, in get_updates
    updates = super().get_updates(
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/bot.py&quot;, line 130, in decorator
    result = func(*args, **kwargs)
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/bot.py&quot;, line 3054, in get_updates
    self._post(
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/bot.py&quot;, line 295, in _post
    return self.request.post(
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/utils/request.py&quot;, line 361, in post
    result = self._request_wrapper(
  File &quot;/usr/local/lib/python3.10/site-packages/telegram/utils/request.py&quot;, line 265, in _request_wrapper
    raise NetworkError(f&#x27;urllib3 HTTPError {error}&#x27;) from error
telegram.error.NetworkError: urllib3 HTTPError (&#x27;Connection aborted.&#x27;, RemoteDisconnected(&#x27;Remote end closed connection without response&#x27;))
</pre>

Expected Behavior

Short problems in the connection should be handled with a retry mechanism, longer outages should be logged more succinctly as an error.

Steps To Reproduce

No response

Anything else?

No response

Discord addon for Brayanbot

Is your feature request related to a problem? Please describe.

There is a modular bot called Brayanbot, which is developed in Javascript. It would be nice to have an addon that could send notifications to members of what offers are available, which could be used on command and automatically sent as soon as a new offer arrived.

Describe the solution you'd like

No response

Anything else?

No response

Docker image from Readme not publicly visible

Docker needs to be installed first of course. If you want to skip the build step, you can use https://hub.docker.com/r/eikowagenknecht/lootscraper as the image.

The referenced image is not publicly visible (404).

This happens even when logged in with a Docker account.

Additional feed for games that are always free

This feed currently only includes offers, meaninggames that usually cost money but currently are available for free.

We could add an additional feed for games that are always free.
E.g.

https://www.gog.com/partner/free_games
Many games from https://itch.io

Sometimes Steam information gets scraped in other languages

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Steam genres are sometimes in the wrong language
999220 -> Приключенческие игры, Инди
268500-> Strategické

Expected Behavior

Everything should be in english

Steps To Reproduce

No response

Anything else?

No response

Generate a web page from the database

It would be nice to have a web page that displays the information from the RSS feed as well. This could be done by just generating a static .html file on every feed update. Probably not the longterm solution because it will start to grow really bit, but some JS filtering should go a long way and be a solid way to do this for at least some thousand entries.

Also: Every RSS entry could then link to that web page for details. Maybe that's a good place to put the alternative languages as well? (see #16).

Add support for more sources

This is not the only way of being notified about free offers. Whole reddit channels exist that manually do the work this script does and sometimes a bit more. I'm thinking of https://www.reddit.com/r/FreeGameFindings/ for example or https://new.isthereanydeal.com/giveaways/ / https://isthereanydeal.com/specials/#/filter:&giveaway. Maybe we could use some of that community's work (after checking the legality of course ;-)). At the very least it's a good indicater of what offers other sites may have and then build scrapers for those sites.

Telegram Channel Support

Is your feature request related to a problem? Please describe.

Adicionar suporte para adicionar bot aos Canais do Telegram.

Describe the solution you'd like

Unfortunately when I add the bot to the channel, I can't use the commands.

Anything else?

No response

For amazon loot the game is missing in the Telegram message

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

For amazon loot only the title is shown and not which game it belongs to.

Expected Behavior

Also show the title of the game the loot belongs to.

Steps To Reproduce

No response

Anything else?

No response

Add support for Xbox Game Pass

Not strictly free games because you need a game pass (which costs money). Could still be interesting to make an RSS feed out of the available games there. Would have to not be included in the free main feed, though and be kept in a seperate feed.

Source of the available games: https://www.xbox.com/en-US/xbox-game-pass/games#PCgames

Use flood control in all sends

Is your feature request related to a problem? Please describe.

In some cases we use reply_markdown_v2 instead of our new flood controlled method.

Describe the solution you'd like

Change all usages to honor flooding responses here as well.

Anything else?

No response

Add support for itch.io

On itch.io there are some offers as well that could be included. The difficult thing here is to distinguish between real offers and low quality games. An approach would be to use https://itch.io/games/new-and-popular/on-sale as a starting point and include all games that cost 0€ and also have a discount of 100% (meaning they are not always free).

Unfortunately this mostly includes games with a recommended price of 1.00€ and low quality. This is probably not what users want to see when they look for free offers. So we would neet another metric to determine the "wantedness" of an offer. The recommended selling price? A rating score?

Not sure if it's worth including this site for the few real offers it sometimes contains.

Add support for GOG.com

Every once in a while, new games are added to gog.com as well. Those are found on https://www.gog.com/partner/free_games