Suggestion: future architecture for scale and efficiency about twitchalerts HOT 3 OPEN

Hi,
I'm currently using PM2 but only with one process. Take note that Discord requires to have shards for every group of 2000 servers. So we can use that to have Discord workers (currently I have three shards).
I also want to keep the web server as it's more flexible than Discord (for example, I use Discord's Modal for /setup but they don't support checkboxes).
Currently the alert is updated every ~2 minutes, so if the streamer change his game, it's updated asap on Discord.

In the future, I want to have only EventSub connection with Twitch. So when a user changes the stream title or game, Twitch send the bot a message using this event.

from twitchalerts.

In terms of approach, here's a few random bulletpoints that I can expand on for anything interesting.

• By and large start fresh, with a new repository such as harfeur/twitchalerts-ng. This is mostly due to the next two points, but also allows you to move on from the MIT/GuideBot attribution as it would be a blank start. Granted MIT isn't a shackle, but still, you'd be writing from scratch so attribution not needed.
• Align to ES6 modules and standardise as .mjs / mark package.json as type: module.
• Use eslint to align code to recommended / possibly document up a standard vscode workspace/settings.json that's part of the repo for all contributors to use as standard tooling to enforce consistency.
• Keep the code base for all three in the same repository, but with three entrypoints of something like server-webui.mjs, server-eventsub.mjs, server-discord.mjs for the actual parts needed.
• Create a dev instance of the bot, and a discord server for developers and the dev instance to write testing notifications into. Pick a few streamers that run throughout the day/week so there's always a couple of active live notices.
• Use github actions to deploy code to the dev instance while we're testing and utilise pm2 to gracefully restart workers.
• I don't suggest you comment on where/how you host the current Prod instance in public but for a dev instance we could collaborate on Oracle Cloud as it has an 'always free' tier of upto 24GB RAM and 6 vCPUs on ampere arm which could run the pm2 stack as well as a 0.5gb RAM 1vCPU AMD VM which could run postgres and nginx-proxy-manager.

I just saw your reply come in as well so few more thoughts before hitting submit :)

• Looks like discord.js guidance actually suggests 1000 servers per shard despite discord supporting upto 2,500. Wonder why the disconnect, maybe djs was written before discord increased limits or something.
• Given that information from you (thank you) I guess we start one PM2 worker for the whole discord microservice, but then allow that to shard as needed.
• A second PM2 worker for eventsub and either combine it with webui or a third PM2 worker for webui dedicated. I didn't mean to suggest that webui would go away, only that it could be combined with another microservice since it's only called when needed and has a lower overhead. Still, it would be better practice to have it's worker not clashing with an eventsub notice that needs to be handled.
• I agree about using EventSub only; as you say, channelupdate contains title and category, and there's dedicated stream.online and stream.offline subscriptions. That'll do away with polling entirely.
• Switching to EventSub does then hit into Twitch limits, so you need to start having users sign in with Twitch OAuth to manage the costs, details here https://dev.twitch.tv/docs/eventsub/manage-subscriptions/#subscription-limits hence the change suggested above in the diagram.
• Switching to EventSub also cuts out any duplication at the twitch interaction in the case multiple servers are monitoring the same streamers.
• Switching to EventSub means you can use the event to drive updating the title etc, and maybe just think of a smarter way to do the "stream uptime" counter so it doesn't require overhead.

from twitchalerts.

Also

• Re-write webui using the same template/design as a Nuxt hybrid SSR/SPA app; everything behind /dashboard as an SPA so it renders on the client for high performance and low server impact, but the homepage, privacy policy etc as hybrid SSR/SPA. It'll lower server load for interactive users but maintain spider/search results for the pages that don't require authentication.

from twitchalerts.