spedrickson / ytchat-backend Goto Github PK

Currently ytchat-backend only supports a single YouTube channel per instance, configured through the YTCHAT_CHANNELID environment variable.

Instead, it should be refactored to accept channelID as a query parameter.

Steps needed for this enhancement:

• Remove YTCHAT_CHANNELID env variable
• Refactor Mongoose to provide multiple mongoDB connections (example 1, example 2). For scalability, consider the Flyweight design pattern.
• Refactor API permissions to be at the channel level, rather than the backend instance level. Easiest way would be to add a function to src/user/schemas/user.service.ts that accepts a channelID and API key, and returns the auth details.
• Update ytchat-frontend to use new query parameter.

Outline

Currently there is a scalability problem with the fillAuthorCache() function. As-is, it asks MongoDB to perform the following aggregation:

1. sort all messages by timestamp (should already use index)
2. group messages by channelID with the latest message's userdata and timestamp as extra fields
3. return collection of every unique user to backend,

This means that it's using O(n) space complexity, with N as the number of unique users that have left a message since ingest began. And if the MongoDB host doesn't have enough memory, it'll become IO-bottlenecked by writing partial results to disk.

This also means the backend is using O(n) space to store this information, which may not be best practices.

Currently takes ~3 minutes to run against 1.1m users, and will continue to grow from there.

Proposed Solution

• Change the fillAuthorCache() function to a fillAuthorCollection() function, which uses aggregation to store the author data in the DB as a separate collection named authors.
• Change getAuthorBySearch() endpoint function to search database instead of local authorCache. Will need to investigate possible solutions to maintain fuzzy search, or change it to simple text matching and/or regex.
• Change the refreshAuthorCache() function to update the authors collection, rather than pulling new authors into the backend's authorCache.
• Remove authorCache entirely, now that it is unused.