plastic-labs / tutor-gpt Goto Github PK

add support for latex in web ui, katex comes to mind

Switch to using a summary buffer as context is being lost from early in the conversations.

On a mobile device when using the web ui the drawer is open by default. This led to some customer confusion when first using the application

Could potentially add text too the header in mobile that says see thoughts here. Use a media query to make it hidden otherwise

When Bloom/tutor-gpt is making lists occasionally it will format different entries differently. A few examples were raised by a user in the discord server.

entry 10 ^

And the below instance on the streamlit shows it changing on both 9 & 10

Add some kind of integration to an analytics provider. Examples of these include

• Google Analytics
• Posthog
• Hotjar

Personal preference would be Posthog due to previous usage

We need to remove the dependency on the model following our pattern perfectly for parsing out the thoughts and responses

• it can misspell the following word (like we saw it spell "Studen:" once)
• it can mis-interpret the pattern, thinking it's Thought-Student only

Some quick chatting with ChatGPT got me the following string that parses everything between "Thought:" and the first newline character:

import re
...
result = re.findall(r'Thought: (.*?)(?=\n|$)', string, re.DOTALL)[-1].strip()

This pattern should be applicable across the board. Can't depend on it following the pattern or generating the correct next term.

Using just pip and a requirements.txt isn't a very strong packaging system and as we grow out will get messier. Should switch to something more robust.

Need to do more due diligence to make sure Poetry is the best one, but should switch to something.

Requirements to look for

• Some kind of locking mechanism (lockfiles)
• Proper version control for dependencies
• Scripting (would be nice)

Previously, the /context command just returned a message like "the context has been successfully set!"

We wanted the bot to start the conversation once a user sets the context. This changed the response from static to dynamic, requiring a starter_chain and obviously an OpenAI API hit.

What I found though, was that whenever I'd set the context, the chain would execute without any problems but I'd see in the fly logs this weird discord error:

discord.errors.ApplicationCommandInvokeError: Application Command raised an exception: NotFound: 404 Not Found (error code: 10062): Unknown interaction

After enough googling, I learned that the reason this happens is because, for slash commands, there's a limit on the amount of time you can go without responding. From the Discord docs:

In our case, the time required to run the starter_chain, get a response from OpenAI, and respond via Discord was exceeding this threshold, causing the application to not respond:

The solution is to defer the interaction. This video does a good job outlining the modifications to make.

sometimes users forget to reply or mention the bot, which is their fault -- but they'll edit the message to mention or reply to the bot. i wanna make sure that bb picks up on these edits and responds accordingly so they aren't left hanging.

this probably has something to do with the on_message method

Integrate Sentry across project

Issue Summary:
The web client currently lacks Progressive Web App (PWA) functionality. This issue is created to track the implementation of PWA features in the web client to enhance user experience and accessibility.

Expected Behavior:

•	Users should be able to install the chatbot web client as a PWA on their devices.
•	The PWA should provide offline access to the chatbot, allowing users to interact even without an internet connection.
•	The PWA should offer push notifications for updates and alerts.

Current Behavior:
The web client lacks PWA features, limiting its usability and accessibility, especially in offline scenarios.

Steps to Reproduce:

1.	Visit the chatbot web client on a mobile or desktop device.
2.	Attempt to install the web client as a PWA.
3.	Observe the lack of offline functionality and push notifications.

Proposed Changes:

1.	Implement PWA features to enable the installation of the web client as an app on supported devices.
2.	Implement service workers and caching strategies to enable offline access.
3.	Integrate push notification functionality for updates and alerts.

Additional Information:

•	Consider compatibility with various browsers and devices.
•	Ensure that the PWA features are user-friendly and do not compromise privacy and security.

Ayush pointed out that serializing the history into one string doesn't utilize the chat API that OpenAI has developed. Langchain added support for ChatMessageHistory() that organizes previous messages much better that we can migrate to now.

Oftentimes, users ask the bot what it can do, expecting it to be able to answer that accurately. Currently, it makes stuff up. I think there are interesting ways we could pre-load vector stores for capability answers to be retrieved. Filing this issue for the backlog though.

Common security vulnerabilities associated with containers can be mitigated by creating a different user for the main application logic to run under. Otherwise the container will run using the root user which has unlimited privileges.

LLM Provider Blocking certain inferences on specific topics. Need to handle exception

Sentry Issue: TUTOR-GPT-API-4

CancelledError: Cancelled by cancel scope 7f12682a6c80
  File "starlette/responses.py", line 277, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "starlette/responses.py", line 273, in wrap
    await func()
  File "starlette/responses.py", line 250, in listen_for_disconnect
    message = await receive()
  File "uvicorn/protocols/http/httptools_impl.py", line 587, in receive
    await self.message_event.wait()
  File "asyncio/locks.py", line 214, in wait
    await fut

ExceptionGroup: unhandled errors in a TaskGroup
(9 additional frame(s) were not displayed)
...
  File "starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 69, in app
    await response(scope, receive, send)
  File "starlette/responses.py", line 270, in __call__
    async with anyio.create_task_group() as task_group:
  File "anyio/_backends/_asyncio.py", line 664, in __aexit__
    raise BaseExceptionGroup(

Currently the container will utilize whatever variables are listed in the .env of the builder. This fairly insecure since the variables are then in memory with the container.

Bad actors can access the container and list the environment variables using a simple env call. Considering some items within this are sensitive such as

• Discord Bot Token
• OpenAI Key

Should rearchitect to use a proper secret storage mechanism. Options such as HashiCorp Vault, AWS Secret Store, SOPS etc.

There should additionally be functionality for continuing to use an .env file for local development.

The message content should automatically scroll too the bottom of the content when a new message is received and the screen is already at the bottom.

If a user has scrolled up to read a previous message it should not scroll and respect the users position.

Move deployment of application to AWS ECR from AWS ECS for a more automated method of deploying the application.

Utilize Github Actions to add continuous integration and deployment and multiple versions running.

• Production instance running off code in main branch
• Canary instance running off code in canary branch

Since the implementation of ToM stuff, "objective discovery" has been abandoned. So we don't need the objective/ folder in the data/prompts/ directory.

Any plugin so it can be used with https://open-assistant.io/chat/ & gpt4all?

How it can be used as alternative to anki?
Required:

• SRS algorithm
• Custom-Quiz generator

If a user has a great conversation with Bloom and decides to sign up for the service they will then lose their previous conversations.

Need to add more too the logic to carry those conversations over.

this session id: 1054cad3-3f48-4634-86dd-2879bd84e30e has messages that have no content

The current functionality of the chatbot maintains a single context and history across all instances of user interaction. As the application is scaled out to include more users and we conversations to coincide with each other there exists a requirement to hold multiple contexts.

Towards this end added functionality is required to track this session state. A proposed version 1 of this functionality is to use an in-memory dictionary in python to keep track of the context and history for each parallel conversation.

A session will be denoted by using the channel id of the incoming messages. The channel id will represent one of the following pycord constructs [1]

• TextChannel
• Thread
• DMChannel
• GroupChannel
• PartialMessageable

The enum ChannelType describes all of the possible Channels that exist. Also all base classes should implement the Snowflake Abstract Base Class meaning the id should be unique

Purpose: to return the generated thought of a given response

I could see it working like this:

• command is locked down to a role
• someone that has the proper role uses /thought and pastes in a discord message link to a response
• if we have redis implemented, each row in the db should represent a response, so i could imagine being able to query the message_link column to find the value in the thought column and return that

Need to add some sort of system for profiling different method calls within the bot to collect telemetry and find bottlenecks.

Several options

• https://pyroscope.io/
• https://www.parca.dev/

Currently we are only using one set of chains for a persona shared by all conversations. As requests scale up and coroutines need to wait on the availability of these chains either lag will be introduced or nondeterministic behavior from race conditions.

Need to implement some kind of mechanism for dynamically creating new chains (possibly by persona) as requests increase, and load balance requests between chains. A few different options or stages for how this should be implemented

• Purely as a broker with a default max set of chains pre-created (no scaling down)
• Split into multiple services with instances that can dynamically scale to meet need (k8s style)

Bloombot utilizes coroutines with asyncio to respond to discord messages as they come, however all of these coroutines are access the shared objects in the globals import. This can lead to race conditions and there should be some kind of synchronization method. Below is a possible starting point

https://superfastpython.com/asyncio-lock/

There have been a plethora of inquiries into needing the thought_channel and several users expressing privacy concerns.

Towards this end we should add functionality to selectively forward only certain messages.

Setup login provider for Google.

https://supabase.com/docs/guides/auth/social-login/auth-google

Feature_Spec_-_Workshop_your_Writing_with_Bloom.pdf

Based on writing feature spec. This specification describes an idea to generalize bloombot to have different "personas" of tutoring. Currently, the goal is to have two personas; the existing discussion based one and a new writing based on. Later on this same model could be generalized to other scenarios or types of tutors (scientific papers?)

The current implementation of the bot uses 3 chains that are reused between all conversations. The only thing that is changed between different channels is the memory but the 3 chains are used to maintain the same "persona".

So to generalize this feature set I am suggesting creating an abstraction class of a TutorPersona or just Persona that takes an input of the prompts to base it's behavior on and then hold the 3 chains it will utilize. The memory objects will not need to be changed but the ConversationCache will need to specify the PersonaType to know which one of the Persona's to utilize. These will essentially be global singletons that are reused and the cache will have a key to indicate which one to use per Channel

Currently the bot will try to handle every request that comes to it. Pycord itself has rate-limiting and extra functionalities to handle this with things like adding a cooldown to the slash command

However, the rate-limiting built in is specifically for interacting with the discord api. The other side of this is receiving messages and sending requests to openAI or any future LLM implementation.

There should be some sort of buffer for incoming requests for handling this and limiting the number of concurrent requests. A common option is to use celery

I realized the chains aren't being loaded with memory. We're defining memory separately, sending history to the chain when we want to chat, but the chains aren't ever defined with a memory object so it doesn't key in on the history in order to summarize it. The result is it just accrues chat history until it hits the token limit. I don't think anyone besides courtland has put up with the bot long enough to hit the chat limit, but we should still be robust against this.

The langchain docs show examples of adding memory to an LLMChain, but our issue is we can't give global memory to a chain because we're loading chains when we initialize the app and using them multiple times -- we can't associate a global memory object the same way we do a global chain object.

I think the solution here is to move the chain initialization down to the local level the same way we do with the memories. So a ConversationCache contains the chain object as well. cc @VVoruganti thoughts?

The discord bot says it's typing and then stops, often a significant amount of time before it responds. We want to make sure users know it's still working and hasn't just forgotten about them to prevent people from sending many messages before it's technically done.

Sentry Issue: TUTOR-GPT-WEB-7

TypeError: Failed to fetch
  at getMessages (./app/page.tsx:126:20)
  at d (./app/page.tsx:143:7)
...
(45 additional frame(s) were not displayed)

The type of error that arises especially with OpenAI is

openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.

Possibility of langchain built-in support

The team wants to be able to DM the bot. I figured it wouldn't be a huge lift, since DMs are just specific types of channels. However, they make things hard because of this note in the API ref for channels:

Note that due to a Discord limitation, DM channels are not resolved since there is no data to complete them. These are PartialMessageable instead.

This is where I get kinda lost. They're saying that DM channels are these discord.PartialMessageable attributes. Cool. Not sure what that means though...

In the on_message event, it successfully detects that a message came from DMs with this if statement:

# if the channel the message came from is a DM channel...
if isinstance(message.channel, discord.channel.DMChannel):
    await message.channel.send("GOTCHA BITCH")

However, when I try a similar thing in the button_callback_factory, it's unsuccessful (likely due to the note above)

if isinstance(interaction.channel, discord.channel.DMChannel):
    dm_interaction = discord.PartialMessageable(interaction.channel.id)
    await dm_interaction.send("GOTCHA ROUND 2")

The /context command successfully works in DMs. However, when you try to continue the conversation, it throws errors.

Ignoring exception in on_message
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/discord/client.py", line 378, in _run_event
await coro(*args, **kwargs)
File "/app/bot/core.py", line 209, in on_message
reply_msg = await self.bot.get_channel(message.channel.id).fetch_message(message.reference.message_id)
AttributeError: 'NoneType' object has no attribute 'fetch_message'

On the workshop branch, it fails when you press a button.

Ignoring exception in view <ContextView timeout=180.0 children=2> for item <Button style=<ButtonStyle.primary: 1> url=None disabled=True label='Discuss' emoji=<PartialEmoji animated=False name='💬' id=None> row=None>:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/discord/ui/view.py", line 414, in _scheduled_task
    await item.callback(interaction)
  File "/app/bot/context.py", line 147, in discuss_button_callback
    await self.button_callback_factory(button, interaction, config)
  File "/app/bot/context.py", line 125, in button_callback_factory
    async with interaction.channel.typing():
  File "/usr/local/lib/python3.10/site-packages/discord/context_managers.py", line 81, in __aenter__
    await channel._state.http.send_typing(channel.id)
AttributeError: 'Object' object has no attribute '_state'

I'm sure this is something simple that I'm not putting together from the API ref, but if you have some time I'd appreciate help on this @VVoruganti

When it sees its ID number in a message it replies. I believe this is because Discord parses "@bloom" on the backend as "@1098230..."

new chat requires user to select the chat after one is created, which is odd ux, in web ui

Sometimes, someone with Nitro will set a long context (> 2k chars). Then, I think because the bot doesn't have nitro, it can't respond with a message greater than that threshold. So whenever you use the /context command with no text in order to see the context that was set by someone else, the application can't respond. Traceback:

File "/usr/local/lib/python3.10/site-packages/discord/commands/core.py", line 132, in wrapped
raise ApplicationCommandInvokeError(exc) from exc
discord.errors.ApplicationCommandInvokeError: Application Command raised an exception: HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
In data.content: Must be 2000 or fewer in length.

Fly.io has released a v2 of their application platform that aims to improve reliabilities. Related posts

• https://fly.io/docs/reference/apps/
• https://community.fly.io/t/fly-migrate-to-v2-automatic-migration-to-apps-v2/11984

Need to investigate if it's right to migrate and if so complete the migration

You can make subcommand groups for slash commands and we just added numerous example passages for the bot to have a discussion on. I'd like to figure out the right way to move all those example commands (and maybe even the regular ones) to their own files to clean up the app.py file.

In order to help streamline things a bit more as the classroom demos are getting scheduled, Courtland said he was going to make categories specific for classrooms, and was wondering if we could have a thought channel for each category.

I think this would be relatively straightforward to implement. I believe the bot has admin rights, so it should be able to create channels. If not, I'll find out which intents it needs for that. But I imagine it checking if a thought channel exists for the category the message is coming from. If it doesn't, create it + construct thought message + send -- if it does, construct the "thought" message and send it.

The security tab shows several vulnerabilities associated with the currently installed version of langchain so need too upgrade the latest patch.

Should also add a set a scripts for running these processes that will re-occur

Occasionally Bloombot responds with the exact same content twice. Example https://discord.com/channels/1076192451997474938/1148639021573095526/1148647322406895616