The thread name currently appears as just 'Audio' in the sidebar instead of an actual title like text threads.

Running the app without the env var loaded caused a null error in the app, we are catching it now on initialization.

Ideally, we should handle all this in the assistant handler, but as a stopgap we'll just throw an exception on startup if the env is missing.

Before we close this, I want some functionality in the assistant handler that will default to something, even if it's the default template agent or even just whatever's first in the assistant list.

We should just be able to import the audio features from chainlit with no issues and integrate that into the app

Right now we handle file uploads to the UX by passing them to the thread message object, which uses a thread-specific file store. These stores have a 7 day unused expiration threshold. If a user resumes a thread which is more than seven days old, this file store will be deleted. A user might assume the file is still available.

To resolve this, we might get the thread from literal check user messages for attachments. Use the last message timing details to check whether we're past this expiration. We might also have a check against the datastore manager -- might be able to map the threads w these temp data stores.

In the event that the datastore is unavailable, inform the user on chat resume - throw an exception.

We might also prompt the user via actions whether to re-upload the files associated with the thread, which we may be able to pull from Literal, although they just seem to have longer expiries on their data objects.

set_citations calls the OAI client to pull file names. This is less than ideal as CL causes it to be rendered on page refresh, leading to a bunch of unecessary calls. I'd like to swap this out to use our FileHandler, currently in my file uploads fork: https://github.com/dahifi/chainlit-template/pull/1

Assistant name should be dynamic, imported from the agent config.

We've managed to get this isolated to a single env var within the app file, but we'll start using chat profiles to handle this in the user session.

We are seeing strange behavior in our streaming event handler. The handler is returning an openai.APIError, which renders to the UX, but then I saw everything logging underneath. It is as if the call failed and then immediately retried.

Description

Bug in main where it can't get the assistant bc it's getting a list

async def on_start_chat_logic(client, assistant_name):
    await assistant_handler.init()
    assistant = assistant_handler.find_by_name(assistant_name)
    cl.user_session.set("assistant_id", assistant.id)

Acceptance Criteria

• Modify assistant_handler.find_by_name to return a single assistant object instead of a list.
• Ensure the function properly handles cases where multiple assistants might match the name.
• Update the relevant unit tests to cover this case.
• Verify that the assistant ID is correctly set in the user session.

This is mostly AFGE related stuff, and we're going to get rid of this in lieu of the new datastores

Pulling the URL up fails, the asst ID seems to be getting mangled with the thread ID.

chainlit_utils.allowed_mime is a list of types that we are allowing in the file uploads.

Note that these files are already uploaded to literal via the message attachments, so we have an additional constraint here.

Description

To avoid client-side caching issues and ensure efficient metadata management, we need to enhance the OpenAIHandler superclass and its subclasses (e.g., FileHandler) to interact with a scalable backend.

Acceptance Criteria

• Design a scalable backend to store metadata.
• Update documentation
• Write unit and integration tests.
• Update OpenAIHandler to include methods for interacting with the backend.
• Modify FileHandler and other subclasses to utilize backend methods for CRUD operations.
• Ensure the backend supports high-volume data efficiently.

Tasks

• Design database schema for metadata storage.
• Implement backend server with necessary API endpoints.
• Write unit and integration tests.
• Update OpenAIHandler to interact with the backend.
• Modify FileHandler to use backend methods.
• Update documentation.

We will be changing assistant api keys and the like due to the new project groups, and will lose the ability to resume threads that were created under old API keys.

openai.NotFoundError: Error code: 404 - {'error': {'message': "No thread found with id 'thread_xxxx'

Here is the CL Docs on Streaming. This example uses the chat.completions endpoint, while we use beta.assistants, so we have a lot of refactors to do before we can even start this.

https://platform.openai.com/docs/assistants/overview?context=with-streaming

We will need to heavily refactor our code to do this, moving from a polling method to an event driven one using AssistantStreamEvent

Start by swapping out client.beta.threads.runs.stream.until_done() without a handler, and verify that the code still works. Then we can swap in the text creations and deltas in another commit, and move on to the tool calls and exception handling as well.

Since OAI has released assistants v2, we have some changes to the way citations are rendered. Since we're not seeing quotes anymore, we need to change the logic so that we at least display the file names and build references.

Generate a script that can pull config of existing agents.

Currently we are doing:
images = [file for file in elements if "image" in file.mime]

This should be moved into handle tool call and sent back to the model

CL Posted a V2 Assistant code interpreter example: https://github.com/Chainlit/cookbook/blob/main/openai-assistant-v2-code-interpreter/app.py

There are a couple of places we should do refactors in our code:

On Text

Right now we stream tokens and call msg.update(), which sends a Literal API call. Refactor to use stream_token like they do here.

On Chat Start

load the assistant in the user session cl.user_session.set("assistant_id", assistant.id)

Tool Calls

We should incorporate the code_interpreter functionality.

thread.run.requires.action

Via Haze on Discord: https://discord.com/channels/1088038867602526210/1241138329991385088/1242120806474186752

 @override
    async def on_event(self, event: AssistantStreamEvent):
        """Callback that is fired whenever an event is encountered. Use to handle tool calls and other events."""
        if event.event == 'thread.run.requires_action':
            logger.info('----------------REQUIRED ACTION-------------------')
            run_id = event.data.id  # Retrieve the run ID from the event data
          async with self.assistant.llm.beta.threads.runs.submit_tool_outputs_stream(
              thread_id=self.current_run.thread_id,
              run_id=self.current_run.id,
              tool_outputs=tool_outputs,
              event_handler=AssistantEventHandler() #Here is where we call itself and start a new stream
          ) as stream:
              await stream.until_done()

This should probably be in on_tool_call_done call

During testing, I've found that I have to keep track of .env files in several different places. I had to make some modification to the .env path handling for our various tests, and found that we don't have a centralized place where the env is loaded, which can lead to some strangeness. We might need a singleton object so that we don't wind up making multiple clients with different .env files.

We'll need to check all these references to make sure we're doing this right:

Was running a cleanup routine on the vector stores and got a rate limit error, so I was thinking about a way to wrap all of the AsyncOpenAI client calls so that we can use retries and catch errors.

When a user uploads a file there is a chance for it to fail, but this does not show in the UI or get handled.

Right now longer stream responses are collapsed after the complete. We would like to know what can be done to prevent this, whether it's a config option or in source.

The OAI Platform files UX is pretty lacking. https://platform.openai.com/storage/files

Right now it shows the following information in the UX:

We would like a better interface to manage files between our local file systems, git repos, Chainlit front end, &c, and make sure we're not loading the same file in multiple places. A lot of times we get random file names. And most of the other endpoints such as vector stores and annotation file citations refer to the OAI file ID. We need a way to manage all this better.

The vector storage has a slightly better interface:

It at least shows us the file names (most of the time) and now shows what assistants and threads a datastore is attached to.

I'm not sure I want to rebuild the OAI UX for all this, but we do need to do checks for file uploads to do file hashes, as well as some sort of summary or descriptive details about a file and why it was added. These can be used for rollups and the like.

We might also use some of this metadata for storing the original download URL or source, this is crucial when we start building ingestion pipelines for youtube videos and other datasources that aren't natively supported by retrieval.

Been thinking about the message.send in both on chat start and chat resume. I think it's probably more annoying long run, and seems like a toast notification would be better served. Maybe save the welcome messages based on a user env, or maybe have some sort of action check.

Probably cleanest to incorporate whatever we're using for the agent instructions; that would make it truly dynamic.

Recent models aren't returning quotes, but will give you fifty citations to the same file. This is not a great UX, and it's stupid to create all these Literal elements.

For the first change here we need to get rid of the element rendering. Let's pick a style that we want to use here, and then go from there.

Using the following in a chainlit.md file to test MD rendering:

# Welcome to Chainlit! 🚀🤖

# Version 1

Message content, blah blah [1] blah [2]

> [Source doc 1](doc1.pdf): Quote or no quote
> [Source doc 2](doc2.pdf): Quote or no quote
 
# Version 2
Some text with [a link][1] and
another [link][2].

[1]: http://example.com/ "Title"
[2]: http://example.org/ "Title"

# Version 3

I have more [^1] to say up here.
[^1]: To say down here.

I just finished cleaning out our empty vector stores, right now we've got several that aren't empty, except for file upload that failed.

We should clean these out, and add logic to our upload functions to remove them if they fail. This should be simple enough on our vector store files API, and we can add additional logic on the file API to clean those up as well.

    async def set_citations(self) -> None:
        for annotation in self.annotations:
            citation = annotation.file_citation
            try:
                retrieved_file: FileObject = await self.client.files.retrieve(
                    annotation.file_citation.file_id
                )

To cache the client retrieve call, you can use a dictionary to store the results of previous calls. Before making a call, you can check if the result is already in the cache. If it is, you can return the cached result instead of making a new call. Here's how you can modify the set_citations method to include caching:

class OpenAIAdapter:
    def __init__(self, message: ThreadMessage) -> None:
        # ...
        self.file_cache = {}

    async def set_citations(self) -> None:
        for annotation in self.annotations:
            citation = annotation.file_citation
            try:
                if citation.file_id in self.file_cache:
                    retrieved_file = self.file_cache[citation.file_id]
                else:
                    retrieved_file: FileObject = await self.client.files.retrieve(
                        citation.file_id
                    )
                    self.file_cache[citation.file_id] = retrieved_file

                # rest of your code...
            except Exception as e:
                # handle exception...

In this code, self.file_cache is a dictionary that stores the results of previous calls to self.client.files.retrieve. Before making a call, we check if the result is already in self.file_cache. If it is, we use the cached result instead of making a new call. If it's not, we make the call and store the result in self.file_cache for future use.

Description

Related to #56

There is a higher-level issue in our system where duplicate assistant names exist, causing functions to return a list instead of a single assistant object. We need to update the assistant handler utility functions to identify and handle these duplicates safely. Rather than purging, we should rename them to ensure uniqueness.

Acceptance Criteria

• Implement a function to identify duplicate assistant names in the assistant_handler.
• Update assistant_handler.find_by_name to handle duplicate names and return a single, unique assistant object.
• Create a utility function to automatically rename duplicates (e.g., appending a number suffix).
• Ensure the renaming function logs the changes for audit purposes.
• Update unit tests to cover the new functionality.
• Verify that the system functions correctly with the updated assistant names.