More options in the UI user to modify how the model will process the data. This can help adjust the tool to the situation.

• More sentence ingestion from transcript (filter out fewer sentences)
• Modify the "buffer" surrounding the similar sentences chosen
• Model selection for generation
• Select file for config (data/transcripts to be used)
• Verification
  • Estimated number of tokens to be used (within limits of chosen model?)
• User supplied API key

Ideally these options could be determined automatically and/or globally. However, this might give more flexibility and improve the user experience.

LLM should output a format that can be processed easily (such as JSON, YAML, etc.). This way the output of the LLM can be used directly by a script to include extra details such as time stamps to the video.

This will likely creating prompts that can be more reliably produce JSON of the actual results. However, fine-tuned LLMs are notorious in actually giving pure JSON responses without careful prompting .

As mentioned in #3 multiple transcripts could prove to be too large for the context window. Without addressing this, the model will not be able to take in the full context needed to properly answer user provided question.

Methods could include

• Brute-force cut-off of long inputs (keeping the same prompt but cutting off the transcripts piece)
• #7
• Feed transcripts iteratively to LLM

Anthropic changed their python sdk - making this code line outdated.

Would love to know if this might help - https://github.com/BerriAI/litellm

~Simple I/O library, that standardizes all the llm api calls to the OpenAI call

from litellm import completion

## set ENV variables
# ENV variables can be set in .env file, too. Example in .env.example
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["ANTHROPIC_API_KEY"] = "anthropic key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# anthropic call
response = completion("claude-v-2", messages)

Using a proven prompting method to help the LLM know that sections belong to different episodes/transcripts and that there are different sections of the transcript used in t he prompt.

Unknown if this will change the quality of the output and to what extent and under what circumstances.

Ultimately want to use the "evidence" for each "key point" generated by the model to be used to create a timestamp from the transcript.

The most brute force solution is check each sentence of the transcript to see if the evidence sentence is contained.

However, sometimes the model will slightly alter the quoted evidence. Thus something like a similarity metric could be used to compare the sentences. This can help find modified quotes and reduce "hallucinated" quotes.

Could be useful for user to select relevant transcript to ask questions on.

Context window can be very large with Anthropic's claude-v1-100k model (100,000 tokens as context), so more than one podcast could be given.

However, there would have to be some sort of limiting criteria since choosing something like "ALL" transcripts could end up being larger than the fairly large context.

Processing multiple transcripts to get the sentence embeddings can take quite some time so speed up for multiple runs on the same transcripts can be beneficial to user experience.

The Streamlit app already uses a local cache but something more persistent would be ideal.

These are potential steps to help with multiple runs:

• App uses local cache (considers the batch of transcripts as identifier to cache)
• App uses local cache per transcript loaded
  • Don't repeat whole process when loading just additional transcript
• Store embeddings as local file
  • Ensures that something like get_embeddings gets the embeddings
• Use vector database
  • Could be deployed but also used locally for testing
  • Can help with filtering sentences by similarity (though that has yet been a bottleneck)
  • Could be leveraged for other uses (e.g. similar questions already asked)

Multiple transcripts from the same series can have time dependence in answering the user provided question. Transcripts should be fed into context in order and/or identify the order within the context.