Question about the compressor library about gpt-researcher HOT 2 CLOSED

Hey @djuillard, thanks for bringing this up! Just to clarify, this IS a RAG architecture. And the compressor does use a vector DB logic. Basically we chunk all documents -> embed them -> retrieve only the relevant ones to the query. This is done to optimize for context with only the most relevant content. I do agree though that retrieval hit rate might be lower than simply passing most of the scraped content, and definitely requires more refinement. Happy to hear suggestions or PRs that can improve it! As for local documents, we're actually working on one here by the one and only @proy9714 : #429

from gpt-researcher.

Thanks for your answer and for the clarification. OK, I understand better. This IS RAG, with vector logics, but not based on vector db at this stage, because not immediately necessary.
I will check whether I can contribute, maybe while adding a vector DB like qdrant with asynchronous capabilities.

from gpt-researcher.