[Feature] Using metadata to filter declarative memory enhances response accuracy about core HOT 2 OPEN

Currently, all documents uploaded into the rabbit hole are utilized to generate a user's response. However, in certain situations, there's a need to prioritize specific documents, especially when working with a large number of documents. This is where metadata can play a crucial role in facilitating this process.

I propose two necessary actions:

• Introduce the capability to set metadata during the document upload API process;

Can be done via before_rabbithole_stores_documents or before_rabbithole_insert_memory hooks, not yet directly via API, have a look here:

The difference is, in the first hook you access all the chunks and can change them (included metata), summarize them, delete them.
In the second you deal with a single chunk.
In those chunk metadata you should use the same key you will use recall side (see below)

• Introduce the capability to apply filters when a user sends a message.

You can do it via hook, using before_cat_recalls_<collection_name>_memories.
You can see how it is used in Cat Advanced Tools plugin

In the hook you should be able to add "metadata": {....} to the recall config and that stuff should filter the retrieval, because it ends up here:

If necessary the metadata you pass can be obtained via websocket, accessible via cat.user_message_json.xxx, or as an output of an entity extraction chain (that you can also place in the hook).

Let me know if all clear!
Thanks

from core.

Everything is clear @pieroit , thank you. However, using a plugin to set metadata implies that I won't have the ability to customize values depending on the file I'm uploading. Correct?

For the project I'm working on, we're uploading various documents categorized by type. If I create a plugin to set the documents' metadata, I would need to change the plugin's settings with every upload. On the other hand, if I could set the metadata directly via API, the process would be much faster.

Regarding filtering, I believe I've got it. I had modified the core of Cheshire Cat to automatically handle filters based on user messages, but perhaps this can also be achieved through a plugin. I'll give it a try. Thank you!

from core.