[Feature] add a new hook before that the document is parsed about core HOT 6 CLOSED

Ah ok I got it.
it makes sense

from core.

Correct me if I'm misunderstanding, but I'm not sure adding it before the parsing happens is the right place.
At that point, you have file bytes and you should parse them to remove the noise.

Hence, you could post process the text before the rabbit hole splits the text or inside the parse itself.
In the last case, you could override the BeautifulSoup parser and perform parsing and post processing all at once

from core.

Hi @nicola-corbellini, at that point the HTML for example is already stripped, so I can't do parsing to remove noise based on some tag or attribute.

from core.

Ah my bad, didn't think about that. Than a custom parser is the best solution, I think.
We are relying on Langchain, which, in turn, uses BeautifulSoup (ref here).
Maybe tuning some params could be enough, but implementing a new one guarantees full control.

from core.

If you agree I could do a proposal on this ;-)

from core.

Mmm ok...in my mind the solution was to just make a plugin with your own parser and replace the default one with the rabbit_hole_instantiates_parsers hook, but if you have a proposal I'm happy to hear it

from core.