Scraping n levels deep about scrapegraph-ai HOT 4 CLOSED

Hei @rawmean, we will add it in the to-do list for feature requests! It would be interesting to create a new graph for this and maybe calling it CrawlerGraph or DeepScraperGraph

from scrapegraph-ai.

I'll try to take a stab at it.
This is what I'm thinking:
Input: URL

1. FetchNode
2. ParseNode
3. RAGNode
4. SearchLinkNode -> Get all the links on the page
5. (new) LinkFilterNode -> Filter out potentially relevant links
6. (new) RepeaterNode -> Executes graph from child node onwards once for each of the input link in parallel
7. FetchNode
8. ParseNode
9. RAGNode
10. (new) ContainsAnswerNode -> A new node type that can tell if the currect content contains the answer
11. (new) ConditionalNode -> A new node with two children, if parent returns true, pick child 1 or else pick child 2
    12a. GenerateAnswerNode
    12b. Go to step 4 for next level of depth

Let me know if this looks reasonable or if you have any other plan/better alternative that you can think of

from scrapegraph-ai.

Yeah, pls contact me thorough email ([email protected])

from scrapegraph-ai.

Sounds really intresting.

from scrapegraph-ai.