Comments (4)
Hei @rawmean, we will add it in the to-do list for feature requests! It would be interesting to create a new graph for this and maybe calling it CrawlerGraph
or DeepScraperGraph
from scrapegraph-ai.
I'll try to take a stab at it.
This is what I'm thinking:
Input: URL
- FetchNode
- ParseNode
- RAGNode
- SearchLinkNode -> Get all the links on the page
- (new) LinkFilterNode -> Filter out potentially relevant links
- (new) RepeaterNode -> Executes graph from child node onwards once for each of the input link in parallel
- FetchNode
- ParseNode
- RAGNode
- (new) ContainsAnswerNode -> A new node type that can tell if the currect content contains the answer
- (new) ConditionalNode -> A new node with two children, if parent returns true, pick child 1 or else pick child 2
12a. GenerateAnswerNode
12b. Go to step 4 for next level of depth
Let me know if this looks reasonable or if you have any other plan/better alternative that you can think of
from scrapegraph-ai.
Yeah, pls contact me thorough email ([email protected])
from scrapegraph-ai.
Sounds really intresting.
from scrapegraph-ai.
Related Issues (20)
- refactoring of merge_answer_node HOT 2
- gpt4o does not return a json output and broke the pipeline
- Omit Certain parts of the HTML HOT 1
- Set default temperature to 0
- `NameError: name 'input_data' is not defined` is raised when trying to use smart_scraper_graph.run() HOT 3
- OMP: Error #15: Initializing libiomp5md.dll, but found libomp140.x86_64.dll already initialized HOT 7
- NameError: name 'input_data' is not defined HOT 1
- Refactoring of prompts HOT 2
- probable bug HOT 2
- refactoring of nodes HOT 1
- ImportError: cannot import name 'OpenAI' from 'scrapegraphai.models' (/usr/local/lib/python3.10/dist-packages/scrapegraphai/models/__init__.py) HOT 3
- Ability to add headers to source ( better integration with Jina AI) HOT 2
- The Dockerfile doesn't work well HOT 2
- Context length exceeded HOT 1
- Scrapegraph returns relative path URLs instead of absolute path **Possible Bug?** HOT 12
- `embedder_model` AttributeError in `/examples/openai/deep_scraper_openai.py` HOT 1
- Different prompt_tokens size between Azure OpenAI and OpenAI HOT 5
- Unable to join discord on readme HOT 1
- Add support for other OpenAI models like gpt-4o-2024-08-06 HOT 1
- improve tokenization function
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapegraph-ai.