Comments (3)
mike442144
commented on June 2, 2024
Look, I totally understand, but it'll break the current API using, also if one does not await queue, 'unhandled promise' warning will always be there. What's worse, it to confuse the API when providing promise and callback at the same time. To be hoest, it is better to deduplicate outside the crawler, which means should be handled by the developer. That keeps the flexibility and consistency. Hope it helpful.
from node-crawler.
CristianMR
commented on June 2, 2024
Thanks for your answer Mike. I already did it. It took some hours to find out this issue so probably others will do it too. Have a nice year btw ✨
from node-crawler.
mike442144
commented on June 2, 2024
Sorry to hear that you spent hours on this issue, so let's keep the issue details here to help others. Thanks, and the same to you, have a nice year.
from node-crawler.
Related Issues (20)
- Detect Timeout Pages HOT 1
- How to skip URL with ECONNRESET or ETIMEDOUT errors HOT 3
- Can this tool actually crawl/spider, or just scrape pages? HOT 5
- maybe a bug ? HOT 1
- Maintained? HOT 1
- Crash ERR_INVALID_URL on Redirect HOT 1
- Queue size goes up and doesn't crawl new pages
- No retry when status code 404 HOT 1
- How to use with await? HOT 3
- Iconv-lite warning: decode()-ing strings is deprecated. HOT 2
- I would like to report a security concern HOT 1
- Jquery not work if response body is buffer HOT 2
- node-crawler.org url hijacked HOT 4
- Security concern HOT 1
- the encoding declaration in @types/crawler HOT 1
- cannot get section node HOT 4
- HTTP Agent with keepAlive HOT 1
- Limit queue size HOT 2
- Dependency vulnerabilities
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-crawler.