Project created by Daniel Kravec, on May 17, 2022.

This is a simple web scrapper, you can save images and see the process of getting every link in the website.

Setup on line 12-19

This lets you control what you want to crawl on the web.

12 | const startURL = 'https://novapro.net';
13 | const stay = true;
14 | const onePage = false;
15 | const images = false;
16 | const toDownloadImages = false;
17 | const useMongo = false;
18 | const mongoLink = process.env.MONGO_URL;
19 | const maxAmount = 500;

• basic webcrawler (doesnt save links).
• save images.
• make sure it stays on the same host.
• checks if the link actually exists before trying to fetch.

• added option to download image (but still search for images).
• seperated download image into a seperate function.
• changed order of saving.

• changed way start data is entered.
• you can now connect to mongo.
• there is now an option to have stay on one page.
• fixed getURL, when url starts with //.
• console logs error if fetch is not res.ok.
• created foundSearchDataSchema, currently unused.

• created foundImageDataSchema and foundLinkDataSchema
• created a function that seperates a link into 3, protocol, main, and path
• you can now have a set maxAmount for amount of linksk to crawl
• updated saveImageToMongo, startd using foundImageDataSchema, doesnt work 100% currently

• Now shows the timestamp of when each link is being crawled.
• Now shows amount of links crawled.
• Saves link to mongo
• created function to save url and links (will redo)
• added upsert to mongo image saving (so it saves the data properly)
• everything isnt required in mongo schema.

• ungitignored /images (to be ignored again)
• moved configuration to config.json

• removed txt in /images

• added /images/* files to gitignore

1- Eventually fails due to different problems, its not consistent.

reasons found.

• reason: socket hang up.
• reason: read ECONNRESET.
• reason: getaddrinfo ENOTFOUND.
• reason: connect ECONNREFUSED
• maybe when trying to fetch a .pdf

some docs

• https://stackoverflow.com/questions/59591058/econnreset-error-while-fetching-an-url-in-node-js

2-

trying to write a lot of images at once.

Error: EISDIR: illegal operation on a directory, open 'images/'.

Emitted 'error' event on WriteStream instance at:

1-

program thinks some links dont exist that exists, doesnt.

1-

Error: ENAMETOOLONG: name too long, open 'images/220px-%D0%9A%D0%BE%D0%BC%D0%BF%D0%BB%D0%B5%D0%BA%D1%81%D0%BD%D0%B0%D1%8F_%D0%BF%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0_%D0%B2%D0%BE%D0%B9%D1%81%D0%BA_%D0%92%D0%BE%D1%81%D1%82%D0%BE%D1%87%D0%BD%D0%BE%D0%B3%D0%BE_%D0%B2%D0%BE%D0%B5%D0%BD%D0%BD%D0%BE%D0%B3%D0%BE_%D0%BE%D0%BA%D1%80%D1%83%D0%B3%D0%B0_10.jpeg'

Emitted 'error' event on WriteStream instance at:

2-

node:internal/deps/undici/undici:4819 throw new TypeError("Request cannot be constructed from a URL that includes credentials: " + input); ^

TypeError: Request cannot be constructed from a URL that includes credentials: http://[email protected]

crawling http://[email protected]

1-

E11000 duplicate key error collection: search-engine.found-images-schemas index: _id_ dup key:

fix: set a new _id naming scheme, maybe encrypt url, so its exactly always the same (instead of adding "..." at the end automatically )

• https://www.youtube.com/watch?v=5THvDt_sbfQ