Puppeteer-based tool for collecting different types of data:

• links
• screenshots
• code snippets

Describe tasks in configs and tool will allow you to run it from web interface:

The result will be printed to page:

Can be useful for old large sites with vague structure. You can found something unexpected.

Config example: tasks/urls.example.js

Tool can make screenshots of given pages with given dimensions and device emulation. You can run task twice to compare result with previous.

Config example: tasks/screens.example.js

Useful if you need download all your demos from external service.

Config example: tasks/snippets.example.js

1. Clone:

git clone [email protected]:yoksel/harvester.git --depth 1 && cd harvester

2. Run npm i

3. Rename credits-example.js to credits.js and fill it with real logins and passwords. It'll allow you to log in and visit a site as a logged in user.

4. Take needed example file in tasks, rename it without example (screens.example.js -> screens.js) and fill it with real data.

5. Run npm start and open localhost:3007

You'll see page wich allows you to start and stop tasks, see collected data and to download it in archive.

credits.js and task files are in gitignore and will not be commited. Don't push your passwords to the public repository.

Collected links

Collected links with screenshots

Full view of the screenshot

Full view of the screenshot with diff

Tool is in development. If you find a bug, fill an issue