cfpb / cfgov-crawler-app Goto Github PK

It would be helpful to document some useful searches one could make in SQLite with the output of a crawl.

For example, "find all pages that link to /about-us/", and save as CSV:

sqlite> .mode csv
sqlite> .output about_us_links.csv
sqlite> SELECT DISTINCT url, json_each.value link FROM cfpb, json_each(json(cfpb.contentLinks)) WHERE json_each.value LIKE '/about-us/%' ORDER BY url, link;
sqlite> .output stdout

If the folder for the database doesn't exist, the db creation fails silently and all the results do not get saved.

It would be nice to be able to start a crawl from the command line, without needing to interact with a UI. This will allow use of this project as part of regularly scheduled automation.

A nice-to-have would be to have text-based console output like the graphical interface, to give an idea how far along we are in the crawl. An even nicer-to-have would be the ability to pause/resume the crawl as we can do in the graphical UI.

Presumably there's some nice Node CLI package we could leverage to support the above? Ping @mistergone @contolini.

I've added some initial content for instructions for how to add custom searches. Please check and make sure if it makes sense!

@chosak and @contolini - Do you mind looking it over?

https://github.com/cfpb/cfgov-crawler-app/blob/main/README.md#adding-custom-search-parameters