danielkelshaw / filterbyfilter Goto Github PK

Once a Coffee record has been defined, as per #5, a function is needed to extract the relevant information from a given ProductPage. This function needs to be bespoke for each shop.

At the moment a coffee record can be constructed - it would be useful if this could be printed to stdout to show the results of the web-scraping.

Provide basic dune file to allow for build / execute of .ml files.
Add a .gitignore to prevent unnecessary files being added.

Add a line in the README.md to explain that this is a project for me to learn OCaml - it will be improved as time goes on.

At the moment the coffee record can only be printed to stdout - it would be useful to write the data to file. With this it would be nice to read the contents of the file and load them into coffee records which can be operated on.

The information that we want to extract should be consistent across different sites and should follow the same template - a record is a good way to implement this in OCaml. This should also allow for easy JSON interactions later down the line if I want to save the results to file.

Some information I may want to include:

• Name
• URL
• Origin
• Prices / Masses
• Description
• Process
• Tasting Notes

To provide a generic case study, I shall be focusing on extracting information from SquareMileCoffee. This can be extended to a number of online shops once I have gained a better appreciation of how to do web scraping with OCaml.

The homepage of SquareMileCoffee lists a number of products - extracting the relevant links will allow us to scrape individual product pages to extract the information that we need. For now this can be a: Uri.t list but perhaps in the future it would be wise to define a custom type.