https://www.youtube.com/playlist?list=PLfsGiW3iUfri1L7eCM-N1FYHjh1IsFTZZ

The following tasks were involved :

1. Identify the classification path from request to content – what categories of pages were part of the request, and what named entities were present on the arrived at page? a. Produce a D3 visualization showing this iteration. b. Did the crawler find the most relevant pages? Why, why not?

2. Derive File size diversity of CCA dataset by MIME type. a. Compute size ratio of Solr index and/or ElasticSearch to file size original diversity. b. Produce a D3 visualization of these metrics.

3. Develop a program and associated D3 visualization that demonstrates Parser call chain and how much text and metadata was actually obtained a. Plot Parser Hierarchy versus i. Amount of Text retrieved per file size per MIME type ii. Amount of Metadata retrieved per file size per MIME type

4. Compute Language identification and diversity across the dataset. a. Produce D3 visualizations of the language diversity.

5. Produce a Word Cloud D3 visualization of your text, metadatea, and language to find maximal occurring topics in your dataset. This should include relevant SWEET topics from Assignment.

6. Download and install the four Named Entity Recognition toolkits i. For Grobid Quantities, produce a Tika NER implementation that invokes Grobid Quantities via its REST service. ii. Produce a visualization in D3 for evaluating the maximal joint agreement NER between the most frequently occurring entities. iii. Analyze whether your new joint agreement produces any update NER for your metadata records, and if so, add new maximal joint agreement NER to the metadata records in Solr and/or ElasticSearch.

7. Identify the Spectrum (range, min/max) of measurements