Authors :

• Gildas HOULMANN
• Thibaud FRANCHETTI

1. What are the types of the fields in the index?

TextField (contents), StringField (path) and LongPoint (modified)

2. What are the characteristics of each type of field in terms of indexing, storage and tokenization?

In this case (some fields can be stored or not), we have:

3. Does the command line demo use stopword removal? Explain how you find out the answer.

No, search index returns less results than the index.

4. Does the command line demo use stemming? Explain how you find out the answer.

No, step returns 49 results. steps only 19.

5. Is the search of the command line demo case insensitive? How did you find out the answer?

No, index and INDEX return the same number of results.

6. Does it matter whether stemming occurs before or after stopword removal? Consider this as a general question.

The order should not have any incidence on the results, however processing the stemming before stopwords removal would be wasting time and resources on those stopwords that will be removed anyway.

1. Find out what is a “term vector” in Lucene vocabulary?

A term vector is a list of all terms in a field for a given document, possibly with their frequency, positions and offsets

2. What should be added to the code to have access to the “term vector” in the index? Have a look at the different methods of the class FieldType. Use Luke to check that the “term vector” is included in the index.

The following methods must be called on our custom FieldType:

    summaryFieldType.setStoreTermVectors(true);
    summaryFieldType.setStoreTermVectorPositions(true);
    summaryFieldType.setStoreTermVectorOffsets(true);

3. Compare the size of the index before and after enabling "term vector", discuss the results.

The size of the index before and after enabling "term vector" is the following (all fields saved) :

As expected, it takes more space after than before. This is quite logical since we add data in the index.

    // Field type for the summary
    FieldType summaryFieldType = new FieldType();
    summaryFieldType.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
    summaryFieldType.setTokenized(true);
    summaryFieldType.setStored(true);
    summaryFieldType.setStoreTermVectors(true);
    summaryFieldType.setStoreTermVectorPositions(true);
    summaryFieldType.setStoreTermVectorOffsets(true);
    summaryFieldType.freeze();

    // Add id
    doc.add(new LongPoint(ID_FIELD, id));
    doc.add(new StoredField(ID_FIELD, id));

    // Add all authors if not null
    if (authors != null && !authors.isEmpty()) {
        Arrays.stream(authors.split(AUTHOR_SEP)).forEach(author ->
                doc.add(new StringField(AUTHORS_FIELD, author, Field.Store.YES)
            );
    }

    // Add title
    if (title != null && !title.isEmpty()) doc.add(new TextField(TITLE_FIELD, title, Field.Store.YES));

    // Add summary
    if (summary != null && !summary.isEmpty()) doc.add(new Field(SUMMARY_FIELD, summary, summaryFieldType));

All the indexes have been computed using an instance of ClassicSimilarity.

2. Explain the difference of these five analyzers.

WhitespaceAnalyzer

Trivial analyzer. The text is divided at every whitespace, no additional treatment is done.

EnglishAnalyzer

Common stopwords from the English language are removed (the list could have been provided but in our case we used the default one), some treatment is done (lower case, removing possessive 's) on the tokens before.

ShingleAnalyzerWrapper

Allows to consider n-grams (n between two bounds min and max, both included, given as argument) as tokens in addition to the unigrams. N-grams are built with adjacent words.

By default, used in combination with a StandardAnalyzer (lower case filter).

StopAnalyzer

Apply a lower case filter and remove all words from the given file.

3. Look at the index using Luke and for each created index find out the following information:
   1. The number of indexed documents and indexed terms.
   2. The number of indexed terms in the summary field.
   3. The top 10 frequent terms of the summary field in the index.
   4. The size of the index on disk.
   5. The required time for indexing (e.g. using System currentTimeMillis() before and after the indexing).

General data:

Top 10 frequent terms for the summary field:

4. Make 3 concluding statements bases on the above observations.

1. The EnglishAnalyzer obviously uses stemming, the others probably don't (it's certain for the StopAnalyzer since computer isn't stemmed as comput but unclear for the others, although the previous description suggest they don't either).
2. The best performances are achieved by the stopwords-based strategies (i.e. EnglishAnalyzer and StopAnalyzer), they also produce the most relevant tokens. A trivial analyzer like the WhitespaceAnalyzer is totally
3. Shingle-based strategies can really quickly become time and resources consuming with the growth of the shingle size, so we should carefully think if we need it or not. By default, use a StandardAnalyzer (lower case filter)

1. What is the author with the highest number of publications? How many publications does he/she have?

The author with the highest number of publications is Thacher Jr., H. C. , with 38 publications

2. List the top 10 terms in the title field with their frequency.

When using the standard analyzer, we obtained these results:

Code used to obtain these results:

public void printTopRankingTerms(String field, int numTerms) {
    // This methods print the top ranking term for a field.
    // See "Reading Index".
    
    try {
        //Obtain the terms statistics
        HighFreqTerms.TotalTermFreqComparator cmp = new HighFreqTerms.TotalTermFreqComparator();
        TermStats[] statsTable = HighFreqTerms.getHighFreqTerms(indexReader, numTerms, field, cmp);
        
        //print
        System.out.println("Top ranking terms for field ["  + field +"] are: ");
        for (TermStats stats : statsTable){
            System.out.println(new String(stats.termtext.bytes) + ", " + stats.totalTermFreq + " occurrences");
        }

    } catch (Exception e) {
        e.printStackTrace();
    }
}

public void query(String q) {
    // See "Searching" section
    try {
        System.out.println("Searching for [" + q +"]");

        //Parse the query
        QueryParser parser = new QueryParser("summary", analyzer);
        Query query = parser.parse(q);

        //obtain the hits
        ScoreDoc[] hits = indexSearcher.search(query, indexReader.maxDoc()).scoreDocs;
        int nbHits = hits.length;

        //print
        System.out.println(nbHits + " results");
        System.out.println("Top 10 results: ");
        for(int i = 0; i < 10; ++i){
            Document doc = indexSearcher.doc(hits[i].doc);
            System.out.println(doc.get("id") + ": " + doc.get("title") + " (" +
                    hits[i].score + ")");
        }

    } catch (ParseException | IOException e) {
        e.printStackTrace();
    }
}

All queries have been computed using an instance of ClassicSimilarity.

1. Searching for publications containing the term “Information Retrieval”.

Parsed query : summary:"inform retriev"
11 results

Top 10 results:

2. Searching for publications containing both “Information” and “Retrieval”.

Parsed query : +summary:inform +summary:retriev
23 results

Top 10 results:

3. Searching for publications containing at least the term “Retrieval” and, possibly “Information” but not “Database”.

Parsed query : +summary:retriev summary:inform -summary:databas
54 results

Top 10 results:

4. Searching for publications containing a term starting with “Info”.

Parsed query : summary:info*
193 results

Top 10 results:

5. Searching for publications containing the term “Information” close to “Retrieval” (max distance 5).

Parsed query : summary:"inform retriev"~5
15 results

Top 10 results:

5. Compare the query "compiler program" with the ClassicSimilarity and MySimilarity

With the ClassicSimilarity :

578 results Top 10 results:

With the MySimilarity :

578 results Top 10 results:

We can see that the results are very different. However, the publications 2652 and 1459 are in the top 10 with both Similarities. That shows that even if the parameters are different, the same kind of publications are found.

We can observe that on the top-rated publication with the ClassicSimilarity (3189), "program" is found 2 times and "compil" 4 times in the summary. For a total of 389 chars, this is a ratio of 6/389 = 0.0154.
On the other hand, on the top-rated publication with the MySimilarity (2923), "program" is found 5 times and "compil" 3 times in the summary. For a total of 940 chars, this is a ratio of 8/940 = 0.00851. The ratio is higher with ClassicSimilarity, but the total number of occurences of the searched terms is higher with MySimilarity.

    @Override
    public float tf(float freq) {
        return (float) (1 + Math.log10(freq));
    }

    @Override
    public float idf(long docFreq, long docCount) {
        return (float) (Math.log10(docCount / (docFreq + 1.0)) + 1);
    }

    @Override
    public float lengthNorm(int numTerms) {
        return 1;
    }