pip install keyness
The log_likelihood() function compares the frequencies of tokens or ngrams in a corpus and reference corpus, rank ordering the output by the log likelihood values of the corpus.
The corpus and reference corpus must be iterables of tokenized texts, such as a list of lists.
The log_likelihood() function will return a rank ordered list with the following data in each item: - the item - its log likelihood value - its frequency in the corpus - its frequency in the reference corpus
from keyness import log_likelihood
corpus = [['here', 'is', 'an', 'example', 'of', 'the', 'first', 'text', 'in', 'the', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'second', 'text', 'in', 'the', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'third', 'text', 'in', 'the', 'corpus']]
reference_corpus = [['here', 'is', 'an', 'example', 'of', 'the', 'first', 'text', 'in', 'the', 'reference', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'second', 'text', 'in', 'the', 'reference', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'third', 'text', 'in', 'the', 'reference', 'corpus']]
log_likelihood(corpus, reference_corpus)
Output
[('reference', 3.904, 0, 3), ('the', 0.023, 6, 6), ('an', 0.011, 3, 3), ('example', 0.011, 3, 3), ('text', 0.011, 3, 3), ('here', 0.011, 3, 3), ('is', 0.011, 3, 3), ('corpus', 0.011, 3, 3), ('in', 0.011, 3, 3), ('of', 0.011, 3, 3), ('second', 0.004, 1, 1), ('first', 0.004, 1, 1), ('third', 0.004, 1, 1)]
from keyness import log_likelihood, type_dist
corpus = [['here', 'is', 'an', 'example', 'of', 'the', 'first', 'text', 'in', 'the', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'second', 'text', 'in', 'the', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'third', 'text', 'in', 'the', 'corpus']]
reference_corpus = [['here', 'is', 'an', 'example', 'of', 'the', 'first', 'text', 'in', 'the', 'reference', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'second', 'text', 'in', 'the', 'reference', 'corpus'], ['here', 'is', 'an', 'example', 'of', 'the', 'third', 'text', 'in', 'the', 'reference', 'corpus']]
log_likelihood(corpus, reference_corpus, dist_func=type_dist)
Output
[('reference', 3.88, 0, 3), ('the', 0.014, 3, 3), ('in', 0.014, 3, 3), ('text', 0.014, 3, 3), ('corpus', 0.014, 3, 3), ('example', 0.014, 3, 3), ('of', 0.014, 3, 3), ('is', 0.014, 3, 3), ('an', 0.014, 3, 3), ('here', 0.014, 3, 3), ('second', 0.005, 1, 1), ('first', 0.005, 1, 1), ('third', 0.005, 1, 1)]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent