Documents are objects, so let's treat them as such...

NLOOP is a python package that provides a convenient interface for processing, analyzing, and modeling text data using popular natural language processing libraries such as spaCy, gensim, and sometimes NLTK!

Install NLOOP by running the following in the command line:

git clone https://github.com/syasini/NLOOP.git

cd NLOOP

pip install [-e] .

NOTE: Don't forget the . at the end! The -e argument will install the package in editable mode which is suitable for developement. Since currently the code is being regularly updated , using this option is recommended.

(optional)

If you use conda, consider creating a new environment prior to the previous steps to ensure a clean installation of NLOOP:

conda create -n nloop_env python=3.7

conda activate nloop_env

import os
import pandas as pd
import numpy as np

from nloop import Text

Let's create a corpus with 3 documents.

>>> docs = ["This is my first document.", 
            "And here's another boring one.", 
            "And let's add a third one just in case!"]

Create a Text object with docs

>>> text = Text(docs, fast=False)

spacy_model: "en"
Only keeping: ['ADJ', 'NOUN', 'PROPN', 'VERB']
nlp.pipe_names = ['tagger', 'parser', 'ner', 'textrank']

Access the processed document using .docs.

>>> text.docs

[This is my first document., 
And here's another boring one., 
And let's add a third one just in case!]

Each element is a spacy Doc object, so we have a list of lists here. The first index refers to each document, and the second index to the tokens within that document. For example to see the first document we can use

>>> text.docs[0]

This is my first document.

And to see the first token in the first document use

>>> text.docs[0][0]

This

The raw tokens can are collectively assembled in .raw_tokens. Each element inside each list is a spacy Token.

>>> text.raw_tokens

[[This, is, my, first, document, .],
 [And, here, 's, another, boring, one, .],
 [And, let, 's, add, a, third, one, just, in, case, !]]

The clean and processed token are saved in .tokens.

>>> text.tokens

[['document'], 
 ['boring'], 
 ['let', 'add', 'case']]

The elements are just strings and not spacy Tokens. These are basically the .lemma_s of the raw_tokens , excluding any stop words and parts of speech other than ['ADJ', 'NOUN ', 'PROPN', 'VERB'] which was indicated upon instantiation of text.

The bag of words (bow) and TFiDF vectors are also calculated and stored in

>>> text.corpus_bow

[[(0, 1)], 
 [(1, 1)], 
 [(2, 1), (3, 1), (4, 1)]]

and

>>>text.corpus_tfidf

[[(0, 1.0)],
 [(1, 1.0)],
 [(2, 0.577350), (3, 0.577350), (4, 0.577350)]]

Other thing to try: text.dictionary, text.token_counter, text.word_cloud(), text.keywords, text.sentences, text.docs.noun_chunks, ...

Checkout the demo.ipynb notebook for more examples.