Documents are objects, so let's treat them as such...
NLOOP is a python package that provides a convenient interface for processing, analyzing, and modeling text data using popular natural language processing libraries such as spaCy, gensim, and sometimes NLTK!
Install NLOOP by running the following in the command line:
git clone https://github.com/syasini/NLOOP.git
cd NLOOP
pip install [-e] .
NOTE: Don't forget the .
at the end!
The -e
argument will install the package in editable mode which is suitable
for developement. Since currently the code is being regularly updated
, using this option is recommended.
(optional)
If you use conda, consider creating a new environment prior to the previous steps to ensure a clean installation of NLOOP:
conda create -n nloop_env python=3.7
conda activate nloop_env
import os
import pandas as pd
import numpy as np
from nloop import Text
Let's create a corpus with 3 documents.
>>> docs = ["This is my first document.",
"And here's another boring one.",
"And let's add a third one just in case!"]
Create a Text
object with docs
>>> text = Text(docs, fast=False)
spacy_model: "en"
Only keeping: ['ADJ', 'NOUN', 'PROPN', 'VERB']
nlp.pipe_names = ['tagger', 'parser', 'ner', 'textrank']
Access the processed document using .docs
.
>>> text.docs
[This is my first document.,
And here's another boring one.,
And let's add a third one just in case!]
Each element is a spacy Doc
object, so we have a list of lists here. The first
index refers to each document, and the second index to the tokens within
that document. For example to see the first document we can use
>>> text.docs[0]
This is my first document.
And to see the first token in the first document use
>>> text.docs[0][0]
This
The raw tokens can are collectively assembled in .raw_tokens
. Each element
inside each list is a spacy Token
.
>>> text.raw_tokens
[[This, is, my, first, document, .],
[And, here, 's, another, boring, one, .],
[And, let, 's, add, a, third, one, just, in, case, !]]
The clean and processed token are saved in .tokens
.
>>> text.tokens
[['document'],
['boring'],
['let', 'add', 'case']]
The elements are just strings
and not spacy Token
s. These are basically the .lemma_
s of the raw_tokens
, excluding any stop words and parts of speech other than ['ADJ', 'NOUN ', 'PROPN', 'VERB']
which was indicated upon instantiation of text.
The bag of words (bow) and TFiDF vectors are also calculated and stored in
>>> text.corpus_bow
[[(0, 1)],
[(1, 1)],
[(2, 1), (3, 1), (4, 1)]]
and
>>>text.corpus_tfidf
[[(0, 1.0)],
[(1, 1.0)],
[(2, 0.577350), (3, 0.577350), (4, 0.577350)]]
Other thing to try:
text.dictionary
,
text.token_counter
,
text.word_cloud()
,
text.keywords
,
text.sentences
,
text.docs.noun_chunks
,
...
Checkout the demo.ipynb notebook for more examples.