koaning / cluestar Goto Github PK

Stargazers

Forkers

morenoluisg wildone manikant92 evolution99 techthiyanes emanuelaboros buckeyes2019 eware-godaddy vishalbelsare jsgro martysteer manika-lamba jaedukseo apjanco

# Transforming data:
import re
import pandas as pd
import nltk
df = pd.read_csv('buscas_consecutivas.csv', sep = ',')
df["full"] = df["query"] + ' ' + df["next_query"]
rule = r"(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s"
texts = df.loc[:5000, "full"].apply(lambda x: re.split(rule, x))
texts = [''.join(ele) for ele in texts]```

# Transformed data:
texts[:3]
['lg k 22 celular lg k 22',
 'estantes estantes de acoestante colorida',
 'porquinho cozinha porquinho saleiro']

# Runing the pipeline:
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
import umap
from whatlies.language import UniversalSentenceLanguage
from cluestar import plot_text
pipe = make_pipeline(TfidfVectorizer(),
                     UniversalSentenceLanguage(variant='multi'),
                     umap.UMAP()
                    )
        

X = pipe.fit_transform(texts)

plot_text(X, texts, color_words=["vinho", "camisa", "tinta", "whiskey"])