centre-for-humanities-computing / tweetopic Goto Github PK

Blazing fast topic modelling for short texts.

Home Page: https://centre-for-humanities-computing.github.io/tweetopic/

License: MIT License

Python 100.00%

dirichlet-process-mixtures gibbs-sampling machine-learning nlp topic-modeling tweet tweet-analysis gsdmm dmm mcmc python scikit-learn visualization

tweetopic's Introduction

tweetopic

⚡ Blazing Fast topic modelling over short texts in Python

Features

Fast ⚡
Scalable 💥
High consistency and coherence 🎯
High quality topics 🔥
Easy visualization and inspection 👀
Full scikit-learn compatibility 🔩

🛠 Installation

Install from PyPI:

pip install tweetopic

👩‍💻 Usage (documentation)

Train your a topic model on a corpus of short texts:

from tweetopic import DMM
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline

# Creating a vectorizer for extracting document-term matrix from the
# text corpus.
vectorizer = CountVectorizer(min_df=15, max_df=0.1)

# Creating a Dirichlet Multinomial Mixture Model with 30 components
dmm = DMM(n_components=30, n_iterations=100, alpha=0.1, beta=0.1)

# Creating topic pipeline
pipeline = Pipeline([
    ("vectorizer", vectorizer),
    ("dmm", dmm),
])

You may fit the model with a stream of short texts:

pipeline.fit(texts)

To investigate internal structure of topics and their relations to words and indicidual documents we recommend using topicwizard.

Install it from PyPI:

pip install topic-wizard

Then visualize your topic model:

import topicwizard

topicwizard.visualize(pipeline=pipeline, corpus=texts)

🎓 References

Yin, J., & Wang, J. (2014). A Dirichlet Multinomial Mixture Model-Based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 233–242). Association for Computing Machinery.

tweetopic's People

Contributors

Stargazers

Forkers

patrikmolnar99 ankylohryax carmonajca

tweetopic's Issues

Relax limitation with number of unique words and max number of occurrances.

test suite

deprecation warning fra pyldaviz

Dependency clash with topic-wizard

As discussed in the ML team meeting last week, there is a dependency clash with topic-wizard related to numpy versions.

Expected Behavior

Running tweetopic and then visualising the results using topic-wizard should reproduce the kind of output described in the README.md.

Current Behavior

In a virtual environment, installing tweetopic via pip and then installing topic-wizard results in the following error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tweetopic 0.2.0 requires numpy<1.23.0,>=1.22.4, but you have numpy 1.24.2 which is incompatible.
tweetopic 0.2.0 requires scikit-learn<1.2.0,>=1.1.1, but you have scikit-learn 1.2.1 which is incompatible.
numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.2 which is incompatible.

This means that the workflow outlined in README.md is not possible.

(The inverse is obviously the case, if topic-wizard is installed before tweetopic.)

Possible Solution

Not a possible solution, but I'm wondering why exactly the two packages require different versions of numpy in the first place. Have there been breaking changes in numpy between minor versions which mean that the two packages cannot use the same dependencies? I haven't had a chance to dig into the code, so might be missing something!

Steps to Reproduce

Error has been reproduced on Ubuntu and MacOS, and for Python versions 3.9, 3.10, and 3.11.

python3.9 -m venv env
pip install --upgrade pip
source .env/bin/activate
pip install tweetopic
pip install topic-wizard