self._c_tf_idf can make more efficient use of vectorizer model about bertopic HOT 3 CLOSED

Hi Maarten,

You can use below code to reproduce my results. I increased the size of the dataset to see how this thing scales for somewhat larger datasets.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer

# Multiply by 5 to create larger dataset to have a more robust estimate
docs = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))['data']*5

# fit + transform
pre_vectorizer_model = CountVectorizer(ngram_range=(1,2), stop_words="english")
pre_vectorizer_model.fit(docs)
X = pre_vectorizer_model.transform(docs)

# more efficient method
pre_vectorizer_model = CountVectorizer(ngram_range=(1,2), stop_words="english")
X = pre_vectorizer_model.fit_transform(docs)

from bertopic.

Thanks for sharing this! Any speed-up is highly appreciated.

. However, calling self.vectorizer_model.fit_transform() is more efficient (can be roughly twice as efficient)

If you have the time, I would love to see some (perhaps existing) benchmarks on this! Twice as efficient seems rather large but would be a great speed-up.

from bertopic.

Awesome, thanks for testing this! As soon as the PR passes, I'll make sure to merge this.

from bertopic.