PartOfSpeech representation reproducibility and word with index 0 about bertopic HOT 2 OPEN

I found that the values remained constant until the PartOfSpeech representation module and switching it to another one resolved the issue. The problem appears to be initially caused by the deduplication method (list(set())) used at lines 121 and 130. Because the hash function used for generating set keys is seeded at the start of the interpreter (seed can be overridden using PYTHONHASHSEED env variable), the output of such deduplication is different with each run. This behavior causes word_indices at line 144 to change with each run. Which is later problematic when sorting keywords with the same c-TF-IDF as they are arranged differently.

Amazing, great catch! That's also a nasty habit of mine so I wouldn't be surprised if that happens in other places as well.

Part 1
Sort the word_indices at line 144 using numpy. This will ensure consistent ordering of words, should be faster than built-in sort, and will transform them into numpy array for further operations.
Remove the numpy array creation at line 145, as its handled by previous step.

Sounds good and a minimal change as well, which I prefer!

When looking at the PoS code, I noticed that word_indices at line 144 are generated using the following condition if words_lookup.get(keyword) which ignores the first word returned by get_feature_names_out. It looks like an error.

I'm not sure if I understand correctly. Why would the first word be ignored?

from bertopic.

I'm not sure if I understand correctly. Why would the first word be ignored?

That's because of how the values are converted to booleans. At line 140 a lookup is created that maps each word to its index (0 based), which is later used at line 144 to extract the indices. This lookup output is filtered using the condition if words_lookup.get(keyword) to prevent None values, as .get method on a dictionary returns None as a default value if the key isn't found. However, the same condition will also evaluate to False for the index 0, causing it to be ignored.

Example

[v for v in [1, None, 2, 0, 3] if v]
# [1, 2, 3]

from bertopic.