Tokenizing single words in English gets tokenizer cwazi about hebrew-tokenizer HOT 4 CLOSED

please give me extra info as i cant reproduce on python 3.8.12

from hebrew-tokenizer.

Seems it's not a hebrew_tokenizer bug after all! but an odd, and not a very stable implementation of beautifulsoup4 find_all() which emits a (not very stable pointer?) string which even changes value between debug runs (!)
So, no bug here, unless you feel the urge to support other peoples non stable code :-)

from hebrew-tokenizer.

How to recreate this issue - just so it's clear why was this bug opened in the first place:

from bs4 import BeautifulSoup
from hebrew_tokenizer import tokenize

sent = 'לצדה פועלות חברות כמו  <a href="%D0%9C%D0%A2%D0%A1">МТС</a> <a href="Beeline">Beeline</a>'

# GOOD - this goes pretty well:
one_word = list(tokenize('לצדה פועלות חברות כמו  '))
print(one_word)
one_word = list(tokenize(sent))
print(one_word)
one_word = list(tokenize('MTC'))
print(one_word)

# BAD - this does NOT go well at all:
soup = BeautifulSoup(sent, features="html.parser")
tags = soup.find_all('a')
for t in tags:
    al_arr = list(tokenize(t.text))
    print(one_word)

from hebrew-tokenizer.

I am opening a new issue instead of that one...

After opening a bug in BeautifulSoup4 repository, the good people maintaining it have explained to me what went wrong. Seems that the 'MTC' letters above AREN'T ENGLISH. It's Cryllic letters which make Hebrew-tokenizer return weird results (internally the REGEX simply does not know of any Cryllic).
I will put my suggested resolution for that in the new bug description.

from hebrew-tokenizer.