Is there a good way to detect non-Japanese text? about cutlet HOT 5 CLOSED

I appreciate the advice. This is definitely a case of bad input and not cutlet's fault... and short of raising a NotJapaneseError or something when presented with unrecognized characters, there isn't much cutlet could do about it.

I'm currently thinking I should run unknown text through chardet or something first, before deciding what to do with it.

Thanks!

from cutlet.

Got it, thanks for the clarification!

from cutlet.

There is not any specific cutlet feature for detecting non-Japanese text.

Besides checking the input string yourself using regexes or something, MeCab has a feature called char_type which is present on the Nodes you get in fugashi. It doesn't recognize hangul specifically, but it has categories like ALPHA and SYMBOL (separate from categories for kanji and kana) that should let you detect it.

I also haven't tried this before, but you could maybe add hangul to the mapping tables cutlet uses internally.

You might also be able to make use of unihandecode, which handles Korean and Japanese.

from cutlet.

Having an option to throw an error on text that would be rendered as ? actually sounds like it might be a good idea, I'll think about it!

To give me a little more idea of your use case, are you using this on:

• text for URLs
• titles (of books, articles, etc.)
• prose sentences / paragraphs
• something else?

from cutlet.

I'm using it for book titles. Mostly Japanese light novels, but there are occasional non-Japanese ones thrown in too.

from cutlet.