It detects german when it's an english phrase about lande HOT 2 OPEN

It's hard to say, these neural networks are black boxes.

Definitely the sentence being quite short makes the job tougher for it, the longer it is the easier it should be to get an accurate detection. Also not only it's small, but almost half of it is just some random name, which surely must mess up its internal calculations, the longer the sentence is the less unlikely it would be that like half of it is just names. Here the network sees some ngrams that may just happen to be from frequent in german than elsewhere, ngrams that should just be ignored because they are part of somebody's name.

Maybe it could be fixed by making the network bigger. Mainly I think I need more data, but the problem is I need about the same amount of data for every supported language, and the dataset I'm using doesn't have tens of thousands of sentences or more for every language.

In general it won't be 100% accurate though, the more language are supported, the shorter each sentence, the smaller the network, and the higher the inaccuracy.

from lande.

Maybe a way to sort of fix this would be to get 100k english sentences, getting them translated for all supported languages, and using those for the training instead 🤔 It might be an interesting approach.

from lande.