Norwegian profile: Bokmål or Nynorsk? about language-detector HOT 2 CLOSED

You are right. But most of the applications I have used detection for only care about detecting whether we have Norwegian, English or Chinese etc, not the "sub" language, so the ISO-639 macro language code "no" is good enough for many. Besides, Arabic is also a macro language, but do people care wether it is Tunisian or Egyptian arabic or any of the other 30 forms (according to https://en.wikipedia.org/wiki/ISO_639_macrolanguage)? I don't know.

So if someone create (good enough) detectors for individual languages like nb and nn, maybe the library should also be able to return both the language code and the macro language code?

from language-detector.

Thanks Jan for your comments. You're from Oslo, and in the search business, so your opinion is very relevant.

I see a small difference between Norwegian and Arabic.

Starting from the English Wikipedia article about the Eiffel Tower https://en.wikipedia.org/wiki/Eiffel_Tower you can get to the Arabic one https://ar.wikipedia.org/wiki/%D8%A8%D8%B1%D8%AC_%D8%A5%D9%8A%D9%81%D9%84 and yes there is just one for all Arabic variations.

For Norwegian we have 2:
https://no.wikipedia.org/wiki/Eiffelt%C3%A5rnet
https://nn.wikipedia.org/wiki/Eiffelt%C3%A5rnet

Wikipedia is cheating, they label the first "Norsk bokmål" but use the (common) macro language code "no" instead of "nb".

This is similar to what we have with German:
https://de.wikipedia.org/wiki/Eiffelturm
https://als.wikipedia.org/wiki/Eiffelturm
Just that in German the code "de" has a unique meaning.

I guess that this answers the question:
The language profile we have for "Norwegian" must have been created from plain Bokmål content. Just like the language profile for German used "de" pages and no other writing forms or dialects.

Then this is good enough. It's documented now. It matches the Wikipedia use of the language. And if one ever needed to detect Nynorsk separately, then he could create a profile for it based on Wikipedia "nn" content.

from language-detector.