"nondeterministic word segmentation" about docs HOT 8 CLOSED

"nondeterministic" also caught my attention, although not so much to raise the question. But it is misleading. I think it should read "dependent on morphosyntax of the given language".

from docs.

Related in updated http://universaldependencies.github.io/docs/tokenization.html: "word segmentation is itself a complex non-deterministic process", "word segmentation is simple and deterministic", ", in languages where there is a nondeterministic mapping".

from docs.

Thanks for picking my nits. I agree that the examples are not good for illustrating the issue of nondeterminism (which I nevertheless think is important). They were originally chosen to illustrate the need to extract words out of whitespace separated tokens (whether nondeterministic or not) but have now ended up in the wrong context. I will try to rework the text.

from docs.

I revised the text. Let me know what you think!

from docs.

Thank you, this fully clarifies the issue for me! (Closing)

from docs.

I also like the current version, thanks!

from docs.

I did one more edit to remove the final use of "deterministic". I think that what Joakim means by "deterministic" is that there's a simple python script that could do the job, without lexicons, morphological analysis, etc. I agree that "deterministic" seems the wrong term, since for something like Arabic, you might use tools like SAMA and Amira, for which the output for a token is deterministic, but it would be very useful to have a two-level representation.

from docs.

Thanks, guys. I removed some redundancy in the sentence that had two "simple" close to each other, but was careful not to reinsert the word "deterministic". :) What I meant was simply that the mapping can be performed by a deterministic transducer that does not consider anything but the character sequence itself. If ABCD always splits into AB and CD, then it is deterministic. But if ABCD can correspond either to AB and CD, or to A and BCD, then it is nondeterministic. But I agree that it is better not to use this terminology.

from docs.