Comments (8)
"nondeterministic" also caught my attention, although not so much to raise the question. But it is misleading. I think it should read "dependent on morphosyntax of the given language".
from docs.
Related in updated http://universaldependencies.github.io/docs/tokenization.html: "word segmentation is itself a complex non-deterministic process", "word segmentation is simple and deterministic", ", in languages where there is a nondeterministic mapping".
from docs.
Thanks for picking my nits. I agree that the examples are not good for illustrating the issue of nondeterminism (which I nevertheless think is important). They were originally chosen to illustrate the need to extract words out of whitespace separated tokens (whether nondeterministic or not) but have now ended up in the wrong context. I will try to rework the text.
from docs.
I revised the text. Let me know what you think!
from docs.
Thank you, this fully clarifies the issue for me! (Closing)
from docs.
I also like the current version, thanks!
from docs.
I did one more edit to remove the final use of "deterministic". I think that what Joakim means by "deterministic" is that there's a simple python script that could do the job, without lexicons, morphological analysis, etc. I agree that "deterministic" seems the wrong term, since for something like Arabic, you might use tools like SAMA and Amira, for which the output for a token is deterministic, but it would be very useful to have a two-level representation.
from docs.
Thanks, guys. I removed some redundancy in the sentence that had two "simple" close to each other, but was careful not to reinsert the word "deterministic". :) What I meant was simply that the mapping can be performed by a deterministic transducer that does not consider anything but the character sequence itself. If ABCD always splits into AB and CD, then it is deterministic. But if ABCD can correspond either to AB and CD, or to A and BCD, then it is nondeterministic. But I agree that it is better not to use this terminology.
from docs.
Related Issues (20)
- Misidentified Lemmas in Spanish HOT 1
- clausal appos HOT 36
- Flat:foreign with Typo=Yes HOT 3
- acl vs xcomp vs advcl HOT 1
- Deprel of list item enumerators HOT 11
- English nominal subtypes: merge :npmod and :tmod as :unmarked HOT 18
- Create treebank without syntactic dependencies HOT 2
- How to document script used for the data in treebank? HOT 7
- Some Broken or missing treebank links HOT 4
- NPs in head-marking languages HOT 19
- Standardizing ExtPos (at least for fixed expressions) HOT 36
- New enhanced dependencies - Propagation of nsubj for ccomp and advcl in pro-drop languages HOT 3
- Annotation of Classifiers in the Egyptian-UJaen Treebank HOT 33
- English mischievous nominals involving names and numbers HOT 7
- Repository for new treebank HOT 1
- Transitive vs intransitive verb features? HOT 1
- `as X as` expressions as `fixed` with ExtPos - what qualifies? HOT 8
- Ellipsis in UD HOT 2
- docs site has stopped building HOT 2
- How to differentiate DET for quantifiers and DET for demonstrative determiners for isolating languages like Thai HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docs.