bjascob / lemminflect Goto Github PK
View Code? Open in Web Editor NEWA python module for English lemmatization and inflection.
License: MIT License
A python module for English lemmatization and inflection.
License: MIT License
Please add pronouns support.
lemminflect.getAllLemmas('his') returns {}
spacy.load("en_core_web_sm")(word)[0]._.inflect("NNS")
gives me "Can't retrieve unregistered extension attribute 'inflect'. Did you forget to call the set_extension
method?".
If I follow the message and add "spacy.tokens.Token.set_extension('inflect', method=Inflections().spacyGetInfl)", I now get " Extension 'inflect' already exists on Token. To overwrite the existing extension, set force=True
on Token.set_extension
"
If I add force=True, it gives me what I want, but there's no mention of this in the tests.
If you'd like, I can add a test to this extent or a note in the README.
Contracts are not in the dictionary lookups
They show up in the LEXICON and in the english_dict.txt but not the forms_table.csv.gz. Likely they are being eliminated by the ASCII checks and shouldn't be.
words = ["'d", "'ll", "'m", "'re", "'s", "'ve"]
words = [would, will, am, are, is, have]
lemmas = [will, will, be, be, be, have
I'm not sure how you used the lemma annotator for CoreNLP to test the lemmatizer, but the Morphology class definitely does use POS tags if available:
https://github.com/stanfordnlp/CoreNLP/blob/main/src/edu/stanford/nlp/process/Morphology.java
For example, the WordTag stemStatic(String word, String tag)
interface
FWIW, the next version of CoreNLP will cover ADJ & ADV as well
LemmInflect currently preserves casing for "all lower", "all upper" and "first upper" casing styles. For words like "McDonald", after lemma/inflection the returned word will be "Mcdonald" since capitalization of individual letters is not maintained.
Is it appropriate to report single words that are incorrect as bugs? I realize the dictionary can't be 100% complete.
Lemminflect 0.2.1:
getAllLemmas('install')
{'VERB': ('install', 'instal')}
I think it should just return 'install'.
For the test case 'quilting/NOUN' and 'plastering/NOUN', the words are not in the lemma lookup so OOV rules are called.
getAllLemmasOOV('quilting`, 'NOUN')` returns 'quilt' (it selects rule "ing,,False")
getAllLemmasOOV('plastering`, 'NOUN') returns 'plastering' (it selects rule ",,False")
In the case of 'quilting' the model selects a verb rule. To prevent this consider...
In addition, the model classes include the ending letters to remove. However, similar above, there is nothing to prevent it selecting a "remove ing" rule for a word ending in something else. I'm not aware of this causing issues but it should be investigated when looking into the first issue.
There are some use cases where users would like to find the base word (aka lemma) but don't know what part-of-speech the word is. This is problematic for words like "painting" which could either be "paint" for a verb or "painting" for a noun. Regardless, it may be useful to simply return "paint" for use in Neural Network sentence classification, etc..
Proposed approach is to use the dictionary to find the shortest word. If the word is not in the dictionary then try OOV for Nouns and Verbs and choose the shortest.
Hi,
Thanks for providing this library.
Have you ever thought of implementing other languages?
If not, I'm currently looking for a tool that does Russian word inflection and I was wondering whether you know of any such tools?
Best,
Eva
Hi, thanks for building this amazing tool!
Currently, it doesn't seem to handle inflections of special adjectives like beautiful and handsome correctly.
Example:
from lemminflect import getLemma, getInflection
lemma = getLemma('beautiful', upos='ADJ')
inflection1 = getInflection(lemma[0], tag='JJR')
inflection2 = getInflection(lemma[0], tag='JJS')
print(inflection1, inflection2)
gives ('beautifuler',)
and ('beautifulest',)
. It'd be great if lemminflect can output something like ('more', 'beautiful',)
or ('more beautiful',)
!
>>> import lemminflect as li
>>> li.getInflection('have', 'VBZ')
('haves',)
Shouldn't that be has
? What am I doing wrong?
import spacy
import lemminflect
results in the last import being reported as unused by linters.
import spacy
import lemminflect
lemminflect.extend_spacy()
or something similar would've been much better.
['somewhat', ####], ['somew', 'ADJ']
[['his', ####], ['hi', 'PROPN'], ['hi', 'ADJ'], ['hi', 'ADV']],
['her', ####], ['he', 'ADJ'], ['h', 'ADV']],
[['could', ####], ['coul', 'ADV']
[['another', ####], ['anoth', 'ADJ'], ['anoth', 'ADV']],
[['question', ####], ['quest', 'ADV']],
[['vs', ####], ['v', 'NOUN'], ['v', 'PROPN'], ['v', 'VERB'], ['v', 'ADJ'], ['v', 'ADV']],
Hi, thanks for the amazing tool!
I'm using lemminflect in my project at uni (which may become a paper later). Do you have a paper on it which I could cite?
When using getAllInflections()
on some adjective, the adverb forms given are the exact same as the adjective and vice versa. Is there a possible fix for this? I'm trying to generate the adjective form of adverbs and vice versa
Is it possible to extend the functionality of the inflection to other languages, and if so, what would be required to do so?
Hi all,
It would be great to have arrival from arrive. I checked the following function, but it did not return arrival
getAllInflections('arrive')
{'VBD': ('arrived',), 'VBG': ('arriving',), 'VBZ': ('arrives',), 'VB': ('arrive',), 'VBP': ('arrive',)}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.