Form-normalization strategies about wn HOT 2 CLOSED

I pushed a commit for v0.7.0 that does normalization by default, where normalization is downcasing and NFKD-based diacritic removal. This cannot be configured by the convenience methods wn.words(), wn.synsets(), etc., but it can be configured when creating a wn.Wordnet object via the normalize parameter. I think morphological analysis, such as morphy, would be a separate process, perhaps configurable via a lemmatize parameter, since the purpose is to find a lemmatic form.

from wn.

My thinking now is that this hypothetical lemmatize parameter is a function with the following type signature:

Lemmatizer = Callable[[str, Optional[str]], Iterator[str]]

...where the two arguments are the word form and optional part of speech. For example:

def lemmatize(form: str, pos: str = None) -> Iterator[str]:
    ...

If the pos is None, it yields values for all parts of speech. Also this function does not need to yield valid lemmatic forms. The results will be used for queries against the database, and these queries act as the filter, so there's no point in doing it twice. Something that requires a lexicon, such as Morphy's exception lists, can be instantiated to make use of a wordnet and still follow the signature:

class Morphy:
    def __init__(self, wordnet: wn.Wordnet):
        ...
    def __call__(self, form: str, pos: str = None) -> Iterator[str]:
        ...

I'm not sure if lemmatize should take the output of any normalize function, or vice versa. I can see arguments both ways. Maybe it needs some preprocessing pipeline instead: preprocessor=[normalize, lemmatize].

from wn.