crack521 / dkpro-keyphrases Goto Github PK

This unit test uses code from stanfordcorenlp, which is not under asl. Also, it 
does not follow camel case convention.

There is no filter which removes from the indexes keyphrases candidates that 
are not in a filter corpus.

DKPro Lab is now available at Maven central, however it is not being declared 
in the poms of some modules.

Since textgraphs module uses MTJ dependency, its license should be changed to 
ASL.

There is no filtering for obtaining the keyphrases from a specific range from 
the text.

The class KeyphrasePerformanceCounter should be refactored because it is too 
big.

Since decompounding might affect the lemmas from each compound part, it is 
necessary to have the lemmatization after the decompounding is done. Then, the 
preprocessing task should be changed.

There is no filter based on the character length from the keyphrases.

There is no filter based on the frequency of the keyphrase.

Textgraphs module's pom file does not declare a dependency on the parameter 
module from DKPro Core.

The package core.candidate from the core module needs a better coverage. At 
least 70% would be reasonable to have.

Some dependencies being used are out of date.

KeyphraseCandidate type is not necessary since the candidate annotator and the 
filters can work on the Keyphrase type without any problem.

- Move DictionaryFilter from module wikipediafilter to core
- It should extend from AbstractFilter

This module uses stanfordnlp module from dkpro core. Its license shall be 
changed to gpl.

Lab module uses code from Stanford CoreNLP wrapper from dkpro. For this reason, 
its license cannot be ASL.

Currently there is no information about the precision at 5, 10 and 15 at the 
report.

Implement a filter which removes candidates that either do or do not match a 
pos pattern.

Hi,

I'm using DKPro Keyphrases' CooccurrenceGraphExtractor to extract keyphrases 
from various texts. The keyphrase extraction for the texts is
performed sequentially. However, my Windows Task-Manager reports that the 
tree-tagger processes do not terminate. So although I process the texts 
sequentially, a growing number of tree-tagger processes accumulates in my RAM 
until my RAM is used up completely.

My code that invokes the keyphrase extraction looks like this:


For(String text : allTexts){
    CooccurrenceGraphExtractor extractor = new CooccurrenceGraphExtractor();
    extractor.setMinKeyphraseLength(2);
    extractor.setCandidate(new Candidate(CandidateType.Token, PosType.N));
    List<Keyphrase> keyphrases = extractor.extract(text);
    keyphrases = getTopRankedUniqueKeyphrases(keyphrases, keyphrases.size());

    // save text 
    ...
}

Is there a way to avoid this accumulation of tree-tagger processes?
Thanks in advance.

Sincerely yours,
Laura




What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

KeyphraseWriter purpose is to be used for evaluation. However, if we run a 
keyphrase extracton pipeline for several configurations using dkpro lab, it is 
not that simple to analyze the outcome print in the console. KeyphraseWriter 
should offer the option of using a file to print out the keyphrases extracted.

KeyphrasePatternCounter should not be an inner-class in 
KeyphraseDatasetStatistics

KeyphraseDatasetStatistics needs to have unit test

The mtj dependency is used only by the textgraphs module. Then, it should be 
only in the textgraphs module pom file.

The core module license should be changed to ASL.

The weka dependency is used only in the wrappers module. Then, it should be 
used only in the wrappers module.

The kea dependency is used only in the wrappers module. Then, it should be used 
only in the wrappers module.

Kea dependency is not on maven central, release should not be blocked by that.

Would be good if the filters extended an abstract type representing a candidate 
filter.

This is class is not used by any other class.

DKPro Core ASL and GPL 1.6.1 are already available on maven central. Their 
dependencies should be updated wherever they are declared.

The current frequency filter only works on the document text.

Since this module uses dkpro stanfordnlp module, its license should be changed 
to gpl.

No model for tf-idf is provided.

Wikipediafilter uses JWPL, so it cannot be licensed under ASL. 

Several deprecated methods from uimaFIT are still being used in dkpro-keyphrases

Should we replace that with OpenNLP?

Filter factories should be moved to a factory package.

Only keyphrase candidates should be removed from the indexes, not the 
keyphrases themselves. The idea of a keyphrase candidate is that it is only a 
candidate, so, it might be removed from indexes after the filtering phase. If 
the keyphrase candidate becomes a keyphrase, then it does not make sense to 
remove it through a filter. Then, the mode from StructureFilter which filters 
out Keyphrase types should be removed.