This repo is for internal use of KoNLP HanNanum Analyzer. Because KoNLP will require some modification on HanNanum Analyzer.
If you want to get original version of HanNanum Analyzer, visit HanNanum Official Website
HanNanum Analyzer for KoNLP
License: GNU General Public License v3.0
This repo is for internal use of KoNLP HanNanum Analyzer. Because KoNLP will require some modification on HanNanum Analyzer.
If you want to get original version of HanNanum Analyzer, visit HanNanum Official Website
'KoNLP' R라이브러리에서 발생한 이슈관련입니다. 링크: haven-jeon/KoNLP#54 (comment)
KoNLP 라이브러리에서 추적하여보니 Hannanum의 SentenceSegmentor2와 관련있음을 확인했습니다.
SentenceSegementor2에서 java BreakIterator를 이용할시에 조건에 맞지 않는 문장이 들어올 경우가 있습니다. 현재까지 확인된 것은 아래와 같습니다.
'가식적인 내용전개와 캐릭터들이. . .;;;;' in NSMC data. ratings_test 35480 line.
쉼표뒤에 세미콜론이 붙어 있어 그런지 문장분할이 정확히 되지않고 BreakIterator가 아래와 같이 분할합니다. Sp = White Space, FS = FullStop
- 가식적인 내용전개와 캐릭터들 이 2)FS 3)Sp 4)FS 5)Sp 6)FS 7);;;;
공백이 생겼기에 SentenceSegmentor2의 doProcess()에서 70번째 라인의 substring을 하려하면 StringIndexOutOfBounds에러가 발생합니다.
KoNLP쪽에서 예외처리가 잘되어있으나, 여러 문장들을 순서대로 처리시에 (sapply를 이용하여) 위의 에러가 발생하고나면 bufSents가 초기화 되지않고, 이전의 값을 가지고 있습니다. 따라서 다음의 시나리오가 발생합니다.
SegtenceSegmentor2.scala파일의 doProcess()함수안에 bufSents를 초기화하는 예외처리문을 넣어야 합니다.
발견된 케이스.
예문: 가식적인 내용전개와 캐릭터들이. . .;;;; => 가식적인 내용전개와 캐릭터들이{FS}{Sp}{FS}{Sp}{Semicolonx4}
Cause of take() method on blocking queue.
'take()' needs to change to 'poll()'
Started with "kr.ac.kaist.swrc.jhannanum.plugin.MajorPlugin.MorphAnalyzer.ChartMorphAnalyzer.ChartMorphAnalyzer.initialize"
check with
gogamza@gogamza-desktop:~/work/HanNanum-Analyzer/src$ grep -R "FileInputStream" *
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/Trie.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/Trie.java: BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(dictionaryFileName)));
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/ConnectionNot.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/ConnectionNot.java: BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(filePath)));
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/AnalyzedDic.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/AnalyzedDic.java: BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(dictionaryFileName), "UTF-8"));
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/Connection.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/plugin/MajorPlugin/MorphAnalyzer/ChartMorphAnalyzer/Connection.java: BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(filePath)));
kr/ac/kaist/swrc/jhannanum/share/TagSet.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/share/TagSet.java: BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(filePath)));
kr/ac/kaist/swrc/jhannanum/share/Code.java:import java.io.FileInputStream;
kr/ac/kaist/swrc/jhannanum/share/Code.java: BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(srcFileName), "UTF-8"));
사용자가 입력한 단어사전의 데이터 POS 태깅시 강제 적용
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.