Code Monkey home page Code Monkey logo

Comments (2)

hexists avatar hexists commented on August 22, 2024

실험 내용 정리

multi processing 으로 인한 오류인지 확인하고, 해결 방법을 찾아봅니다.

실험 과정

  • 같은 코드로 증상이 재현되는지 확인
  • 오류가 출력될 경우, 오류를 해결할 수 있는지 확인
  • 해결 방법을 정리 후 공유

실험 내용

  • 동일한 코드를 실행하는 경우, 몇가지 오류를 출력합니다.

  • 환경 변수 설정 오류

objc[47789]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[47789]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[47790]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[47790]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
  • 환경 변수를 설정해서 해결했습니다. ref

    export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
    
  • 입력 데이터가 빈 경우(NaN)

Concurrent tagging:
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "./test_mt.py", line 17, in do_concurrent_tagging
    l = [k.pos(lines[i]) for i in range(start, end)]
  File "./test_mt.py", line 17, in <listcomp>
    l = [k.pos(lines[i]) for i in range(start, end)]
  File "/usr/local/lib/python3.7/site-packages/konlpy/tag/_kkma.py", line 55, in pos
    sentences = self.jki.morphAnalyzer(phrase)
TypeError: No matching overloads found for kr.lucypark.kkma.KkmaInterface.morphAnalyzer(float), options are:
	public java.util.List kr.lucypark.kkma.KkmaInterface.morphAnalyzer(java.lang.String) throws java.lang.Exception
  • NaN인 경우를 필터링 했습니다. ref

    filt_condition = dataset['document'].str.contains("", na=False)
    dataset = dataset[filt_condition]
    
  • 입력 데이터 분석에 오랜 시간이 걸리는 경우(OOM이 나거나 계속 실행됩니다.)

Traceback (most recent call last):
  File "./test_mp.py", line 69, in <module>
    results = pool.map(morphRow, dataset.iloc[:, 0].values)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x114512b70>'. Reason: 'PicklingError("Can't pickle <java class 'java.lang.OutOfMemoryError'>: attribute lookup java.lang.OutOfMemoryError on konlpy.tag._kkma failed",)'
  • OOM 발생 원인을 찾기 위해, 데이터를 출력하면서 문제가 되는 값들을 정리했습니다.
    konlpy에서 발생한 문제이기 보다는 kkma에서 발생한 문제입니다.
    이미 이슈로 등록되어 있습니다. 해당되는 내용을 추가로 정리할 예정입니다.
    • konlpy/konlpy#172
    • 문제가 되는 값
      진짜 조낸 재밌다 굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿굿
      몰라 그냥 영화도 안봤는데 쓰레기얌 ㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂㅂ
      황시욬황시욬황시욬황시욬황시욬황시욬황시욬황시욬황시욬황시욬
      ㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁㄹㅇㄴㅁ
      

실험 결과 및 결론

  • 오류를 확인해보니 kkma를 multiprocessing해서 나는 오류는 아니였습니다.
  • kkma에 있는 오류(분석이 오래걸리는, 안되는)로 인해 multiprocessing시에도 문제가 발생한 것으로,
    kkma의 오류를 수정하거나, kkma에서 문제가 될만한 입력을 제외하고 입력해야 합니다.
  • 위 내용들은 기존 이슈에 공유하고, 추가로 오류가 나는 입력들 또한 이슈에 댓글로 남기겠습니다.

from test_konlpy.

hexists avatar hexists commented on August 22, 2024

참고

from test_konlpy.

Related Issues (3)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.