Code Monkey home page Code Monkey logo

Comments (7)

zhangyimi avatar zhangyimi commented on August 25, 2024

这个错误能够复现么?
错误看起来是创建线程时资源不够导致的,跟你机器当时的运行状态或者配置比较相关,我本地重复调用25w次未复现。
可以考虑修改本地进程允许的最大线程数避免一下。

from ddparser.

WyHy avatar WyHy commented on August 25, 2024

这个错误能够复现么? 错误看起来是创建线程时资源不够导致的,跟你机器当时的运行状态或者配置比较相关,我本地重复调用25w次未复现。 可以考虑修改本地进程允许的最大线程数避免一下。

可以重现,每次都会导致进程崩掉,已经这个样子一周情况了。我用的是 mac monterey 12.0.1。
这个解释中我理解的是paddle会不断创建线程,导致线程资源耗尽,我如果调高本地最大线程数,会不会在更高短句数据量下也崩溃掉?
你这边追踪bug的话还需要哪些数据,我这边尽量提供给你

from ddparser.

zhangyimi avatar zhangyimi commented on August 25, 2024

我如果调高本地最大线程数,会不会在更高短句数据量下也崩溃掉?

存在这种可能,因为这涉及到paddle的dataloader内部对资源的调度。
但目前未见过用户报类似的case,paddle下也未见相关issue,建议你先换台空闲机器或者调大最大线程数看是否可以稳定复现。
如果还不行,可以提供下可以复现的数据及脚本。

from ddparser.

only-yao avatar only-yao commented on August 25, 2024

我也遇到了同样的问题, 会不停的创建线程, 使用的api 是 ddp.parse_seg

paddlepaddle==2.1.0
ddparser==1.0.6
Resource temporarily unavailable

from ddparser.

WyHy avatar WyHy commented on August 25, 2024

我用这个脚本本地运行,大概在160000+次的时候会报新的错误,无法创建线程

# 运行脚本
from ddparser import DDParser

ddp = DDParser()
count = 0
for i in range(500000):
    text = "两侧额顶叶深部白质区及侧脑室周围多发点状、小斑片及片状异常信号"
    ddp_res = ddp.parse(text)
    count += 1
    if i % 10000 == 0:
        print("==>", i)

# 错误信息
==> 170000
Traceback (most recent call last):
  File "/Users/wangy/PycharmProjects/test/tests.py", line 40, in <module>
    ddp_res = ddp.parse(text)
  File "/Users/wangy/anaconda3/lib/python3.7/site-packages/ddparser/run.py", line 399, in parse
    pred_arcs, pred_rels, pred_probs = epoch_predict(self.env, self.args, self.model, dataset.loader)
  File "<decorator-gen-270>", line 2, in epoch_predict
  File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 276, in __impl__
    return func(*args, **kwargs)
  File "/Users/wangy/anaconda3/lib/python3.7/site-packages/ddparser/parser/model.py", line 185, in epoch_predict
    for batch, inputs in enumerate(loader(), start=1):
  File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 961, in __iter__
    self._start()
  File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 947, in _start
    self._thread.start()
  File "/Users/wangy/anaconda3/lib/python3.7/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

@zhangyimi 辛苦帮忙看看是什么原因,这里的 ddparser==1.0.6,paddlepaddle==2.1.3
非常感谢!

from ddparser.

zhangyimi avatar zhangyimi commented on August 25, 2024

这个问题会在近期修复。

from ddparser.

WyHy avatar WyHy commented on August 25, 2024

这个问题会在近期修复。

辛苦,非常期待!

from ddparser.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.