BigQuery Storage API: RuntimeError: ("can't start new thread", 'occurred at index 0') about python-bigquery HOT 3 CLOSED

@megancooper This is more of an educated guess, but the following bit indeed seems relevant:

  ...
  File "/home/hadoop/conda/lib/python3.7/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: ("can't start new thread", 'occurred at index 0')

An error is raised when Python cannot start a new thread. Since you mention a high number of executors, could it be that a max thread limit is hit and no more threads can be created?

You could log the number of active threads at key points in the program and see if the number gets too high just before the crash?

import threading
threading.active_count()   # this should be logged

If there are indeed too many threads created, one possible would be reducing the number of concurrent queries or somehow throttle them (I'm not familiar with Spark, unfortunately).

from python-bigquery.

Hi, how do you solve this problem?

from python-bigquery.

Hi @shell-no-shell — we ended up closing this since the OP did not respond. If you have a similar issue, can you please file a new ticket (if you think it's a bug in the library) or a support issue (if it would require digging into project details, since we don't have access to that info here). Thanks!

from python-bigquery.