workthread.join()
is still freezing on occasion, unsure what the workthread is doing or why it's freezing as nothing is being logged, it is the waitKill
workthread, and factorio did already get killed and it did fully die (no remnants of its process is still running).
I'm not entirely sure of the purpose of that workthread as it is spawned up and then immediately joined on.
When I canceled it after leaving it at that point for 20 minutes it printed this when I Ctrl+c'd it:
^CProcess ForkPoolWorker-55:
Process ForkPoolWorker-57:
Process ForkPoolWorker-64:
Process ForkPoolWorker-51:
Process ForkPoolWorker-66:
Process ForkPoolWorker-59:
Process ForkPoolWorker-53:
Process ForkPoolWorker-65:
Process ForkPoolWorker-56:
Process ForkPoolWorker-54:
Process ForkPoolWorker-58:
Process ForkPoolWorker-60:
Process ForkPoolWorker-61:
Process ForkPoolWorker-62:
Process ForkPoolWorker-63:
Process ForkPoolWorker-52:
sh: 1: taskkill: not found
killed factorio
disabling FactorioMaps mod
cleaning up
Traceback (most recent call last):
File "auto.py", line 339, in <module>
auto(*sys.argv[1:])
File "auto.py", line 250, in auto
crop(outFolder, otherInputs[0], otherInputs[1], otherInputs[2], basepath)
File "/home/overminddl1/Games/Factorio/tmp/mods/L0laapk3_FactorioMaps_2.4.2/crop.py", line 66, in crop
if progressQueue.get(True):
File "<string>", line 2, in get
File "/usr/lib/python3.6/multiprocessing/managers.py", line 757, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 335, in get
res = self._reader.recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
Interestingly I only hit Ctrl+c once, I'm guessing it's printing KeyboardInterrupt
for each subprocess. At this point, however, it was frozen again with no further output, so after waiting a while I hit Ctrl+c again and it printed this and finally died:
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers
finalizer()
File "/usr/lib/python3.6/multiprocessing/util.py", line 186, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 597, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 582, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
A few interesting things to notice:
First, when Ctrl+c was hit it tried running taskkil /im factorio.exe
, which is a Windows command (the heck?! Python should not need those.)...
It seems the subprocesses were all stuck waiting on messages, the only one different was:
File "/home/overminddl1/Games/Factorio/tmp/mods/L0laapk3_FactorioMaps_2.4.2/crop.py", line 66, in crop
if progressQueue.get(True):
For note mp.cpu_count()
is returning 16 (the number of cores in this system).
Some testing shows that in crop.py
this section:
while len(files) > 0:
workers = pool.map_async(partial(work, imgsize=imgsize, folder=folder, queue=progressQueue), files, 128)
for _ in range(len(files)):
print(repr(files))
if progressQueue.get(True):
doneSize += 1
progress = float(doneSize) / originalSize
tsiz = tsize()[0]-15
print("\rcrop {:5.1f}% [{}{}]".format(round(progress * 100, 1), "=" * int(progress * tsiz), " " * (tsiz - int(progress * tsiz))), end="")
workers.wait()
files = [x for x in workers.get() if x]
if len(files) > 0:
time.sleep(10 if len(files) > 1000 else 1)
I put a print(repr(files))
just after the for _ in range(len(files)):
line and it was only called once, printing out:
['-12 -18 0 0 1\n', '-12 -17 0 0 3\n', '-11 -18 0 0 0\n', '-11 -17 0 0 1\n', '-12 -16 0 72 3\n', '-12 -15 0 0 3\n', '-11 -16 0 72 3\n', '-11 -15 0 328 2\n', '-12 -14 0 392 2\n', '-11 -14 0 392 0\n', '-10 -18 8 0 5\n', '-10 -17 8 0 f\n', '-9 -18 0 0 0\n', '-9 -17 232 0 5\n', '-10 -16 72 72 f\n', '-10 -15 72 328 f\n', '-9 -16 232 72 f\n', '-9 -15 232 328 a\n', '-10 -14 72 488 a\n', '-9 -14 0 488 0\n', '-8 -18 360 0 5\n', '-8 -17 360 0 f\n', '-7 -18 0 0 0\n', '-7 -17 264 0 5\n', '-8 -16 424 72 f\n', '-8 -15 424 232 f\n', '-7 -16 264 40 f\n', '-7 -15 264 488 b\n', '-8 -14 424 488 a\n', '-7 -14 0 488 3\n', '-7 -13 0 424 2\n', '-6 -18 424 0 5\n', '-6 -17 424 0 f\n', '-5 -18 0 0 1\n', '-5 -17 488 0 7\n', '-6 -16 456 232 f\n', '-6 -15 456 488 f\n', '-5 -16 488 232 f\n', '-5 -15 488 488 f\n', '-6 -14 456 488 b\n', '-6 -13 0 424 2\n', '-5 -14 264 488 f\n', '-5 -13 264 488 b\n', '-5 -12 0 328 2\n', '-4 -18 104 0 5\n', '-4 -17 104 0 f\n', '-3 -18 40 0 5\n', '-3 -17 328 0 f\n', '-4 -16 104 232 f\n', '-4 -15 264 488 f\n', '-3 -16 328 232 f\n', '-3 -15 328 456 f\n', '-4 -14 264 488 f\n', '-4 -13 264 488 b\n', '-3 -14 456 456 f\n', '-3 -13 456 488 e\n', '-4 -12 0 328 2\n', '-3 -12 456 328 8\n', '-2 -18 72 0 5\n', '-2 -17 296 0 f\n', '-1 -18 232 0 5\n', '-1 -17 232 0 f\n', '-2 -16 296 232 f\n', '-2 -15 296 456 f\n', '-1 -16 296 168 f\n', '-1 -15 296 360 e\n', '-2 -14 296 456 f\n', '-2 -13 296 424 e\n', '-1 -14 296 168 c\n', '-1 -13 232 424 8\n', '-2 -12 136 104 8\n', '0 -18 360 0 5\n', '0 -17 360 0 f\n', '1 -18 456 0 4\n', '1 -17 456 0 c\n', '0 -16 360 200 f\n', '0 -15 264 456 e\n', '1 -16 456 200 c\n', '1 -15 168 456 8\n', '0 -14 264 168 c\n', '0 -13 264 424 8\n', '2 -18 232 0 4\n', '2 -17 232 0 c\n', '2 -16 232 200 c\n', '2 -15 232 456 8']
Upon which it seems to permanently freeze at progressQueue.get(True)
.
It only seems to be happening on 'night' as well, these is the log up to that point:
GregOver2
GregOver2c
factorio path: /home/overminddl1/Games/Factorio/tmp/bin/x64/factorio
output folder: script-output/FactorioMaps/GregOver2
checking for updates
enabling FactorioMaps mod
[GAME] 138.417 Created atlas bitmap 8192x2432 [linear-minification, mipmap, terrain] libpng warning: iCCP: profile 'ICC PROFILE': 3042AF0Dh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': E042E80Fh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': F097D00Fh: exceeds application limits
libpng warning: iCCP: CRC error
libpng warning: iCCP: profile 'ICC PROFILE': D00DCD0Dh: exceeds application limits
libpng warning: iCCP: CRC error
[GAME] 168.751 Warning LuaGameScript.cpp:1545: luaremove_path error: Scripts can only use file space inside of script-output Processing GregOver2/10/nauvis/day (1 of 2)
crop 100.0% [================================================================================================================================================================================================]
ref 100.0% [================================================================================================================================================================================================]
zoom 100.0% [================================================================================================================================================================================================]
Processing GregOver2/10/nauvis/night (2 of 2)
crop 0.0% [ ]['-12 -18 0 0 1\n', '-12 -17 0 0 3\n', '-11 -18 0 0 0\n', '-11 -17 0 0 1\n', '-12 -16 0 72 3\n', '-12 -15 0 0 3\n', '-11 -16 0 72 3\n', '-11 -15 0 328 2\n', '-12 -14 0 392 2\n', '-11 -14 0 392 0\n', '-10 -18 8 0 5\n', '-10 -17 8 0 f\n', '-9 -18 0 0 0\n', '-9 -17 232 0 5\n', '-10 -16 72 72 f\n', '-10 -15 72 328 f\n', '-9 -16 232 72 f\n', '-9 -15 232 328 a\n', '-10 -14 72 488 a\n', '-9 -14 0 488 0\n', '-8 -18 360 0 5\n', '-8 -17 360 0 f\n', '-7 -18 0 0 0\n', '-7 -17 264 0 5\n', '-8 -16 424 72 f\n', '-8 -15 424 232 f\n', '-7 -16 264 40 f\n', '-7 -15 264 488 b\n', '-8 -14 424 488 a\n', '-7 -14 0 488 3\n', '-7 -13 0 424 2\n', '-6 -18 424 0 5\n', '-6 -17 424 0 f\n', '-5 -18 0 0 1\n', '-5 -17 488 0 7\n', '-6 -16 456 232 f\n', '-6 -15 456 488 f\n', '-5 -16 488 232 f\n', '-5 -15 488 488 f\n', '-6 -14 456 488 b\n', '-6 -13 0 424 2\n', '-5 -14 264 488 f\n', '-5 -13 264 488 b\n', '-5 -12 0 328 2\n', '-4 -18 104 0 5\n', '-4 -17 104 0 f\n', '-3 -18 40 0 5\n', '-3 -17 328 0 f\n', '-4 -16 104 232 f\n', '-4 -15 264 488 f\n', '-3 -16 328 232 f\n', '-3 -15 328 456 f\n', '-4 -14 264 488 f\n', '-4 -13 264 488 b\n', '-3 -14 456 456 f\n', '-3 -13 456 488 e\n', '-4 -12 0 328 2\n', '-3 -12 456 328 8\n', '-2 -18 72 0 5\n', '-2 -17 296 0 f\n', '-1 -18 232 0 5\n', '-1 -17 232 0 f\n', '-2 -16 296 232 f\n', '-2 -15 296 456 f\n', '-1 -16 296 168 f\n', '-1 -15 296 360 e\n', '-2 -14 296 456 f\n', '-2 -13 296 424 e\n', '-1 -14 296 168 c\n', '-1 -13 232 424 8\n', '-2 -12 136 104 8\n', '0 -18 360 0 5\n', '0 -17 360 0 f\n', '1 -18 456 0 4\n', '1 -17 456 0 c\n', '0 -16 360 200 f\n', '0 -15 264 456 e\n', '1 -16 456 200 c\n', '1 -15 168 456 8\n', '0 -14 264 168 c\n', '0 -13 264 424 8\n', '2 -18 232 0 4\n', '2 -17 232 0 c\n', '2 -16 232 200 c\n', '2 -15 232 456 8']
killed factorio
At which point it freezes. Interesting that the file list doesn't output for the day crop...
Replacing the work
function to just return False
does not change anything, setting output shows that it looks like it's not even being called... interesting...
Tried calling map_async
and even map
in a variety of different ways and no callback executes, everything is getting locked on a semaphore and never being released.
Just to get it 'working' for now, commenting out all the multiprocess stuff in crop.py (haven't run into any issues with it in zoom.py yet for whatever reason) and just calling work
directly in a loop on files
and that works, only takes a second as well with about a hundred chunks (zoom is the one that takes a few seconds) and works fine.