Normal to Save Checkpoint for 10 minutes on Training? about deepdanbooru HOT 14 CLOSED

SELECT Count(1) FROM posts WHERE posts.tag_count_general >= 10

da3dsoul@THE-THRONE:/media/da3dsoul/Golias/DeepDanbooru$ deepdanbooru train-project /media/da3dsoul/Golias/DeepDanbooru/unbooru_model/
Using Adam optimizer ... 
Loading tags ... 
Creating model (resnet_custom_v2) ... 
Model : (None, 299, 299, 3) -> (None, 14176)
Loading database ... 
No checkpoint. Starting new training ... (2021-08-31 00:47:17.383431)
Shuffling samples (epoch 0) ... 
Trying to change learning rate to 0.001 ...
Learning rate is changed to <tf.Variable 'learning_rate:0' shape=() dtype=float32, numpy=0.001> ...
2021-08-31 00:47:31.829958: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:592] remapper failed: Not found: Op type not registered '_CopyFromGpuToHost' in binary running on THE-THRONE. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: /opt/rocm-4.3.0/miopen/share/miopen/db/gfx803_32.HIP.fdb.txt
Traceback (most recent call last):
  File "/usr/local/bin/deepdanbooru", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/deepdanbooru/__main__.py", line 52, in train_project
    dd.commands.train_project(project_path, source_model)
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/deepdanbooru/commands/train_project.py", line 196, in train_project
    step_result = model.train_on_batch(
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1727, in train_on_batch
    logs = self.train_function(iterator)
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2942, in __call__
    return graph_function._call_flat(
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/home/da3dsoul/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted:  OOM when allocating tensor with shape[1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node resnet_custom_v2/batch_normalization_63/FusedBatchNormV3 (defined at home/da3dsoul/.local/lib/python3.8/site-packages/deepdanbooru/commands/train_project.py:196) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[Func/assert_greater_equal/Assert/AssertGuard/else/_1/input_control_node/_62/_83]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted:  OOM when allocating tensor with shape[1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node resnet_custom_v2/batch_normalization_63/FusedBatchNormV3 (defined at home/da3dsoul/.local/lib/python3.8/site-packages/deepdanbooru/commands/train_project.py:196) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_52220]

Function call stack:
train_function -> train_function

image_dir = *pull it from the database dir*
if not os.isdir(image_dir):
    throw new Exception("Missing images folder!")