I am trying to reproduce your amazing work, but I am getting an out of mermory issue for some reasons.
I have tried to reduce batch_size, and reduce prefetch, none of these worked.
Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9605 MB memory) -> physical GPU (device: 0, name: TITAN
Xp COLLECTORS EDITION, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into D:\StageVYNCK\GenderClass\VyncketUn\cnn_model\model.ckpt.
2019-05-17 12:19:47.423066: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.425793: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:47.428146: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 483183872 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.430317: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 483183872
2019-05-17 12:19:47.432615: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 434865664 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.435390: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 434865664
2019-05-17 12:19:47.437544: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:47.439808: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.455649: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:57.457975: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.459555: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 536870912 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-05-17 12:19:57.461229: W .\tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 536870912
2019-05-17 12:19:57.462758: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (cuda_host_bfc) ran out of memory trying to allocate 392.00MiB. Current allocation summary follows.
2019-05-17 12:19:57.464715: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 12, Chunks in use: 12. 3.0KiB allocated for chunks. 3.0KiB in use in bin. 120B client-requested in use in bin.
2019-05-17 12:19:57.466972: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.2KiB client-requested in use in bin.
2019-05-17 12:19:57.469227: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.472371: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.474488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2019-05-17 12:19:57.476930: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 3, Chunks in use: 3. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2019-05-17 12:19:57.479308: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.482225: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.484920: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.487328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.489671: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 2, Chunks in use: 2. 808.0KiB allocated for chunks. 808.0KiB in use in bin. 808.0KiB client-requested in use in bin.
2019-05-17 12:19:57.492187: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 1, Chunks in use: 1. 581.0KiB allocated for chunks. 581.0KiB in use in bin. 404.0KiB client-requested in use in bin.
2019-05-17 12:19:57.494839: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.498008: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.500185: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 6, Chunks in use: 6. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 24.00MiB client-requested in use in bin.
2019-05-17 12:19:57.502562: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.504796: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.507932: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.510289: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.512693: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-17 12:19:57.514864: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 1, Chunks in use: 1. 491.59MiB allocated for chunks. 491.59MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-05-17 12:19:57.517460: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 392.00MiB was 256.00MiB, Chunk State:
2019-05-17 12:19:57.518808: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400000 of size 256
2019-05-17 12:19:57.520036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400100 of size 256
2019-05-17 12:19:57.521273: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400200 of size 256
2019-05-17 12:19:57.522748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400300 of size 256
2019-05-17 12:19:57.524784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400400 of size 256
2019-05-17 12:19:57.526171: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400500 of size 256
2019-05-17 12:19:57.527399: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400600 of size 256
2019-05-17 12:19:57.528672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400700 of size 256
2019-05-17 12:19:57.529895: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400800 of size 256
2019-05-17 12:19:57.531114: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203400900 of size 8192
2019-05-17 12:19:57.532375: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203402900 of size 8192
2019-05-17 12:19:57.534155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203404900 of size 8192
2019-05-17 12:19:57.535444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406900 of size 256
2019-05-17 12:19:57.536714: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406A00 of size 256
2019-05-17 12:19:57.537947: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406B00 of size 256
2019-05-17 12:19:57.539200: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203406C00 of size 4096
2019-05-17 12:19:57.540521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203407C00 of size 4096
2019-05-17 12:19:57.541766: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203408C00 of size 4096
2019-05-17 12:19:57.543185: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203409C00 of size 413696
2019-05-17 12:19:57.544617: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000020346EC00 of size 594944
2019-05-17 12:19:57.545895: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203600000 of size 4194304
2019-05-17 12:19:57.547194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203A00000 of size 4194304
2019-05-17 12:19:57.548500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000203E00000 of size 4194304
2019-05-17 12:19:57.549777: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204200000 of size 413696
2019-05-17 12:19:57.551093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265000 of size 512
2019-05-17 12:19:57.552500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265200 of size 512
2019-05-17 12:19:57.553739: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265400 of size 512
2019-05-17 12:19:57.554978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204265600 of size 4194304
2019-05-17 12:19:57.556259: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204665600 of size 4194304
2019-05-17 12:19:57.557575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204A65600 of size 4194304
2019-05-17 12:19:57.559832: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E65600 of size 4096
2019-05-17 12:19:57.561373: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E66600 of size 4096
2019-05-17 12:19:57.562672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E67600 of size 4096
2019-05-17 12:19:57.563905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000000204E68600 of size 515471872
2019-05-17 12:19:57.565208: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2019-05-17 12:19:57.566461: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 12 Chunks of size 256 totalling 3.0KiB
2019-05-17 12:19:57.567741: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-05-17 12:19:57.569629: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 4096 totalling 24.0KiB
2019-05-17 12:19:57.570891: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 8192 totalling 24.0KiB
2019-05-17 12:19:57.572133: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 413696 totalling 808.0KiB
2019-05-17 12:19:57.573406: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 594944 totalling 581.0KiB
2019-05-17 12:19:57.574675: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 4194304 totalling 24.00MiB
2019-05-17 12:19:57.575958: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 515471872 totalling 491.59MiB
2019-05-17 12:19:57.577300: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 517.00MiB
2019-05-17 12:19:57.579081: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 68719476736
InUse: 542113792
MaxInUse: 542113792
NumAllocs: 67
MaxAllocSize: 536870912
2019-05-17 12:19:57.581960: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *********************************************************************************xxxxxxxxxxxxxxxxxxx
Traceback (most recent call last):
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[{{node dense/kernel/Adam}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\StageVYNCK\GenderClass\VyncketUn\bin\train.py", line 45, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 471, in train_and_evaluate
return executor.run()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 611, in run
return self.run_local()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 712, in run_local
saving_listeners=saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 358, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1124, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model_default
saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1403, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 508, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 934, in init
stop_grace_period_secs=stop_grace_period_secs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 648, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1122, in init
_WrappedSession.init(self, self._create_session())
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1127, in _create_session
return self._sess_creator.create_session()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\monitored_session.py", line 812, in create_session
hook.after_create_session(self.tf_sess, self.coord)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 568, in after_create_session
self._save(session, global_step)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 599, in _save
self._get_saver().save(session, self._save_path, global_step=step)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\saver.py", line 1171, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[node dense/kernel/Adam (defined at D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py:99) ]]
Caused by op 'dense/kernel/Adam', defined at:
File "D:\StageVYNCK\GenderClass\VyncketUn\bin\train.py", line 45, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 471, in train_and_evaluate
return executor.run()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 611, in run
return self.run_local()
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 712, in run_local
saving_listeners=saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 358, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1124, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1154, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1112, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py", line 30, in model_fn
return get_training_spec(learning_rate, joint_loss)
File "D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py", line 99, in get_training_spec
global_step=tf.train.get_global_step())
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 413, in minimize
name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 595, in apply_gradients
self._create_slots(var_list)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\adam.py", line 135, in _create_slots
self._zeros_slot(v, "m", self._name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\optimizer.py", line 1153, in _zeros_slot
new_slot_variable = slot_creator.create_zeros_slot(var, op_name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 183, in create_zeros_slot
colocate_with_primary=colocate_with_primary)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 157, in create_slot_with_initializer
dtype)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\training\slot_creator.py", line 65, in _create_slot_var
validate_shape=validate_shape)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1479, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1220, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 547, in get_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 499, in _true_getter
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 911, in _get_single_variable
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 213, in call
return cls._variable_v1_call(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 176, in _variable_v1_call
aggregation=aggregation)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 155, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2495, in default_variable_creator
expected_shape=expected_shape, import_scope=import_scope)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 217, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 1395, in init
constraint=constraint)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\variables.py", line 1509, in _init_from_args
name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\state_ops.py", line 79, in variable_op_v2
shared_name=shared_name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 1425, in variable_v2
shared_name=shared_name, name=name)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "D:\Programs\Anaconda3\envs\TF-GPU\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
InternalError (see above for traceback): Dst tensor is not initialized.
[[node dense/kernel/Adam (defined at D:/StageVYNCK/genderclass/vyncketun\age_gender\cnn_estimator.py:99) ]]