rvignav / simclr Goto Github PK

WARNING:tensorflow:From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
I0705 11:17:09.415343 140462500476736 dataset_info.py:361] Load dataset info from /scratch/users/rvignav/datadir/diabetic_retinopathy_detection/btgraham-300/3.0.0
I0705 11:17:09.425307 140462500476736 dataset_builder.py:282] Reusing dataset diabetic_retinopathy_detection (/scratch/users/rvignav/datadir/diabetic_retinopathy_detection/btgraham-300/3.0.0)
WARNING:tensorflow:From run.py:407: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W0705 11:17:09.425994 140462500476736 module_wrapper.py:139] From run.py:407: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From run.py:409: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W0705 11:17:09.426208 140462500476736 module_wrapper.py:139] From run.py:409: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From run.py:410: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W0705 11:17:09.426391 140462500476736 module_wrapper.py:139] From run.py:410: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From run.py:420: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W0705 11:17:09.426871 140462500476736 module_wrapper.py:139] From run.py:420: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

WARNING:tensorflow:Estimator's model_fn (<function build_model_fn.<locals>.model_fn at 0x7fbfd4e3c400>) includes params argument, but params are not passed to Estimator.
W0705 11:17:09.427150 140462500476736 estimator.py:1994] Estimator's model_fn (<function build_model_fn.<locals>.model_fn at 0x7fbfd4e3c400>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Using config: {'_model_dir': '/scratch/users/rvignav/simclr_test', '_tf_random_seed': None, '_save_summary_steps': 69, '_save_checkpoints_steps': 69, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fbfd4e5f2e8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=69, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I0705 11:17:09.427832 140462500476736 estimator.py:212] Using config: {'_model_dir': '/scratch/users/rvignav/simclr_test', '_tf_random_seed': None, '_save_summary_steps': 69, '_save_checkpoints_steps': 69, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fbfd4e5f2e8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=69, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I0705 11:17:09.428189 140462500476736 tpu_context.py:220] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W0705 11:17:09.428814 140462500476736 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False.
INFO:tensorflow:Waiting for new checkpoint at /scratch/users/rvignav/simclr_test
I0705 11:17:09.428961 140462500476736 checkpoint_utils.py:124] Waiting for new checkpoint at /scratch/users/rvignav/simclr_test
INFO:tensorflow:Found new checkpoint at /scratch/users/rvignav/simclr_test/model.ckpt-687
I0705 11:17:09.432511 140462500476736 checkpoint_utils.py:133] Found new checkpoint at /scratch/users/rvignav/simclr_test/model.ckpt-687
WARNING:tensorflow:From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0705 11:17:09.438420 140462500476736 deprecation.py:506] From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
I0705 11:17:09.439526 140462500476736 dataset_builder.py:478] Constructing tf.data.Dataset for split test, from /scratch/users/rvignav/datadir/diabetic_retinopathy_detection/btgraham-300/3.0.0
HERE INPUT_FN
INFO:tensorflow:Calling model_fn.
I0705 11:17:13.008392 140462500476736 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Running eval on CPU
I0705 11:17:13.008868 140462500476736 tpu_estimator.py:3124] Running eval on CPU
WARNING:tensorflow:From /home/users/rvignav/simclr/resnet.py:277: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0705 11:17:13.012213 140462500476736 deprecation.py:323] From /home/users/rvignav/simclr/resnet.py:277: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
WARNING:tensorflow:From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0705 11:17:13.013813 140462500476736 deprecation.py:323] From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /home/users/rvignav/simclr/model_util.py:134: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
W0705 11:17:13.573580 140462500476736 deprecation.py:323] From /home/users/rvignav/simclr/model_util.py:134: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0705 11:17:13.692904 140462500476736 deprecation.py:323] From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/losses/losses_impl.py:121: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
INFO:tensorflow:===============Variables to train (begin)===============
I0705 11:17:13.737640 140462500476736 model.py:104] ===============Variables to train (begin)===============
INFO:tensorflow:[<tf.Variable 'base_model/conv2d/kernel:0' shape=(3, 3, 3, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_1/kernel:0' shape=(1, 1, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_1/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_1/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_2/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_2/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_2/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_3/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_3/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_3/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_4/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_4/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_4/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_5/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_5/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_5/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_6/kernel:0' shape=(1, 1, 64, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_6/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_6/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_7/kernel:0' shape=(3, 3, 64, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_7/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_7/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_8/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_8/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_8/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_9/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_9/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_9/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_10/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_10/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_10/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_11/kernel:0' shape=(1, 1, 128, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_11/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_11/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_12/kernel:0' shape=(3, 3, 128, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_12/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_12/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_13/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_13/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_13/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_14/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_14/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_14/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_15/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_15/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_15/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_16/kernel:0' shape=(1, 1, 256, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_16/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_16/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_17/kernel:0' shape=(3, 3, 256, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_17/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_17/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_18/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_18/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_18/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_19/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_19/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_19/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_20/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_20/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_20/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/dense/kernel:0' shape=(512, 512) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/batch_normalization/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/batch_normalization/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/dense/kernel:0' shape=(512, 512) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/batch_normalization/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/batch_normalization/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_2/dense/kernel:0' shape=(512, 128) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_2/batch_normalization/gamma:0' shape=(128,) dtype=float32_ref>]
I0705 11:17:13.737865 140462500476736 model.py:105] [<tf.Variable 'base_model/conv2d/kernel:0' shape=(3, 3, 3, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_1/kernel:0' shape=(1, 1, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_1/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_1/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_2/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_2/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_2/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_3/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_3/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_3/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_4/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_4/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_4/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_5/kernel:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_5/gamma:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_5/beta:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_6/kernel:0' shape=(1, 1, 64, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_6/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_6/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_7/kernel:0' shape=(3, 3, 64, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_7/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_7/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_8/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_8/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_8/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_9/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_9/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_9/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_10/kernel:0' shape=(3, 3, 128, 128) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_10/gamma:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_10/beta:0' shape=(128,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_11/kernel:0' shape=(1, 1, 128, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_11/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_11/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_12/kernel:0' shape=(3, 3, 128, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_12/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_12/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_13/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_13/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_13/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_14/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_14/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_14/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_15/kernel:0' shape=(3, 3, 256, 256) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_15/gamma:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_15/beta:0' shape=(256,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_16/kernel:0' shape=(1, 1, 256, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_16/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_16/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_17/kernel:0' shape=(3, 3, 256, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_17/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_17/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_18/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_18/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_18/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_19/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_19/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_19/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/conv2d_20/kernel:0' shape=(3, 3, 512, 512) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_20/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'base_model/batch_normalization_20/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/dense/kernel:0' shape=(512, 512) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/batch_normalization/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_0/batch_normalization/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/dense/kernel:0' shape=(512, 512) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/batch_normalization/gamma:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_1/batch_normalization/beta:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_2/dense/kernel:0' shape=(512, 128) dtype=float32_ref>, <tf.Variable 'head_contrastive/nl_2/batch_normalization/gamma:0' shape=(128,) dtype=float32_ref>]
INFO:tensorflow:================Variables to train (end)================
I0705 11:17:13.739330 140462500476736 model.py:106] ================Variables to train (end)================
WARNING:tensorflow:From /home/users/rvignav/simclr/model_util.py:72: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0705 11:17:13.739619 140462500476736 deprecation.py:323] From /home/users/rvignav/simclr/model_util.py:72: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /home/users/rvignav/simclr/model.py:218: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

W0705 11:17:13.756437 140462500476736 module_wrapper.py:139] From /home/users/rvignav/simclr/model.py:218: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.

WARNING:tensorflow:From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/metrics_impl.py:2639: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
W0705 11:17:13.886275 140462500476736 deprecation.py:323] From /home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/metrics_impl.py:2639: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
INFO:tensorflow:Done calling model_fn.
I0705 11:17:13.992630 140462500476736 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-07-05T11:17:14Z
I0705 11:17:14.008369 140462500476736 evaluation.py:255] Starting evaluation at 2021-07-05T11:17:14Z
INFO:tensorflow:Graph was finalized.
I0705 11:17:14.412155 140462500476736 monitored_session.py:240] Graph was finalized.
2021-07-05 11:17:14.412763: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-07-05 11:17:14.418010: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2021-07-05 11:17:14.418185: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5e00100 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-07-05 11:17:14.418201: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
INFO:tensorflow:Restoring parameters from /scratch/users/rvignav/simclr_test/model.ckpt-687
I0705 11:17:14.419870 140462500476736 saver.py:1284] Restoring parameters from /scratch/users/rvignav/simclr_test/model.ckpt-687
INFO:tensorflow:Running local_init_op.
I0705 11:17:14.745326 140462500476736 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0705 11:17:14.792443 140462500476736 session_manager.py:502] Done running local_init_op.
ERROR:tensorflow:Error recorded from evaluation_loop:  Input to reshape is a tensor with 874800 values, but the requested shape has 3072
	 [[{{node Reshape}}]]
	 [[IteratorGetNext]]
E0705 11:17:15.808625 140462500476736 error_handling.py:75] Error recorded from evaluation_loop:  Input to reshape is a tensor with 874800 values, but the requested shape has 3072
	 [[{{node Reshape}}]]
	 [[IteratorGetNext]]
INFO:tensorflow:evaluation_loop marked as finished
I0705 11:17:15.808902 140462500476736 error_handling.py:101] evaluation_loop marked as finished
WARNING:tensorflow:Reraising captured error
W0705 11:17:15.809075 140462500476736 error_handling.py:135] Reraising captured error
Traceback (most recent call last):
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __inference_Dataset_map_map_fn_180}} Input to reshape is a tensor with 874800 values, but the requested shape has 3072
	 [[{{node Reshape}}]]
	 [[IteratorGetNext]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 456, in <module>
    app.run(main)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "run.py", line 437, in main
    checkpoint_path=ckpt)
  File "run.py", line 329, in perform_evaluation
    name=FLAGS.eval_name)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3056, in evaluate
    rendezvous.raise_errors()
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors
    six.reraise(typ, value, traceback)
  File "/share/software/user/open/py-jupyter/1.0.0_py36/lib/python3.6/site-packages/six.py", line 686, in reraise
    raise value
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3051, in evaluate
    name=name)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
    name=name)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
    return _evaluate()
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1619, in _evaluate_run
    config=self._session_config)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/share/software/user/open/py-jupyter/1.0.0_py36/lib/python3.6/site-packages/six.py", line 686, in reraise
    raise value
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/users/rvignav/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Input to reshape is a tensor with 874800 values, but the requested shape has 3072
	 [[{{node Reshape}}]]
	 [[IteratorGetNext]]