tensorfreitas / temporal-ensembling-for-semi-supervised-learning Goto Github PK

I changed the code to make it run on my data. My images are also 32x32x3 patches. There are 18000+ labelled samples. I am using a GPU with 8GB memory. I set the batch size to 128. When training starts, everything looks good, the GPU utilization is about 50%, but as the training going on, the GPU utilization decreases slowly. After 100+ epoches, the training becomes very slow and the GPU utilization is about 1% to 5%. Does anyone have this problem? Thank you!

Hi,

First of all, I'd like to thank you tremendously for sharing this clean TensorFlow implementation - it saved me a lot of time!

Second, I am wondering if by any chance you have compared your pi-model results with the original paper. I tried training the pi-model on SVHN using the same hyperparameters reported with 1000 labels, but for some reason the model overfits around the 50th-60th epoch, and the final test accuracy of the best ckpt is only 85%, versus almost 95% in the paper.

then i find that the training time of this code
current_outputs, loss_val, grads = temporal_ensembling_gradients(X_labeled_train, y_labeled_train, X_unlabeled_train,model, unsupervised_weight, current_ensemble_targets)
become more and more long after every epoch
anyone know how to deal with this problem. thanks

How much effort would it be to convert this to handle row vectors instead of images?

thanks

hi,
when l use the real ratio: labeled_batch_fraction = num_labeled_samples / num_train_unlabeled_sample(it should be num_labeled_samples / (num_train_unlabeled_sample+num_labeled_samples?)), after training about more than 50 epochs, l find that the both train and val acc is always between 9%-10%. it means that the model doesn't work at all. do you have the same question? besides, train many times your code, the best test acc is only 89% far away from the paper. is it same with you?
thank you.

Whenever I try to execute train_pi_model.py or train_temporal_model.py I get this same error. Can someone help me diagnose the problem?

(tf-env) C:\Users\crm0376\Documents\ONR\Temporal-Ensembling-for-Semi-Supervised-Learning-master>C:/Users/crm0376/.local/conda_root/envs/tf-env/python.exe c:/Users/crm0376/Documents/ONR/Temporal-Ensembling-for-Semi-Supervised-Learning-master/train_temporal_ensembling_model.py 2018-11-08 17:23:21.615533: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Traceback (most recent call last): File "c:/Users/crm0376/Documents/ONR/Temporal-Ensembling-for-Semi-Supervised-Learning-master/train_temporal_ensembling_model.py", line 199, in <module> main() File "c:/Users/crm0376/Documents/ONR/Temporal-Ensembling-for-Semi-Supervised-Learning-master/train_temporal_ensembling_model.py", line 111, in main model, unsupervised_weight, current_ensemble_targets) File "c:\Users\crm0376\Documents\ONR\Temporal-Ensembling-for-Semi-Supervised-Learning-master\pi_model.py", line 54, in temporal_ensembling_gradients model, unsupervised_weight, ensembling_targets) File "c:\Users\crm0376\Documents\ONR\Temporal-Ensembling-for-Semi-Supervised-Learning-master\pi_model.py", line 25, in temporal_ensembling_loss z_labeled = model(X_train_labeled) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 757, in __call__ outputs = self.call(inputs, *args, **kwargs) File "c:\Users\crm0376\Documents\ONR\Temporal-Ensembling-for-Semi-Supervised-Learning-master\pi_model.py", line 264, in call h = self.__apply_image_augmentation(h) File "c:\Users\crm0376\Documents\ONR\Temporal-Ensembling-for-Semi-Supervised-Learning-master\pi_model.py", line 248, in __apply_image_augmentation image, random_translation.numpy, 'NEAREST') File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\contrib\image\python\ops\image_ops.py", line 125, in translate translations_to_projective_transforms(translations), File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\contrib\image\python\ops\image_ops.py", line 191, in translations_to_projective_transforms translations, name="translations", dtype=dtypes.float32) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\framework\ops.py", line 1050, in convert_to_tensor as_ref=False) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\framework\ops.py", line 1146, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\framework\constant_op.py", line 229, in _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\framework\constant_op.py", line 179, in constant t = convert_to_eager_tensor(value, ctx, dtype) File "C:\Users\crm0376\.local\conda_root\envs\tf-env\lib\site-packages\tensorflow\python\framework\constant_op.py", line 114, in convert_to_eager_tensor return ops.EagerTensor(value, context=handle, device=device, dtype=dtype) ValueError: Attempt to convert a value (<bound method _EagerTensorBase.numpy of <tf.Tensor: id=233, shape=(1, 2), dtype=float32, numpy=array([[-0.76205397, 0.42147446]], dtype=float32)>>) with an unsupported type (<class 'method'>) to a Tensor.

tf.losses.softmax_cross_entropy is softmax(output) and then cross_entropy, while the model's last layer--dense also has the softmax activation. It means that the final loss will softmax twice. is it wrong?

Hi,
Thanks for your code.
In your implement, the ensemble targets are network output before softmax. However, I think the target is the ensemble of output after softmax. Since in table 5 in the paper, output is the softmax.

I'm sorry,..., but your codes seem to have several bugs, and I think it's why it can't reach the accuracy that the paper Temporal Ensembling... reported.

1. you normalize the SVHN images with axes [1,2,3], but they are [32, 32, 3, Num] before you did the reshape option.
2. your model's output is classification possiblility after Softmax activation, but you computed the loss with another rebundant softmax(e.g. tf.losses.softmax_cross_entropy...)
3. your data augmentation is a translation op,
   Args: translations: A 2-element list representing [dx, dy] or a matrix of 2-element lists representing [dx, dy] to translate for each image (for a batch of images). The rank must be statically known (the shape is not TensorShape(None). name: The name of the op.
   but you only give an element for each image instead of a 2-element list [dx, dy]...
   4.It seems that your weight normalization's reimplementations also have some bugs which I don't know. In addition, I use it with a simple NN mnist supervised classification task, it got a poor test accuracy (53%), which a 97.6% test accuracy can be obtained without using your Weight norm under the same condition.

Much appreciated for your succinct implementation. I'm trying to reproduce the result using your code. However, when I try to produce the baseline result, the supervised version with 1000 labels, using this code, it gives rather poor result, ~30%. Did you validate this by yourself? Or am I doing something stupid?
Thank you again.