gabrieleangeletti / deep-learning-tensorflow Goto Github PK

View Code? Open in Web Editor NEW

965.0 91.0 380.0 18 MB

Ready to use implementations of various Deep Learning algorithms using TensorFlow.

Home Page: http://blackecho.github.io

License: MIT License

Python 100.00%

deep-learning tensorflow

deep-learning-tensorflow's Introduction

Deep Learning algorithms with TensorFlow

This repository is a collection of various Deep Learning algorithms implemented using the TensorFlow library. This package is intended as a command line utility you can use to quickly train and evaluate popular Deep Learning models and maybe use them as benchmark/baseline in comparison to your custom models/datasets. If you want to use the package from ipython or maybe integrate it in your code, I published a pip package named yadlt: Yet Another Deep Learning Tool.

Requirements:

tensorflow >= 1.0

List of available models:

Convolutional Network
Restricted Boltzmann Machine
Deep Belief Network
Deep Autoencoder as stack of RBMs
Denoising Autoencoder
Stacked Denoising Autoencoder
Deep Autoencoder as stack of Denoising Autoencoders
MultiLayer Perceptron
Logistic Regression

Installation

Through pip:

pip install yadlt

You can learn the basic usage of the models by looking at the command_line/ directory. Or you can take a look at the documentation.

Note: the documentation is still a work in progress for the pip package, but the package usage is very simple. The classes have a sklearn-like interface, so basically you just have to create the object (e.g. sdae = StackedDenoisingAutoencoder()) and call the fit/predict methods, and the pretrain() method if the model supports it (e.g. sdae.pretrain(X_train, y_train), sdae.fit(X_train, y_train) and predictions = sdae.predict(X_test))

Through github:

cd in a directory where you want to store the project, e.g. /home/me
clone the repository: git clone https://github.com/blackecho/Deep-Learning-TensorFlow.git
cd Deep-Learning-TensorFlow
now you can configure the software and run the models (see the documentation)!

Documentation:

You can find the documentation for this project at this link.

Models TODO list

Recurrent Networks (LSTMs)
Variational Autoencoders
Deep Q Reinforcement Learning

deep-learning-tensorflow's People

Contributors

Stargazers

Watchers

Forkers

ml-ai-nlp-ir kentchun33333 jeffzhengye wanjinchang halofanx techscientist ryanscl buptqitian qiuyuew james97 darren1231 jinxindeep lando-ke yogendratamang48 stephen-xu asilvino veterun aldor007 ljdawn datastark simonxl butterflyaichinese undarmaa xinweizhang ifishlin nagabharat sarenlin siavash-khodadadeh him-28 williemaddox sharare90 madwombat andreas-koukorinis robi56 wellwang wavelets s4sarath vaaale pineking suvigy-epam oldgil xiximeng binbinbian zhangyuancv lhy20 benjamesbabala vtkingdom sufengniu lsq357 0532yangyang zc8340311 gsdgdf yejunbin qgzang gavinzjchao hammerzhang louiss007 chenhaifeng2016 szmgithub inessus yaokaichun huanyu-liao ytian81 jeanwang325 lixiangbao collector-m stevenlol lyk125 oftensmile lab-x nh007cs minghui phynex forforp xsongx bjyanli xulukai intmmsp xcbat eternallovelin salehinejad yebi2013 weibogit thesby chirayukong lexmao kevinmtian lanternying likesetaria scylla bgfurfeature yangqiokay hitflame qingaidexin cihankayacihan sawyergao92 solertis xiaohu2015 ozgurgundogan luckytina

deep-learning-tensorflow's Issues

Error while training DBN

I'm trying to train a DBN with a custom dataset. However I get the following error message

Training layer 1...
Traceback (most recent call last):
  File "run_dbn.py", line 118, in <module>
    srbm.pretrain(trX, vlX)
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/models/boltzmann/dbn.py", line 124, in pretrain
    train_set=train_set, validation_set=validation_set)
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/core/model.py", line 57, in pretrain_procedure
    layer_graphs[l])
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/core/model.py", line 74, in _pretrain_layer_and_gen_feed
    validation_set, validation_set, graph=graph)
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/core/unsupervised_model.py", line 58, in fit
    self.build_model(train_X.shape[1])
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/models/boltzmann/rbm.py", line 153, in build_model
    self.cost = self.loss.compile(vprob, self.input_data, regterm=regterm)
  File "/home/nishanth/miniconda2/lib/python2.7/site-packages/yadlt-0.0.5-py2.7.egg/yadlt/core/trainers.py", line 135, in compile
    tf.summary.scalar(self.lfunc, cost)
AttributeError: 'module' object has no attribute 'summary'

Further information
I installed tensorflow 0.8.0 through conda.
I'm not sure if my installation of tensorflow is incorrect. The validate your installation section in their installation documentation works.
I cloned the yadlt github repo.
I'm executing the following command

python run_dbn.py  --dataset custom --verbose 1 --rbm_layers 512,256 --rbm_learning_rate 0.005 --rbm_num_epochs 15 --rbm_batch_size 25 --finetune_batch_size 25 --finetune_learning_rate 0.001 --finetune_num_epochs 10 --finetune_loss_func softmax_cross_entropy --finetune_dropout 0.7 --finetune_act_func relu

Not sure what's going wrong. Maybe you could help me?
Thanks for your time.

Is pip up to date?

I'm not seeing yadlt/models/rnn_models in pip. How often does it get updated?

Thanks.

get_weights_as_images

I think the first argument to the function "yadlt/utils/utilities/get_weights_as_images" should not be "self"? Correct me if I'm wrong. Also, could you provide an example of how to use this function. It seems pretty straight forward except for the "weights" argument, not sure what to pass here. Can we simply use one of the .npy output files from your examples (e.g. one of the output layers from command_line/run_stacked_autoencoder_unsupervised.py)?

The implement of CDK algorithm

Hi! I checked the CDK algorithm carefully, which is as below:

My question is: when using gaussin visible units, why the function 'compute_positive_association' uses 'hidden_probs'?And why it uses 'hidden_states' on binary visible units?
Thank you!

TypeError: Expected binary or unicode string, got 38

When I run the run_dbn.py code, I get a TypeError: Expected binary or unicode string, got 38.
My data set consists of 38 float features and the class number is 5.
I do not know how to fix this and any help would be greatly appreciated.

MacBook-Pro:Deep-Learning-TensorFlow-master$ python run_dbn.py --main_dir dbn-models --model_name my-deeper-dbn --verbose 1 --rbm_layers 512,38 --rbm_learning_rate 0.005 --rbm_num_epochs 15 --rbm_batch_size 25 --finetune_batch_size 25 --finetune_learning_rate 0.001 --finetune_num_epochs 10 --finetune_loss_func softmax_cross_entropy --finetune_dropout 0.7 --finetune_act_func relu
4999
trX.shape: (4999, 38)
trY.shape: (4999, 5)
vlX.shape: (4999, 38)
vlY.shape: (1999, 5)
Training layer 1...
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Tensorboard logs dir for this run is .yadlt/logs/run98
Reconstruction loss: 0.807519: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00, 2.86it/s]
Training layer 2...
Tensorboard logs dir for this run is .yadlt/logs/run99
Reconstruction loss: 0.128697: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00, 2.40it/s]
Start deep belief net finetuning...
#features: 38
#classes: 5
prev_layer: Tensor("encode-1/dropout/mul:0", shape=(?, 38), dtype=float32)
in_dim: 38
out_dim: 5
Traceback (most recent call last):
File "run_dbn.py", line 166, in
srbm.fit(trX, trY, vlX, vlY)
File "Python/Deep-Learning-TensorFlow-master/yadlt/core/supervised_model.py", line 54, in fit
self.build_model(train_X.shape[1], num_classes)
File "Python/Deep-Learning-TensorFlow-master/yadlt/models/boltzmann/dbn.py", line 175, in build_model
self.mod_y, _, _ = Layers.linear(next_train, n_classes)
File "Python/Deep-Learning-TensorFlow-master/yadlt/core/layers.py", line 37, in linear
W = tf.Variable(tf.truncated_normal(shape=[in_dim, out_dim], stddev=0.1))
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 167, in truncated_normal
shape_tensor = _ShapeTensor(shape)
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 41, in _ShapeTensor
return ops.convert_to_tensor(shape, dtype=dtype, name="shape")
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 651, in convert_to_tensor
as_ref=False)
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 716, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "anaconda3/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 65, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got 38

Issues with the documentation

I found a few issues with the project's documentations

The neurons per layer numbers in the "stacked denoising autoencoder" seem to be wrong . Currently they are 784 <-> 1024, 1024 <-> 784, 784 <-> 512, 512 <-> 256 but I think they should be changed to 784 <-> 1024, 784 <-> 512, 512 <-> 256. It probably also makes sense to include the decoder layers.
In the "stacked deep autoencoder" it says This command trains a Stack of Denoising Autoencoders. Denoising should probably be removed. The neurons per layer numbers described in the section contain both the encoder and decoder layers, but in the stacked denoising section only the encoder layers are described.

I am happy to contribute these changes if you would like.

Issues with encoder parameters

I am running into issues with the encoder parameters.

running

encoder = StackedDeepAutoencoder(dae_layers=[60,30])

raises an IndexError

IndexError                                Traceback (most recent call last)
<ipython-input-90-77a8bc4b22b1> in <module>()
----> 1 encoder = StackedDeepAutoencoder(dae_layers=[60,30])
      2 output = encoder.pretrain(train_img)

/python2.7/site-packages/yadlt/models/autoencoder_models/stacked_deep_autoencoder.pyc in __init__(self, dae_layers, model_name, main_dir, models_dir, data_dir, summary_dir, dae_enc_act_func, dae_dec_act_func, dae_loss_func, dae_num_epochs, dae_batch_size, dataset, dae_opt, dae_learning_rate, momentum, finetune_dropout, dae_corr_type, dae_corr_frac, verbose, finetune_loss_func, finetune_enc_act_func, tied_weights, finetune_dec_act_func, dae_l2reg, finetune_batch_size, do_pretrain, finetune_opt, finetune_learning_rate, finetune_num_epochs)
     72                 n_components=layer, main_dir=self.main_dir, model_name=self.model_name + '-' + dae_str,
     73                 models_dir=os.path.join(self.models_dir, dae_str), data_dir=os.path.join(self.data_dir, dae_str),  summary_dir=os.path.join(self.tf_summary_dir, dae_str),
---> 74                 enc_act_func=dae_enc_act_func[l], dec_act_func=dae_dec_act_func[l], loss_func=dae_loss_func[l],
     75                 opt=dae_opt[l], learning_rate=dae_learning_rate[l], l2reg=dae_l2reg,
     76                 momentum=self.momentum, corr_type=dae_corr_type, corr_frac=dae_corr_frac,

IndexError: list index out of range

There are a couple of issues:

Why is index 1 being used from dae_enc_act_func? Shouldn't it use index 0? This will fix the problem.
Why are the __init__ parameters initialized as lists in the first place? Wouldn't it make more sense to have them as strings?
If we are going to have them as a list then they should be of the form [tanh] not list(['tanh']).

Issue pretraining without validation set

When trying to pretrain a DeepAutoencoder without a validation_set I get the following error:

Traceback (most recent call last): File "multimodal-mnist/mnist_autoencoder.py", line 34, in <module> output = encoder.pretrain(train_x) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/yadlt/models/rbm_models/deep_autoencoder.py", line 109, in pretrain self.rbm_graphs[l]) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/yadlt/models/rbm_models/deep_autoencoder.py", line 131, in _pretrain_rbm_and_gen_feed next_valid = rbm.transform(validation_set, graph=graph) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/yadlt/models/rbm_models/rbm.py", line 252, in transform encoded_data = self.sample_hidden_from_visible(data)[0].eval() File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/yadlt/models/rbm_models/rbm.py", line 190, in sample_hidden_from_visible hprobs = tf.nn.sigmoid(tf.matmul(visible, self.W) + self.bh_) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 1023, in matmul a = ops.convert_to_tensor(a, name="a") File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 566, in convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/tensorflow/python/ops/constant_op.py", line 179, in _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/tensorflow/python/ops/constant_op.py", line 162, in constant tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape)) File "/Users/Jackson/development/ai_experiments/venv/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 325, in make_tensor_proto raise ValueError("None values not supported.") ValueError: None values not supported.

Problem about using trained model to predict

I trained my own network model with custom data and supervised sDAE. When I finished training, I got good results on the test set. So I want to use a well-trained model in the later process to predict large amounts of data. But according to the documentation, I use the command line python run_stacked_autoencoder_supervised.py --restore_previous_model sdae --do_pretrain True --dae_num_epochs 0 --finetune_num_epochs 0 , only got a 24% accuracy.

So I checked the code carefully and found in /yadlt/core/supervised_model.py that you're going to build the model, initialize tensorflow stuff, train the model, and save the model in the fit () method.

Your code works well in pretraining - finetuning - prediction process. But if I want to directly restore the already trained model to predict, I will encounter the above situation.

So I made a change in the score () method, joined the building model, initialize tensorflow stuff, the specific code is as follows:

def score(self, test_X, test_Y):
        if len(test_Y.shape) != 1:
            num_classes = test_Y.shape[1]
        else:
            raise Exception("Please convert the labels with one-hot encoding.")
        with self.tf_graph.as_default():
            # Build model
            self.build_model(test_X.shape[1], num_classes)
            with tf.Session() as self.tf_session:
                # Initialize tf stuff
                summary_objs = tf_utils.init_tf_ops(self.tf_session)
                self.tf_merged_summaries = summary_objs[0]
                self.tf_summary_writer = summary_objs[1]
                self.tf_saver = summary_objs[2]
                # Restore the model
                self.tf_saver.restore(self.tf_session, self.model_path)
                feed = {
                    self.input_data: test_X,
                    self.input_labels: test_Y,
                    self.keep_prob: 1
                }
                return self.accuracy.eval(feed)

Then I can directly restore the model to predict. Hope this helps others.

ImportError: No module named 'tqdm'

Hi, i followed the instruction to clone project:

i run this in Windows 10 with Gitbash software.

pip install
i'm quite sure it is already installed because if i install it again i found this message.
Requirement already satisfied: yadlt in c:\users\lab2\appdata\local\programs\python\python35\lib\site-packages
i didn't change my directory and clone the project by insterting:
git clone https://github.com/blackecho/Deep-Learning-TensorFlow.git
when i run cmd_line/boltzmann/run_dbn.py i got this following error.

Traceback (most recent call last):
File "<pyshell#4>", line 1, in
from yadlt.models.boltzmann import dbn
File "C:\Users\LAB2\AppData\Local\Programs\Python\Python35\lib\site-packages\yadlt\models\boltzmann\dbn.py", line 8, in
from tqdm import tqdm
ImportError: No module named 'tqdm'

Thank you for any help

Where is model_predictions ?

When I use the method predict,I get a AttributeError
AttributeError: 'DeepBeliefNetwork' object has no attribute 'model_predictions'
any idea?

Check the requirements to run on Python 3.5

I was trying to run Deep Belief networks on python 3.5 and TensorFlow on windows, but when a I run the example code of the DBN (run_dbn.py), I found an issue with the module cPickle, are you aware of that error? Are there more issues than prevents the current version to work on python 3.5?

thanks in advance

Using custom data fails (errors with placeholders and feed data sizes)

I'm using the run_stacked_autoencoder_unsupervised..py script with custom data in npy format, and am receiving a failure after the data is loaded, at the point where it's being fed into the training step:

tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 1 2 I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 1: Y Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 2: Y Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 970, pci bus id: 0000:03:00.0) I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 970, pci bus id: 0000:02:00.0) I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:2) -> (device: 2, name: GeForce GTX 970, pci bus id: 0000:01:00.0) Tensorboard logs dir for this run is /home/oem/.yadlt/logs/un_sdae/dae-1/run2 Traceback (most recent call last): File "command_line/run_stacked_autoencoder_unsupervised.py", line 185, in <module> encoded_X, encoded_vX = sdae.pretrain(trX, vlX) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/models/autoencoder_models/deep_autoencoder.py", line 142, in pretrain validation_set=validation_set) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/core/model.py", line 180, in pretrain_procedure layer_graphs[l]) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/core/model.py", line 197, in _pretrain_layer_and_gen_feed validation_set, validation_set, graph=graph) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/core/unsupervised_model.py", line 48, in fit train_set, train_ref, validation_set, validation_ref) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/models/autoencoder_models/denoising_autoencoder.py", line 74, in _train_model self._run_train_step(train_set) File "/home/oem/Documents/phdCaseStudy/tensorflow27/local/lib/python2.7/site-packages/yadlt/models/autoencoder_models/denoising_autoencoder.py", line 101, in _run_train_step self.tf_session.run(self.train_step, feed_dict=tr_feed) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 894, in _run % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape()))) ValueError: Cannot feed value of shape (25, 128, 128, 3) for Tensor u'x-input:0', which has shape '(?, 128)'

My image tiles (saved as an npy batch) are 128x128 with 3 channels (rgb). The error seams to indicate that the script understands the shape of the data, but that this u'x-input:0 has a shape of (?, 128). I've tried to find where to fix this, without much luck. I've tried using np.reshape but I've only been able to change the shape of the (25,128,128,3) array and not the shape of the u'x-input:0...

Any help on this would be greatly appreciated!

duplicate finetune_learning_rate

In the examples for both the stacked denoising autoencoder and stacked deep autoencoder, the option --finetune_learning_rate is listed twice. Just fyi.

Problem with DeepBeliefNetwork

Hi,

I am having problem in prediction after training a DBN. I get the following error?

model.predict(X_test)
INFO:tensorflow:Restoring parameters from C:\itemp.yadlt\models/dbn
Traceback (most recent call last):

File "", line 1, in
model.predict(X_test)

File "C:\Anaconda3\lib\site-packages\yadlt\core\supervised_model.py", line 95, in predict
return self.model_predictions.eval(feed)

AttributeError: 'DeepBeliefNetwork' object has no attribute 'model_prediction

Any idea?

AttributeError: 'Graph' object has no attribute 'SerializeToString' -- Version problems

Hi, backecho. I got these tracebacks when running

python command_line/run_stacked_autoencoder_unsupervised.py --layers 512,256,128 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.001 --finetune_num_epochs 25 --opt gradient_descent --learning_rate 0.05 --enc_act_func sigmoid --dec_act_func sigmoid --loss_func cross_entropy --finetune_act_func tanh --dropout 0.7

I googled it and found this is a problem concerning the version of tensorflow. I am using r0.7.1 but the attribute SerializeToString seems to be in r0.8.
Is there any other way to solve this problem apart from upgrading tensorflow?

Traceback (most recent call last):
  File "command_line/run_stacked_autoencoder_unsupervised.py", line 178, in <module>
    encoded_X, encoded_vX = sdae.pretrain(trX, vlX)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 132, in pretrain
    next_train, next_valid = self._pretrain_autoencoder_and_gen_feed(autoenc, next_train, next_valid)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 149, in _pretrain_autoencoder_and_gen_feed
    autoenc.fit(train_set, validation_set)
  File "./models/autoencoder_models/denoising_autoencoder.py", line 90, in fit
    self._initialize_tf_utilities_and_ops(restore_previous_model)
  File "./models/autoencoder_models/denoising_autoencoder.py", line 109, in _initialize_tf_utilities_and_ops
    self.tf_summary_writer = tf.train.SummaryWriter(self.tf_summary_dir, self.tf_session.graph)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/summary_io.py", line 104, in __init__
    self.add_graph(graph_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/summary_io.py", line 168, in add_graph
    graph_bytes = graph_def.SerializeToString()
AttributeError: 'Graph' object has no attribute 'SerializeToString'

training on raw images

I noticed your examples read in binary data from mnist, cifar, etc. but I'm not finding any examples that just train on a bunch of raw image files. Does yadlt have this capability?

Thanks

DBN finetuning issue

Hi, I started exploring the package yesterday. Started with the command line DBN example (same network and hyperparameters) as given in the documentation.

python command_line/run_dbn.py --dataset mnist --main_dir dbn-models --model_name my-deeper-dbn  --verbose 1 --rbm_layers 512,256 --rbm_learning_rate 0.005 --rbm_num_epochs 15 --rbm_batch_size 25 --finetune_batch_size 25 --finetune_learning_rate 0.001 --finetune_num_epochs 10 --finetune_loss_func softmax_cross_entropy --finetune_dropout 0.7 --finetune_act_func relu

Pretraining worked well as in the reconstruction error reduced over the epochs.

Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
Creating stored_models/dbn-models directory to save/restore models
Creating data/dbn-models directory to save model generated data
Creating logs/dbn-models directory to save tensorboard logs
Creating stored_models/dbn-models/rbm-1 directory to save/restore models
Creating data/dbn-models/rbm-1 directory to save model generated data
Creating logs/dbn-models/rbm-1 directory to save tensorboard logs
Creating stored_models/dbn-models/rbm-2 directory to save/restore models
Creating data/dbn-models/rbm-2 directory to save model generated data
Creating logs/dbn-models/rbm-2 directory to save tensorboard logs
Training layer 1...
Tensorboard logs dir for this run is logs/dbn-models/rbm-1/run6
Reconstruction loss at step 0: 0.121678
Reconstruction loss at step 1: 0.103993
Reconstruction loss at step 2: 0.094516
Reconstruction loss at step 3: 0.0880938
Reconstruction loss at step 4: 0.0832195
Reconstruction loss at step 5: 0.0794863
Reconstruction loss at step 6: 0.0764901
Reconstruction loss at step 7: 0.0738373
Reconstruction loss at step 8: 0.0716205
Reconstruction loss at step 9: 0.0696231
Reconstruction loss at step 10: 0.0680134
Reconstruction loss at step 11: 0.0665949
Reconstruction loss at step 12: 0.0651861
Reconstruction loss at step 13: 0.0640741
Reconstruction loss at step 14: 0.0630116
Training layer 2...
Tensorboard logs dir for this run is logs/dbn-models/rbm-2/run5
Reconstruction loss at step 0: 0.172686
Reconstruction loss at step 1: 0.144182
Reconstruction loss at step 2: 0.129113
Reconstruction loss at step 3: 0.119446
Reconstruction loss at step 4: 0.112444
Reconstruction loss at step 5: 0.107379
Reconstruction loss at step 6: 0.103563
Reconstruction loss at step 7: 0.100636
Reconstruction loss at step 8: 0.0982092
Reconstruction loss at step 9: 0.0960361
Reconstruction loss at step 10: 0.0942256
Reconstruction loss at step 11: 0.0926428
Reconstruction loss at step 12: 0.0913576
Reconstruction loss at step 13: 0.0902729
Reconstruction loss at step 14: 0.0891995

But when it came to fine-tuning, the accuracy remained the same across epochs (i.e. network was not trained).

Start deep belief net finetuning...
Tensorboard logs dir for this run is logs/dbn-models/run5
Accuracy at step 0: 0.0958
Accuracy at step 1: 0.0958
Accuracy at step 2: 0.0958
Accuracy at step 3: 0.0958
Accuracy at step 4: 0.0958
Accuracy at step 5: 0.0958
Accuracy at step 6: 0.0958
Accuracy at step 7: 0.0958
....
....
....

I also tried with other parameter configurations and datasets, but had the same problem. Then I tried without pretraining and this time the test accuracy got better as expected.

python command_line/run_dbn.py --dataset mnist --main_dir dbn-models --model_name my-deeper-dbn --verbose 1 --rbm_layers 512,256 --rbm_learning_rate 0.005 --rbm_num_epochs 15 --rbm_batch_size 25 --finetune_batch_size 25 --finetune_learning_rate 0.001 --finetune_num_epochs 10 --finetune_loss_func softmax_cross_entropy --finetune_dropout 0.7 --finetune_act_func relu --do_pretrain False

Accuracy at step 0: 0.5808
Accuracy at step 1: 0.7334
Accuracy at step 2: 0.7884
Accuracy at step 3: 0.8222
Accuracy at step 4: 0.8392
Accuracy at step 5: 0.8544
Accuracy at step 6: 0.8642
Accuracy at step 7: 0.8722
Accuracy at step 8: 0.8782
....
....
....

Did anybody else get the DBN with pretraining to work from command line? I know the parameters in the documentation are not the best ones, but I still expect the network to train. I will now look into the code to find the issue, but any pointers would be appreciated. thanks!

Always show Test accuracy 1.0 for run_stacked_autoencoder_supervised.py

I just try with the file after a slight modifying for my custom data run_stacked_autoencoder_supervised.py with doc configuation

After running the code the output shows always test accuracy 1.0 . I put some wrong output label in testlabel.txt file , but at this case it also shows accuracy 1.0.

The CommandLine Output LookLike

Kingfisher:Deep-Learning-TensorFlow Nandi$ python practice/run_stacked_autoencoder_supervised.py --dae_layers 1024,784,512,256 --dae_batch_size 25 --dae_num_epochs 5 --verbose 1 --dae_corr_type masking --dae_corr_frac 0.0 --finetune_num_epochs 25 --finetune_opt momentum --momentum 0.9 --finetune_learning_rate 0.05 --dae_enc_act_func sigmoid --dae_dec_act_func sigmoid --dae_loss_func cross_entropy --finetune_act_func relu --finetune_loss_func softmax_cross_entropy --dropout 0.7
Creating /Users/Nazib/.yadlt/stored_models/sdae/ directory to save/restore models
Creating /Users/Nazib/.yadlt/data/sdae/ directory to save model generated data
Creating /Users/Nazib/.yadlt/logs/sdae/ directory to save tensorboard logs
Creating /Users/Nazib/.yadlt/stored_models/sdae/dae-1 directory to save/restore models
Creating /Users/Nazib/.yadlt/data/sdae/dae-1 directory to save model generated data
Creating /Users/Nazib/.yadlt/logs/sdae/dae-1 directory to save tensorboard logs
Creating /Users/Nazib/.yadlt/stored_models/sdae/dae-2 directory to save/restore models
Creating /Users/Nazib/.yadlt/data/sdae/dae-2 directory to save model generated data
Creating /Users/Nazib/.yadlt/logs/sdae/dae-2 directory to save tensorboard logs
Creating /Users/Nazib/.yadlt/stored_models/sdae/dae-3 directory to save/restore models
Creating /Users/Nazib/.yadlt/data/sdae/dae-3 directory to save model generated data
Creating /Users/Nazib/.yadlt/logs/sdae/dae-3 directory to save tensorboard logs
Creating /Users/Nazib/.yadlt/stored_models/sdae/dae-4 directory to save/restore models
Creating /Users/Nazib/.yadlt/data/sdae/dae-4 directory to save model generated data
Creating /Users/Nazib/.yadlt/logs/sdae/dae-4 directory to save tensorboard logs
Training layer 1...
Tensorboard logs dir for this run is /Users/Nazib/.yadlt/logs/sdae/dae-1/run5
Reconstruction loss at step 0: -8036.11
Reconstruction loss at step 1: -9851.44
Reconstruction loss at step 2: -11081.3
Reconstruction loss at step 3: -11957.8
Reconstruction loss at step 4: -12676.6
Training layer 2...
Tensorboard logs dir for this run is /Users/Nazib/.yadlt/logs/sdae/dae-2/run5
Reconstruction loss at step 0: 0.659456
Reconstruction loss at step 1: 0.658065
Reconstruction loss at step 2: 0.656675
Reconstruction loss at step 3: 0.655287
Reconstruction loss at step 4: 0.653903
Training layer 3...
Tensorboard logs dir for this run is /Users/Nazib/.yadlt/logs/sdae/dae-3/run5
Reconstruction loss at step 0: 0.788582
Reconstruction loss at step 1: 0.787858
Reconstruction loss at step 2: 0.787136
Reconstruction loss at step 3: 0.786414
Reconstruction loss at step 4: 0.785694
Training layer 4...
Tensorboard logs dir for this run is /Users/Nazib/.yadlt/logs/sdae/dae-4/run5
Reconstruction loss at step 0: 0.743701
Reconstruction loss at step 1: 0.743354
Reconstruction loss at step 2: 0.743006
Reconstruction loss at step 3: 0.742658
Reconstruction loss at step 4: 0.742312
Tensorboard logs dir for this run is /Users/Nazib/.yadlt/logs/sdae/run5
Accuracy at step 0: 1.0
Accuracy at step 1: 1.0
Accuracy at step 2: 1.0
Accuracy at step 3: 1.0
Accuracy at step 4: 1.0
Accuracy at step 5: 1.0
Accuracy at step 6: 1.0
Accuracy at step 7: 1.0
Accuracy at step 8: 1.0
Accuracy at step 9: 1.0
Accuracy at step 10: 1.0
Accuracy at step 11: 1.0
Accuracy at step 12: 1.0
Accuracy at step 13: 1.0
Accuracy at step 14: 1.0
Accuracy at step 15: 1.0
Accuracy at step 16: 1.0
Accuracy at step 17: 1.0
Accuracy at step 18: 1.0
Accuracy at step 19: 1.0
Accuracy at step 20: 1.0
Accuracy at step 21: 1.0
Accuracy at step 22: 1.0
Accuracy at step 23: 1.0
Accuracy at step 24: 1.0
Test set accuracy: 1.0

I have create a folder Named Data1 and keep my data
testdata.txt
testlabel.txt
traindata.txt
trainlabel.txt

The modified code I use here
run_stacked_autoencoder_supervised.txt

Produces an out-of-range result for MNIST

Produces an out-of-range result (reconstruction loss) for MNIST.

Here is the console output:

[vagrant@machinelearning DeepAutoencoder-TensorFlow]$ make run
python ../Reference/Deep-Learning-TensorFlow/cmd_line/boltzmann/run_deep_autoencoder.py
--dataset mnist
--cifar_dir ../Reference/Datasets/MNIST/mnist.pkl
--main_dir deep-autoencoder
--model_name deeper-is-better
--rbm_layers 1000,500,250,100,30
--rbm_batch_size 128
--finetune_batch_size 128
--finetune_loss_func cross_entropy
--verbose 1
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
Training layer 1...
2017-11-09 02:43:59.295164: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-09 02:43:59.317348: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-09 02:43:59.317387: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run6
Reconstruction loss: nan: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [08:26<00:00, 50.62s/it]
Training layer 2...
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run7
Reconstruction loss: nan: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [06:20<00:00, 38.10s/it]
Training layer 3...
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run8
Reconstruction loss: nan: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [01:25<00:00, 8.56s/it]
Training layer 4...
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run9
Reconstruction loss: nan: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:25<00:00, 2.55s/it]
Training layer 5...
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run10
Reconstruction loss: nan: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:08<00:00, 1.14it/s]
Tensorboard logs dir for this run is /home/vagrant/.yadlt/logs/run11
Reconstruction loss: nan: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [17:52<00:00, 107.29s/it]
Test set reconstruction loss: nan
[vagrant@machinelearning DeepAutoencoder-TensorFlow]$

accuray always drops when testing Stacked Denoising Autoencoder

tried several sets of hyperparameters:
first tried those parameters in your documentation:

python command_line/run_stacked_autoencoder_supervised.py --layers 1024,784,512,256 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.002 --finetune_num_epochs 25 --opt momentum --momentum 0.9 --learning_rate 0.05 --enc_act_func sigmoid --finetune_act_func relu --dropout 0.7

Accuracy at step 0: 0.1502
Accuracy at step 1: 0.1126
Accuracy at step 2: 0.1126
Accuracy at step 3: 0.1126
Accuracy at step 4: 0.1126
Accuracy at step 5: 0.1126
Accuracy at step 6: 0.1126
Accuracy at step 7: 0.1126
Accuracy at step 8: 0.1126
Accuracy at step 9: 0.1126
Accuracy at step 10: 0.1126
Accuracy at step 11: 0.1126
Accuracy at step 12: 0.1126
Accuracy at step 13: 0.1126

changed the finetuning learning rate:

python command_line/run_stacked_autoencoder_supervised.py --layers 1024,784,512,256 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.0002 --finetune_num_epochs 25 --opt momentum --momentum 0.9 --learning_rate 0.05 --enc_act_func sigmoid --finetune_act_func relu --dropout 0.7

Accuracy at step 0: 0.1002
Accuracy at step 1: 0.1002
Accuracy at step 2: 0.0868
Accuracy at step 3: 0.0868
Accuracy at step 4: 0.0868
Accuracy at step 5: 0.0868
Accuracy at step 6: 0.0868
Accuracy at step 7: 0.0868
Accuracy at step 8: 0.0868
Accuracy at step 9: 0.0868
Accuracy at step 10: 0.0868
Accuracy at step 11: 0.0868
Accuracy at step 12: 0.0868
Accuracy at step 13: 0.0868
Accuracy at step 14: 0.0868
Accuracy at step 15: 0.0868

changed dropout rates:

python command_line/run_stacked_autoencoder_supervised.py --layers 1024,784,512,256 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.0002 --finetune_num_epochs 25 --opt momentum --momentum 0.9 --learning_rate 0.05 --enc_act_func sigmoid --finetune_act_func relu --dropout 0.9

Accuracy at step 0: 0.094
Accuracy at step 1: 0.0992
Accuracy at step 2: 0.11
Accuracy at step 3: 0.11
Accuracy at step 4: 0.11
Accuracy at step 5: 0.11
Accuracy at step 6: 0.11
Accuracy at step 7: 0.11
Accuracy at step 8: 0.11
Accuracy at step 9: 0.11
Accuracy at step 10: 0.11
Accuracy at step 11: 0.11
Accuracy at step 12: 0.11
Accuracy at step 13: 0.11
Accuracy at step 14: 0.11

changed network structure:

python command_line/run_stacked_autoencoder_supervised.py --layers 512,128 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.002 --finetune_num_epochs 25 --opt momentum --momentum 0.9 --learning_rate 0.05 --enc_act_func sigmoid --finetune_act_func relu --dropout 1.0

Accuracy at step 0: 0.1804
Accuracy at step 1: 0.1986
Accuracy at step 2: 0.1786
Accuracy at step 3: 0.1614
Accuracy at step 4: 0.1468
Accuracy at step 5: 0.1346
Accuracy at step 6: 0.1192
Accuracy at step 7: 0.1066
Accuracy at step 8: 0.101
Accuracy at step 9: 0.1002
Accuracy at step 10: 0.1002

Has this something to do with network structure I chose?
By the way, cost in the pretraining process are decreasing which is reasonable.

how to train a regression model by using stacked_autoencoder_supervised?

hi Blackecho,
Deep-Learning-TensorFlow is really a wonderful tool for us to train nn models based on tensorflow. I wonder whether we can train a regression model by using stacked_autoencoder_supervised? Since existing supervised model requires
'train_Y : array_like, shape (n_samples, n_classes)'? May I know whether existing package support regression dae model? thanks.

Jianju

Add to PyPi

Thanks for the great repository! 👍

I'm wondering if you could add this repository to PyPi so that it can be installed using PIP. This would be particularly useful for people who are not using it from the command line and need to import its models into their existing projects.

Cheers

assertion error assert optimizer in ["sgd", "adagrad", "adam", "momentum"]

File "tensorFlow/Deep-Learning-TensorFlow/yadlt/core/trainers.py", line 27, in init
assert optimizer in ["sgd", "adagrad", "adam", "momentum"]
AssertionError

Is that due to a program bug? Anyone knows how to fix it?

Thanks.

AttributeError: 'NpzFile' object has no attribute 'shape'

Hello!

I've just started using yadlt, and am working with the github version, cloned 2 days ago. I'm loading my own dataset (.npz files) with the command_line/run_stacked_autoencoder_unsupervised.py model.

Here's the error:

Training layer 1...
Traceback (most recent call last):
File "command_line/run_stacked_autoencoder_unsupervised.py", line 185, in
encoded_X, encoded_vX = sdae.pretrain(trX, vlX)
File "/usr/local/lib/python2.7/dist-packages/yadlt/models/autoencoder_models/deep_autoencoder.py", line 142, in pretrain
validation_set=validation_set)
File "/usr/local/lib/python2.7/dist-packages/yadlt/core/model.py", line 180, in pretrain_procedure
layer_graphs[l])
File "/usr/local/lib/python2.7/dist-packages/yadlt/core/model.py", line 196, in _pretrain_layer_and_gen_feed
layer_obj.fit(train_set, validation_set, graph=graph)
File "/usr/local/lib/python2.7/dist-packages/yadlt/core/unsupervised_model.py", line 44, in fit
self.build_model(train_set.shape[1])

AttributeError: 'NpzFile' object has no attribute 'shape'

The initial dataset is a bunch of 256x256 rgb images, and I saved them to npz as a
(21280, 256, 256, 3)
sized array.

At the moment, I'm saving only 1 array in the npz file (not multiple arrays).

I'm having difficulty figuring out how train_set has not attribute shape...

Thanks for your help!

Some problems when running 'Deep Autoencoder' on both CIFAR10 and MNIST

After downloading and unzipping cifar10 dataset, I tried deep autoencoder with:

python command_line/run_deep_autoencoder.py --dataset cifar10 --cifar_dir /home/data/cifar-10-batches-py/ --main_dir deep-autoencoder --model_name deeper-is-better --layers 8192,2048,512,256

something goes wrong at the fifth step in 1st layer pretraining:

Training layer 1...

Validation cost at step 0: 137.183
Validation cost at step 1: 137.183
Validation cost at step 2: 137.183
Validation cost at step 3: 137.183
Validation cost at step 4: 137.183
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Traceback (most recent call last):
  File "command_line/run_deep_autoencoder.py", line 121, in <module>
    srbm.pretrain(trX, vlX)
  File "./models/rbm_models/deep_autoencoder.py", line 111, in pretrain
    next_train, next_valid = self._pretrain_rbm_and_gen_feed(rboltz, next_train, next_valid)
  File "./models/rbm_models/deep_autoencoder.py", line 135, in _pretrain_rbm_and_gen_feed
    return rboltz.transform(train_set), rboltz.transform(validation_set)
  File "./models/rbm_models/rbm.py", line 260, in transform
    encoded_data = self.sample_hidden_from_visible(data)[0].eval()
  File "./models/rbm_models/rbm.py", line 201, in sample_hidden_from_visible
    hprobs = tf.nn.sigmoid(tf.matmul(visible, self.W) + self.bh_)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1036, in matmul
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 911, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 486, in apply_op
    _Attr(op_def, input_arg.type_attr))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 59, in _SatisfiesTypeConstraint
    ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: DataType uint8 for attr 'T' not in list of allowed values: float32, float64, int32, complex64

Then I tried MNIST with:

python command_line/run_deep_autoencoder.py --dataset mnist --main_dir deep-autoencoder --model_name deeper-is-better --layers 512,256 --verbose 1 --rbm_num_epochs 5 --rbm_batch_size 100

The pretraining process went fine:

Training layer 1...
Validation cost at step 0: 0.140783
Validation cost at step 1: 0.122195
Validation cost at step 2: 0.111817
Validation cost at step 3: 0.104732
Validation cost at step 4: 0.0994828

Training layer 2...
Validation cost at step 0: 0.162714
Validation cost at step 1: 0.13298
Validation cost at step 2: 0.117772
Validation cost at step 3: 0.108244
Validation cost at step 4: 0.101742

But at finetuning stage:

Start deep autoencoder finetuning...
Traceback (most recent call last):
  File "command_line/run_deep_autoencoder.py", line 127, in <module>
    srbm.build_model(trX.shape[1])
  File "./models/rbm_models/deep_autoencoder.py", line 203, in build_model
    encode_output = self._create_encoding_layers()
  File "./models/rbm_models/deep_autoencoder.py", line 245, in _create_encoding_layers
    hstates = utilities.sample_prob(hprobs, self.hrand[l])
  File "./utils/utilities.py", line 20, in sample_prob
    return tf.nn.relu(tf.sign(probs - rand))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 518, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 1683, in sub
    return _op_def_lib.apply_op("Sub", x=x, y=y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2156, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1612, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1390, in _BroadcastShape
    % (shape_x, shape_y))
ValueError: Incompatible shapes for broadcasting: (?, 512) and (?, 256)

Could you figure out what's wrong?

hidden layer representation

Hello, after training the stacked denoising autoencoders, I want to extract the hidden layer representation of the given data, can someone help me please how can I do it?

Problem with run_conv_net.py

Traceback (most recent call last):
File "/home/saber/Desktop/Deep-Learning-TensorFlow-master/cmd_line/convolutional/run_conv_net.py", line 90, in
convnet.fit(trX, trY, vlX, vlY)
File "/usr/local/lib/python2.7/dist-packages/yadlt/core/supervised_model.py", line 62, in fit
self.build_model(train_X.shape[1], num_classes)
File "/usr/local/lib/python2.7/dist-packages/yadlt/models/convolutional/conv_net.py", line 109, in build_model
self.cost = self.loss.compile(self.mod_y, self.input_labels)
AttributeError: 'ConvolutionalNetwork' object has no attribute 'mod_y'
I got this problem,have no idea, Any help will be appreciated! Thank you!
I directly run run_conv_net.py. It leaded the error?

Bug: ImportError: No module named model

When running:

python /home/maddoxw/git/Deep-Learning-TensorFlow/command_line/run_deep_autoencoder.py --dataset cifar10 --cifar_dir /media/maddoxw/EXTRA/CIFAR-10/cifar-10-batches-py/ --main_dir deep-autoencoder --model_name deeper-is-better --rbm_layers 8192,2048,512,256 --rbm_batch_size 128 --finetune_batch_size 128 --finetune_loss_func cross_entropy

I get the following error:

Traceback (most recent call last):
  File "/home/maddoxw/git/Deep-Learning-TensorFlow/command_line/run_deep_autoencoder.py", line 7, in <module>
    from yadlt.models.rbm_models import deep_autoencoder
  File "/home/maddoxw/.local/lib/python2.7/site-packages/yadlt/models/rbm_models/deep_autoencoder.py", line 11, in <module>
    from yadlt.core.unsupervised_model import UnsupervisedModel
  File "/home/maddoxw/.local/lib/python2.7/site-packages/yadlt/core/unsupervised_model.py", line 9, in <module>
    import model
ImportError: No module named model

unsupervised_model.py should see model.py since it's in the same folder. Not sure why it doesn't.

yadlt.utilities.seq_data_iterator() indent problem

Hello,
I noticed a little error problem the indent in the function: seq_data_iterator(),
the last line 'yield (x,y)' needs one more tab to make it inside the 'for' loop.

Thank you for your code which really helps me a lot.

Proble with different models!!

Hi,
I am having problem in implementation of different models even in Example Usage!

for example==>python run_dbn.py --dataset mnist --main_dir dbn-models --model_name my-deeper-dbn --verbose 1 --rbm_layers 512,256 --rbm_learning_rate 0.005 --rbm_num_epochs 15 --rbm_batch_size 25 --finetune_batch_size 25 --finetune_learning_rate 0.001 --finetune_num_epochs 10 --finetune_loss_func softmax_cross_entropy --finetune_dropout 0.7 --finetune_act_func relu

output:
Traceback (most recent call last):
File "run_dbn.py", line 71, in
trX, trY, vlX, vlY, teX, teY = datasets.load_mnist_dataset(mode='supervised')
File "/home/h3kt0rz/.local/lib/python3.5/site-packages/yadlt/utils/datasets.py", line 27, in load_mnist_dataset
mnist = input_data.read_data_sets("MNIST_data/", one_hot=one_hot)
File "/home/homayoon/h3kt0rz/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 235, in read_data_sets
SOURCE_URL + TRAIN_IMAGES)
File "/home/h3kt0rz/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 208, in maybe_download
temp_file_name, _ = urlretrieve_with_retry(source_url)
File "/home/h3kt0rz/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 165, in wrapped_fn
return fn(*args, **kwargs)
File "/home/h3kt0rz/tensorflow/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 190, in urlretrieve_with_retry
return urllib.request.urlretrieve(url, filename)
File "/usr/lib/python3.5/urllib/request.py", line 188, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Please help me

'DeepBeliefNetwork' object has no attribute 'model_predictions'

Hi, I saw that this problem had been resolved and the issue closed in April -- but I just pip installed yadlt today and continue to get this error.

Thanks, blackecho, for your great work!

Encountered an issue when running the latest 'stacked deep autoencoder' code.

I'm on commit 365fe44. After running the command in the document, the stacked deep autoencoder stopped unexpectedly.
commands:

python command_line/run_stacked_autoencoder_unsupervised.py --dae_layers 512,256,128 --dae_batch_size 25 --dae_num_epochs 5 --verbose 1 --dae_corr_type masking --dae_corr_frac 0.0 --finetune_learning_rate 0.0001 --finetune_num_epochs 25 --finetune_opt gradient_descent --finetune_learning_rate 0.05 --dae_enc_act_func sigmoid --dae_dec_act_func sigmoid --dae_loss_func cross_entropy --finetune_enc_act_func tanh --finetune_dec_act_func sigmoid --finetune_loss_func cross_entropy --dropout 0.7

tracebacks:

I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Traceback (most recent call last):
  File "command_line/run_stacked_autoencoder_unsupervised.py", line 178, in <module>
    encoded_X, encoded_vX = sdae.pretrain(trX, vlX)
  File "./yadlt/models/autoencoder_models/stacked_deep_autoencoder.py", line 106, in pretrain
    next_train, next_valid = self._pretrain_autoencoder_and_gen_feed(autoenc, next_train, next_valid)
  File "./yadlt/models/autoencoder_models/stacked_deep_autoencoder.py", line 123, in _pretrain_autoencoder_and_gen_feed
    autoenc.fit(train_set, validation_set)
  File "./yadlt/models/autoencoder_models/denoising_autoencoder.py", line 67, in fit
    self.tf_saver.save(self.tf_session, self.model_path)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 827, in __exit__
    context_manager.__exit__(exec_type, exec_value, exec_tb)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3215, in get_controller
    assert self.stack[-1] is default
IndexError: list index out of range

Using trained DAE

After setting up the package and running the following command
python run_autoencoder.py --n_components 1024 --batch_size 64 --num_epochs 20 --verbose 1 --learning_rate 0.05 --corr_type masking --corr_frac 0.5 --enc_act_func sigmoid --dec_act_func sigmoid --loss_func cross_entropy --opt momentum --momentum 0.9 --main_dir dae-models --model_name dae1024
The folder .yadlt was created with folders 'data', 'logs' and 'models'
The 'models' folder has 'checkpoint ' , 'dae.data-00000-of-00001' , 'dae.index' and 'dae.meta' files in it.
Did the code run properly?
If yes where is the trained model?
How to use the trained model to denoise images?

Thank You

Issue of using DBN

I am trying to use your module in the scripts instead of the command line. Here I want to use the DBN.

When I run

import tensorflow as tf
from yadlt.models.boltzmann import dbn

from tensorflow.examples.tutorials.mnist import input_data

# load the data
mnist = input_data.read_data_sets("tensorflow_learning/MNIST_data/", one_hot = True)
x_train = mnist.train.images
y_train = mnist.train.labels
x_val = mnist.validation.images
y_val = mnist.validation.labels

# create a DBN
my_DBN = dbn.DeepBeliefNetwork(rbm_layers = [50, 30, 20],
                               finetune_loss_func = 'softmax_cross_entropy',
                               finetune_dropout = 1,
                               finetune_learning_rate = 0.001,
                               finetune_act_func = tf.nn.sigmoid,
                               finetune_num_epochs = 20,
                               finetune_batch_size = 20,
                               do_pretrain = True)

my_DBN.pretrain(x_train, x_val)

my_DBN.build_model(784, 10)

my_DBN._train_model(x_train, y_train, x_val, y_val)

The last line my_DBN._train_model(x_train, y_train, x_val, y_val) gives the following error:

File "D:\Program Files\Anaconda3\lib\site-packages\yadlt\models\boltzmann\dbn.py", line 147, in _train_model

AttributeError: 'NoneType' object has no attribute 'run'

When I go back to the line 147 of dbn.py, that is
self.tf_session.run

By the way, I am not quite sure the scheme I used above is correct or not. Could you write a kind of script usage in the documentation?

DenoisingAutoencoder._train_model args mismatch with parent UnsupervisedModel.

I noticed that the args signature in the DenoisingAutoencoder._train_model differs from that of its' parent UnsupervisedModel._train_model. Specifically, the second and third arguments are swapped.

def _train_model(self, train_set, validation_set, train_ref=None, validation_ref=None):

vs.

def _train_model(self, train_set, train_labels, validation_set, validation_labels):

Is this the intended behavior?

Thanks

Some op name has changed in latest version of Tensorflow

The op 'tf.mul' has been renamed to 'tf.multiply' and 'tf.sub' to 'tf.subtract' in the 1.0.0 tensorflow, please check some other op

conv net fails in build model

Hi! I really like this repository it's super useful! I'm working on a CNN and wanted to use yours as starting position. Now there is a problem in the conv_net.py (line 111). Commit: a038e15

self.cost = self.loss.compile(self.mod_y, self.input_labels)

AttributeError: 'ConvolutionalNetwork' object has no attribute 'mod_y'

Tried to run it on the MNIST dataset with all the default settings. Hope you can help!

How does this model fine tune?

In the model of "run_stacked_autoencoder_unsupervised.py", we set some parameters about the fine tuning, I wonder since it is an unsupervised learning, how does it fine tune?

Error with stacked autoencoder_supervised

I've noticed the error using Tensorflow-gpu(1.0.0)
There were two problems:

-in the file Trainers,line 128:
tf.square(tf.sub(ref_y, mod_y))))
should be tf.square(tf.subtract(ref_y, mod_y))))

-in the file Trainers,line 48:
use_nesterov from the dict wasn't available,i fixed using the default value "False"
elif optimizer == "momentum":
self.opt_ = tf.train.MomentumOptimizer(
d("learning_rate"), d("momentum"),
use_nesterov=False)

That was with your last run_stacked_autoencoder_supervised.

questions about run_conv_net.py

Hello again,

I am trying to run run_conv_net.py. Here is my command:

time python command_line/run_conv_net.py 
--main_dir covnet-mnist-models 
--model_name covnet-mnist 
--layers conv2d-5-5-32-1,maxpool-2,conv2d-5-5-64-1,maxpool-2,full-1024,softmax 
--batch_size 50 
--verbose 1 
--learning_rate 1e-4 
--opt adam 
--dropout 0.5

And here is my output:

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.7.5 locally
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
Creating /home/maddoxw/.yadlt/stored_models/covnet-mnist-models directory to save/restore models
Creating /home/maddoxw/.yadlt/data/covnet-mnist-models directory to save model generated data
Creating /home/maddoxw/.yadlt/logs/covnet-mnist-models directory to save tensorboard logs
Start Convolutional Network training...
Building Convolutional layer with 1 input channels and 32 5x5 filters with stride 1
Building Max Pooling layer with size 2
Building Convolutional layer with 32 input channels and 64 5x5 filters with stride 1
Building Max Pooling layer with size 2
Building fully connected layer with 3136 in unitsand 1024 out units
Building softmax layer with 1024 in units and10 out units
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:03:00.0
Total memory: 12.00GiB
Free memory: 10.34GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:843] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Tensorboard logs dir for this run is /home/maddoxw/.yadlt/logs/covnet-mnist-models/run4
Accuracy at step 0: 0.0958
Accuracy at step 1: 0.0958
Accuracy at step 2: 0.0958
Accuracy at step 3: 0.0958
Accuracy at step 4: 0.0958
Accuracy at step 5: 0.0958
Accuracy at step 6: 0.0958
Accuracy at step 7: 0.0958
Accuracy at step 8: 0.0958
Accuracy at step 9: 0.0958
I tensorflow/core/common_runtime/gpu/gpu_device.cc:843] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Test set accuracy: 0.0979999899864

real    2m25.073s
user    2m53.696s
sys     0m27.416s

However the accuracy isn't changing. Should it be?

This paper uses the same layers architecture as the one found in your readthedocs example, so I don't think that is the problem.

Also, here are the saved results from the run:

maddoxw@wmdevbox:~$ ls -la /home/maddoxw/.yadlt/stored_models/covnet-mnist-models
total 13M
drwxrwxr-x 2 maddoxw maddoxw 4.0K Sep 14 11:14 ./
drwxrwxr-x 8 maddoxw maddoxw 4.0K Sep  6 14:39 ../
-rw-rw-r-- 1 maddoxw maddoxw  191 Sep 14 11:14 checkpoint
-rw-rw-r-- 1 maddoxw maddoxw  13M Sep 14 11:14 covnet-mnist
-rw-rw-r-- 1 maddoxw maddoxw  56K Sep 14 11:14 covnet-mnist.meta

Can I load covnet-mnist directly into tensorflow?

maddoxw@wmdevbox:~$ ls -la /home/maddoxw/.yadlt/data/covnet-mnist-models
total 8.0K
drwxrwxr-x 2 maddoxw maddoxw 4.0K Sep  1 18:24 ./
drwxrwxr-x 8 maddoxw maddoxw 4.0K Sep  6 14:39 ../

Should this directory be empty?

maddoxw@wmdevbox:~$ ls -la /home/maddoxw/.yadlt/logs/covnet-mnist-models/run4
total 88K
drwxr-xr-x 2 maddoxw maddoxw 4.0K Sep 14 10:56 ./
drwxrwxr-x 7 maddoxw maddoxw 4.0K Sep 14 11:12 ../
-rw-rw-r-- 1 maddoxw maddoxw  77K Sep 14 10:58 events.out.tfevents.1473868563.wmdevbox

Could you provide a few pointers on how to use Tensorboard to read the binary log files?

Thanks for sharing.

Error when running run_deep_autoencoder.py

I'm getting the following error when running run_deep_autoencoder.py.

Traceback (most recent call last):
  File "/home/maddoxw/git/Deep-Learning-TensorFlow/command_line/run_deep_autoencoder.py", line 149, in <module>
    noise=FLAGS.rbm_noise, stddev=FLAGS.rbm_stddev)
  File "/home/maddoxw/.local/lib/python2.7/site-packages/yadlt/models/rbm_models/deep_autoencoder.py", line 77, in __init__
    visible_unit_type=expanded_args['noise'][l],
TypeError: 'bool' object has no attribute '__getitem__'

One possible issue I found is in the run_deep_autoencoder.py file on line:

flags.DEFINE_boolean('rbm_noise', 'gauss', 'Type of noise. Default: "gauss".')

The types don't match. I'm not sure if it should be boolean or string?

Thanks

deep bolztman machine reconstruction error

How do we fix the error that stem from the following line:
https://github.com/blackecho/Deep-Learning-TensorFlow/blob/f859d6e6cc58162363d8076e56c005545927c0c7/cmd_line/boltzmann/run_deep_autoencoder.py#L155

Import Error on example with convolutional net

I'm running this:

python cmd_line/convolutional/run_conv_net.py --dataset custom --main_dir convnet-models --model_name my.Awesome.CONVNET --train_dataset path/to/train_set.npy --valid_dataset path/to/valid_set.npy --test_set path/to/test_set.npy --layers conv2d-5-5-32-1,maxpool-2,conv2d-5-5-64-1,maxpool-2,full-1024,softmax --batch_size 50 --verbose 1 --learning_rate 1e-4 --opt adam --dropout 0.5

and get:

Traceback (most recent call last):
File "cmd_line/convolutional/run_conv_net.py", line 4, in
from yadlt.models.convolutional import conv_net
ImportError: No module named 'yadlt'

I'm a Pynoob trying to import this to python 3 but get stuck at this error.

zip argument must support iteration

When running the command below on r0.8, I encounter a problem with zip function.

 python command_line/run_stacked_autoencoder_unsupervised.py --layers 512,256,128 --batch_size 25 --num_epochs 5 --verbose 1 --corr_type masking --corr_frac 0.0 --finetune_learning_rate 0.001 --finetune_num_epochs 25 --opt gradient_descent --learning_rate 0.05 --enc_act_func sigmoid --dec_act_func sigmoid --loss_func cross_entropy --finetune_act_func tanh --dropout 0.7

Traceback (most recent call last):
  File "command_line/run_stacked_autoencoder_unsupervised.py", line 178, in <module>
    encoded_X, encoded_vX = sdae.pretrain(trX, vlX)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 132, in pretrain
    next_train, next_valid = self._pretrain_autoencoder_and_gen_feed(autoenc, next_train, next_valid)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 149, in _pretrain_autoencoder_and_gen_feed
    autoenc.fit(train_set, validation_set)
  File "./models/autoencoder_models/denoising_autoencoder.py", line 91, in fit
    self._train_model(train_set, validation_set)
  File "./models/autoencoder_models/denoising_autoencoder.py", line 122, in _train_model
    self._run_train_step(train_set)
  File "./models/autoencoder_models/denoising_autoencoder.py", line 137, in _run_train_step
    shuff = zip(train_set, x_corrupted)
TypeError: zip argument #2 must support iteration

this happens when run run_stacked_autoencoder_unsupervised.py with these command line parameters:

 --corr_type masking

which makes the function masking_noise() in utilities be called.

# models/autoencoder_models/denoising_autoencoder.py
 def _corrupt_input(self, data):

        """ Corrupt a fraction of 'data' according to the
        noise method of this autoencoder.
        :return: corrupted data
        """

        corruption_ratio = np.round(self.corr_frac * data.shape[1]).astype(np.int)

        if self.corr_type == 'none':
            return np.copy(data)

        if self.corr_frac > 0.0:
            if self.corr_type == 'masking':
                return utilities.masking_noise(data, self.tf_session, self.corr_frac)

            elif self.corr_type == 'salt_and_pepper':
                return utilities.salt_and_pepper_noise(data, corruption_ratio)

        return None

# utils/utilities.py
def masking_noise(data, sess, v): 

    """ Apply masking noise to data in X, in other words a fraction v of elements of X
    (chosen at random) is forced to zero.
    :param data: array_like, Input data
    :param sess: TensorFlow session
    :param v: fraction of elements to distort, float
    :return: transformed data
    """

    data_noise = data.copy()
    rand = tf.random_uniform(data.shape)
    data_noise[sess.run(tf.nn.relu(tf.sign(v - rand))).astype(np.bool)] = 0 

    return data_noise

So I found 'where' the bug is, but cannot figure out 'what' is going wrong. what should be done to fix this problem?

Issue of nan Reconstruction loss when hidden layer is of dimension 1000

When I run the following command from command line, I get reconstruction loss of nan

python run_deep_autoencoder.py --rbm_layers 1000 --batch_size 10 --rbm_num_epochs 30 --verbose 1 --finetune_learning_rate 0.1 --finetune_num_epochs 50 --opt momentum --learning_rate 0.05 --enc_act_func sigmoid --dec_act_func sigmoid --loss_func mse --finetune_act_func sigmoid --rbm_noise gauss

I noticed this only occurs if hidden layer dimensions are more than visible feature dimensions.

Installation issue

Hi Gabriele,
I am trying to install your nice package with pip on Ubuntu 14.04 with tf v0.8. Unfortunately, I receive the following message:

$pip install yadlt
Downloading/unpacking yadlt
  Could not find a version that satisfies the requirement yadlt (from versions: 0.0.1rc2, 0.0.1rc3, 0.0.1rc4, 0.0.2rc3, 0.0.2rc2, 0.0.2rc1, 0.0.1rc1)
Cleaning up...
No distributions matching the version for yadlt
Storing debug log for failure in /home/ubuntu/.pip/pip.log

The log file is:

$ cat /home/ubuntu/.pip/pip.log
------------------------------------------------------------
/usr/bin/pip run on Wed Jun  8 14:49:22 2016
Downloading/unpacking yadlt
  Getting page https://pypi.python.org/simple/yadlt/
  URLs to search for versions for yadlt:
  * https://pypi.python.org/simple/yadlt/
  Analyzing links from page https://pypi.python.org/simple/yadlt/
    Found link https://pypi.python.org/packages/25/23/c32d5b944dc58e33143ea4661c8bfb03f8cf284dba502bc60c8275692317/yadlt-0.0.1rc2.tar.gz#md5=f4ca41e2fa61c25db8011881d30d72bf (from https://pypi.python.org/simple/yadlt/), version: 0.0.1rc2
    Found link https://pypi.python.org/packages/47/f5/d1fbf73f7218e8c6dd5d5f8af9f9f820ec0de80060f6f3fc4b0049c57c83/yadlt-0.0.1rc3.tar.gz#md5=c923f7b9825c4b707b530099a967f757 (from https://pypi.python.org/simple/yadlt/), version: 0.0.1rc3
    Found link https://pypi.python.org/packages/6c/5a/1183a1312e4cd7e5ec0480db4377b993c29880c8951a1fa4c41e59a8387e/yadlt-0.0.1rc4.tar.gz#md5=3facee3891530f9ccd4d1e4ebf871dff (from https://pypi.python.org/simple/yadlt/), version: 0.0.1rc4
    Found link https://pypi.python.org/packages/7c/59/33dc32f10680e93b54766fee15eba1a0d129e2afe2938382d7a6c42694f1/yadlt-0.0.2rc3.tar.gz#md5=2d024185c431a213bd5a7bfa185b605e (from https://pypi.python.org/simple/yadlt/), version: 0.0.2rc3
    Found link https://pypi.python.org/packages/92/9b/71d5d596d8aba19eeacf2effa121827ae595dc22f6d0b0f27ef3b6a17fe4/yadlt-0.0.2rc2.tar.gz#md5=e1c239ccb5ed4657289cd5e156930ec7 (from https://pypi.python.org/simple/yadlt/), version: 0.0.2rc2
    Found link https://pypi.python.org/packages/d2/a9/ec489b5a0c3e067f6afa290b09ba37718fe1c663e089e45d8e8e039d5201/yadlt-0.0.2rc1.tar.gz#md5=94d4468191fb7c1d89e1a8cd0636e7ef (from https://pypi.python.org/simple/yadlt/), version: 0.0.2rc1
    Found link https://pypi.python.org/packages/e7/f5/74f45107ac5cb0ad6a95f6aae40ac9d3066cd32d6d4a3972b52b40ed4d71/yadlt-0.0.1rc1.tar.gz#md5=973996e4cf36a9d74de108b4321ef35c (from https://pypi.python.org/simple/yadlt/), version: 0.0.1rc1
  Ignoring link https://pypi.python.org/packages/25/23/c32d5b944dc58e33143ea4661c8bfb03f8cf284dba502bc60c8275692317/yadlt-0.0.1rc2.tar.gz#md5=f4ca41e2fa61c25db8011881d30d72bf (from https://pypi.python.org/simple/yadlt/), version 0.0.1rc2 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/47/f5/d1fbf73f7218e8c6dd5d5f8af9f9f820ec0de80060f6f3fc4b0049c57c83/yadlt-0.0.1rc3.tar.gz#md5=c923f7b9825c4b707b530099a967f757 (from https://pypi.python.org/simple/yadlt/), version 0.0.1rc3 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/6c/5a/1183a1312e4cd7e5ec0480db4377b993c29880c8951a1fa4c41e59a8387e/yadlt-0.0.1rc4.tar.gz#md5=3facee3891530f9ccd4d1e4ebf871dff (from https://pypi.python.org/simple/yadlt/), version 0.0.1rc4 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/7c/59/33dc32f10680e93b54766fee15eba1a0d129e2afe2938382d7a6c42694f1/yadlt-0.0.2rc3.tar.gz#md5=2d024185c431a213bd5a7bfa185b605e (from https://pypi.python.org/simple/yadlt/), version 0.0.2rc3 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/92/9b/71d5d596d8aba19eeacf2effa121827ae595dc22f6d0b0f27ef3b6a17fe4/yadlt-0.0.2rc2.tar.gz#md5=e1c239ccb5ed4657289cd5e156930ec7 (from https://pypi.python.org/simple/yadlt/), version 0.0.2rc2 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/d2/a9/ec489b5a0c3e067f6afa290b09ba37718fe1c663e089e45d8e8e039d5201/yadlt-0.0.2rc1.tar.gz#md5=94d4468191fb7c1d89e1a8cd0636e7ef (from https://pypi.python.org/simple/yadlt/), version 0.0.2rc1 is a pre-release (use --pre to allow).
  Ignoring link https://pypi.python.org/packages/e7/f5/74f45107ac5cb0ad6a95f6aae40ac9d3066cd32d6d4a3972b52b40ed4d71/yadlt-0.0.1rc1.tar.gz#md5=973996e4cf36a9d74de108b4321ef35c (from https://pypi.python.org/simple/yadlt/), version 0.0.1rc1 is a pre-release (use --pre to allow).
  Could not find a version that satisfies the requirement yadlt (from versions: 0.0.1rc2, 0.0.1rc3, 0.0.1rc4, 0.0.2rc3, 0.0.2rc2, 0.0.2rc1, 0.0.1rc1)
Cleaning up...
  Removing temporary dir /tmp/pip_build_ubuntu...
No distributions matching the version for yadlt
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1178, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 322, in find_requirement
    raise DistributionNotFound('No distributions matching the version for %s' % req)
DistributionNotFound: No distributions matching the version for yadlt

ValueError: Dimensions 128 and 512 are not compatible && Reconstruction loss at step 0: nan

When running

 python command_line/run_stacked_autoencoder_unsupervised.py --layers 512,256,128 --batch_size 25 --num_epochs 5 --verbose 1  --corr_frac 0.0 --finetune_learning_rate 0.001 --finetune_num_epochs 25 --opt gradient_descent --learning_rate 0.05 --enc_act_func sigmoid --dec_act_func sigmoid --loss_func cross_entropy --finetune_act_func tanh --dropout 0.7

, I got this:

Traceback (most recent call last):
  File "command_line/run_stacked_autoencoder_unsupervised.py", line 181, in <module>
    sdae.build_model(trX.shape[1], encodingw, encodingb)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 296, in build_model
    self.reconstruction = self._create_decoding_layers(next_train)
  File "./models/autoencoder_models/stacked_deep_autoencoder.py", line 415, in _create_decoding_layers
    y_act = tf.matmul(next_decode, dec_w) + dec_b
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1036, in matmul
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 911, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2156, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1612, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/common_shapes.py", line 89, in matmul_shape
    inner_a.assert_is_compatible_with(inner_b)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 98, in assert_is_compatible_with
    % (self, other))
ValueError: Dimensions 128 and 512 are not compatible

After debugging I found the problem lies in:

# models/autoencoder_models/stacked_deep_autoencoder.py
def _create_decoding_layers(self, last_encode):

        """ Create the decoding layers for reconstruction finetuning.
        :param last_encode: output of the last encoding layer
        :return: output of the final encoding layer.
              """                                                                                                                                                                             

        next_decode = last_encode

        for l, layer in enumerate(reversed(self.layers)):

            with tf.name_scope("decode-{}".format(l)):

                # Create decoding variables
                dec_w = tf.Variable(tf.transpose(self.encoding_w_[l].initialized_value()))
                dec_b = tf.Variable(tf.constant(0.01, shape=[dec_w.get_shape().dims[1].value]))
                self.decoding_w.append(dec_w)
                self.decoding_b.append(dec_b)

                print dec_w.get_shape().dims[0], dec_w.get_shape().dims[1] 
                print next_decode.get_shape().dims[0], next_decode.get_shape().dims[1]  
                y_act = tf.matmul(next_decode, dec_w) + dec_b

                if self.finetune_act_func == 'sigmoid':
                    layer_y = tf.nn.sigmoid(y_act)

                elif self.finetune_act_func == 'tanh':
                    layer_y = tf.nn.tanh(y_act)

                elif self.finetune_act_func == 'relu':
                    layer_y = tf.nn.relu(y_act)

                else:
                    layer_y = None

                # the input to the next layer is the output of this layer
                next_decode = tf.nn.dropout(layer_y, self.keep_prob)

            self.layer_nodes.append(next_decode)

        return next_decode

dec_w = tf.Variable(tf.transpose(self.encoding_w_[l].initialized_value()))

should be

dec_w = tf.Variable(tf.transpose(self.encoding_w_[len(self.layers) - l -1].initialized_value()))

Then I got classical nan problem in finetuning stage:

Reconstruction loss at step 0: nan
Reconstruction loss at step 1: nan
Reconstruction loss at step 2: nan
Reconstruction loss at step 3: nan
Reconstruction loss at step 4: nan
Reconstruction loss at step 5: nan
...

Is it because I didn't update to the newest code?

load_cifar10_dataset data_batch = pickle.load(fo) UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 6: ordinal not in range(128)

I tried to run ruen_autoencoder.py
I got this error >>

"D:\Anaconda3\lib\site-packages\yadlt\utils\datasets.py", line 71, in load_cifar10_dataset
    data_batch = pickle.load(fo)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 6: ordinal not in range(128)

how can I fix it? pls

ImportError: No module named 'tensorflow.models'

The version of TF that I installed is 1.0. I met the error when I ran run_dbn.py. Maybe there is something different in the 1.0 TF?

Traceback (most recent call last):
  File "run_dbn.py", line 5, in <module>
    from yadlt.utils import datasets, utilities
  File "/usr/local/lib/python3.5/site-packages/yadlt/utils/datasets.py", line 8, in <module>
    from tensorflow.models.rnn.ptb import reader
ImportError: No module named 'tensorflow.models'

gabrieleangeletti / deep-learning-tensorflow Goto Github PK

deep-learning-tensorflow's Introduction

Deep Learning algorithms with TensorFlow

Requirements:

List of available models:

Installation

Through pip:

Through github:

Documentation:

Models TODO list

deep-learning-tensorflow's People

Contributors

Stargazers

Watchers

Forkers

deep-learning-tensorflow's Issues

Here's the error:

AttributeError: 'NpzFile' object has no attribute 'shape'

Recommend Projects

Recommend Topics

Recommend Org