mila-iqia / blocks-examples Goto Github PK

Hello,

While playing with the machine translation example I came across an unexpected crash. I can't reload the previous train model and see Warning.

When I come across the code in checkpoints.py

In Line 138:

        params_all = self.load_parameters()
        params_this = main_loop.model.get_parameter_dict()

I print the content of params_all and params_this.

I find that params_all contains many '-' but params_this only contains '/',

so it will cause load parameter fails.

so can anyone fix this problems?

Thanks in advance

Hello,

I found out that the best_bleu_model....npz file is much larger than params.npz. It's storing different parameters than params.npz, how can I restore it?
Hope to get your help.

I'm getting the following errors when I try to run the blocks machine translation example (repost from blocks-users, maybe this is a real bug?)

python -m machine_translation
< ... >
INFO:machine_translation:Total number of parameters: 37
INFO:machine_translation:Building model
INFO:machine_translation:Initializing extensions
INFO:machine_translation:Building sampling model
INFO:machine_translation:Building sampler
INFO:machine_translation:Building bleu validator
INFO:machine_translation:Initializing training algorithm
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
INFO:machine_translation:Initializing main loop
INFO:blocks.main_loop:Entered the main loop
INFO:blocks.algorithms:Initializing the training algorithm
<<!! BUG IN FGRAPH.REPLACE OR A LISTENER !!>> <type 'exceptions.Exception'> Constant{-1} is already owned by another fgraph local_shape_to_shape_i
ERROR (theano.gof.opt): Optimization failure due to: local_shape_to_shape_i
ERROR:theano.gof.opt:Optimization failure due to: local_shape_to_shape_i
ERROR (theano.gof.opt): node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20)
ERROR:theano.gof.opt:node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20)
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace
self.change_input(node, i, new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input
self.import_r(new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r
self.import(variable.owner, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import
raise Exception("%s is already owned by another fgraph" % r)
Exception: Constant{-1} is already owned by another fgraph

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace
self.change_input(node, i, new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input
self.import_r(new_r, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r
self.import(variable.owner, reason=reason)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import
raise Exception("%s is already owned by another fgraph" % r)
Exception: Constant{-1} is already owned by another fgraph

ERROR (theano.gof.opt): Optimization failure due to: local_subtensor_lift
ERROR:theano.gof.opt:Optimization failure due to: local_subtensor_lift
ERROR (theano.gof.opt): node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2})
ERROR:theano.gof.opt:node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2})
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift
if not u.owner or len(u.clients) > 1:
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift
if not u.owner or len(u.clients) > 1:
AttributeError: 'TensorVariable' object has no attribute 'clients'

< ... many more AttributeErrors for local_add_canonizer, local_mul_canonizer, etc >

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/state/partition1/localdata/bjt/blocks-examples/machine_translation/main.py", line 41, in
get_dev_stream(*_configuration), args.bokeh)
File "machine_translation/init.py", line 175, in main
main_loop.run()
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 197, in run
reraise_as(e)
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/utils/init.py", line 225, in reraise_as
six.reraise(type(new_exc), new_exc, orig_exc_traceback)
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 172, in run
self.algorithm.initialize()
File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/algorithms/init.py", line 253, in initialize
self.inputs, [], updates=all_updates, *_self.theano_func_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/pfunc.py", line 526, in pfunc
output_keys=output_keys)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1777, in orig_function
output_keys=output_keys).create(
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1457, in init
optimizer_profile = optimizer(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 101, in call
return self.optimize(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 2211, in apply
lopt_change = self.process_node(fgraph, node, lopt)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1830, in process_node
self.failure_callback(e, self, repl_pairs, lopt, node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1688, in warn_inplace
return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1674, in warn
raise exc
AssertionError:

Original exception:
AssertionError

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Here are the versions of things that seems relevant:

python --version
Python 2.7.10 :: Anaconda 1.9.1 (64-bit)

python -c "import theano; print theano.version"
0.7.0.dev-80197cf574b62fa7d869f4de800700d2f75b7f4e

pip install git+git://github.com/mila-udem/blocks.git -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt
Collecting git+git://github.com/mila-udem/blocks.git
Cloning git://github.com/mila-udem/blocks.git to /tmp/pip-VSApNQ-build
Requirement already satisfied (use --upgrade to upgrade): blocks==0.1.1 from git+git://github.com/mila-udem/blocks.git in /home/bjt/anaconda/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): numpy==1.9.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): picklable-itertools==0.1.1 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): progressbar2==2.7.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): pyyaml==3.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): toolz==0.7.2 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): theano from git+https://github.com/Theano/Theano.git#egg=theano in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): fuel from git+https://github.com/mila-udem/fuel#egg=fuel in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from theano->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): h5py in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): tables in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pyzmq in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): pillow in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): requests in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Wed_Aug_27_10:36:36_CDT_2014
Cuda compilation tools, release 6.5, V6.5.16

When I run the reverse_words and machine_translation examples, the cost does not decrease, and (in the MT example) the generated samples are still gibberish after 80 epochs. The sqrt example works correctly, which is why I'm suspecting it's to do with the sequence models.

I'm using very recent (yesterday's) git checkouts of blocks-examples, blocks, fuel and theano (installed with pip). Scipy, numpy, etc, are from standard pip install libraries. Using python 2.7 on CPUs; I've replicated this behaviour on two different machines.
theano.version '0.7.0.dev-49b554843f47f1b2bc83bb1cbf64dbcbfc70484a'

Is this a known issue?

Can anyone make an example of how to do a simple RNN? One with Back propagation through time would be sufficient. Basically, ones similar to the one in this paper (LM RNN).

http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf

It seems that the lenet example doesn't work with the current fuel api. I make it works by changing

MNIST(("train",)) into MNIST("train")

Hello,

While playing with the machine translation example I came across an unexpected crash. I tried to enable weight noise as done in the paper "On Using Monolingual Corpora in Neural Machine Translation". I assigned a value higher than 0 to the configuration parameter "weight_noise_ff" and got a crash in line 78 of init.py:

enc_params = Selector(encoder.lookup).get_params().values()

I replaced get_params() with get_params(), according to the API documentation, but then I get the following error:

File "machine_translation/init.py", line 85, in main
dec_params += Selector(decoder.state_init).get_parameters().values()
AttributeError: 'Decoder' object has no attribute 'state_init'

Any advice?

Thanks in advance

Currently live plotting is disabled for testing. The reason for that are difficulties emerging when testing Blocks-examples in the context of testing Blocks, see the PR mila-iqia/blocks#748.

Examples meant to be instructive ought to be more thoroughly commented. There are some non-obvious bits even in the MNIST example.

What is sentence mask used for in the program ? I see source sentence mask and target sentence mask. In the block documentation, it says the mask is (batch,) size and is 1 when data is available and 0 otherwise. When is the data not available ? Or are they any other use of the sentence mask?

My program raise NaN error after several iterations. I want to monitor the automatically added auxiliary variables (W_norm and b_norm). I use TrainingDataMonitoring(fg.auxiliary_variables, after_batch=True). But the program raise ValueError: variables should have different names! Duplicates: W_norm, b_norm. My model is a multiple layer neural network. So, there are duplicate auxiliary variable names.
What should I do to monitor these variables? Besides how to monitor the gradient?

Hi,

I am running the machine translation examples, however the best BLEU score reported is 7.71 with more than 400000 iterations' training. However the paper neural machine translation by jointly learning to align and translate reports over 17.82 or 26.75.

Any idea on that?

Not sure if this works if sqrt is not a module but is a package. See - http://stackoverflow.com/questions/4042905/what-is-main-py

 python -m sqrt --num-batches 1000 sqrt/saved_state

I was able to make it work with after adding blocks to PYTHONPATH

 python sqrt --num-batches 1000 sqrt/saved_state

Hi everyone,

I get the following error:

Training status:
batch_interrupt_received: False
epoch_interrupt_received: False
epoch_started: True
epochs_done: 0
iterations_done: 48
received_first_batch: True
resumed_from: f899e453ab7e4d66acbe7558d9dcedaf
training_started: True
Log records from the iteration 48:
decoder_cost_cost: 406.997089365

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
File "main.py", line 45, in
get_dev_stream(**configuration), args.bokeh)
File "../machine_translation/init.py", line 175, in main
main_loop.run()
File "/usr/local/lib/python2.7/dist-packages/blocks-0.0.1-py2.7.egg/blocks/main_loop.py", line 196, in run
reraise_as(e)
File "/usr/local/lib/python2.7/dist-packages/blocks-0.0.1-py2.7.egg/blocks/utils/init.py", line 225, in reraise_as
six.reraise(type(new_exc), new_exc, orig_exc_traceback)
File "/usr/local/lib/python2.7/dist-packages/blocks-0.0.1-py2.7.egg/blocks/main_loop.py", line 182, in run
while self.run_epoch():
File "/usr/local/lib/python2.7/dist-packages/blocks-0.0.1-py2.7.egg/blocks/main_loop.py", line 231, in run_epoch
while self._run_iteration():
File "/usr/local/lib/python2.7/dist-packages/blocks-0.0.1-py2.7.egg/blocks/main_loop.py", line 244, in _run_iteration
batch = next(self.epoch_iterator)
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).__next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 32, in next
data = self.data_stream.get_data()
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 138, in get_data
data = next(self.child_epoch_iterator)
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 30, in next
data = self.data_stream.get_data(next(self.request_iterator))
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 591, in get_data
data, next(self.child_epoch_iterator)):
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 32, in next
data = self.data_stream.get_data()
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 639, in get_data
return self.get_data()
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 633, in get_data
data = next(self.child_epoch_iterator)
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 32, in next
data = self.data_stream.get_data()
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 214, in get_data
data = next(self.child_epoch_iterator)
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 30, in next
data = self.data_stream.get_data(next(self.request_iterator))
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 591, in get_data
data, next(self.child_epoch_iterator)):
File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
return type(self).next(self)
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/iterator.py", line 32, in next
data = self.data_stream.get_data()
File "/usr/local/lib/python2.7/dist-packages/fuel-0.0.1-py2.7-linux-x86_64.egg/fuel/transformers/init.py", line 215, in get_data
image = self.mapping(data)
File "../machine_translation/stream.py", line 83, in call
for x in sentence_pair[0]],
IndexError: tuple index out of range

Original exception:
IndexError: tuple index out of range

the training data is download from WMT15 cs-en following prepare_data.py description.
and the version of blocks is 0.0.1

Could anyone help me with it?

I think we are really missing an example for those here.

@harmdevries89, @dmitriy-serdyuk, do you have anything ready for that purpose?

hi all,
would this framework be useful for multivariate time series.
Best,
Andrew

Is there any way to run sampling other than during training? I am looking for a mechanism where I could load the model and run translation based on my input.

Hi,

I am trying to run the machine translation example, but I face the memory allocation error:
ImportError: ('The following error happened while compiling the node, Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n, /hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory\n\nOriginal exception:\n\tImportError: The following error happened while compiling the node, Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), \n, /hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory', Elemwise{Composite{Switch(i0, i1, Switch(AND(LT((i2 + i3), i1), GT(i4, i1)), i5, minimum((i2 + i3), i6)))}}(Elemwise{le,no_inplace}.0, TensorConstant{0}, TensorConstant{-1}, Elemwise{Composite{Switch(LT(Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10), Composite{Switch(LT(i0, i1), i1, i0)}(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{(i0 - Switch(LT(i1, i2), i2, i1))}(i0, Composite{(i0 - Switch(GE(i1, i2), i2, i1))}(i1, Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(LT(i0, i1), (i0 + i2), i0)}(Composite{Switch(i0, i1, Switch(AND(LT(i2, i1), GT(i3, i1)), i4, maximum(i5, i2)))}(i2, i3, (i4 - i5), i5, i6, i7), i3, i8), i3, i9), i8), i3), i3, i1), i3), i10)}}.0, Elemwise{sub,no_inplace}.0, Elemwise{sub,no_inplace}.0, Elemwise{switch,no_inplace}.0), '\n', '/hltsrv0/farajian/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-redhat-7.2-Nitrogen-x86_64-2.7.11-64/tmpPi8duk/94aaeae5119dfd3722a2721c3fce5069.so: failed to map segment from shared object: Cannot allocate memory')

I tried with different settings, but no success. I also checked the maximum memory which was used in running the script and realized that it was less than 1GB (Max vmem = 830.188M).
Any idea about this problem and how it can be solved?

I am not sure how to modify the example for either of the reverse_words or machine_translation examples to be able to use it with a dataset to do say speech synthesis.

There is no lookup table for each phoneme/character and each utterance maybe of dimension [num of frames x feature dimension]. So say utt1 is say dimension 700 x 50, utt2 is 600 x 50 etc. At each time I would like the encoder to see the entire utterance sequence without using a lookup table .

Is there a simple pointer on how to train an attention network with this kind of input?

I am stuck with mainly the following:

1. How to load this kind of data into a file using fuel, since all the examples I have seen so far are converting it to one stream where it iterates over rows, when I need it to iterate over matrices with variable number of rows.
2. If I do use the stream how to make the attention network know the end of utterance without using a lookup table or an eos tag. Instead of a row vector can I give a matrix as input to it?

Any pointers would be helpful.

ran the machine translation and got this error at validation:

INFO:machine_translation.sampling:Started Validation:
ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
...
...
...
Original exception:
UnusedInputError: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 3 is not part of the computational graph needed to compute the outputs: sequencegenerator_generate_outputs.
To make this error into a warning, you can pass the parameter on_unused_input='warn' to theano.function. To disable it completely, use on_unused_input='ignore'.
Use of uninitialized value $length_reference in numeric eq (==) at ./data/multi-bleu.perl line 148.

blocks 0.1.1
fuel 0.1.1
bleeding edge Theano 0.7.0

...or it will keep breaking Blocks tests, like here

@orhanf , @dmitriy-serdyuk , can you please fix this quickly?

Is there in quick way for adding files for new MT task.

Running the example in the README does not work with theano 0.7 but requires a bleeding-edge (requires at least PersistentCudaNdarrayID from misc/pkl_utils.py ). Could be good to mention this or make the main example compatible with 0.7.

We should have a ready demo to reproduce results from http://karpathy.github.io/2015/05/21/rnn-effectiveness/

This is mostly matter of changing the input in Markov chain examples. And also we don't have deep RNNs so far, but it should kind of work without them as well.

Most of example in this repo only explain how to train a model. I think it would be also helpful to show how to make predictions in some of them, at least in the basic MNIST one.

Hi,

I have a very weird problem going on with the machine translation example. In stream.py, get_data_from_batch method was reimplemented for PaddingWithEOS class.

On a system with up-to-date Theano, blocks and fuel, the RNNSearch model runs correctly. I'm trying to implement another model on top of this example which also uses the stream.py module for stream processing but with masks disabled: All sequences are padded with EOS and the sequences are all 32 length.

The problem is that for this code, I'm getting a Theano exception. I traced it back to the following problem: Blocks was never getting the padded batch which should have been returned by PaddingWithEOS.get_data_from_batch() method. I then checked through this method in the fuel repository but it wasn't mentioned anywhere in the code. Actually, I couldn't find any caller of this method at all in blocks, fuel, blocks-examples code tree :) Then I discovered that this method was renamed to get_batch() in:

commit 667e81fd1e4c02dece0ef0abc71a7870b18506bc
Author: Vincent Dumoulin <[email protected]>
Date:   Tue Jul 7 15:26:40 2015 -0400

    Adapt Filter, Cache, Batch, Unpack, Padding to new Transformer interface

Now when I rename it to get_batch in stream.py, it started to work. The holy question is how come the current RNNSearch code is working without exception?

Thanks.

Oops, I first created this one empty.

Blocks-examples is an extremely important repository in Blocks family. As @dwf said in an off-line conversation, there is very little sense in having a framework without working examples of how it can be used.

That said, we will have to update all the examples here every time anything is changed in Blocks. This is why it is essential to limit the growth of this repo. One way to do it is to stick to examples for which ratio 'educational value/size' is the largest one.

Of the examples currently available machine translation demo is by far the largest one, and I think should live in a separate repository. I do not know why it was moved from https://github.com/kyunghyuncho/NMT to Blocks-examples in the first place, but I think this historical error should be fixed.

I didn't find the external weights initialization of lookuptable. Does you just setup it randomly ?

Hi all,

First of all, thanks for the incredible work! I was running the reverse_words example, and everything works, but when I run:

python -m reverse_words beam_search <path_to_my_model>

I get an error:

Traceback (most recent call last):
  File "/ichec/home/users/chokamp/programs/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/ichec/home/users/chokamp/programs/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/ichec/home/users/chokamp/projects/blocks-examples/reverse_words/__main__.py", line 35, in <module>
    main(**vars(args))
  File "reverse_words/__init__.py", line 327, in main
    batch_size, axis=1))
  File "reverse_words/__init__.py", line 285, in generate
    ComputationGraph(generated[1]))
ValueError: too many values to unpack

After digging around a bit, I can fix it by changing these lines to:

                (initial_states, samples) = VariableFilter(
                    bricks=[reverser.generator], name="outputs")(
                        ComputationGraph(generated[1]))

I.e. adding the intial_states so that samples is the correct variable. I'd love to submit a pull request but wanted to check first if there's anything I'm missing.

Cheers!

Should it be deleted or should be support live plotting in this demo?

Following #20.

Stuff to do:

• Add a bit more tests.
• Refactor imports, make them more consistant, maybe split big files.
• Find a better way to use configurations.

When running the bleu evaluation after 80000 runs, it crashes.
please don't mind the row numbers on main and init as I have changed them a bit.
you can reproduce the error if you set the bleu_val_freq to something relatively low (default is 80k).

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
File "machine_translation/main.py", line 41, in
get_dev_stream(**configuration), args.bokeh)
File "machine_translation/init.py", line 175, in main
main_loop.run()
File "venv/local/lib/python2.7/site-packages/blocks/main_loop.py", line 197, in run
reraise_as(e)
File "venv/local/lib/python2.7/site-packages/blocks/utils/init.py", line 258, in reraise_as
six.reraise(type(new_exc), new_exc, orig_exc_traceback)
File "venv/local/lib/python2.7/site-packages/blocks/main_loop.py", line 183, in run
while self._run_epoch():
File "venv/local/lib/python2.7/site-packages/blocks/main_loop.py", line 232, in _run_epoch
while self._run_iteration():
File "venv/local/lib/python2.7/site-packages/blocks/main_loop.py", line 255, in _run_iteration
self._run_extensions('after_batch', batch)
File "venv/local/lib/python2.7/site-packages/blocks/main_loop.py", line 263, in _run_extensions
extension.dispatch(CallbackName(method_name), *args)
File "venv/local/lib/python2.7/site-packages/blocks/extensions/init.py", line 338, in dispatch
self.do(callback_invoked, *(from_main_loop + tuple(arguments)))
File "machine_translation/sampling.py", line 174, in do
self._save_model(self._evaluate_model())
File "machine_translation/sampling.py", line 206, in _evaluate_model
ignore_first_eol=True)
File "venv/local/lib/python2.7/site-packages/blocks/search.py", line 266, in search
self.compile()
File "venv/local/lib/python2.7/site-packages/blocks/search.py", line 130, in compile
self._compile_next_state_computer()
File "venv/local/lib/python2.7/site-packages/blocks/search.py", line 113, in _compile_next_state_computer
self.contexts + self.input_states + next_outputs, next_states)
File "venv/local/lib/python2.7/site-packages/theano/compile/function.py", line 320, in function
output_keys=output_keys)
File "venv/local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 479, in pfunc
output_keys=output_keys)
File "venv/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1776, in orig_function
output_keys=output_keys).create(
File "venv/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1415, in init
self._check_unused_inputs(inputs, outputs, on_unused_input)
File "venv/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1553, in _check_unused_inputs
i.variable, err_msg))
theano.compile.function_module.UnusedInputError: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 3 is not part of the computational graph needed to compute the outputs: sequencegenerator_generate_outputs.
To make this error into a warning, you can pass the parameter on_unused_input='warn' to theano.function. To disable it completely, use on_unused_input='ignore'.

Original exception:
UnusedInputError: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 3 is not part of the computational graph needed to compute the outputs: sequencegenerator_generate_outputs.
To make this error into a warning, you can pass the parameter on_unused_input='warn' to theano.function. To disable it completely, use on_unused_input='ignore'.
Use of uninitialized value $length_reference in numeric eq (==) at ./data/multi-bleu.perl line 148.

There are classes defined in __main__. We could fix this by moving them to a separate file.

sample0_heatmap.pdf
sample1_heatmap.pdf
HI,
I draw the align heatmap during training. The heatmap is took from training iteration 18000. I cannot get an align map close to the result in the paper.

I'm trying to use a different encoder output, let's call it memory, to initialize decoder state.
I added it to the contexts of AttentionRecurrent, just like how attended is added.
Then I get the memory in GRU.initial_state() and do a transformation just like how attended is used.
It's only used for initialization so I pop it out at compute_states(), as attended gets popped out.
Training goes fine. But when I do testing, with mode=translate, I have a problem.
It appears that memory is not included in the inner_cg of SequenceGenerator.
How can I fix this?

I think the bug is easy to reproduce, just modify line11 in machine_translation/configuration.py to
config['dec_nhids'] = 25
The error message is:

ValueError: total size of new array must be unchanged
Apply node that caused the error: Reshape{4}(Elemwise{add,no_inplace}.0, MakeVector{dtype='int64'}.0)
Toposort index: 983
Inputs types: [TensorType(float64, 3D), TensorType(int64, vector)]
Inputs shapes: [(10, 80, 25), (4,)]
Inputs strides: [(16000, 200, 8), (8,)]
Inputs values: ['not shown', array([10, 80, 12,  2])]
Outputs clients: [[Reduce{maximum}{3}(Reshape{4}.0), Elemwise{Composite{(EQ(i0, i1) \* i2)}}(InplaceDimShuffle{0,1,2,x}.0, Reshape{4}.0, InplaceDimShuffle{0,1,2,x}.0)]]

In this example, dropout is effectively ignored because:

• The cost used in GradientDescent should be cg.outputs[0] rather than 'cost':
  https://github.com/mila-udem/blocks-examples/blob/12013e6c737171ce285b27381f0f5aacfdff7cab/machine_translation/__init__.py#L160
• Also the cost used in TrainingDataMonitoring:
  https://github.com/mila-udem/blocks-examples/blob/12013e6c737171ce285b27381f0f5aacfdff7cab/machine_translation/__init__.py#L112
  is not the dropped out one, so instead one should use here cg.outputs[0].

Hi,
we are having a hard time to reproduce the results we got with GroundHog using Blocks.
Given the exact same training data, vocabulary, test set and settings we are 3 Bleu points behind GroundHog on a German to English translation task. We tried many different setups and number of iterations, but we can't reach it.
The GroundHog translation costs also seem to have a higher correlation between good and bad sentences then blocks. e.g.:

"vielen Dank ." translated to "thank you ."
a perfect translation and a common phrase which should have a low cost.
GroundHog cost: 0.000250929
Blocks cost: 0.357417

"fliegende Katze ." is translated to "fly away , cat ." not wrong but kind of a strange/unusual sentence.
GroundHog: 0.280177
Blocks cost: 0.267061

Blocks gives "thank you ." a higher cost to then "fly away , cat ." which seems strange to me. I take this as a hint that the problem is mainly related to the model and not to the search. The last comment here:
kyunghyuncho/NMT#21
seems to have the same issue. Has there been any progress on this?
Any tips where the Blocks computation graph differs from the GroundHog graph (it's to large to just look at it an see a difference)? Or other hints what the problem could be?

Thanks,

I am looking for an example that does machine translation. I have run the training for one day with the following result

Training status:
         batch_interrupt_received: False
         epoch_interrupt_received: False
         epoch_started: True
         epochs_done: 0
         iterations_done: 11
         received_first_batch: True
         resumed_from: None
         training_started: True
Log records from the iteration 11:
         decoder_cost_cost: 350.873748779

Epochs done is still showing 0.

Anyway got the model saved in search_model_cs2en . Looking for how to do translation based on this.

I just started to learn blocks. How to print alignment weight between encode and decode steps in sampling.Sampler ?
Thank you

How to run machine_translation examples??when i run by command:"python -m machine_translation machine_translation/saved_state", i get this error:
import configurations - ImportError: No module named 'configurations'.
so how to run it with other commands or need some parameters???

In installation documentation, i see that block has supported for python 2.6>>But when i run examples for python 2.6,i get this error:
RuntimeError: module compiled against API version a but this version of numpy is 9
RuntimeError: module compiled against API version a but this version of numpy is 9
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
loader, code, fname = _get_module_details(mod_name)
File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
loader = get_loader(mod_name)
File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
return find_loader(fullname)
File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
import(pkg)
File "sqrt/init.py", line 6, in
from blocks.algorithms import GradientDescent, Scale
File "/usr/lib/python2.6/site-packages/blocks/algorithms/init.py", line 14, in
from blocks.graph import ComputationGraph
File "/usr/lib/python2.6/site-packages/blocks/graph.py", line 158
return {var.name: var for var in self.inputs}
^
SyntaxError: invalid syntax
So i need a new version for python,or i need change code?Or some other ways

I'm worried that people will be confused by all the example scripts being called __init__.py, and think this is required or something. Maybe we should mention this in the readme or something?

HI
I use the command : python -m machine_translation but it occur some error lists like these:
ERROR:theano.gof.opt:Optimization failure due to: local_mul_canonizer
ERROR (theano.gof.opt): node: Elemwise{true_div}(Elemwise{add,no_inplace}.0, DimShuffle{x,0}.0)
ERROR:theano.gof.opt:node: Elemwise{true_div}(Elemwise{add,no_inplace}.0, DimShuffle{x,0}.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/media/big/Workspace/machine_translation/Theano/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/media/big/Workspace/machine_translation/Theano/theano/tensor/opt.py", line 4456, in transform
for c, c_idx in out.clients:
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/media/big/Workspace/machine_translation/Theano/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/media/big/Workspace/machine_translation/Theano/theano/tensor/opt.py", line 4456, in transform
for c, c_idx in out.clients:
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR (theano.gof.opt): Optimization failure due to: local_mul_canonizer
ERROR:theano.gof.opt:Optimization failure due to: local_mul_canonizer
ERROR (theano.gof.opt): node: Elemwise{mul}(Elemwise{add,no_inplace}.0, Elemwise{mul,no_inplace}.0)
ERROR:theano.gof.opt:node: Elemwise{mul}(Elemwise{add,no_inplace}.0, Elemwise{mul,no_inplace}.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/media/big/Workspace/machine_translation/Theano/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/media/big/Workspace/machine_translation/Theano/theano/tensor/opt.py", line 4456, in transform
for c, c_idx in out.clients:
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/media/big/Workspace/machine_translation/Theano/theano/gof/opt.py", line 1787, in process_node
replacements = lopt.transform(node)
File "/media/big/Workspace/machine_translation/Theano/theano/tensor/opt.py", line 4456, in transform
for c, c_idx in out.clients:
AttributeError: 'TensorVariable' object has no attribute 'clients'
.....
.....
.....
How can I solve this problem?

When collecting vocabulary,
vocab = {'UNK': 1, '': 0, '': 0}
bos, eos tag are all set to 0, then vocab would collect vocab_size - 2 normal words.
But in steam.py get_tr_stream method,

    src_vocab = _ensure_special_tokens(
        src_vocab if isinstance(src_vocab, dict)
        else cPickle.load(open(src_vocab)),
        bos_idx=0, eos_idx=src_vocab_size - 1, unk_idx=unk_id)

the eos tag is set src_vocab_size - 1. Since the vocab already contains vocab_size words, eos and the last word are map to the same idx.
I think it is a bug, although it is of little significance.

Hello Guys,

I am trying to understand machine translation example by parts of code over the terminal. However I am stuck with below strange error while generating sequence through decode.generate().

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function.py", line 308, in function
output_keys=output_keys)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/pfunc.py", line 536, in pfunc
output_keys=output_keys)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function_module.py", line 1766, in orig_function
output_keys=output_keys).create(
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function_module.py", line 1422, in init
accept_inplace)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function_module.py", line 177, in std_fgraph
update_mapping=update_mapping)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/gof/fg.py", line 171, in init
self.import_r(output, reason="init")
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/gof/fg.py", line 315, in import_r
self.import(variable.owner, reason=reason)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/gof/fg.py", line 420, in import
detailed_err_msg)
theano.gof.fg.MissingInputError: A variable that is an input to the graph was neither provided as an input to the function nor given a value. A chain of variables leading from this input to an output is [sequencegenerator_initial_states_states[t-1], sequencegenerator_generate_states, readout_readout_states, merge_apply_states, merge_apply_states, transform_states_apply_input_, dot.0, transform_states_apply_output, merge_apply_states, Elemwise{add,no_inplace}.0, Elemwise{add,no_inplace}.0, Elemwise{add,no_inplace}.0, merge_apply_output, initializablefeedforwardsequence_apply_args_0, maxout_bias_apply_input_, Elemwise{add,no_inplace}.0, maxout_bias_apply_output, maxout_apply_input_, Reshape{3}.0, max, maxout_apply_output, softmax0_apply_input_, dot.0, softmax0_apply_output, softmax1_apply_input_, dot.0, Elemwise{add,no_inplace}.0, softmax1_apply_output, initializablefeedforwardsequence_apply_output, readout_readout_output_0, readout_emit_readouts, softmaxemitter_emit_readouts, softmaxemitter_probs_readouts, ndimensionalsoftmax_apply_args_0, ndimensionalsoftmax_apply_input_, Softmax.0, ndimensionalsoftmax_apply_output, ndimensionalsoftmax_apply_output, softmaxemitter_probs_output_0, Reshape{2}.0, Subtensor{::, int64}.0, Shape.0, Elemwise{Cast{int32}}.0, mrg_uniform{TensorType(float64, vector),no_inplace}.0]. This chain may not be unique
Backtrace when the variable is created:
File "/home/karn_s/.local/lib/python2.7/site-packages/blocks/bricks/recurrent.py", line 230, in recurrent_apply
brick.name, application.application_name))

The error appears while running following piece of codes:

from model import BidirectionalEncoder
from model import Decoder
import configurations
config = getattr(configurations,'get_config_cs2en')()

source_sentence = tensor.lmatrix('source')
source_sentence_mask = tensor.matrix('source_mask')
encoder = BidirectionalEncoder(config['src_vocab_size'], config['enc_embed'], config['enc_nhids'])
decoder = Decoder(config['trg_vocab_size'], config['dec_embed'], config['dec_nhids'], config['enc_nhids'] * 2)
target_sentence = tensor.lmatrix('target')
target_sentence_mask = tensor.matrix('target_mask')
cost = decoder.cost(encoder.apply(source_sentence, source_sentence_mask), source_sentence_mask, target_sentence, target_sentence_mask)
cst_fn = theano.function([source_sentence,source_sentence_mask,target_sentence,target_sentence_mask],cost)

representation = encoder.apply(source_sentence,source_sentence_mask)
generated = decoder.generate(source_sentence, representation)
generate_fn = theano.function([source_sentence,source_sentence_mask], generated)
<<<<< MissingInputError>>>>>

sampling_input = tensor.lmatrix('input')
sampling_representation = encoder.apply(sampling_input, tensor.ones(sampling_input.shape))
generated = decoder.generate(sampling_input, sampling_representation)
generate_fn = theano.function([sampling_input],generated)
<<<<< MissingInputError>>>>>

Please assist.

When I run block examples, I have the error
/usr/bin/python: No module named cffi
I did install cffi successfully
pip install cffi

Switch travis config when mila-iqia/blocks-extras#15 is fixed. See #27 .

I use block version 0.1.1 and theano version 0.7.0
And when i try to run examples from blocks-examples i get this error:ImportError: cannot import name PersistentCudaNdarrayID.(https://github.com/mila-udem/blocks-examples : command : "python -m sqrt --num-batches 1000 sqrt/saved_state").How can i fix error?

Hi all,

I am wondering if there is a proper way to resume training using a new corpus. Let's say, I have a model already trained using corpus A, and now I want to adapt it to corpus B.
I tried this with simply modifying the configuration file to point to the new corpus, and re-started the training. But, I realized it is still using the old corpus. So, I removed the old files and re-ran the code.
This time I got an error saying that it can not find the training corpus (it was still looking for the old corpus). By renaming corpus B to A, the error disappears.
But, I see an strange behavior: The bleu score constantly decrease (in the first validation it was 29.33 and now after 70K iterations it is 21.56).
If I just train a system using corpus B from scratch (using the same configurations) the bleu score reaches to 24.81.
I would assume that in the first case (i.e. adapting a pre-trained model to a new corpus) the performance should be higher or in the worst case, the same as the the baseline. So, I can not find any answer to this drop of bleu score (3 points lower than the baseline).
Do you have any idea?

p.s. I use Europarl and TED corpus as corpus A and B, respectively (English-Italian pair). Since TED corpus is very small, I want to improve the performance of the system by first training a system on Europarl, and then adapting it to TED (similar to the University of Stanford's submission in IWSLT 2015).