Comments (4)
Thanks! Good catch. Just fixed a typo in the ipynb.
from theano_lstm.
Hi Jonathan,
I am trying to implement dropout in the theano.scan, but without
using your package, since I have written most part of the rnn. Do you
have any suggestions? I read your codes and instructions, but still have
some questions. E.g, since masks is non_sequences, what will be the
input in the theano.function?
Thanks very much!
Cosmo
On 04/20/2015 10:18 PM, Jonathan Raiman wrote:
Thanks! Good catch. Just fixed a typo in the ipynb.
—
Reply to this email directly or view it on GitHub
#11 (comment).
Best,
Cosmo Zhang
from theano_lstm.
Hey Cosmo,
To get dropout to work I've done the following:
Get some random number generator:
import theano, theano.tensor as T
srng = theano.tensor.shared_randomstreams.RandomStreams(1234)
Then create the mask you want to use:
drop_prob = T.scalar()
shape = (300, 300)
mask = T.cast(
srng.binomial(n=1, p=1-drop_prob, size=shape),
theano.config.floatX
)
And then in your scan:
result, updates = theano.scan(fn = step,
sequences = [x],
outputs_info = etc...,
non_sequences = [mask])
error = etc...
Your theano function can now take as inputs:
func = theano.function([x, target, drop_prob], (result[-1] - target)**2)
As long as the random variables aren't generated within the scan op you should be fine. I've had problems whenever I tried created a new set of binomials on every time step or something else that's fancy. But the frozen dropout values as shown above worked for me.
from theano_lstm.
Thank you very much, I solved it last night in another way:
import theano, theano.tensoras T
srng= theano.tensor.shared_randomstreams.RandomStreams(1234)
drop_prob= T.scalar()
shape= (shape1, 300,300) #shape1 is the length of input sequence
mask= T.cast(
srng.binomial(n=1,p=1-drop_prob,size=shape),
theano.config.floatX
)
and then
|result, updates = theano.scan(fn = step,
sequences = [x, mask],
outputs_info = etc...,
non_sequences = None)|
Do you think it is also plausible?
And in your setting, how did yo write step function?
Thank you very much!
On 4/21/2015 10:47 PM, Jonathan Raiman wrote:
Hey Cosmo,
To get dropout to work I've done the following:
Get some random number generator:
import theano, theano.tensoras T
srng= theano.tensor.shared_randomstreams.RandomStreams(1234)Then create the mask you want to use:
drop_prob= T.scalar()
shape= (300,300)
mask= T.cast(
srng.binomial(n=1,p=1-drop_prob,size=shape),
theano.config.floatX
)And then in your scan:
|result, updates = theano.scan(fn = step,
sequences = [x],
outputs_info = etc...,
non_sequences = [mask])error = etc...
|Your theano function can now take as inputs:
|func = theano.function([x, target, drop_prob], (result[-1] - target)**2)
|As long as the random variables aren't generated within the scan op
you should be fine. I've had problems whenever I tried created a new
set of binomials on every time step or something else that's fancy.
But the frozen dropout values as shown above worked for me.—
Reply to this email directly or view it on GitHub
#11 (comment).
from theano_lstm.
Related Issues (17)
- Speed Benchmark HOT 3
- Optimization failure in Tutorial HOT 5
- Bidirectional RNN HOT 1
- Examples are probably outdated HOT 1
- Sequences of vectors HOT 2
- Question in Example Code
- Problem running the example code in 32-bit OS
- Question about fixing node value
- super() syntax for python 2 HOT 1
- Issue with running tutorial HOT 2
- Error with tutorial local_argmax_pushdown
- Masking operation errornous
- Setting the learning rate
- license? HOT 4
- Simple example HOT 16
- LSTM model equations HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from theano_lstm.