add model examples about edward HOT 19 OPEN

Ah, just saw https://github.com/blei-lab/edward/pull/557/files / https://github.com/blei-lab/edward/blob/master/examples/probabilistic_matrix_factorization.py from @siddharth-agrawal

Still happy to make a notebook using the netflix data in / notebooks

from edward.

I was browsing the Stan-users mailing list at 3am in the morning. Someone implemented a Bayesian neural network in Stan(!). There are two different implementations available: [1], [2a, 2b], discussion here

We should include one of these as a model example, and also implement the same model in TensorFlow (which is very easy).

from edward.

@dustinvtran I want to contribute to Edward and feel that working on this issue will help me more as well. Do you have any suggestions as I go forward? I'm using this guide for the dev workflow.

from edward.

perfect! i think a good way to start is maybe to select one of the models you're interested in above and then to implement a working version of it using some data and edward's inference.

from edward.

@dustinvtran I'm thinking of starting with 'Poisson process' next. Do you have any references that might help me? I'm not familiar with many variational inference models.

from edward.

@siddharth-agrawal: Cool. You can try building off #294, which I think @diengadji was working on but is no longer(?). It defines a Cox process for spatial data.

You can look into using the same example or apply a simpler version (e.g., Poisson vs Cox, i.i.d. data instead). I also recommend using a real data set, perhaps one typically applied with Poisson processes. For a more vanilla Poisson process, I also recommend point estimation via MAP instead of VI.

from edward.

@dustinvtran After I play a bit more with Edward and look through the code in more depth, I'd like to have a go at implementing a markov random field. Aware of anyone working on undirected models?

from edward.

Undirected is difficult. We haven't focused on it, although there is limited support such as for undirected models that can be conditionally specified (see https://github.com/mariru/exponential_family_embeddings).

We don't really know how to expose an undirected model's graph structure in the computational graph. But that would be the first step.

from edward.

Please use gibbs sampling as inference method

from edward.

@dustinvtran I'd be happy to take a stab at PMF using the original netflix ratings. Is anyone else on this already?

from edward.

from edward.

I've been trying to get vanilla PCF working on the larger movie lens dataset (20 million ratings, ~20k movies ~130k users) without much luck.

#682

Would greatly appreciate any tips!

from edward.

@patrickeganfoley i looked your notebook. a couple suggestions:

• start simple: use a MAP point estimate rather than going full variational. under coordinate ascent setting, for Gaussian MF, MAP point estimate matches variational mean, and you won't get much from the (under-estimated) approximating variances anyway.

• start simple 2: use a global regularization rather than per-user/item, especially when fitting only on observed ratings -- the data is actually very sparse and probably cannot afford giving you reasonable estimate on a per-user/item basis.

(Updated)

• (slightly) more complex model: K = 2 is too simplistic. I would at least start with K = 5.

from edward.

Thanks @dawenl ! Will try MAP now.

on 2 - I'm not sure if I completely understand. I am pretty sure I am only setting up 4 regularization terms (user offset, movie offset, user vec, movie vec) which doesn't seem to be too much to ask of 20m ratings.

from edward.

OK, I should have looked more closely. I thought sigma_user_betas and sigma_movie_betas are set up on a per-user/item basis.

from edward.

@dawenl thank you for your help! (discussing here #682 so as to not block discussion of other examples)

from edward.

@dustinvtran I would like to start working on Bayesian word embedding model. Do you have any references on how to implement a variational inference model ? I saw the LSTM language model in Edward examples. Thanks.

from edward.

@iliemihai I know @mariru was extending https://github.com/mariru/exponential_family_embeddings to work on probabilistic embedding models with variational inference. In particular, normal priors over embedding and context vectors, and normal variational distributions to approximate their posteriors. It could be useful to start from there. (alternatively, if you want just Bayesian RNNs, you can take the LSTM language model and replace MAP)

from edward.

@iliemihai check out the branch feature/edward in the https://github.com/mariru/exponential_family_embeddings repo. It uses edward's MAP inference

from edward.