Hi,

Some of the files seem to be miss and the code is not functioning without them.

Can you please upload them back so we can continue using this wonderful framework?

Thanks,
EB

I've ran a PixelCNN in Colab last week, but I didn't know when it would finish (4 hours) because there was no in-training output. If you're concerned about flooding a cell with outputs, you could use tqdm over all the epochs that way the user can still see progress while printing only a few lines.

Thank you for this code. I was able to get it to run in https://github.com/sorokine/docker-colab-local

I came across neural-style-gui that uses neural-style-pt and one of the features I'm lamenting that neural-style-gui has that your method does not is the ability to load multiple styles.

Is there anyway to restore that feature in your method?

Hi Eugen,
I have successfully generated the image with NADE model,but i don't konw how to generate the label of the image.I don't see relative functions in codes.Can i get the label by some functions?

forgive me if you've already made this possible.
I was looking at a small gif and I thought to myself it would be cool to stylize each picture in the gif.

I realized I could do this myself by saving the pictures individually and then applying the process one at a time. I remember you saying something about input images can be processed similar to how you did the multiple style images.

Forgive me if the feature is already there and I just don't know how to set it up properly. But if it's not there (or is) can you assist in either an explanation or save this as a pending feature?

It would be pretty cool to redo gif's using this.

Hi Eugen,

I've followed the readme and successfully train imageGPT model by reproduce, but I have no idea how to generate an image with the trained model. I tend to train the model by 1, 2, 3, ... epochs and generate images for each stage to see what happens.

I checked the following code, but also have no idea what sample_fn should I assigned (it is None by default). Would you please add a simple example in the readme?

Style transfer notebook won't open in colab

Notebook not found
There was an error loading this notebook. Ensure that the file is accessible and try again.
Ensure that you have permission to view this notebook in GitHub and authorize Colaboratory to use the GitHub API.

Also, manually loading the notebook locally or on colab fails in the secondcell

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-15-7857ada21f3b> in <module>
      2 
      3 import kornia
----> 4 from nn_hallucinations import colab_utils
      5 import torch
      6 from torch import nn

ImportError: cannot import name 'colab_utils' from 'nn_hallucinations' (unknown location)

Notebooks are pretty prone to getting out of date and breaking (e.g. #13). We should create some lightweight tests for notebooks which have no canonical implementations in the library (currently style_transfer.py, and cppn.py).

Thank you for sharing good codes!
and Sorry for asking you about the original paper, but I can not find where to ask. 😭

I am trying to understand the Pixel CNN and I don't know how the mode generates images from zero tensor input.
Most of the codes feed the zero tensor into the models for generations.

I think,
If zero input comes into the model, and model may compute only the bias values in layers ( 0 tensor + bias ).
And results of the model are the same because the same bias values are computed every try for generating an image.
So the model generates the same image each tries to generate.

But the model looks like generates a different image on every try.
How does the model generate an image from zero tensor input? Is there are any noise values or random sampling functions thing?

In the reproduce function of VQ-VAE-2, you use VectorQuantizedVAE rather than VectorQuantizedVAE2.

Hi,

I've just noticed a typo in your username in README:

To figure out:

• Should we log the average train loss (or maybe a moving average)?
• What output should be logged (e.g. epoch, percent done, train_loss, eval_loss, batches/s, anything else we need)?

Hi Eugen,
I want to use multiple GPUs to train the NADE model at the same time. I'm trying to write device_id=[0,1,2,3] or device_id =(0,1,2,3) in trainer.Trainer( ) ,but error message appears,I don't know how to write it correctly.

Hi!

I try to load the ImageNet dataset to the PixelSnail Model.

after fixing this lines in the pixel_snail.py scripts :

 `class PixelSNAIL(base.AutoregressiveModel):
  """The PixelSNAIL model.
  Unlike [1], we implement skip connections from each block to the output.
  We find that this makes training a lot more stable and allows for much faster
  convergence.
  """
  def __init__(self, 
               in_channels=3 ####### <------ I changed this, 
               out_dim=1,
               probs_fn=torch.sigmoid,
               sample_fn=lambda x: distributions.Bernoulli(probs=x).sample(),
               n_channels=64,
               n_pixel_snail_blocks=8,
               n_residual_blocks=2,
               attention_key_channels=4,
               attention_value_channels=32,
               head_channels=1):
    """Initializes a new PixelSNAIL instance.`

I still get this error :

RuntimeError: Given groups=1, weight of size [64, 1, 3, 3], expected input[128, 3, 64, 64] to have 1 channels, but got 3 channels instead

Am I missing something? should I do anything else when trying to load 3 channels images?

Thanks,
Eyal

Could you please let me know, how can I replicate NLL results in the paper. For how many epochs should I train.
Could you please give me script to generate an image using the trained model?

Readme is out of date and not very informative.

Hey,

I wrapped your implementation of ImageGPT for a project I am doing in collab.
I have single channel images of size 64 with the intention of scaling it to 128.
Running you code crashes (out of memory) even if I set batch_size to 2.
It runs only with batch_size=1 and it doesn't use that much of memory around 3GB, while being extremely slow 60 hours for ~200Kpictures (20 s per picture).
Is this normal or is there something wrong?

It will probably make experiment tracking a lot simpler.

We should still keep around TensorBoard logging for people who don't want to use W&B.

Everything is manually tested right now... not great.

Currently, the only way to measure model quality during training is by monitoring the evaluation loss. However, a low loss does not always correspond to high quality generation (e.g. see https://arxiv.org/abs/1511.01844). Furthermore, for autoregressive models, if the model is not correctly masked, it will get very low loss by peeking into the future, but the generation quality will be extremely bad.

In #16, one suggestion was to periodically sample from the model as it trains (e.g. every 10 epochs) and display these samples using TensorBoard. We should be able to easily support this in the Trainer.

This seems trivial and might be something I get to myself. But having slider bars for various options would be very nice.

Hi

I trained PixelSnail on CIFAR10 and try to sample from the model with this function that I wrote (based on the code from this project notebook) :

        print("Sample")
        
        if self._epoch % 10 == 0 :
            print("Epoch Number: " + str(self._epoch))
            print("sampling")
            curr_path = 'sample_from_epoch_' + str(self._epoch) + '.png'
            print(curr_path)
            sampleTensor=self._model.sample((10, 3, 32, 32))
            sampleTensor=sampleTensor.cpu()
            cu.imsave(sampleTensor, figsize=(50, 5),filename = curr_path)
            
      self._summary_writer.close()

Where cu.imsave is :

def imsave(batch_or_tensor, title=None, figsize=None, filename="sample.png"):
  """Renders tensors as an image using Matplotlib.
  Args:
    batch_or_tensor: A batch or single tensor to render as images. If the batch
      size > 1, the tensors are flattened into a horizontal strip before being
      rendered.
    title: The title for the rendered image. Passed to Matplotlib.
    figsize: The size (in inches) for the image. Passed to Matplotlib.
  """
  batch = batch_or_tensor
  for _ in range(4 - batch.ndim):
    batch = batch.unsqueeze(0)
  n, c, h, w = batch.shape
  tensor = batch.permute(1, 2, 0, 3).reshape(c, h, -1)
  image = _IMAGE_UNLOADER(tensor)

  plt.figure(figsize=figsize)
  plt.title(title)
  plt.axis('off')
  plt.imsave(filename,image)

For some reason the output after 90 epochs looks bad :

am I missing anything?

In the style_transfer notebook, there is a size argument in the colab_utils.load_image function that resizes the images into squares (i.e., makes the dimensionality of the length and width the same). This leads to the outputs having square dimensions and I'd rather avoid this, but when I remove this argument and execute the run_style_transfer function, I get the following error:

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:32: UserWarning: Using a target size (torch.Size([1, 256, 256, 161])) that is different to the input size (torch.Size([1, 256, 311, 256])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-64-2ce60ed3a286> in <module>()
     23     content_weight=1.,
     24     style_weight=1000.,
---> 25     log_steps=250)

7 frames
/usr/local/lib/python3.7/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     72     if has_torch_function(tensors):
     73         return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 74     return _VF.broadcast_tensors(tensors)  # type: ignore
     75 
     76 

RuntimeError: The size of tensor a (256) must match the size of tensor b (161) at non-singleton dimension 3

I'm wondering if there is a way to run the style transfer (which is of excellent quality) without resizing the content and style images into square dimensions.

Right now, everything is implemented in colab notebooks. While great for early implementation/experimentation, this is not very maintainable/reusable.

Hi!

Is there a way to use the sample function that appears in the notebook within the py scripts / add sampling option to the tensorboard logs?

I tried to convert the imshow to imsave and run it after finish training, but the output image is currrpted for some reason.

BTW - this project is great! thank you for this.

Regards.

Most Colab notebooks provide the same exact implementation of different models as the code does. They also quickly fall out of date and become broken.

It's probably a good idea to just have one canonical tutorial notebook which can be used to train all our models.

However, some of the notebooks (such as Neural Style Transfer and CPPN) need to be kept around.

Epochs is dependent on the dataset size. This complicates things like how long to train for, how often to evaluate, how often to log, etc. Currently we evaluate/log at the end of each epoch which is not ideal for large datasets. While the logging/evaluation is not strictly tied to using epochs, and we could support both, things might get pretty messy.

https://github.com/EugenHotaj/pytorch-generative/blob/master/pytorch_generative/nn/attention.py

Hi! I was wondering why the shape of the positional encoding is what it is.

I thought 2 channels were enough, one channel for the encoding in the first dimension, and one channel for the encoding in the second dimension. Now, the positional encoding has C duplicates of each.

Thanks!

Right now, the Trainer object creates logs after every step (although logs get flushed every 100 steps). This (probably) slows down our training. We need to first measure if this is the case (e.g. turn off all logs, see how fast the model trains), and tune logging rate accordingly.

Hi!

In trainer.py the defult settings are for device=torch.device('cpu')).

when I change it to 'cuda' I get this error :

RuntimeError: All input tensors must be on the same device. Received cpu and cuda:0

Do you know what should I do in order to fix this?

Thanks!

Hi,
When I use KDE I get a density whose integral is not 1.

import torch
from pytorch_generative.models import KernelDensityEstimator

# Random 2D data
data = torch.normal(torch.zeros((100, 2)), torch.ones((100, 2)))
estimator = KernelDensityEstimator(data)

# Build 
# [[-8.0, -8.0], [-8.0, -7.9], [-8.0, -7.8], ..., [-8.0,  7.9],
#  [-7.9, -8.0], [-7.9, -7.9], [-7.9, -7.8], ..., [-7.9,  7.9],
#   ...
#  [ 7.9, -8.0], [ 7.9, -7.9], [ 7.9, -7.8], ..., [ 7.9,  7.9]]
dx = 0.1
X = torch.arange(-8, 8, dx)
Y = torch.arange(-8, 8, dx)
xx, yy = torch.meshgrid(X, Y)
a = torch.stack((xx, yy), axis=2).view(-1, 2)

probs = estimator(a)
print(torch.sum(probs*dx**2)) # -> tensor(0.1251)

Am I missing something? When I do the same thing with scikit, I don't have this task:

from sklearn.neighbors import KernelDensity

# Random 2D data
data = np.random.normal(np.zeros((100, 2)), np.ones((100, 2)))
kde = KernelDensity(bandwidth=1.0, kernel="gaussian")
kde.fit(data)

# Build 
# [[-8.0, -8.0], [-8.0, -7.9], [-8.0, -7.8], ..., [-8.0,  7.9],
#  [-7.9, -8.0], [-7.9, -7.9], [-7.9, -7.8], ..., [-7.9,  7.9],
#   ...
#  [ 7.9, -8.0], [ 7.9, -7.9], [ 7.9, -7.8], ..., [ 7.9,  7.9]]
dx = 0.1
X = np.arange(-8, 8, dx)
Y = np.arange(-8, 8, dx)
xx, yy = np.meshgrid(X, Y)
a = np.stack((xx, yy), axis=2).reshape(-1, 2)

probs = np.exp(kde.score_samples(a))
print(np.sum(probs*dx**2)) # -> 0.9999999970275351

This will also make it easy to create new trainers from colab without having to monkey patch colab_utils.

Some things we could add:

• [easy] Use down/right shifted 3x3 conv instead of a right shifted 1x3 conv in the horizontal stack.
• [medium] Use a U-Net architecture with skip connections (this has shown improvement in some early experiments)
• [medium] [optional] Use gated ResNet layers in both the vertical and horizontal stacks instead of the adhoc GatedPixelCNNLayer.
• [hard] [optional] Implement the discretized logistic mixture for non-binary outputs