Comments (7)
Yes, of course! As this is further/additional improvements to the model, it would be nice to have it as a optional/additional model choices.
from alias-free-gan-pytorch.
@MHRosenberg It should be possible, and the changes are quite simple:
First, make it so the self.affine_fourier
linear layer returns 6 values instead of 4, two for rotation, two for shift, and two for scale (initialize the ones for the scale to 1).
Next, in the FourierFeature
class you need to divide coord_h
and coord_w
by these new scale values, but first, make sure you don't normalize them, and second, you need to threshold them so they can't take a value lower than 1 (if they do, then you'll end up with frequencies above the cutoff and you'll have aliasing)
I did my own custom implementation of the FourierFeatures class based on the one from this repo (for personal purposes) here https://github.com/pabloppp/pytorch-tools/blob/master/torchtools/nn/fourier_features.py#L42 and I included the scale as an optional affine transformation, maybe you can use some of this code as a base.
The main issue that I found is that, for some reason, after training the scale was not homogeneous, meaning that if I scaled x1.5 on both axes, somehow the face ended up being longer, or wider, so I needed to maybe scale x1.5 on the height and x1.2 on the width or something like that. I also think that since when scaling the Fourier features you're changing the frequencies, in order for it to work seamlessly you'd need to also change the parameters of the AliasFree activation, to reflect those frequency changes, but I didn't try any of that yet. A 'hacky' solution is to train your GAN with images with a lot of RandomResidedCropping augmentation, so the model is forced to learn to properly make faces at different scales.
I would love to hear if you make any progress with this! 🙇
from alias-free-gan-pytorch.
Interesting! But I wonder it will compatible with antialiasing.
from alias-free-gan-pytorch.
Yeah, that could be a problem because by scaling the Fourier features you are altering the frequencies.
My thought was that, if we keep the frequencies below the cutoff value it should be fine, so I only allow scaling to values > 1 (so you can zoom in but not zoom out) by using a threshold.
The face does get distorted when zooming, but I don't think that is caused by aliasing, since I don't see wave/sticky effects. To me, this deformation seems to be caused by not applying properly the scaling, like if some frequencies were being scaled more than others, or idk...
pe.mp4
from alias-free-gan-pytorch.
I think achieving scale equivariance will be hard. (If it is possible) But definitely it will be very interesting & important development.
I think it will be nice if model could have options for frequency modulation.
from alias-free-gan-pytorch.
Even if we do not achieve equivariance, and we do have those weird deformations, allowing the model to modify the scale can be beneficial, in the same way as your current configuration gives the model freedom to modify the rotation, but it's not rotation equivariant, but having this freedom allows the model to be much more flexible, right?
For example, take a look at this interpolation, the model is clearly taking advantage of the scale to deform the face into the shape it wants, and the result is still pretty smooth!
pe.mp4
pe.mp4
from alias-free-gan-pytorch.
This last movie looks closer to what I'm aiming for. Is it possible to use that method to transition between entirely distinct faces? If so, I'd love to see what modifications to the code are necessary for that.
from alias-free-gan-pytorch.
Related Issues (20)
- Question on Generating Video Samples HOT 1
- Questions on Filter parameters HOT 4
- Training Speed HOT 1
- Required GPU Memory HOT 1
- GPU Scalability Bug (GPU 0 has 4x the vram of all others) HOT 2
- Internal Representations HOT 2
- Any tips or code for generating images and movies from the checkpoints? HOT 6
- Training on 1024 resolution HOT 5
- colab demo
- Jinc function HOT 1
- Artifacts HOT 3
- about input fourier features.
- Train script issue HOT 5
- any body can give a pretrained model?
- Issues regarding the LMDB dataset creating. HOT 1
- Are there still differences to the official implementation?
- Is `conv2d_gradfix` really incompatible with PyTorch > 1.8?
- About train 1024 img
- Augment set to true causes RuntimeError
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alias-free-gan-pytorch.