borgwang / tinynn Goto Github PK

是从知乎上看到的，非常喜欢也这个工程，觉得这个工程对初学者太友好了，比直接使用大深度学习框架，能了解更多的细节。

看了一些transformer 的资料，还是不太理解它是怎么实现的，在网上找的transformer 实现都是基于pytorch 和 tf实现的，没有像这样从头开始实现的。

It seems to me the optimizer methods, given SGD as example, use sum of gradients from a batch to multiply learning rate directly:

def _compute_step(self, grad):
    return - self.lr * grad

It is suggested we should average the grad by batch size, the benefits of doing this is listed in this post. Basically you do not have to adjust learning rate when changing batch size.

If you agree to this, I would create a pull request to add option to use mean gradients and at the same time provide compatibility to use simply sum of gradients (for efficiency consideration).

should SOFTPLUS be Softplus(x)= log(1+exp(x)) ？ The code looks like Softplux(x) = log(1+exp(-x)) + max（x,0.0）.I tried to compute them.And they are equivalent. But why should we use this more complex way?

The MNIST example is like torchvision, it download the dataset programmatically when the data file is not in specific directory. I personally had very bad experience using torchvision because it does not use CDN link and no progress bar is given, in China, it is painfully slow. Moreover, it hard-code the link in the code and add maintenance cost. I think tinynn is doing the same thing.

I imagine if I want to contribute another example, according to the convention, I will need to import urllib and write try except and mkdir logic in my code just to help user download the dataset. Why not make it old-fashion and simpler to provide list of download links in text to make user and contributor life easier?

I love the cleanness of tinynn, I think I can learn a lot from this code. But I suggest to keep the example code small and instead provide documentation or text description to ask the user to download dataset themselves with a list of links (official link, Baidu Yun, etc.).

I think it is a very outdated practice to add information such as author, date and filename into file header. These are tracked by git already (in much better way), and they only add maintenance cost. Just a suggestion, for later new files to be created.

I come across this suggestion because it makes me hesitate whether I should modify your header when I change a minor thing in the code. And it will also cause confusion for potential contributors of this project (if you want more people to participate).

If you agree, I can help to remove those fields in current source code.

Why only one layer of dense net without any activation will have over 80% test accuracy after the first epoch?

I simply comment other layers in dense net, and use dense net to train.

        net = Net([
            # Dense(200),
            # ReLU(),
            Dense(10),
            # ReLU(),
            # Dense(70),
            # ReLU(),
            # Dense(30),
            # ReLU(),
            # Dense(10)
        ])

Is this expected result?

大佬您好，在看到激活函数的时候有一点不清楚，为什么forward和backward要再增加两个函数fun和derivative_func？