GenericBlock Module Issues about n-beats HOT 8 CLOSED

Ad 1. I came here to write the same thing, I also looked at the implementation by the original authors and they do not have ReLu there. Ad 2, I believe there should be bias. The paper is self-contradictory: In 3.3, the authors describe using bias:

This is also reflected in the original authors' code (see: repo by ElementAI). Caution: that repo has been created by the original authors, but is probably not the code they initially wrote, as it's PyTorch and in the paper they talk about using TensorFlow.
In case of the GenericBlock, theta=Linear(hidden_4) and output = Linear_with_bias(theta), as described in the paper if you consided 3.3 is equivalent to output=Linear_with_bias(hidden_4), as in the ElementAI implementation.
Additionally, the ElementAI implementation uses linear with bias to obtain theta for all block types - this is not explained in the paper at all, so I have no idea which way that one is supposed to be.

from n-beats.

@chedatomasz I'm pretty sure the $g^{b}$ and $g^{f}$ blocks should not have a bias. In order for $g^{b}$ and $g^{f}$ to be linear projections, as stated by the authors, the bias needs to be 0: the operation $V^{f}\theta^{f}$ can be viewed as a linear combination where the columns of $V^{f}$ serve as basis vectors and the columns of $\theta^{f}$ are the coefficients.

I'm pretty convinced that there are bugs in both this implementation and the ElementAI implementation of the trend and seasonality blocks. I don't think they construct the weight matrices as described in the paper.

from n-beats.

I agree that if $g^{b}$ and $g^{f}$ are to truly be linear projections, the bias would need to be zero. And I agree that on page three they call it a linear projection, and provide equations for a linear projection. But on page four, in the fragment I quoted, they define it again, this time as fully connected with bias.

Regarding forming the matrices: For generic basis, the matrices are present as parameters of the Linear layers, hidden by the framework. For seasonality and trend, the matrices are formed explicitly in seasonality_model and trend_model

from n-beats.

Based on the implementation in this repo, the following are the backcast and forecast basis vectors given a backcast length of 30, a forecast length of 5, and a theta dimension of 4:

The original paper does not say how $g^{b}$ is constructed for the trend or seasonality blocks. The description is only for the $g^{f}$ block. It turns out that the forecast vectors in the $g_{f}$ trend block form a basis for all $n=\theta-1$ degree polynomials. This is why I believe the backcast vectors should be constructed in the same way, and they should be as follows:

from n-beats.

Regarding this one, I just added another issue. I believe this reflects the code in ElementAI's implementation, where they have flipped time axis for backcast. I will email the original authors to confirm what their intentions were.

from n-beats.

Point 1. fixed: 0e44180. The keras impl did not seem to have this issue.
Point 2. so we should leave the linear with bias=True?
Point 3. cf. the other issue raised.

from n-beats.

Point3: be40395

from n-beats.

I'll close this. Let me know if it LGTY!

from n-beats.