question about Parameter+delta? about mamba HOT 6 OPEN

S4 code has a parameter dt that should be in the range of 1e-3 to 1e-1 (these are hyperparameters that you can change). In Mamba's case we want softplus(x @ weight + dt_bias) to be around that range. We can assume that x @ weight has zero mean at initialization, so we initialize dt_bias so that softplus(dt_bias) is in the range 1e-3 to 1e-1. Ofc this does not guarantee it will stay in this range as the model is trained, but only at initialization.

from mamba.

1. This adds a bias term to ensure $\Delta$ is the right magnitude. See previous SSM papers for details.
2. This is motivated in the paper; the low-rank factorization saves parameters and is a generalization of a "down projection followed by broadcast"
3. That's ultimately what happens in spirit

from mamba.

for question1: in my option, it looks like delta is the length of discretization, in order to get the dependent thing, it has to add + $S_\Delta$, but in the paper, it says
, why remove "Parameter+"，it's not the same to the code:dt = self.dt_proj.weight @ dt.t()

from mamba.

Parameter here is the dt_bias. Linear(x_t) means self.dt_proj.weight @ dt.t() + dt_bias. In the code we separately do self.dt_proj.weight @ dt.t() and then the dt_bias is added in a separate slep (in the (CUDA kernel).

from mamba.

Hello @tridao @albertfgu, I am very interested in this point as well. Could you please elaborate a bit more on how adding this learnable dt_bias to any input dependent $S_{\Delta}(x_t)$ can ensure the $\Delta$ is at the right magnitude? May I ask which previous SSM paper discussed this?

Much appreciated!

from mamba.

Thanks @tridao so much for your explanation!

from mamba.