AssertionError when reproduction of Dlinear model about gluonts HOT 18 OPEN

@hanlaoshi dlinear is an inherently univariate model so there is no need to group the time series etc. just pass the time series individually and predict individually like other univariate models....

from gluonts.

oh i see now you are not using those imports... so the issue seems to be in the dataset preparation helpers you have rather than dlinear... can you confirm that you get the same error with another model like DeepAR?

from gluonts.

@hanlaoshi dlinear is an inherently univariate model so there is no need to group the time series etc. just pass the time series individually and predict individually like other univariate models....

Thank you for getting back to me so quickly. So, the patch_TST and lag_TST are also univariate models, right?

from gluonts.

yes they are... the issue is in the split function i see so even before the model is called... i will try to debug

from gluonts.

oh i see now you are not using those imports... so the issue seems to be in the dataset preparation helpers you have rather than dlinear... can you confirm that you get the same error with another model like DeepAR?

Actually, I solved the problem by rewriting the train_data and test_data code. The modified code is provided below.:

train_data = ListDataset(
    [
        {
            "start": df['date'][0],
            "target": df.iloc[:-pre_len*num_test_dates,1:].values.T,
        }
    ],
    freq="H",
    one_dim_target=False
)
test_data = ListDataset(
    [
        {
            "start": df['date'][0],
            "target": df.iloc[:-pre_len*(num_test_dates-1-i),1:].values.T if i < num_test_dates-1 else df.iloc[:,1:].values.T,#type: ignore
        }
        for i in range(num_test_dates)
    ],
    freq="H",
    one_dim_target=False
)

But I got the other error below (maybe this error arose because the model is inherently univariate)::
Traceback (most recent call last):
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/newdisk/ai/.vscode-server/extensions/ms-python.python-2023.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/home/newdisk/ai/Project/pytorch-ts-master/examples/2022-9-multi_flow/CoST-main/dlinear_test.py", line 114, in
model = estimator.train(train_data, num_workers=0)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/model/estimator.py", line 248, in train
return self.train_model(
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/model/estimator.py", line 211, in train_model
trainer.fit(
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 545, in fit
call._call_and_handle_interrupt(
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 581, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 990, in _run
results = self._run_stage()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/trainer.py", line 1036, in _run_stage
self.fit_loop.run()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/fit_loop.py", line 202, in run
self.advance()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 136, in run
self.advance(data_fetcher)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 240, in advance
batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 187, in run
self._optimizer_step(batch_idx, closure)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 265, in _optimizer_step
call._call_lightning_module_hook(
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/call.py", line 157, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/core/module.py", line 1282, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/core/optimizer.py", line 151, in step
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/strategies/strategy.py", line 230, in optimizer_step
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/plugins/precision/precision_plugin.py", line 117, in optimizer_step
return optimizer.step(closure=closure, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad
ret = func(self, *args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/optim/adam.py", line 183, in step
loss = closure()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/plugins/precision/precision_plugin.py", line 104, in _wrap_closure
closure_result = closure()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 140, in call
self._result = self.closure(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 126, in closure
step_output = self._step_fn()
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 315, in _training_step
training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/lightning/pytorch/strategies/strategy.py", line 382, in training_step
return self.lightning_module.training_step(*args, **kwargs)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/model/d_linear/lightning_module.py", line 86, in training_step
train_loss = self._compute_loss(batch)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/model/d_linear/lightning_module.py", line 76, in _compute_loss
distr = self.model.distr_output.distribution(distr_args, loc, scale)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/distributions/distribution_output.py", line 141, in distribution
return AffineTransformed(distr, loc=loc, scale=scale)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/gluonts/torch/distributions/affine_transformed.py", line 42, in init
super().init(
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/distributions/transformed_distribution.py", line 64, in init
shape = transform.forward_shape(base_shape)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/distributions/transforms.py", line 372, in forward_shape
shape = part.forward_shape(shape)
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/distributions/transforms.py", line 772, in forward_shape
return torch.broadcast_shapes(shape,
File "/home/newdisk/ai/anaconda3/envs/aug/lib/python3.8/site-packages/torch/functional.py", line 126, in broadcast_shapes
raise RuntimeError("Shape mismatch: objects cannot be broadcast to a single shape")
RuntimeError: Shape mismatch: objects cannot be broadcast to a single shape

By the way, If I want to change Dlinear to a multivariate model, where should I make modifications?

from gluonts.

it doesnt make sense for dlinear to be multivariate, you would need to rehape the context_length, multivariate_dim tensor to context_length*multivariate_dim vector and then the nn.linear would be potentially huge:

nn.linear(context_length*multivariate_dim , prediction_length*multivariate_dim)

also this is just to learn the representation, on the emission side you would then need a way to model the joint multivariate_dim distribution for each of the prediction_length time points...

from gluonts.

it doesnt make sense for dlinear to be multivariate, you would need to rehape the context_length, multivariate_dim tensor to context_length*multivariate_dim vector and then the nn.linear would be potentially huge:

nn.linear(context_length*multivariate_dim , prediction_length*multivariate_dim)

also this is just to learn the representation, on the emission side you would then need a way to model the joint multivariate_dim distribution for each of the prediction_length time points...

Thank you for generously addressing my question. I would like to kindly ask if you could assist me once again by helping me correct the code I have written for mean_wQuantileLoss and m_sum_mean_wQuantileLoss. The code is provided below:

def quantile_loss(y_true, y_pred, q):
    loss = 2 * np.sum(np.abs((y_pred - y_true) * ((y_true <= y_pred) - q)))
    return loss 

def mean_wQuantileLoss(forecasts, targets, quantiles=(np.arange(20)/20.0)[1:]):
    B, T, D = targets.shape
    loss = 0.0
    for i in range(B):
        for d in range(D):
            for q in quantiles:
                loss += quantile_loss(targets[i, :, d], forecasts[i, :, d], q) / np.sum(np.abs(targets))
    return loss / (B * D * len(quantiles))

def m_sum_mean_wQuantileLoss(forecasts, targets, quantiles=(np.arange(20)/20.0)[1:]):#
    B, T, D = targets.shape
    forecasts = np.sum(forecasts, axis=-1)
    targets = np.sum(targets, axis=-1)
    total_loss = 0.0
    for i in range(B):
        loss = 0.0
        for q in quantiles:
            loss += quantile_loss(targets[i], forecasts[i], q) / np.sum(np.abs(targets))
        total_loss += loss / len(quantiles)
    return total_loss / B

from gluonts.

so the evaluator returns all these metrics so just use those I would say instead of re-writing these functions

from gluonts.

ok have a look at the implementation here:

huggingface/evaluate#509

from gluonts.

also note that compraring univariate models against multivariate models on the smallish sized open datasets will always result in the univariate methods performing better due to the difficulty of not learning spurious correlations in the multivariate methods (which is why even a simple linear model like dlinear performs "better" than a multivariate version of transformer methods for example) and thus there is no insight in a model's performance when one does such a comparison...

from gluonts.

also note that compraring univariate models against multivariate models on the smallish sized open datasets will always result in the univariate methods performing better due to the difficulty of not learning spurious correlations in the multivariate methods (which is why even a simple linear model like dlinear performs "better" than a multivariate version of transformer methods for example) and thus there is no insight in a model's performance when one does such a comparison...

Thank you again for the insightful answer. After looking at the crps.py code, I still have some follow-up questions. Can the shapes of both predictions and references be (B,T,D)? But in the example on line 61 of crps.py, it seems like the shape of predictions is (B,T*D), while the shape of references should be (B,), is that right? I'd appreciate it if you could clarify the expected shapes of predictions and references for CRPS calculation. Please let me know if I misunderstood anything. Thanks!

from gluonts.

so Crps needs many prediction samples (so 1 more tensor dim) and one ground-truth sample so the shape of the prediction has an extra dim

from gluonts.

so Crps needs many prediction samples (so 1 more tensor dim) and one ground-truth sample so the shape of the prediction has an extra dim

You're right. I remember in your pytorch-ts conditional flow model, your approach was to repeat the first dimension of the tensor to be predicted num_parallel_samples times, so that the final predicted sample shape is (-1,num_parallel_samples,prediction_length,target_dim). At this point, the predictions tensor shape would be (B, num_parallel_samples, prediction_length, target_dim), while the references tensor shape would be (B, prediction_length, target_dim). Is my understanding of this correct?

from gluonts.

yes that is right

from gluonts.

yes that is right

I really appreciate you taking the time to explain the model to me clearly and concisely. It's impressive how quickly you understood my question and provided an accurate description of the shape manipulation. Your knowledgeable response demonstrates your strong grasp of the temporal forecasting concepts. Thanks for sharing your expertise - it is greatly helping my understanding of time series prediction models.

from gluonts.

oh not a problem at all! glad to help!

from gluonts.

oh not a problem at all! glad to help!

Hi， Dr. Kashif, Based on the CRPS calculation code you provided huggingface/evaluate/pull/509 , I have some uncertainties when reproducing it. My main confusion comes from the line "forecast_quantile = np.quantile(y_pred, q, axis=0)".

Assuming the shape of y_truth is (B,T,D) and the shape of y_pred is (B,100,T,D), after running "forecast_quantile = np.quantile(y_pred, q, axis=0)", the shape of forecast_quantile becomes (100,T,D). This shape does not match y_truth and would cause an error, or should we change the axis=1 to match the shape between forecast_quantile and y_truth?

Could you please help explain where the issue might be? I'm unsure if my understanding of the shapes is correct after taking the quantile along axis 0.

def quantile_loss(target: np.ndarray, forecast: np.ndarray, q: float) -> float:
        return 2.0 * np.sum(np.abs((target - forecast) * ((target <= forecast) - q)))
def _compute(
        self,
        predictions,
        references,
        quantiles=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
        sum=False,
        sum_axis=-1,
        multioutput="uniform_average",
    ):
        # if the number of dims of predictions > 2 then sum over sum_axis dimension if sum is True
        if sum and len(predictions.shape) > 1:
            predictions = np.sum(predictions, axis=sum_axis)
            references = np.sum(references, axis=sum_axis)
        abs_target_sum = np.sum(np.abs(references))
        weighted_quantile_loss = []
        for q in quantiles:
            forecast_quantile = np.quantile(predictions, q, axis=0)
            weighted_quantile_loss.append(
                self.quantile_loss(references, forecast_quantile, q) / abs_target_sum
            )
        if multioutput == "raw_values":
            return weighted_quantile_loss
        elif multioutput == "uniform_average":
            return np.average(weighted_quantile_loss)
        else:
            raise ValueError(
                "The multioutput parameter should be one of the following: "
                + "'raw_values', 'uniform_average'"
            )

from gluonts.

sorry for the delay... in the DLinear setting there should not be a D dimensional tensor and also the CRPS is a univariate metric

from gluonts.