sheoyon-jhin / ancde Goto Github PK

Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting

Python 100.00%

time-series deep-learning icdm2021

ancde's Introduction

Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting
for Time-Series Classification and Forecasting
[ICDM 2021]

Neural networks inspired by differential equations have proliferated for the past several years. Neural ordinary differential equations (NODEs) and neural controlled differential equations (NCDEs) are two representative examples of them. In theory, NCDEs provide better representation learning capability for time-series data than NODEs. In particular, it is known that NCDEs are suitable for processing irregular time-series data. Whereas NODEs have been successfully extended after adopting attention, however, it had not been studied yet how to integrate attention into NCDEs. To this end, we present the method of Attentive Neural Controlled Differential Equations (ANCDEs) for time-series classification and forecasting, where dual NCDEs are used: one for generating attention values, and the other for evolving hidden vectors for a downstream machine learning task. We conduct experiments with three real-world time-series datasets and 10 baselines. After dropping some values, we also conduct irregular time-series experiments. Our method consistently shows the best accuracy in all cases by non-trivial margins. Our visualizations also show that the presented attention mechanism works as intended by focusing on crucial information.

The overall architecture of NCDE

Our proposed attentive NCDE In our method, the bottom NCDE produces attention values and the top NCDE produces the last hidden vector.

The detailed architecture of our proposed attentive NCDE. From the raw time-series data x(t), the continuous path X(t) is created by the natural cubic spline algorithm. The bottom NCDE reads X(t) to produce attention values. The path Y(t) is defined as the (element-wise) multiplication of X(t) and the attention value at time t. The top NCDE produces the last hidden vector with Y(t). Our framework supports both classification and regression.

The soft attention visualization in physioNet Sepsis. Blue means strong attention and white means weak attention. Note that the bottom NCDE gives strong attention when there are non-trivial changes on the curves.

The soft attention visualization in Google Stock. Blue means strong attention and white means weak attention. Note that the open price has weak attention always. This is because it shows high correlations to other values. Our attention model strategically ignores the open price.

create conda environments

conda env create --file ancde.yml

activate conda

conda activate neuralcde

UEA_CharacterTrajectories

python3 uea_attentive.py --seed 2021 --missing_rate 0.3 --model="ancde" --h_channel 40 --hh_channel 40 --layer 3 --lr 0.001 --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 10 --step_mode 'valloss'

PhysioNet Sepsis No OI

python3 sepsis_attentive.py --seed 2021 --intensity '' --model="ancde" --h_channel 49 --hh_channel 49 --layer 4 --lr 0.00001  --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 20 --step_mode 'valloss'

PhysioNet Sepsis OI

python3 sepsis_attentive.py --seed 2021 --intensity 'True' --model="ancde" --h_channel 49 --hh_channel 49 --layer 4 --lr 0.00001  --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 20 --step_mode 'valloss'

Google Stock(single-step forecasting)

python3 stock.py --seed 2021 --input_seq 24 --output_seq 1 --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Google Stock(multi-step forecasting)

python3 stock.py --seed 2021 --input_seq 20 --output_seq 5 --model="ancde_forecasting" --h_channel 15 --hh_channel 15 --layer 2 --lr 0.005 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Mujoco

python3 mujoco.py --seed 2021 --input_seq 20 --output_seq 5  --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

Run shell file (ex PhysioNet Sepsis dataset)

sh sepsis.sh

Run shell file, background (ex PhysioNet Sepsis dataset)

nohup sh sepsis.sh 2>sepsis.err 1>sepsis.log &

Citation

@article{jhin2021attentive,
  title={Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting},
  author={Jhin, Sheo Yon and Shin, Heejoo and Hong, Seoyoung and Jo, Minju and Park, Solhee and Park, Noseong},
  journal={ICDM},
  year={2021}
}

ancde's People

Contributors

Stargazers

Watchers

Forkers

freekang tbhacker karlheinzniebuhr trellixvulnteam hit636 luciferjason

ancde's Issues

Dependency issue

While running this command, a dependency exception occurred:
conda env create --file ancde.yml

The conflict is caused by:
    The user requested tensorboard==2.2.1
    tensorflow 2.4.1 depends on tensorboard~=2.4

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict


Pip subprocess error:
ERROR: Cannot install -r /home/ubuntu/ANCDE/condaenv.wor7pxna.requirements.txt (line 74) and tensorboard==2.2.1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

failed

No module named 'sktime.utils.load_data'

Hello, when running this example:
python3 stock.py --seed 2021 --input_seq 24 --output_seq 1 --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

I get this error:
Traceback (most recent call last): File "stock.py", line 2, in <module> import datasets File "/home/ubuntu/ANCDE/experiments/datasets/__init__.py", line 3, in <module> from . import uea File "/home/ubuntu/ANCDE/experiments/datasets/uea.py", line 6, in <module> import sktime.utils.load_data ModuleNotFoundError: No module named 'sktime.utils.load_data'

running Ubuntu 20.04.4 LTS

Please assist thank you

Time augment for Google Stock forecasting experiment

Hi, first of all thanks for introducing this fabulous work!
I have a question regarding the _process_data inside the stock.py.
As shown in https://github.com/patrick-kidger/NeuralCDE/blob/master/example/example.py, shouldn't we augment time before the features of the data?
The final data before computing the coefficients for the natural cubic spline interpolation is shaped
X : [3661,24,6]
y : [3661,1,6] (when assuming single-step forecast of lag 24 and horizon 1)
The last dimension '6' means {Open, High, Low, Close, Adj_Close, Volume} of the stock_data.csv.
I think the last dimension should be 7 including {time, Open, High, Low, Close, Adj_Close, Volume}.
If I am misunderstanding something about the NCDE implementation, please feel free to correct me.
Thanks again!

Too many indices for tensors of dimension 2

When I run 'python3 sepsis_attentive.py --seed 2021 --intensity 'True' --model="ancde" --h_channel 49 --hh_channel 49 --layer 4 --lr 0.00001 --soft 'True' --slope_check '' --timewise 'True' --attention_channel 20 --attention_attention_channel 20 --step_mode 'valloss'', I got this error, the tenser has only two dimensions, but the index is three-dimensional

After I change the index dimension, the following error occurs：

Also, there is no module named 'odesolve'

Use on custom irregular time series

Thanks for your sharing of this great work.

I have an irregular multivariant time series. Intervals of observations are very irregular. I want to make a multi step forecasting model.
I tried to use some imputation methods for missing values, but the accuracy was very poor.
Finally I have found that your approach could be helpful to my case.

I think my dataset is very similar to your google stock dataset.

To use your model for my dataset, what steps should I follow?
Is it possible to use your model for multi-step forecasting? If then, how can I make it?

Thanks in advance.

FileNotFoundError on single step forecast command

Running single step forecasting gives me this error

python3 stock.py --seed 2021 --input_seq 24 --output_seq 1 --model="ancde_forecasting" --h_channel 12 --hh_channel 12 --layer 2 --lr 0.001 --soft '' --slope_check 'True' --timewise '' --attention_channel 4 --attention_attention_channel 8 --step_mode 'valloss'

in X.shape torch.Size([3661, 24, 6])
6

parameter of Neural CDE 1751

parameter of ORIGINAL ODE FUNC 1248

parameter of ATTENTION ODE FUNC 292
valloss
Starting training for model:

ANCDE_forecasting(
  input_channels=6, hidden_channels=12, output_channels=1, initial=True
  (func_f): FinalTanh_ff6(
    input_channels: 6, hidden_channels: 4, hidden_hidden_channels: 8, num_hidden_layers: 2
    (linear_in): Linear(in_features=6, out_features=8, bias=True)
    (linear_test): Linear(in_features=8, out_features=4, bias=True)
    (linears): ModuleList(
      (0): Linear(in_features=4, out_features=4, bias=True)
    )
    (linear_out): Linear(in_features=4, out_features=36, bias=True)
  )
  (func_g): FinalTanh(
    input_channels: 6, hidden_channels: 12, hidden_hidden_channels: 12, num_hidden_layers: 2
    (linear_in): Linear(in_features=12, out_features=12, bias=True)
    (linears): ModuleList(
      (0): Linear(in_features=12, out_features=12, bias=True)
    )
    (linear_out): Linear(in_features=12, out_features=72, bias=True)
  )
  (STE): Hardsigmoid(
    (act): Hardtanh(min_val=-1.0, max_val=1.0)
  )
  (initial_network): Linear(in_features=6, out_features=6, bias=True)
  (feature_extractor): Linear(in_features=6, out_features=12, bias=True)
  (linear): Linear(in_features=12, out_features=6, bias=True)
  (time_attention): Linear(in_features=6, out_features=1, bias=True)
)


/usr/local/envs/neuralcde/lib/python3.7/site-packages/torch/cuda/memory.py:234: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  FutureWarning)
  0%|          | 0/300 [00:00<?, ?it/s]Traceback (most recent call last):
  File "stock.py", line 170, in <module>
    main()
  File "stock.py", line 126, in main
    step_mode=step_mode,
  File "/content/ANCDE/experiments/common.py", line 800, in main_forecasting
    step_mode,
  File "/content/ANCDE/experiments/common.py", line 275, in _train_loop_forecasting
    times, train_coeffs, lengths, slope, stream=True, **kwargs
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/ANCDE/experiments/models/metamodel.py", line 647, in forward
    **kwargs)
  File "/content/ANCDE/experiments/models/../../controldiffeq/cdeint_module.py", line 239, in ancde_bottom
    out = odeint(func=vector_field, y0=z0, t=t, **kwargs)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/adjoint.py", line 129, in odeint_adjoint
    ys = OdeintAdjointMethod.apply(*y0, func, t, flat_params, rtol, atol, method, options)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/adjoint.py", line 18, in forward
    ans = odeint(func, y0, t, rtol=rtol, atol=atol, method=method, options=options)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/odeint.py", line 76, in odeint
    solution = solver.integrate(t)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/solvers.py", line 91, in integrate
    dy = self.step_func(self.func, t0, t1 - t0, y0)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/fixed_grid.py", line 29, in step_func
    return rk_common.rk4_alt_step_func(func, t, dt, y)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/rk_common.py", line 77, in rk4_alt_step_func
    k4 = func(t + dt, tuple(y_ + dt * (k1_ - k2_ + k3_) for y_, k1_, k2_, k3_ in zip(y, k1, k2, k3)))
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/torchdiffeq/_impl/adjoint.py", line 122, in forward
    return (self.base_func(t, y[0]),)
  File "/content/ANCDE/experiments/models/../../controldiffeq/cdeint_module.py", line 70, in __call__
    np.save(self.file, self.h_prime_list.cpu().detach().numpy())
  File "<__array_function__ internals>", line 6, in save
  File "/usr/local/envs/neuralcde/lib/python3.7/site-packages/numpy/lib/npyio.py", line 524, in save
    file_ctx = open(file, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/content/ANCDE/experiments/Google_h_prime/23326.npy'
  0%|          | 0/300 [00:00<?, ?it/s]