awslabs / gluonts Goto Github PK

Probabilistic time series modeling in Python

License: Apache License 2.0

Python 87.63% Shell 0.16% R 0.18% Jupyter Notebook 12.00% Dockerfile 0.01% Just 0.02%

time-series deep-learning forecasting neural-networks machine-learning time-series-prediction time-series-forecasting mxnet pytorch aws

gluonts's Introduction

GluonTS - Probabilistic Time Series Modeling in Python

📢 BREAKING NEWS: We released Chronos, a suite of pretrained models for zero-shot time series forecasting. Chronos can generate accurate probabilistic predictions for new time series not seen during training. Check it out here!

GluonTS is a Python package for probabilistic time series modeling, focusing on deep learning based models, based on PyTorch and MXNet.

Installation

GluonTS requires Python 3.7 or newer, and the easiest way to install it is via pip:

# install with support for torch models
pip install "gluonts[torch]"

# install with support for mxnet models
pip install "gluonts[mxnet]"

See the documentation for more info on how GluonTS can be installed.

Simple Example

To illustrate how to use GluonTS, we train a DeepAR-model and make predictions using the airpassengers dataset. The dataset consists of a single time series of monthly passenger numbers between 1949 and 1960. We train the model on the first nine years and make predictions for the remaining three years.

import pandas as pd
import matplotlib.pyplot as plt

from gluonts.dataset.pandas import PandasDataset
from gluonts.dataset.split import split
from gluonts.torch import DeepAREstimator

# Load data from a CSV file into a PandasDataset
df = pd.read_csv(
    "https://raw.githubusercontent.com/AileenNielsen/"
    "TimeSeriesAnalysisWithPython/master/data/AirPassengers.csv",
    index_col=0,
    parse_dates=True,
)
dataset = PandasDataset(df, target="#Passengers")

# Split the data for training and testing
training_data, test_gen = split(dataset, offset=-36)
test_data = test_gen.generate_instances(prediction_length=12, windows=3)

# Train the model and make predictions
model = DeepAREstimator(
    prediction_length=12, freq="M", trainer_kwargs={"max_epochs": 5}
).train(training_data)

forecasts = list(model.predict(test_data.input))

# Plot predictions
plt.plot(df["1954":], color="black")
for forecast in forecasts:
  forecast.plot()
plt.legend(["True values"], loc="upper left", fontsize="xx-large")
plt.show()

Note, the forecasts are displayed in terms of a probability distribution and the shaded areas represent the 50% and 90% prediction intervals.

Contributing

If you wish to contribute to the project, please refer to our contribution guidelines.

Citing

If you use GluonTS in a scientific publication, we encourage you to add the following references to the related papers, in addition to any model-specific references that are relevant for your work:

@article{gluonts_jmlr,
  author  = {Alexander Alexandrov and Konstantinos Benidis and Michael Bohlke-Schneider
    and Valentin Flunkert and Jan Gasthaus and Tim Januschowski and Danielle C. Maddix
    and Syama Rangapuram and David Salinas and Jasper Schulz and Lorenzo Stella and
    Ali Caner Türkmen and Yuyang Wang},
  title   = {{GluonTS: Probabilistic and Neural Time Series Modeling in Python}},
  journal = {Journal of Machine Learning Research},
  year    = {2020},
  volume  = {21},
  number  = {116},
  pages   = {1-6},
  url     = {http://jmlr.org/papers/v21/19-820.html}
}

@article{gluonts_arxiv,
  author  = {Alexandrov, A. and Benidis, K. and Bohlke-Schneider, M. and
    Flunkert, V. and Gasthaus, J. and Januschowski, T. and Maddix, D. C.
    and Rangapuram, S. and Salinas, D. and Schulz, J. and Stella, L. and
    Türkmen, A. C. and Wang, Y.},
  title   = {{GluonTS: Probabilistic Time Series Modeling in Python}},
  journal = {arXiv preprint arXiv:1906.05264},
  year    = {2019}
}

gluonts's People

Contributors

Stargazers

Watchers

Forkers

hunglethanh9 benidis lostella finvis zwbjtu123 jgasthaus valeman aalexandrov zeyefkey trendingtechnology arsa-nik miaolin rubenszimbres mdastmalchi linhr000 geoalgo burakyilmaz321 vafl neveroldmilk sebastiangj gridl szha mlperf-deepts maxmxf harshvardhan1996 jccarrigan michaelwinnicki shayan-taheri jaheba goriccardo cliu0816 kuperov kallooa nguyenkaos rlucas7 qrebjock vdeuschle mbohlkeschneider strong-roots-capital rmedico cyrusmvahid canerturkmen xyhuang rshyamsundar dvhieu 17801026076 warrenlu happyf bhanuprakash212 heoa idabu tklein23 satarii forecasts-and-nn sahand68 leofrota dxcv nlosse nllosse lennonmwy abiraja2004 happy-face hanifmahboobi zibagandomkar robertren1996 kangking2019 lovvge blindedshooter wh-forker gseneviratna-zz liuliuforph limu1928 tonylibing xf05888 yuanzhaoyz jiacli holacola1985 supporon antoine-schwartz lawrencechen0921 smadalin happiness20 uuuuuext dotgc sdforce lxw4939 papirapi anguoyang syyunn edgbr dcmaddix miyainnyc fpli-mbr statmixedml yejiachen forever211 tdenimal guilherme-guilherme litieops aaronspieler

gluonts's Issues

Flaky distribution inference tests

We used a TOL = 0.2 which caused tests to sometimes fail. The new value is 0.3 which is arguably way too high.

We should think about a better strategy dealing with these flaky tests.

cc @jgasthaus

Provide SageMaker compatible docker container

Description

We should think about creating a GluonTS container, which can be used in Amazon SageMaker. It could behave similar to SageMaker DeepAR w.r.t data-loading (train and test channels) and evaluation of models.

Having these containers could make it a lot easier for customers to try out GluonTS models or even use them in a production setting.

This would require two things:

implementing a “shell” which can interacts with the SageMaker system
building and distribution of the docker images

References

SageMaker DeepAR

Transformation result keys names conflict

Hi,

I am having some issues with transformations

train_tf = transformation(iter(train_ds), is_train=True)

train = next(iter(train_tf))
[temp for temp in train.keys()]
**output**:
['start',
 'feat_static_cat',
 'source',
 'past_feat_dynamic_age',
 'future_feat_dynamic_age',
 'past_feat_dynamic_real',
 'future_feat_dynamic_real',
 'past_observed_values',
 'future_observed_values',
 'past_target',
 'future_target',
 'past_is_pad',
 'forecast_start']

After applying transformation, i sent the train_tf to estimator.train(train_tf)
but it is expecting the key 'target'

`KeyError                                  Traceback (most recent call last)
<ipython-input-92-f57d69ea49bf> in <module>
      1 estimator = DeepAREstimator(freq="4H",context_length = 48, prediction_length=24, distr_output = gluonts.distribution.GaussianOutput(), trainer=Trainer(epochs=25))
----> 2 predictor = estimator.train(training_data=train_tf)

~/.local/lib/python3.7/site-packages/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

~/.local/lib/python3.7/site-packages/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

~/.local/lib/python3.7/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    249 
    250                     with tqdm(train_iter) as it:
--> 251                         for batch_no, data_entry in enumerate(it, start=1):
    252                             if self.halt:
    253                                 break

/opt/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py in __iter__(self)
    977 """, fp_write=getattr(self.fp, 'write', sys.stderr.write))
    978 
--> 979             for obj in iterable:
    980                 yield obj
    981                 # Update and possibly print the progressbar.

~/.local/lib/python3.7/site-packages/gluonts/dataset/loader.py in __iter__(self)
    186         assert self._cur_iter is not None
    187         while True:
--> 188             data_entry = next(self._cur_iter)
    189             self._buffer.add(data_entry)
    190             if (

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    343         self, data_it: Iterator[DataEntry], is_train: bool
    344     ) -> Iterator:
--> 345         for data_entry in data_it:
    346             try:
    347                 for result in self.flatmap_transform(

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    295         self, data_it: Iterator[DataEntry], is_train: bool
    296     ) -> Iterator:
--> 297         for data_entry in data_it:
    298             try:
    299                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    299                 yield self.map_transform(data_entry.copy(), is_train)
    300             except Exception as e:
--> 301                 raise e
    302 
    303     @abc.abstractmethod

~/.local/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    297         for data_entry in data_it:
    298             try:
--> 299                 yield self.map_transform(data_entry.copy(), is_train)
    300             except Exception as e:
    301                 raise e

~/.local/lib/python3.7/site-packages/gluonts/transform.py in map_transform(self, data, is_train)
    312 
    313     def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 314         return self.transform(data)
    315 
    316     @abc.abstractmethod

~/.local/lib/python3.7/site-packages/gluonts/transform.py in transform(self, data)
    437 
    438     def transform(self, data: DataEntry) -> DataEntry:
--> 439         value = data[self.field]
    440         if not isinstance(value, float):
    441             # this lines produces "ValueError: setting an array element with a

KeyError: 'target'

DeepAREstimator unexpected 'target_dim'

Description

I'm trying DeepAREstimator to predict 4 dimensions data.
This address shows a parameter name 'target_dim' to set dimension of the target.
But when I create the estimator like this:
estimator = DeepAREstimator(freq="1D",
prediction_length=20,
target_dim=4,
trainer=Trainer(epochs=15))
Got an error: init() got an unexpected keyword argument 'target_dim'

Please help

multi gpu support

Description

currently estimator functions accept a Context object. Context can support only one gpu. What we need is to be able to process a list of Context object. the same way that we run multi gpu on one machine. and then in train we need to use gluon.util.split_and_load and context is a list of context usually can be coded as ctx=[gpu(i] for i in mx.context.num_gpu()]

Optimization Features for MXNet training

The training is not optimized and it is nice to be able to pass parameters for training optimization rather than inheriting new classes and adding optimization features.
Most wanted list:

num_workers for data loader. Increasing num_workers to mp.cou_count()-3 increases performance by using more cpus for data loading.
option for nvidia amp and casting network to float 16. I know it has a more dramatic effect on CNN and computer vision cases, but might still be useful.
ability to add parameters for hybridization. hybridize(static_aloc=True, static_shape=True). These options improve training performance.

Tutorial notebook Exception

I downloaded the notebook available on https://gluon-ts.mxnet.io/examples/forecasting/tutorial.html, and run it on my local laptop (Darwin Kernel Version 16.7.0, Anaconda3 Distro, Python 3.7.3, Jupyter 4.4.0, gluonts 0.1.1) and get the following exception at:

from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
from gluonts.trainer import Trainer

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/python/anaconda3/lib/python3.7/site-packages/pydantic/validators.py in find_validators(type_, arbitrary_types_allowed)
    261         try:
--> 262             if issubclass(type_, val_type):
    263                 return validators

TypeError: issubclass() arg 1 must be a class

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-13-1fca1cb620ad> in <module>
----> 1 from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
      2 from gluonts.trainer import Trainer

~/python/anaconda3/lib/python3.7/site-packages/gluonts/model/simple_feedforward/__init__.py in <module>
      1 # Relative imports
----> 2 from ._estimator import SimpleFeedForwardEstimator
      3 
      4 __all__ = ['SimpleFeedForwardEstimator']

~/python/anaconda3/lib/python3.7/site-packages/gluonts/model/simple_feedforward/_estimator.py in <module>
      7 # First-party imports
      8 from gluonts.core.component import validated
----> 9 from gluonts.distribution import DistributionOutput, StudentTOutput
     10 from gluonts.model.estimator import GluonEstimator
     11 from gluonts.model.predictor import Predictor, RepresentableBlockPredictor

~/python/anaconda3/lib/python3.7/site-packages/gluonts/distribution/__init__.py in <module>
     14 # Relative imports
     15 from . import bijection
---> 16 from .binned import Binned, BinnedOutput
     17 from .distribution import Distribution
     18 from .distribution_output import DistributionOutput

~/python/anaconda3/lib/python3.7/site-packages/gluonts/distribution/binned.py in <module>
    166 
    167 
--> 168 class BinnedOutput(DistributionOutput):
    169     distr_cls: type = Binned
    170 

~/python/anaconda3/lib/python3.7/site-packages/gluonts/distribution/binned.py in BinnedOutput()
    170 
    171     @validated()
--> 172     def __init__(self, bin_centers: List) -> None:
    173         # cannot pass directly nd.array because it is not serializable
    174         bc = mx.nd.array(bin_centers)

~/python/anaconda3/lib/python3.7/site-packages/gluonts/core/component.py in validator(ctor)
    167                 f'{ctor_clsnme}Model',
    168                 __config__=ConfigBase.Config,
--> 169                 **ctor_fields,
    170             )
    171         else:

~/python/anaconda3/lib/python3.7/site-packages/pydantic/main.py in create_model(model_name, __config__, __base__, **field_definitions)
    408                 annotation=f_annotation,
    409                 class_validators=vg.get_validators(f_name),
--> 410                 config=config,
    411             )
    412 

~/python/anaconda3/lib/python3.7/site-packages/pydantic/fields.py in infer(cls, name, value, annotation, class_validators, config)
    105             required=required,
    106             model_config=config,
--> 107             schema=schema,
    108         )
    109 

~/python/anaconda3/lib/python3.7/site-packages/pydantic/fields.py in __init__(self, name, type_, class_validators, default, required, model_config, alias, allow_none, schema)
     85         self.shape: Shape = Shape.SINGLETON
     86         self._schema: Schema = schema
---> 87         self.prepare()
     88 
     89     @classmethod

~/python/anaconda3/lib/python3.7/site-packages/pydantic/fields.py in prepare(self)
    134 
    135         self._populate_sub_fields()
--> 136         self._populate_validators()
    137 
    138     def schema(self, by_alias=True):

~/python/anaconda3/lib/python3.7/site-packages/pydantic/fields.py in _populate_validators(self)
    264                 *tuple(v.func for v in self.class_validators if not v.whole and v.pre),
    265                 *(get_validators() if get_validators else find_validators(self.type_,
--> 266                                                                           self.model_config.arbitrary_types_allowed)),
    267                 *tuple(v.func for v in self.class_validators if not v.whole and not v.pre),
    268             )

~/python/anaconda3/lib/python3.7/site-packages/pydantic/validators.py in find_validators(type_, arbitrary_types_allowed)
    263                 return validators
    264         except TypeError as e:
--> 265             raise RuntimeError(f'error checking inheritance of {type_!r} (type: {display_as_type(type_)})') from e
    266 
    267     if arbitrary_types_allowed:

RuntimeError: error checking inheritance of ~T (type: T)

Unit tests for distributions methods

Currently there are no unit tests for distributions methods like .mean, .stddev, or .variance. These methods are covered in other tests, see here. However, we should avoid using untested .mean, .stddev, or .variance in the middle of testing other properties.

See also the discussion in #42 for some context.

Add tests for valid transformation chains.

What: Add tests for all valid transformation chains in all models.
How: Write test functions/classes that test valid transformation chain combinations for all models. This needs to be abstracted to run all possible combinations for all models.
Done: All valid transformation chains are tested.

How to add categorical features using the model

Hi, I doubt if we can actually harness the deepar estimator package here.
Even if i add categories to the data, The model is not processing the category variables. I am stuck. Can you anyone explain me how do we add categorical features to the model.

Temp is a dataframe consisting of No. of PART1's(articles) sold from the stores. The stores are label encoded.

train_ds = [{'start': start ,'feat_dynamic_cat':[x], 'target': list(temp[temp['Store']== x]['PART1'])[:-prediction_length] } for x in temp['R Store'].unique()]
test_ds = [{'start': start, 'feat_dynamic_cat':[x] , 'target': list(temp[temp['Store']==x]['PART1']) } for x in temp[' Store'].unique()]

Wavenet Estimator does not take minute frequency

Description

(A clear and concise description of what the bug is.)

To Reproduce

(Please provide minimal example of code snippet that reproduces the error. For existing examples, please provide link.)

Error Message

(Paste the complete error message, including stack trace.)

Environment

Operating system:
Python version:
GluonTS version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

Tutorial notebook gets pushed to webpage with empty dataset folder

When I download the tutorial notebook the archive contains also the empty "m4_hourly" directory. As a result, the cell containing

dataset = get_dataset("m4_hourly", regenerate=False)

fails.

source code should be kept in src folder

having the package directly exposed in the root directory will cause the CI to test the exposed version instead of the version installed by the user. see context dmlc/gluon-nlp#390

Error importing ComplexSeasonalTimeSeries on python 3.7

Hi Gluon TS maintainers

What a great project. I found a little glitch under anaconda and windows 10, when trying to import the ComplexSeasonalTimeSeries class. The issue appears similar to #62, #67, and #68 but those issues show as fixed, and this error occurs on HEAD.

To reproduce, create a new anaconda environment for python 3.7, clone the repo and install from source. Then:

C:\path>python
Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from gluonts.dataset.artificial._base import ComplexSeasonalTimeSeries
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\users\alex\src\gluon-ts\src\gluonts\dataset\artificial\__init__.py", line 2, in <module>
    from ._base import (
  File "c:\users\alex\src\gluon-ts\src\gluonts\dataset\artificial\_base.py", line 20, in <module>
    from gluonts.dataset.artificial.recipe import (
  File "c:\users\alex\src\gluon-ts\src\gluonts\dataset\artificial\recipe.py", line 242, in <module>
    class LinearTrend(Lifted):
  File "c:\users\alex\src\gluon-ts\src\gluonts\dataset\artificial\recipe.py", line 244, in LinearTrend
    def __init__(self, slope_fun: Callable = Constant(1.0)) -> None:
  File "c:\users\alex\src\gluon-ts\src\gluonts\core\component.py", line 352, in validator
    **init_fields,
  File "C:\Users\Alex\.conda\envs\gts\lib\site-packages\pydantic\main.py", line 410, in create_model
    config=config,
  File "C:\Users\Alex\.conda\envs\gts\lib\site-packages\pydantic\fields.py", line 107, in infer
    schema=schema,
  File "C:\Users\Alex\.conda\envs\gts\lib\site-packages\pydantic\fields.py", line 87, in __init__
    self.prepare()
  File "C:\Users\Alex\.conda\envs\gts\lib\site-packages\pydantic\fields.py", line 135, in prepare
    self._populate_sub_fields()
  File "C:\Users\Alex\.conda\envs\gts\lib\site-packages\pydantic\fields.py", line 240, in _populate_sub_fields
    assert issubclass(origin, Mapping)
AssertionError
>>>

But in contrast, repeating the above with python 3.6, we get

C:\path>python
Python 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from gluonts.dataset.artificial._base import ComplexSeasonalTimeSeries
>>>

Cheers
Alex

Multivariate time series raise an error

Description

I'm trying to model 4-dimensional time series using MultivariateGaussianOutput, but training fails due to some invalid loss value.

To Reproduce

import numpy as np

from gluonts.dataset.common import ListDataset
from gluonts.distribution import MultivariateGaussianOutput
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer

train_dataset = ListDataset(
    data_iter=[
        {
            "start": "2019-01-01 00:00:00",
            "target": np.ones(shape=(4, 100)).tolist(),
        },
    ],
    freq="5min",
)

estimator = DeepAREstimator(
    'H', prediction_length=5, trainer=Trainer(epochs=3, hybridize=False),
    distr_output=MultivariateGaussianOutput(dim=4),
)

predictor = estimator.train(train_dataset)

Error Message

Traceback (most recent call last):
  File "/Users/stellalo/gluon-ts/examples/run_multivariate.py", line 15, in <module>
    freq="5min",
  File "/Users/stellalo/gluon-ts/src/gluonts/dataset/common.py", line 259, in __init__
    self.list_data = [process(data) for data in data_iter]
  File "/Users/stellalo/gluon-ts/src/gluonts/dataset/common.py", line 259, in <listcomp>
    self.list_data = [process(data) for data in data_iter]
  File "/Users/stellalo/gluon-ts/src/gluonts/dataset/common.py", line 435, in __call__
    data = t(data)
  File "/Users/stellalo/gluon-ts/src/gluonts/dataset/common.py", line 374, in __call__
    f"JSON array has bad shape - expected {self.req_ndim} "
gluonts.core.exception.GluonTSDataError: JSON array has bad shape - expected 1 dimensions, got -1

Environment

Operating system: macOS
Python version: 3.6.6
GluonTS version: master (35cc49b)

(Add as much information about your environment as possible, e.g. dependencies versions.)

Test minimal viable dependencies

We had the issue that our specified minimal version of mxnet (1.3.0) did not work.

When doing a fresh install with pip, it will install a newer version which still satisfies >=1.3.0 and thus this mistake wasn't detected (fixed with #96).

Thus, we should think about testing the minimal versions of our dependecies.

Use categoical and dynamc features by default in DeepAR

Currently DeepAR does not use categorical and dynamic features by default even though they are present in the dataset. The flags use_feat_dynamic_real and use_feat_static_cat are set to False by default in DeepAREstimator. This is very bad for us in terms of results since people usually run methods with their default options or miss setting these flags explicitly.

There were a couple of arguments against not using them by default but there are better remedies for them than not silently running DeepAR with incorrect options and returning less accurate results:

Issue with not setting cardinality and still using feat_static_cat: we can make cardinality argument compulsory (or ideally derive this from data)
If all time series do not have same features: there is no harm in failing with suitable error when the data is not consistent.

Which case is important/priority for us right now: Running smoothly even if the data is not properly formatted or running with correct options that succeeds only when the data is consistent?

numpy version issue while installing via pip

Ubuntu 16.04.6 LTS, Python 3.7.3 (pyenv).

Using pip install gluonts in a clean environment will get version conflict error about. It will try to install numpy 1.16.4 first:

Collecting numpy>=1.14.0 (from gluonts)
  Downloading https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl (17.3MB)

And later mxnet gives version conflict error:

ERROR: mxnet 1.4.1 has requirement numpy<1.15.0,>=1.8.2, but you'll have numpy 1.16.4 which is incompatible.

Workaround is to run pip install mxnet after pip install gluonts to let pip downupgrade numpy:

$ pip install mxnet
Requirement already satisfied: mxnet in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (1.4.1)
Requirement already satisfied: requests>=2.20.0 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from mxnet) (2.22.0)
Collecting numpy<1.15.0,>=1.8.2 (from mxnet)
  Downloading https://files.pythonhosted.org/packages/18/84/49b7f268741119328aeee0802aafb9bc2e164b36fc312daf83af95dae646/numpy-1.14.6-cp37-cp37m-manylinux1_x86_64.whl (13.8MB)
     |████████████████████████████████| 13.8MB 13.5MB/s
Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from mxnet) (0.8.4)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests>=2.20.0->mxnet) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests>=2.20.0->mxnet) (2019.3.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests>=2.20.0->mxnet) (1.25.3)
Requirement already satisfied: idna<2.9,>=2.5 in ./.pyenv/versions/3.7.3/lib/python3.7/site-packages (from requests>=2.20.0->mxnet) (2.8)
Installing collected packages: numpy
  Found existing installation: numpy 1.16.4
    Uninstalling numpy-1.16.4:
      Successfully uninstalled numpy-1.16.4
Successfully installed numpy-1.14.6

If it's not easy to fix, maybe add some instructions into README.md.

Add link to Arxiv preprint in readme

Please consider mentioning and adding a link to the corresponding Arxiv preprint https://arxiv.org/abs/1906.05264 in the readme of this repo. Thanks.

Quick Start Example with "NegativeBinomialOutput" throw exception

System: Ubuntu 16.04
Python: 3.6.4
mxnet: 1.4.1

Code to reproduce:

import pandas as pd
url = "https://raw.githubusercontent.com/numenta/NAB/master/data/realTweets/Twitter_volume_AMZN.csv"
df = pd.read_csv(url, header=0, index_col=0)
from gluonts.dataset.common import ListDataset
training_data = ListDataset(
    [{"start": df.index[0], "target": df.value[:"2015-04-05 00:00:00"]}],
    freq = "5min"
)
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
from gluonts.distribution import NegativeBinomialOutput, StudentTOutput
estimator = DeepAREstimator(freq="5min", prediction_length=12, distr_output=NegativeBinomialOutput(),
                            trainer=Trainer(epochs=10))
predictor = estimator.train(training_data=training_data)

Exception:
DeferredInitializationError               Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in <listcomp>(.0)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

/usr/local/lib/python3.6/site-packages/mxnet/gluon/parameter.py in data(self, ctx)
    493                                "instead." % (self.name, str(ctx), self._stype))
--> 494         return self._check_and_get(self._data, ctx)
    495 

/usr/local/lib/python3.6/site-packages/mxnet/gluon/parameter.py in _check_and_get(self, arr_list, ctx)
    207                 "You can also avoid deferred initialization by specifying in_units, " \
--> 208                 "num_features, etc., for network layers."%(self.name))
    209         raise RuntimeError(

DeferredInitializationError: Parameter 'deepartrainingnetwork0_lstm0_i2h_weight' has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters. You can also avoid deferred initialization by specifying in_units, num_features, etc., for network layers.

During handling of the above exception, another exception occurred:

MXNetError                                Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    788         try:
--> 789             self.infer_shape(*args)
    790         except Exception as e:

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in infer_shape(self, *args)
    861         """Infers shape of Parameters from inputs."""
--> 862         self._infer_attrs('infer_shape', 'shape', *args)
    863 

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in _infer_attrs(self, infer_fn, attr, *args)
    850             arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
--> 851                 **{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
    852             if arg_attrs is None:

/usr/local/lib/python3.6/site-packages/mxnet/symbol/symbol.py in infer_shape(self, *args, **kwargs)
    995         try:
--> 996             res = self._infer_shape_impl(False, *args, **kwargs)
    997             if res[1] is None:

/usr/local/lib/python3.6/site-packages/mxnet/symbol/symbol.py in _infer_shape_impl(self, partial, *args, **kwargs)
   1125             ctypes.byref(aux_shape_data),
-> 1126             ctypes.byref(complete)))
   1127         if complete.value != 0:

/usr/local/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
    251     if ret != 0:
--> 252         raise MXNetError(py_str(_LIB.MXGetLastError()))
    253 

MXNetError: Error in operator deepartrainingnetwork0__mul1: [10:04:54] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node deepartrainingnetwork0__mul1 at 1-th input: expected [32,24], got [32,32,24]

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23d55a) [0x7f44332ff55a]
[bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23dbc1) [0x7f44332ffbc1]
[bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4ee78d) [0x7f44335b078d]
[bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x5587d6) [0x7f443361a7d6]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x762338) [0x7f4433824338]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2be2cea) [0x7f4435ca4cea]
[bt] (6) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2be5658) [0x7f4435ca7658]
[bt] (7) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f4435c140ca]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f44791201da]
[bt] (9) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x26b) [0x7f447911c5cb]



During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-6-7512b68a283e> in <module>()
      1 estimator = DeepAREstimator(freq="5min", prediction_length=12, distr_output=NegativeBinomialOutput(),
      2                             trainer=Trainer(epochs=10))
----> 3 predictor = estimator.train(training_data=training_data)

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    256 
    257                             with mx.autograd.record():
--> 258                                 output = net(*inputs)
    259 
    260                                 # network can returns several outputs, the first being always the loss

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args)
    538             hook(self, args)
    539 
--> 540         out = self.forward(*args)
    541 
    542         for hook in self._forward_hooks.values():

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args)
    905             with x.context as ctx:
    906                 if self._active:
--> 907                     return self._call_cached_op(x, *args)
    908 
    909                 try:

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:
--> 805             self._deferred_infer_shape(*args)
    806             cargs = []
    807             for is_arg, i in self._cached_op_args:

/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    791             error_msg = "Deferred initialization failed because shape"\
    792                         " cannot be inferred. {}".format(e)
--> 793             raise ValueError(error_msg)
    794 
    795     def _call_cached_op(self, *args):

ValueError: Deferred initialization failed because shape cannot be inferred. Error in operator deepartrainingnetwork0__mul1: [10:04:54] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node deepartrainingnetwork0__mul1 at 1-th input: expected [32,24], got [32,32,24]

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23d55a) [0x7f44332ff55a]
[bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23dbc1) [0x7f44332ffbc1]
[bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4ee78d) [0x7f44335b078d]
[bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x5587d6) [0x7f443361a7d6]
[bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x762338) [0x7f4433824338]
[bt] (5) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2be2cea) [0x7f4435ca4cea]
[bt] (6) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2be5658) [0x7f4435ca7658]
[bt] (7) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f4435c140ca]
[bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f44791201da]
[bt] (9) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x26b) [0x7f447911c5cb]

ComplexSeasonalTimeSeries doesn't work with min/hour/month frequencies

Hello,

I am getting an error when trying to use ComplexSeasonalTimeSeries with min/hour/month frequencies (works fine for day and week).

ComplexSeasonalTimeSeries(
    num_series=100,
    prediction_length=5000,
    freq_str='H',
    length_low=10000,
    length_high=10001,
    min_val=0,
    max_val=1e10,
    is_integer=True,
    proportion_missing_values=0,
    is_noise=True,
    is_scale=True,
    percentage_unique_timestamps=0.,
    is_out_of_bounds_date=False
).make_timeseries()

gives

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_str_to_tsobject()

pandas/_libs/tslibs/np_datetime.pyx in pandas._libs.tslibs.np_datetime._string_to_dts()

ValueError: Hours out of range in datetime string "2018-08-06 24:00:00"

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_str_to_tsobject()

pandas/_libs/tslibs/parsing.pyx in pandas._libs.tslibs.parsing.parse_datetime_string()

/miniconda3/lib/python3.6/site-packages/dateutil/parser/_parser.py in parse(timestr, parserinfo, **kwargs)
   1355     else:
-> 1356         return DEFAULTPARSER.parse(timestr, **kwargs)
   1357 

/miniconda3/lib/python3.6/site-packages/dateutil/parser/_parser.py in parse(self, timestr, default, ignoretz, tzinfos, **kwargs)
    652 
--> 653         ret = self._build_naive(res, default)
    654 

/miniconda3/lib/python3.6/site-packages/dateutil/parser/_parser.py in _build_naive(self, res, default)
   1226 
-> 1227         naive = default.replace(**repl)
   1228 

ValueError: hour must be in 0..23

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-62-abc29a91ead8> in <module>
----> 1 data = csts.make_timeseries()

~/Documents/Swist/swist/dataset/artificial/_base.py in make_timeseries(self, seed)
    508             t = np.arange(length)
    509             idx = pd.DatetimeIndex(
--> 510                 start=start, freq=self.freq_str, periods=length
    511             )
    512             special_tp_indicator = self._special_time_point_indicator(idx)

/miniconda3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py in __new__(cls, data, freq, start, end, periods, tz, normalize, closed, ambiguous, dayfirst, yearfirst, dtype, copy, name, verify_integrity)
    280                 start, end, periods,
    281                 freq=freq, tz=tz, normalize=normalize,
--> 282                 closed=closed, ambiguous=ambiguous)
    283             warnings.warn("Creating a DatetimeIndex by passing range "
    284                           "endpoints is deprecated.  Use "

/miniconda3/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py in _generate_range(cls, start, end, periods, freq, tz, normalize, ambiguous, nonexistent, closed)
    409 
    410         if start is not None:
--> 411             start = Timestamp(start)
    412 
    413         if end is not None:

pandas/_libs/tslibs/timestamps.pyx in pandas._libs.tslibs.timestamps.Timestamp.__new__()

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_to_tsobject()

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.convert_str_to_tsobject()

ValueError: could not convert string to Timestamp

Split dependencies

We should think about splitting the dependencies. For example, matplotlib is only useful if you want to display data, but not needed when running training on a server.

We could use "extras" in setup.py and have something like:

    # barebone
    $ pip install gluonts

    # full-installation
    $ pip install "gluonts[notebook]"

Undefined name

flake8 testing of https://github.com/awslabs/gluon-ts on Python 3.7.1

$ flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics

./src/gluonts/model/predictor.py:677:35: F821 undefined name 'gluonts'
    def __init__(self, estimator: 'gluonts.model.estimator.Estimator'):
                                  ^
1     F821 undefined name 'gluonts'
1

Address FutureWarning related to Timestamp

Currently our tests produce some FutureWarnings on newer Pandas versions, as timestamp + int will no longer be allowed and needs to be replaced with timestamp + int * freq.

Cannot predict using NegativeBinominalOutput

Description

I am training example model with NegativeBinominalOutput. After fixing #91 it trains well, but I cannot make a prediction. The Minimal reproducible example is attached. It is the same as in #91, but just have a prediction code.

To Reproduce

import pandas as pd
from gluonts.dataset.common import ListDataset

url = "https://raw.githubusercontent.com/numenta/NAB/master/data/realTweets/Twitter_volume_AMZN.csv"
df = pd.read_csv(url, header=0, index_col=0)

training_data = ListDataset(
    [{"start": df.index[0], "target": df.value[:"2015-04-05 00:00:00"]}],
    freq="5min"
)
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
from gluonts.distribution import NegativeBinomialOutput

estimator = DeepAREstimator(freq="5min", prediction_length=12,
                            distr_output=NegativeBinomialOutput(),
                            trainer=Trainer(epochs=1))
predictor = estimator.train(training_data=training_data)

for forecast in predictor.predict(training_data):
    print(forecast.mean)

Error Message

Traceback (most recent call last):
  File "/home/ubuntu/projects/mwe_neg_binominal_predict.py", line 19, in <module>
    for forecast in predictor.predict(training_data):
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/model/predictor.py", line 277, in predict
    outputs = self.prediction_net(*inputs).asnumpy()
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/gluon/block.py", line 548, in __call__
    out = self.forward(*args)
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/gluon/block.py", line 925, in forward
    return self.hybrid_forward(ndarray, x, *args, **params)
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/model/deepar/_network.py", line 526, in hybrid_forward
    begin_states=state,
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/model/deepar/_network.py", line 463, in sampling_decoder
    new_samples = distr.sample()
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/distribution/neg_binomial.py", line 92, in sample
    s, mu=self.mu, alpha=self.alpha, num_samples=num_samples
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/distribution/distribution.py", line 241, in _sample_multiple
    samples = sample_func(*args_expanded, **kwargs_expanded)
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/gluonts/distribution/neg_binomial.py", line 88, in s
    x = F.minimum(F.random.gamma(r, theta), 1e6)
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/ndarray/random.py", line 383, in gamma
    [alpha, beta], shape, dtype, ctx, out, kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/ndarray/random.py", line 38, in _random_helper
    return sampler(*params, shape=shape, dtype=dtype, out=out, **kwargs)
  File "<string>", line 68, in _sample_gamma
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/mxnet/base.py", line 253, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: vector::_M_range_insert

Environment

Operating system: Ubuntu
Python version: 3.7
GluonTS version: 0.4 (--pre installation)

Regression tests

Description

Evaluations stored in https://github.com/awslabs/gluon-ts/tree/master/evaluations are generated manually. This causes several issue: 1. manual mistakes can happen 2. it can take a long time before we detect an issue impacting accuracy. Such evaluations could be done at a regular basis to make it easier to check when accuracy regression happens.

flaky test test_multivariate_sampling

http://ci.mxnet.io/blue/organizations/jenkins/GluonTS-py3-master-cpu-unittest/detail/PR-40/1/pipeline/#step-143-log-937

Note that this might indicate an issue with mxnet master branch/nightly.

Test Issue

Testing notification chain.

shape assertions

Adding shape assertions for MXNet arrays would be good for readability, debugging, and to get useful error messages. Shape assertions may be used e.g. when constructing distributions, or throughout the hybrid_forward of the models networks. One caveat of this is that this can only be done when F=mx.nd.

The simplest thing to do would be to have a utility function like:

def assert_shape(F, x, shape):
    assert F == mx.sym or x.shape == shape

We may also need something similar for asserting that an array is broadcast-compatible with some other shape:

def assert_broadcastable(F, x, shape):
    if F == mx.sym:
        return
    x_shape = x.shape
    assert len(x_shape) <= len(shape)
    x_shape = (1,) * (len(shape) - len(x_shape)) + x_shape
    x_shape_arr = np.array(x_shape)
    broadcasting_axes = np.nonzero(x_shape_arr != np.array(shape))
    assert (x_shape_arr[broadcasting_axes] == 1).all()

(The latter is essentially copied from MXNet's broadcast_to. Actually, shouldn't MXNet factor out and expose this utility?)

canonical estimator results in shape error

Description

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

To Reproduce

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

Error Message

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

Environment

Operating system:
Python version:
GluonTS version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

How to save the trained estimator

HI
I ddin't find any documentation related to saving the trained model. I want to save my model for further usage.

Thanks

Working with Weekly or Montly time series

Hello,

I am having difficulties to work with Weekly and Montly frequency.

When I run this code using Daily frequenct, it works properly, however, if I resample to weekly or montly I get the following error:

TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'

Here is my code:

from gluonts.dataset.util import to_pandas
from gluonts.trainer import Trainer
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
from gluonts.dataset import common
from gluonts.model import deepar
import pandas as pd

!pip install quandl

import quandl

dataset = pd.DataFrame(quandl.get("FRED/DEXBZUS"))

dataset = dataset.resample('M').apply(lambda ser: ser.iloc[-1,])

training_data = common.ListDataset([{"start": dataset.index[0],
                            "target": dataset.Value[:"2017"]}],
                          freq="M")

estimator = deepar.DeepAREstimator(freq="M", prediction_length=40, trainer=Trainer( epochs=5))
predictor = estimator.train(training_data=training_data)

for test_entry, forecast in zip(training_data , predictor.predict(training_data )):
    to_pandas(test_entry)[-60:].plot(linewidth=2)
    forecast.plot(color='g', prediction_intervals=[50.0, 90.0])
plt.grid(which='both')

Dynamical Feature with Different Length will Throw Exception

System: Ubuntu 16.04
Python: 3.6.4
mxnet: 1.4.1

Code to reproduce:

import pandas as pd
url = "https://raw.githubusercontent.com/numenta/NAB/master/data/realTweets/Twitter_volume_AMZN.csv"
df = pd.read_csv(url, header=0, index_col=0)
from gluonts.dataset.common import ListDataset
training_data = [
    {"start": pd.Timestamp(df.index[0], freq='5min'), "target": df.value[:"2015-04-05 00:00:00"], 
     "feat_dynamic_real": pd.to_datetime(df[:"2015-04-05 00:00:00"].index).dayofweek.values},
    {"start": pd.Timestamp(df.index[0], freq='5min'), "target": df.value[:"2015-04-10 00:00:00"], 
     "feat_dynamic_real": pd.to_datetime(df[:"2015-04-10 00:00:00"].index).dayofweek.values}
]
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
from gluonts.distribution import NegativeBinomialOutput, StudentTOutput
estimator = DeepAREstimator(freq="5min", prediction_length=12, distr_output=NegativeBinomialOutput(),
                            trainer=Trainer(epochs=10))
predictor = estimator.train(training_data=training_data)

Exceptions:

KeyError                                  Traceback (most recent call last)
<ipython-input-9-7512b68a283e> in <module>()
      1 estimator = DeepAREstimator(freq="5min", prediction_length=12, distr_output=NegativeBinomialOutput(),
      2                             trainer=Trainer(epochs=10))
----> 3 predictor = estimator.train(training_data=training_data)

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    249 
    250                     with tqdm(train_iter) as it:
--> 251                         for batch_no, data_entry in enumerate(it, start=1):
    252                             if self.halt:
    253                                 break

/usr/local/lib/python3.6/site-packages/tqdm/_tqdm.py in __iter__(self)
   1003                 """), fp_write=getattr(self.fp, 'write', sys.stderr.write))
   1004 
-> 1005             for obj in iterable:
   1006                 yield obj
   1007                 # Update and possibly print the progressbar.

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/dataset/loader.py in __iter__(self)
    200             ):
    201                 for batch in self._emit_batches_while_buffer_larger_than(
--> 202                     self.batch_size - 1
    203                 ):
    204                     yield batch

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/dataset/loader.py in _emit_batches_while_buffer_larger_than(self, thresh)
    167             self._buffer.shuffle()
    168         while len(self._buffer) > thresh:
--> 169             yield self._buffer.next_batch()
    170 
    171     def _iterate_forever(

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/dataset/loader.py in next_batch(self)
     58             #print(np.asarray(v[:n]).dtype)
     59             #print(v[:n])
---> 60             batch[k] = self.stack(v[:n])
     61         #batch = {k: self.stack(v[:n]) for k, v in self._buffers.items()}
     62         for key in self._buffers.keys():

/usr/local/lib/python3.6/site-packages/gluonts-0.1.1-py3.6.egg/gluonts/dataset/loader.py in stack(self, xs)
     70             if data.dtype.kind == 'f':
     71                 data = data.astype(self.float_type)
---> 72             return mx.nd.array(data, dtype=data.dtype, ctx=self.ctx)
     73         elif isinstance(xs[0], mx.nd.NDArray):
     74             return mx.nd.stack(*xs)

/usr/local/lib/python3.6/site-packages/mxnet/ndarray/utils.py in array(source_array, ctx, dtype)
    144         return _sparse_array(source_array, ctx=ctx, dtype=dtype)
    145     else:
--> 146         return _array(source_array, ctx=ctx, dtype=dtype)
    147 
    148 

/usr/local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in array(source_array, ctx, dtype)
   2486             except:
   2487                 raise TypeError('source_array must be array like object')
-> 2488     arr = empty(source_array.shape, ctx, dtype)
   2489     arr[:] = source_array
   2490     return arr

/usr/local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in empty(shape, ctx, dtype)
   3875     if dtype is None:
   3876         dtype = mx_real_t
-> 3877     return NDArray(handle=_new_alloc_handle(shape, ctx, False, dtype))
   3878 
   3879 

/usr/local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in _new_alloc_handle(shape, ctx, delay_alloc, dtype)
    137         ctypes.c_int(ctx.device_id),
    138         ctypes.c_int(int(delay_alloc)),
--> 139         ctypes.c_int(int(_DTYPE_NP_TO_MX[np.dtype(dtype).type])),
    140         ctypes.byref(hdl)))
    141     return hdl

KeyError: <class 'numpy.object_'>

Error comes from: https://github.com/awslabs/gluon-ts/blob/3c6d1106c28143a78def634f23c43e46b5fd047d/src/gluonts/dataset/loader.py#L62

remove pandas frequency workaround

As of pandas 0.24.2, some workarounds are needed because of a bug, see #79. These are apparently solved in pandas master, and should be included in the upcoming 0.25.0 release.

To do, after pandas 0.25.0 has been released:

bump requirement to >=0.25.0
remove workarounds such as the one added in #79

Test tutorial notebooks in CI.

When having notebooks, we should make sure that new changes don't break them.

It's possible to extract the code from a notebook using https://pypi.org/project/nbless/ (nbextract).

/cc @mbohlkeschneider

Rethink Exception hierarchy

The exception / error classes defined in gluonts.core.exception and their usages do not follow a consistent pattern.

We need to rethink the hierarchy and the intended use of each exception class. Ideally, with the revised model we should be able to clearly differentiate between errors:

that are caused by bad user input, e.g.
- bad hyperparameters;
- malformed input data
that are caused by something else (e.g. missing expected directory).

Get continuous integration on Windows

Some problems would have been easily detected had we had CI in place on Windows.

Maybe we can use AppVeyor?

add tests for gluonts.shell

Tests for the gluonts.shell package are missing, and should be added.

simple_feedforward throws an exception at import

I was just going through the tutorial given at https://gluon-ts.mxnet.io/examples/forecasting/tutorial.html and encountered this problem at step #11 (I am working on macOS and Python 3.7). What shall I do?

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/validators.py", line 262, in find_validators
if issubclass(type_, val_type):
TypeError: issubclass() arg 1 must be a class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "test.py", line 31, in
from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/model/simple_feedforward/init.py", line 2, in
from ._estimator import SimpleFeedForwardEstimator
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/model/simple_feedforward/_estimator.py", line 9, in
from gluonts.distribution import DistributionOutput, StudentTOutput
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/distribution/init.py", line 16, in
from .binned import Binned, BinnedOutput
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/distribution/binned.py", line 168, in
class BinnedOutput(DistributionOutput):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/distribution/binned.py", line 172, in BinnedOutput
def init(self, bin_centers: List) -> None:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gluonts/core/component.py", line 169, in validator
**ctor_fields,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/main.py", line 410, in create_model
config=config,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/fields.py", line 107, in infer
schema=schema,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/fields.py", line 87, in init
self.prepare()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/fields.py", line 136, in prepare
self.populate_validators()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/fields.py", line 266, in populate_validators
self.model_config.arbitrary_types_allowed)),
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydantic/validators.py", line 265, in find_validators
raise RuntimeError(f'error checking inheritance of {type!r} (type: {display_as_type(type)})') from e
RuntimeError: error checking inheritance of ~T (type: T)

pip install error

I get compilation error during pip install process on 18.04 when it build the ujson extension:

Running setup.py bdist_wheel for ujson ... error
Complete output from command /home/captain/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-74754z_z/ujson/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-puziaq_8 --python-tag cp36:
running bdist_wheel
running build
running build_ext
building 'ujson' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/python
creating build/temp.linux-x86_64-3.6/lib
gcc -pthread -B /home/captain/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/captain/anaconda3/include/python3.6m -c ./python/ujson.c -o build/temp.linux-x86_64-3.6/./python/ujson.o -D_GNU_SOURCE
In file included from /home/captain/anaconda3/lib/gcc/x86_64-unknown-linux-gnu/4.8.5/include-fixed/syslimits.h:7:0,
from /home/captain/anaconda3/lib/gcc/x86_64-unknown-linux-gnu/4.8.5/include-fixed/limits.h:34,
from /home/captain/anaconda3/include/python3.6m/Python.h:11,
from ./python/py_defines.h:39,
from ./python/ujson.c:39:
/home/captain/anaconda3/lib/gcc/x86_64-unknown-linux-gnu/4.8.5/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */

Tutorial [WinError 3] The system cannot find the path specified

I am trying to run the tutorial. I get FileNotFoundError when running the below piece of code. I have installed gluonts in an Anaconda environment with Python 3.6 on Windows Home 10 version 1803 OS Build 17134.765.

`from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer

estimator = DeepAREstimator(freq="5min", prediction_length=12, trainer=Trainer(epochs=10))
predictor = estimator.train(training_data=training_data)`

FileNotFoundError Traceback (most recent call last)
in
3
4 estimator = DeepAREstimator(freq="5min", prediction_length=12, trainer=Trainer(epochs=10))
----> 5 predictor = estimator.train(training_data=training_data)

~\Anaconda3\envs\GluonTS\lib\site-packages\gluonts\model\estimator.py in train(self, training_data)
187 def train(self, training_data: Dataset) -> Predictor:
188
--> 189 training_transformation, trained_net = self.train_model(training_data)
190
191 # ensure that the prediction network is created within the same MXNet

~\Anaconda3\envs\GluonTS\lib\site-packages\gluonts\model\estimator.py in train_model(self, training_data)
180 net=trained_net,
181 input_names=get_hybrid_forward_input_names(trained_net),
--> 182 train_iter=training_data_loader,
183 )
184

~\Anaconda3\envs\GluonTS\lib\site-packages\gluonts\trainer_base.py in call(self, net, input_names, train_iter)
177
178 with tempfile.TemporaryDirectory(
--> 179 prefix='gluonts-trainer-temp-', dir='/tmp'
180 ) as gluonts_temp:
181

~\Anaconda3\envs\GluonTS\lib\tempfile.py in init(self, suffix, prefix, dir)
790
791 def init(self, suffix=None, prefix=None, dir=None):
--> 792 self.name = mkdtemp(suffix, prefix, dir)
793 self._finalizer = _weakref.finalize(
794 self, self._cleanup, self.name,

~\Anaconda3\envs\GluonTS\lib\tempfile.py in mkdtemp(suffix, prefix, dir)
368 file = _os.path.join(dir, prefix + name + suffix)
369 try:
--> 370 _os.mkdir(file, 0o700)
371 except FileExistsError:
372 continue # try again

FileNotFoundError: [WinError 3] The system cannot find the path specified: '/tmp\gluonts-trainer-temp-nodyqafj'

Investigate build process on Windows

The package build seems to be failing on Windows, apparently once again due to some hard-coded path.

Context is in #75

Fix issues in the doc builds and re-enable warnings as failure

Basically do the necessary work to revert the changes made in #72.

get_model_summary

Description

I would like to print my model summary (keras style) or access the network and take a look at layers, weights, collect_params, and number of parameters.
It would be nice to add a get_network_summary or an immutable network object returned by the estimator.

Streamline "freq", "time_granularity" naming

Find a naming that is consistent and change it throughout the code for freq/time_granularity.

Reduce the Conda dependencies listed in `env`

Packages that are duplicated between a requirements*.txt and a *.yml file in env should be removed from the *.yml file.

Add build-system requirement to documentation.

See #74

Some of our dependencies might need to be installed locally. This can cause pip install gluonts to fail, since not all dependencies were installed successfully.

We should make this point clear in the docs.

Alternatively, we could think about making ujson (which caused most troubles) optional or offer docker-images which have gluon-ts already installed.

Enable Black in the Jenkins pipelines

Requesting ensemble of models

Dear Team,

Similar to sagemaker current feature, requesting ensemble of models for forecasting or anomaly detection.

Thanks!

website follow-up items

enforce lint in sanity check: #10 (comment)
enforce warning as error in doc builds: #10 (comment)

awslabs / gluonts Goto Github PK

gluonts's Introduction

GluonTS - Probabilistic Time Series Modeling in Python

Installation

Simple Example

Contributing

Citing

Links

Documentation

References

Tutorials and Workshops

gluonts's People

Contributors

Stargazers

Watchers

Forkers

gluonts's Issues

Description

References

Description

Description

Description

To Reproduce

Error Message

Environment

Description

To Reproduce

Error Message

Environment

Description

To Reproduce

Error Message

Environment

Description

Description

To Reproduce

Error Message

Environment

Description

Recommend Projects

Recommend Topics

Recommend Org