Thank you for sharing this project !

I was testing load_gunpoint_dataframe and got an str as type for x_train.

I used X_train, y_train = load_gunpoint_dataframe(split='TRAIN', return_X_y=True).

Thanks

multiple integration tests rely on downloading data from the internet - making connection problems to their source (more specifically timeseriesclassification.com) a potential failure risk. I suggest removing this implicit outside dependency for cleanness.

Including an abstract parent class
and 1-3 simple examples, optimally with test cases (test notebook or CI script)

Conditional on
#1

I thought it worth spelling out the structure of how people can use and implement classification algorithms in sktime. The balance is between modularity and efficiency. This is summarising what we have, but I think it worth it as a we try to widen the development base.

Modular approach:
I propose two classifiers as pipelines
TimeSeriesEnsemble (or time_series_ensemble? need to resolve naming conventions. Another ticket!). This pipeline is a set of transforms and an estimator/classifier as the last element. The logic is that the transforms are applied independently to each member of the ensemble, and the final estimator is the base classifier for the ensemble. The transformers are not applied sequentially. They are applied independently to the data and the results are concatenated. They must be Randomizable, so that a different transform is applied for each ensemble member. The transformers and the base classifier should be seedable for reproducability.

Error: fatal error: Python.h: No such file or directory
OR
Error: .so : undefined symbol: _Py_ZeroStruct

Both are caused by incorrect packages / setup under linux. You need the python3-dev package to be installed to fix the former issue, and ensure setup.py is called with python3 rather than python2. To build cython run: python3 setup.py build_ext -i

Cython should then successfully build '.so' and '.c' files

reference bake-off in the literature for classical forecasting: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194889

xpandas provides functionality for fusing series-to-tabular transformers with tabular supervised learning methods.

This should be interfaced, or replicated in the more general interface.

In this interface, it should be possible to chain series-to-series transformers (e.g., truncation) with series-to-tabular transformers to obtain a series-to-tabular transformer, or series-to-series with series-to-tabular with a tabular SL method to obtain a TSC/TSR method.

Conditional on a consolidated interface design as in #5 and #6.

If easy interfacing is not possible, it may also require separate pipeline design, in this case please raise issue of pipeline API design.

• Make existing Orchestrator and evaluation framework compatible with classical forecasting

Some series-to-series transformers that would be useful.

unfitted, single-series, simple

• binning/aggregation transformer

Behaviour:
returns the sequence of [aggregator application] (e.g., count) in the bins. Index is start time, end time, or index (from start) of bin, depending on index hyper-parameter

Hyper-parameters:
bin specs - start: time/index, end : time/index, numbins : integer
index - 'start', 'end', or 'bin'
aggregator - function to apply to values within bin, default = count

alternative to bin specs: index sequence

• truncation transformer

Behaviour:
cuts off any entry in the sequence with index outside [lower, upper]

Hyper-parameters:
lower, upper : time

• simple equal spacing transformer

Behaviour:
intra/extrapolates series to the nodes by the specified strategy, e.g., fill in nearest or next (careful with boundaries)

Hyper-parameters:
node specs - start: time/index, end : time/index, numsteps : integer
index - 'start', 'end', or 'bin'
strategy - 'nearest', 'last' , 'next', 'pw_linear'

alternative to node specs: index sequence

• re-indexing transformer

Behaviour:
changes the index by a the strategy indicated in the reindexing parameter
integer = replace with ascending count
field = get from data frame column

Hyper-parameters:
strategy - 'integer', 'field'

• index extractor transformer

Behaviour:
creates a series from the index of the series

• NA remover transformer

Behaviour:
removes sequence elements that are numpy.nan

• padding transformer

Behaviour:
pads a sequence/series with value at start or end until it has the desired length

Hyper-parameters:
where - 'start', 'end'
what - value
length - integer
optional: index treatment

• NA imputer

Behaviour:
Fills in NA values by the specified strategy

Hyper-parameters:
strategy - 'nearest', 'last' , 'next', 'pw_linear'

unfitted, single-series, reduction

• interpolation transformer

Behaviour:
uses a scikit-learn regressor or classifier to interpolate to the specified index set.
Fits series values against series index, and uses the regressor/classifier to predict value from index

Hyper-parameters:
index set
estimator - sklearn regressor

• Supervised NA imputer

Behaviour:
Fills in NA values by the specified strategy by using a scikit-learn regressor or classifier. Fits non-NA series values against series index, and uses the regressor/classifier to predict value from index

Hyper-parameters:
strategy - 'nearest', 'last' , 'next', 'pw_linear'

• advanced: exogeneous or multi-column versions

unfitted, multiple-series

Note: the below are "unfitted" since they run on the entire series

• index homogenization transformer
  Behaviour:
  Looks up the indices for all the series and introduces them for all the series. Fills in values at new nodes by the specified strategy.

Hyper-parameters:
strategy - ' NA', 'nearest', 'last', 'next'

design questions

• would it make sense to create an "interpolator" class which in predict takes a series and an index sequence and returns the values?
• does it make sense to expose dedicated index parameter interfaces in some of the above?

wrapper which implements grid search tuning for TSC/TSR methods

interface should be exactly as GridSearchCV in sklearn except for the TSC/TSR use case, i.e.:

• wrapper method with constructor initialization
• lazy use of data, only in tuned method's fit/predict

Conditional on #3 and a "proper" hyper-parameter interface in #2 since calls evaluation per hyper-parameter choice in grid

Not sure if this is the best way to structure, not that familiar with GIT, so advise if there is a better way. First project is to integrate sktime with keras so that we can reproduce existing research. I've added some suggested tasks on the project following a MoSCoW approach, but please adapt as you see fit. I am keen that we test it all installs properly on windows early on though.

Implement transformer from nested pandas df to tabular data

Sources to investigate for copy/adapt:

sklearn, functionTransformer, by duck typing

pysf (MultiRowTransformer)

tslearn, seglearn, pyts

Worfklow automation, method evaluation, benchmarking and post-hoc analyses
Should be straightforward (but slightly tedious) by adapting mlaut framework interface

Conditional on evaluation framework #3 which in turn is conditional on prediction interface #2

implement forecasting interface according to #18

Not sure if this is the best way to structure, not that familiar with GIT, so advise if there is a better way. First project is to integrate sktime with keras so that we can reproduce existing research. I've added some suggested tasks on the project following a MoSCoW approach, but please adapt as you see fit. I am keen that we test it all installs properly on windows early on though.

This might interest you

https://github.com/alan-turing-institute/MLJ.jl/wiki/2019-MLJ---sktime-tutorial-and-development-sprint

Applications are closing shortly

I will by default use camel case for classifier names etc. This is just my habit. Can we formalise some naming conventions to help me please? Probably best to copy sklearn?

Not necessarily in sktime unified interface.
Priority is scalable/efficient and robust implementation which can be interfaced.

Care needs to be taken with writing code in a way such that it includes:

• a fitting method, to data, which returns a stored models
• a prediction method, which takes a model and returns predictions given the feature
• a method interface which explicitly exposes hyper-parameters (e.g., per declaration or as a return dictionary)

currently loading data is performed in utils.load_data and works for .ts format. It would be good to bring back the methods to load from arff and from long format and add them as methods here

I've been writing these in issues now.

A nicer (and more persistent) place should be created in the wiki for this, potentially updated after subsequent discussions (mirroring the status quo).

This is to provide a solid source for whenever we wish to write a paper/publication or manual.

(maybe best to assign to me, but let's discuss next meeting)

investigate what would need changing to make pysf to interface with a different data container (FK)

sklearn-like interface compatible with xpandas

Specify methods, variables, fit/transform interface

• consistent naming, underscores
• make sure they work

As @sajaysurya pointed out, we may not at all need xpandas for the core use cases.

This is a high-priority issue to be decided by w/c Feb 4 meeting, since it the data container is a central design decision. This thread is to collect pro/con until the decision is made - issue is complete once we decide to remain with, or leave xpandas as the data container solution for the API and implement the alternative (note: in the case of leave the issue is only complete once the alternative is implemented in the existing code).

thread to discuss the proximity forest implementation. Contributors

1. George Oastler (UEA)
2. Jason Lines (UEA)
3. Francois Petitjean (Monash)
4. Ahmed Shifas (Monash)

This should be a full specification document, including:

• class methods, should include at least "fit" and "predict"
• class variables (if applicable)
• hyper-parameter interface methods, e.g., getter/setters

As discussed in #5

Interface UEA algorithms from their dedicated module.

Obviousuly conditional on #9, the algorithms implemented, and #2, the predictor-estimator interface implemented.

• For reduction strategies from classical forecasting to time series regression, tuning is already covered by sklearn's GridSearchCV class.
• For classical forecasters, like ARIMA, we need low-level tuning meta-estimator(s) similar to GridSearchCV and other available tuning strategies.

split off from #1

implementation of rudimentary evaluation functionality, should include:

• computation of methods' predictions on test set, after training on training set
• computation of common average and aggregate losses/scores for classification and regression
• interface to metrics/scoring functionality as in sklearn
• at first, preliminary design of the above

Note: not full-blown orchestration and experiment management

Conditional on #2 since calls the estimator interface which needs to be consolidated first.

to go with the extension guidelines

should check conformity with interface design and extension guidelines

To consider in the first instance

1. basic structure (Tony approach vs Marcus approach)
2. handling missing values (transform vs bespoke)
3. Classifying multivariate data (independent vs dependent).

Point 1 is obviously fundamental. To summarise, this is how I see the differences, I have done a minimal sci-kit learn implementation where the intervals and summary stats are hard coded, but the base classifier is configurable through the constructor. Marcus has implemented it with an internal Pipeline, where the transforms are externally configurable through the constructor, but the classifier is hard coded. I have inherited from RandomForest directly, whereas Marcus has cloned and adapted the base classifiers and the ensembles.

My reasoning is that, whilst a generic configurable TSClassifier with a pipeline is desirable, it should be the base classifier. If we implement a classifier called TSF that directly refers to a paper that describes the algorithm, then we should match the paper description. With a general purpose pipeline, users could build a classifier that is called TSF but is in fact something completely different. I also think transformers may be an over design for this very simple algorithm, and I would like to test what overhead (if any) they introduce.

Obviously, I prefer mine :) I am of course more than happy to discuss and to go the other way if that is the consensus.

For easy loading into workspace.

Similar to existing example code in xpandas but better.

needs to contain:

• description of folder structure
• description of necessary API elements to implement, inheritance
• for atomic transformers (series-to-tabular, series-to-series; one-series, multi-row)
• and for atomic classifiers and regressors
• low and high level interface

Separated from #7.

Reminder for @mloning and I. Cython is currently a dependency, but if you're using non-cython stuff you still have to compile it (which we know can be very painful). I've seen cython compiled on the fly, so maybe that should be the deal for ease of use?

• See pmdarima Python package for a working example
• related to #7 and #44

thread to discuss collaboration with Basque Centre for Applied Mathematics, based on this paper

https://link.springer.com/content/pdf/10.1007%2Fs10618-018-0596-4.pdf

This is arising from #9 and #10, and @TonyBagnall 's suggestion of sub-projects.

I propose to have one branch per algorithm in development, then @jasonlines @TonyBagnall to review & approve pull requests into dev.

Collaborators to be added here - I think with the golden Turing subscription to Github we have fine-grained control over credentials etc.

Describe the bug

When using sktime.utils.load_data.load_from_ucr_tsv_to_dataframe to load one of UCR's TSV files as a DataFrame whose cells contain Series, the indices of those Series start at 1, as opposed to 0, what one would expect.

This leads to problems when fitting the sktime.classifiers.elastic_ensemble.ElasticEnsemble on such a DataFrame since that estimator (as well as some utility components employed by the estimator) expect the Series indices to start at 0.

I have no idea why sktime.utils.load_data.load_from_ucr_tsv_to_dataframe creates 1-indexed Series. For now, I'm working around the problem by manually resetting the indices of all loaded Series after import.

I haven't checked whether other means of loading the data also behave this way.

To Reproduce

from sktime.utils.load_data import load_from_ucr_tsv_to_dataframe
from sktime.classifiers.elastic_ensemble import ElasticEnsemble

X_train, y_train = load_from_ucr_tsv_to_dataframe("GunPoint_TRAIN.tsv")

# what follows is the workaround
#X_train = X_train.applymap(lambda series: series.reset_index(drop=True))

elastic_ensemble = ElasticEnsemble()
elastic_ensemble.fit(X_train, y_train)

Expected behavior

To reiterate, load_from_ucr_tsv_to_dataframe should yield a DataFrame containing Series whose indices always start at 0.

Versions

Bake-off in python toolbox with full set of methods and suite of post-hoc analyses.

Conditional on:

• data loading implemented #8
• baseline methods and transformer principles implemented #7
• UEA favourite methods implemented #10
• grid-tuning implemented #4

for supporting #6.

Triaging based on favourite list.
To be put against existing implementations of transformers in tslearn, tsfresh, pyts, numpy.
Make decision of implement/interface/leave it (with priority perhaps).
To add a time estimate for implement/interface.

1. Use sphinx with doc strings
2. Settle on convention how to write doc strings
3. Disseminate info of how to write doc strings that conforms to it

• For classical forecasting API, see #18, #218 and corresponding API design document
• For exemplar supervised forecasting API, see pysf

• transformers for distribution of missing values (see #6)
• adapting existing algorithms to internally handle missing values if possible
• add details about missing values in meta-data

aka: FK and AG to finish their paper

• AutoARIMA, see existing Python implementation based on original R implementation in forecast package
• Theta method
• "Seasonal last" strategy for DummyForecaster