Requirement Text
Extended Kalman Filter State Estimation

Suggested Solution
EKF using https://filterpy.readthedocs.io/en/latest/kalman/ExtendedKalmanFilter.html

DoD

• Implement feature
• Add new example
• Add tests
• Add to change notes for next release

Note
Suggested by @chetankul

Requirement

Add alternate battery model (from SWS project)

Background

See: https://github.com/nasa/GSAP/blob/SimplifiedEC/src/Models/SimplifiedCircuitBatteryModel.cpp

Definition of Done

• Implement Simplified EC model
• Create test
• Integrate test into automated tests
• Create example using model
• Integrate example into automated tests and all examples
• Add new model to release notes

Requirement Text
Add a basic feed forward neural network option, similar to the lstm

DoD

• Implement feature
• Add tests
• update documentation
• Add to change notes for next release

Describe the bug
For predictors, when setting events = [] should predict until time, ignoring events. This doesn't work for UT Predictor

To Reproduce
Steps to reproduce the behavior:

1. Go to utpredictor example
2. Set events = [] and horizon = 50
3. Run

Requirement Text
Add the ability to sample using the Latin Hypercube sampling method in the UKF, MC, and general sampling

Suggested Solution
Add the ability to specify the sampling method in PF/MC sampling and in the UnweightedSamples sampling. Implement algorithms and put in utils sub package.

DoD

• Implement feature
• Add to example
• Add tests
• update documentation
• Add to change notes for next release

Look at Marcos' work in Automatic adaptive step size. See if marcos is willing to partner and investigate ways that it can be adopted in progpy

Requirement Text
Add option to shuffle the data being used to train the LSTM

DoD

• Add feature
• Add to example

Requirement Text
Create a new kalman filter based predictor that operates on linear models

DoD

• Implement kalman predictor
• Add to example
• Add tests
• Add to change notes for next release

See: https://pomegranate.readthedocs.io

pomegranate is a Python package that implements fast and flexible probabilistic models ranging from individual probability distributions to compositional models such as Bayesian networks and hidden Markov models.

Could be leveraged within ProgPy.

Requirement Text
Create a CNN state transition model

DoD

• implement
• Add example
• Add tests
• Add to change notes for next release

Requirement Text
Restructure storage of data from simulation (e.g., SimResult, *Container) to leverage Pandas as appropriate

Background Information
In nasa/prog_models#482 @mstraut investigated options for data storage. The result was a design decision to use Pandas. This was partially implemented in prog_models v1.5

DoD

• Implement feature
• Update examples
• Update tutorial
• Update tests (as needed)
• update documentation
• Add to change notes for next release

https://github.com/nasa/trick

Requirement Text
Allow the user to specify the number of Dense layers for each output of the LSTM.

Background Information
Additional dense layers could improve the ability to represent complex relationships between the common system state (output of LSTM) and the final output

Suggested Solution
Configuration argument

DoD

• Implement feature
• Add to example
• Add tests
• update documentation
• Add to change notes for next release

Requirement Text
UAV model shall be able to simulate forward at any reasonable dt

Background Information
Right now the flight model requires that you use a set dt- that used to make the waypoints.

Suggested Solution
Interpolate if between points

DoD

• Implement feature
• Add to example
• Add tests
• update documentation for feature

Requirement Text
Implement sizeof method for containers (e.g., prediction, UncertainData, and ToE Prediction Profile), state estimators, and predictors

Background Information
sizeof determines the amount of size an object takes

Suggested Solution
state estimator and predictors: size of object + size of parameters + size of model
containers- size of object + size of data

minimum test - make sure size of state est/pred is greater than size of parameters, and adding new parameter increases size. Then that adding a param to the model increases size
Similar to container, adding data increases size.

DoD

• Implement feature
• Add tests

Requirement

Implement Thermal event for electrochemistry battery model

DoD

• Implement event
• Add documentation
• Add tests
• Add to release notes

Requirement Text
Ensemble model shall allow creation with one model

Background Information
in v1.5, EnsembleModel artificially rejected creation with <2 models

Suggested Solution
Remove check

Note
Suggestion from v1.5 release review

DoD

• Implement feature
• Add to tests

Requirement Text
This feature should be able to return a Warning detailing option keys that do not not exist in the method's options.
The current optimize.show_options function currently either returns a String or prints out the String to the user. However, the current text that is being returned does not necessarily accomplish tasks such that it functions without any worrying

Background Information
The current optimize.show_options function currently either returns a String or prints out the String to the user. However, the current text that is being returned does not necessarily accomplish tasks such that it functions without worrying about future changes done to the documentation/functionality of optimize.show_options. As such, we are mandated to verify that the returned Warnings are working as intended, which is manual behavior that should be abstracted away.

Suggested Solution
Using Regex to parse returned text and return options in some data structure. However, does not accommodate future changes develop via SciPy's side, so this solution is only a temporary one.

Some other solutions may be to track changes done to the optimize.show_options website page, and whenever there are changes done, this will notify us to make changes. This however, still requires us to rework the parser. Furthermore, there could be some downtime between developing changes to the function and documenting updates to their website page.

Looking through the source code and/or the documentation may also provide some inspiration on solving the issue.

optimize.show_options documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.show_options.html

Source Code: https://github.com/scipy/scipy/blob/v1.10.1/scipy/optimize/_optimize.py#L3732-L3951

DoD
edit this to be a list of tasks that need to be completed

• Implement feature
• Add to example
• Add testing to test_estimate_params.py
• Update documentation
• Add to change notes for next release

Requirement

Create a new model that can represent a battery pack - i.e., multiple cells in some combination of series and parallel

Background

Current model is cell-level

DoD

• Implement model
• Add tests
• Add example
• Add documentation
• Add to release notes

Requirement Text
Update lstm to support direct model (i.e., time_of_event)

Background Information
Currently lstm is just state transition

DoD

• Implement feature
• Add to example
• Add tests
• update documentation
• Add to change notes for next release

Requirement Text
Provide a special container to support states containing a window of data and extra data

Background Information
Some models (e.g., LSTM) store a window of historical data as flattened state u_t, z_t, u_t-1, ...

Suggested Solution
Container with add_data(u) method and .extra property

DoD

• Implement feature
• Update LSTM
• Add tests
• update documentation
• Add to change notes for next release

Explore tools for model-based diagnosis determining the likelihood of each event occurring giving a distribution of states (from state estimator)

DoD

• Implement example
• Implement tools (if appropriate)
• Add documentation
• Add to release notes

Requirement

Implement guidance and/or methods for estimating process and sensor noise from data

Background

Process and sensor noise is very important for noise representation. There is currently no guidance on how to estimate this

@portia19 - this is related to a discussion we've been having on validation of prognostics models

DoD

• Implement example demonstrating how to do this
• Implement tools to help, as appropriate
• Add tests (if new tools added)
• Add to release notes (if new tools added)

Requirement Text
Implement logging in progpy

Background Information
Right now it can be difficult to debug because of lack of logging. Some simple logging would help with this

Suggested Solution
Use logging package https://docs.python.org/3/library/logging.html. DONT LOG IN INNER LOOP OF SIMULATION, EXCEPT FOR ERROR CASES

DoD

• Implement feature
• Add to example
• Add to tutorial
• Add tests
• update documentation
• Add to change notes for next release

Requirement Text
Lstm shall provide ability to learn performance metrics

Background Information
Lstm now does everything except pm

Suggested Solution
Same as output

DoD

• Implement feature
• Add to example
• Add tests
• update documentation

Requirement Text
Implement k-fold validation

Background Information
See page 135 of deep learning with python

DoD

• Implement
• Add to example
• Add to change notes for next release

Add step to CI/CD infrastructure that will auto publish the newest version to PyPi (i.e., pip) on each github release. See example here (from chaospy)

Requirement Text
Ensemble model should return a measurement of dispersion

Background Information
Ensemble model aggregates the results of the component models, but does not produce any measurement of the dispersion of the results from the models

Suggested Solution
Allow specification of "dispersion metrics" as parameter. Metrics will be calculated each time values are aggregated. The result is then saved as an attribute of the *Container (e.g., StateContainer, OutputContainer, etc.).

E.g.,
dispersion_metrics = {'std': np.std, 'custom1': some_custom_method}

Then
X = m.next_state(...)
Print(X.dispersion_metrics['std'])
Print(X.dispersion_metrics['custom1'])

Or just one metric
Dispersion = np.std

Then x.dispersion

Another thought:
It would be nice if it's only calculated when you need it (e.g., at save_pts). Maybe we save raw data as member, and calculate when saving

DoD

• Implement feature
• Add to example
• Add tests
• update documentation

Requirement Text
Create functions, classes, or features to simplify future load for data model

Background Information
Currently very complicated.

DoD

• Implement feature
• Add to example
• Add tests
• update documentation
• Add to change notes for next release

Get state estimation working with data models (e.g., LSTMStateTransitionModel)

DoD

• Implement example
• Make any changes required to get it working
• Add documentation to help users
• Add to release notes

Requirements

Explore methods for using time_of_event fin prediction

Potential Solutions

• If no more save_pts, switch to time_of_event (to get final time, since no more states are required)

DoD

• Implement feature
• Add tests

DoD

• Explore compatibility with Simpy
• Add issues, as needed, for follow-up work to ensure or demonstrate compatibility

Currently parameter estimation only supports scalars- this issue is to add support for lists as well

D.o.D.

• Parameter estimation works for lists too
• feature is tested
• Added to release notes

Extend Data Models to include CMAPSS 2 dataset

Requirement Text
Implement methods for reducing LSTM model size

Suggested Solution
.reduce method

DoD

• Implement method
• Add example
• Add tests
• Add to change notes for next release

Requirement Text
Update MC so that different samples can run concurrently

Background Information
Current MC operation is single-threaded. Ran into issues making it multithreaded in the past

DoD

• Implement feature
• update documentation
• Add to change notes for next release

notes
https://www.computer.org/csdl/magazine/cs/2021/06/09658236/1zw1ldymkJW

Python doesn't always parallelize threads as you would expect

Requirement Text
Pre-generate loads before Piecewise.call

Background Information
Right now InputContainers are build when call is called, this is unnecessary

Suggested Solution
Pregenerate in init

DoD

• Implement feature

Add some sort of factor thats dependent on the batteries environment?

For example: If two batteries are being modeled, one is inclosed in a chamber the other exposed to space, the first battery's temperature would likely increase much more with the same loading than the one exposed to space (due to higher ambient temperature, or an ambient temperature highly correlated to the battery temperature).

This task is to add this affect to the model.

Perhaps some sort of ambient correlation factor (tCorr), such that

Tamb = (1.0-params['tCorr']) * params['x0']['Tb'] + params['tCorr'] * Tb
Tbdot = (voltage_eta*i/mC) + ((Tamb - Tb)/tau)

This way correlation of 0 would use the ambient temperature of space, 1 would use the temperature of the battery, and in between would use some proportion

Create interface for integrating models with C++ GSAP framework

Background

GSAP is a C++ framework for implementing prognostics applications. It follows a very similar format. Generally python is highly comparable with C++

D.o.D

• Interface exists in GSAP that integrates a prog_models model into GSAP
• Add to GSAP tests

Requirement Text
support historical window state where not every value is used in state (e.g., u_t, u_t-5, u_t-10)

Suggested Solution
Extend #22

DoD

• Implement feature
• Add to example for lstm
• Add tests
• update documentation
• Add to change notes for next release

Requirement Text
Support the setting of a final (i.e., catch all) loading level for progpy.loading.Piecewise

Background Information
Currently this is done manually by setting a time of float('inf')

Suggested Solution
Allow load to be one longer than times to identify catch-all

DoD

• Implement feature
• Add to example
• Add tests
• update documentation

Suggestion from Rodney Martin: Investigate the trend of Model Cards in AI/ML. Identify if there is value in establishing an ontology for communicating quality and limitation of prognostics models.

See also:

• https://modelcards.withgoogle.com/about
• https://arxiv.org/abs/1810.03993
• https://cloud.google.com/blog/products/ai-machine-learning/create-a-model-card-with-scikit-learn

We want to submit a rc version (ends in .rc#, e.g., .rc0) to PyPi, then test it for each commit to a release branch.

Suggestion - see pypi-version to get latest release version. If to left of rc in the package.version (or setup.py) is the same, then increment rc #, otherwise add rc0 to release number. We might have to move version # to a separate file to get this to work.

See Also: #48

Integrate Capacitor Model from Jose Celaya

See also:

• https://babelfish.arc.nasa.gov/bitbucket/projects/TI_DP/repos/capacitor-model/browse/RAMS2012%20Matlab%20Files
• http://papers.phmsociety.org/index.php/phme/article/view/1423

Checklist:

• Implement model
• Add tests
• Add documentation
• Add simulation example
• Add to release notes

Requested by Portia

Requirement Text
List of additional features to be developed to further improve the trajectory generation

DoD
Features to add/enhance:

• Controllers - separate controllers into separate model and add other controller options
• Add checks for unrealistic waypoint inputs
• Update documentation with more detail on models involved
• Add functionality to support any user-defined time step (currently must be constant)
• Made model stateless - Model is not currently stateless, since state values are saved in the parameters (i.e. 'next_waypoint' in event_state). Edit to achieve stateless model