Code Monkey home page Code Monkey logo

faultmap's Introduction

FaultMap

Travis Build(https://travis-ci.org/SimonStreicher/FaultMap)

Code Coverage(https://coveralls.io/github/SimonStreicher/FaultMap?branch=master)

Code Climate(https://codeclimate.com/github/SimonStreicher/FaultMap)

DOI(https://zenodo.org/badge/latestdoi/14229559)

Introduction

FaultMap is a data-driven, model-free process fault detection and diagnosis tool. Causal links between processes elements are identified using information theory measures (transfer entropy). These links are then used to create a visual representation of the main flows of information (disturbances, etc.) among the process elements as a directed graph. These directed graphs are useful as troubleshooting aids.

Network centrality algorithms are applied to determine the most influential elements based on the strength and quality of their influence on other connected nodes (eigenvector centrality).

Documentation and demonstrations still under development.

Prerequisites

Most of the prerequisites are related to getting JPype to work correctly:

  • Python 3.7+ with compatible C++ compiler
    • On Windows, compiling packages usually requires the VC++ 2015.3 v14.00 (v140) toolset for desktop to be installed from the Visual Studio installer.
  • Java JDK 1.8.201+ (or latest Java 8 SDK)
    • The JAVA_HOME environment variable should point to the installation directory.

Installation

git clone https://github.com/SimonStreicher/FaultMap.git
cd FaultMap
conda create --name faultmap python=3.7
source activate faultmap
pip install -r requirements.txt
pytest

A Docker image is available with all necessary packages and dependencies installed.

docker pull simonstreicher/faultmap

If you want to build locally, the Dockerfile can be found at FaultMapDocker GitHub.

Setup

Create directories for storing the data, configuration files as well as results. Create a file caseconfig.json in the root directory, similar to testconfig.json which comes with the distribution. Enter the full path to the data, configuration and results directories as well as the infodynamics.jar you want to use for Java Information Dynamics Toolkit (JIDT) (the tested version is included in the distribution).

Example caseconfig.json file (also included in example_configs directory):

{
  "dataloc": "~/faultmap/faultmap_data",
  "configloc": "~/repos/faultmapconfigs",
  "saveloc": "~/faultmap/faultmap_results",
  "infodynamicsloc": "~/repos/FaultMap/infodynamics.jar"
}

Configuration

Refer to the example_configs directory in the distribution for the required format of configuration files in order to fully define cases and scenarios. The following configuration files are needed to fully specify a specific case:

  1. weightcalc.json
  2. noderank.json
  3. graphreduce.json
  4. plotting.json

Execution

In order to calculate a full set of results for a specific case, make sure this case name is included in the config_full.json file in the directory defined under configloc in the caseconfig.json file.

Example config_full.json file (also included in example_configs directory):

{
  "mode": "cases",
  "writeoutput": true,
  "cases": [
    "tennessee_eastman"
  ]
}

faultmap's People

Contributors

alchemyst avatar sjstreicher avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

faultmap's Issues

Revisit dependencies

I don't think we still use python-igraph as a dependency (I could find a file which imported it). I think we should revise requirements.txt, conda-requirements.txt and sphinx_requirements.txt.

Better memory management

Better memory management is needed for large datasets.
Writing interim results to file between causal variables should go a great distance to fix this.

Current workaround is simply to split sets such that less than about 5000 variable pairs are evaluated in a single job.

Test suite coverage should be 100%

It is clear that some of the refactoring that has been happening has broken code without anyone noticing. This should not happen. We should have 100% test suite coverage. Even demo_all.py doesn't exercise all the code paths at the moment.

Once we have 100% coverage with the test suite, you should get into the habit of running the test suite before committing to stop non-working code from getting into master (non-working code may be acceptable in clearly marked dev branches).

Undefined key in `test_weightcalc.py`

Traceback (most recent call last):
  File "test_weightcalc.py", line 13, in <module>
    weightcalc(mode, case, writeoutput)
  File "/Users/alchemyst/Documents/Subjects/students/2014/streicher_s/LoopRank/ranking/gaincalc.py", line 467, in weightcalc
    causevarindexes = caseconfig[scenario]['causevarindexes']
KeyError: 'causevarindexes'

Check configuration files consistency

Implement a checksum on configuration parameters used to generate results set. Incorporate a check for identical checksums when looping through cases and deciding whether to re-calculate results.

networkgen.py highly redundant

The test networks should be more efficiently stored. At the moment networkgen.py has a high level of redundancy. At least we should try to avoid the duplication inherent in having connection matrices that are only the 0-1 equivalents of the gain matrices. I think data files are more appropriate here than functions. It also allows us to loop over them if need be.

Error when running demo_weightcalc.py

I encounter the following error when running demo_weightcalc.py:

INFO:root:Running scenario raw_truncated
INFO:root:Method: transfer_entropy
INFO:root:Analysing effect of: D8L1003.PV
Traceback (most recent call last):
  File "demo_weightcalc.py", line 32, in <module>
    weightcalc(mode, case, writeoutput)
  File "/Users/alchemyst/Documents/Subjects/students/2014/streicher_s/LoopRank/ranking/gaincalc.py", line 484, in weightcalc
    causevarindexes, affectedvarindexes)
  File "/Users/alchemyst/Documents/Subjects/students/2014/streicher_s/LoopRank/ranking/gaincalc.py", line 334, in estimate_delay
    teCalc = te_setup()
  File "/Users/alchemyst/Documents/Subjects/students/2014/streicher_s/LoopRank/transentropy.py", line 73, in setup_infodynamics_te
    teCalc = teCalcClass()
  File "/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/jpype/_jpackage.py", line 53, in __call__
    raise TypeError, "Package "+self.__name+" is not Callable"
TypeError: Package infodynamics.measures.continuous.kernel.TransferEntropyCalculatorKernel is not Callable

Travis times out

Travis times out when requested to do multiprocessing. However, the tests takes mere seconds to run locally.

Fixup imports

Don't import jpype and from jpype import * in transentropy.py. Also get rid of unused imports in all the files, it just makes it harder to see what is actually being used.

Completely non-interactive mode for running code

At the moment, my default entry-point for the code is demo_all.py. This pops up a window which I must close manually for all the code to run. We need a flag or a configuration option which kills all interaction and runs through the code without waiting for the user at all. This will make testing easier.

Tests and demos must all run

I have modified demo_all to run all files starting with demo_ or test_. However, not all of these files run without errors. If things have changed and some of these files are now obsolete, please delete them.

TypeError: No matching overloads found for infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(map,map)

Hi, got an exception when attempting to use faultmap with the gpu enabled:
I'm using the latest faultmap from the repo and jidt 1.5 compiled with gpu support

What can be done here? It seems it attempts to call a non-existent method.

Kind Regards,
Frederico Valente

INFO:root:Analysing effect of: raiseXERError[SIF-0127] on LRT::SignalPublisher::publish_signals[LRT-0010] for box number: 1
INFO:root:Now testing delay: 0
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/fault_map_env/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/fault_map_env/lib/python3.7/site-packages/multiprocess/pool.py", line 44, in mapstar
return list(map(*args))
File "/fault_map_env/lib/python3.7/site-packages/pathos/helpers/mp_helper.py", line 15, in
func = lambda args: f(*args)
File "/app/FaultMap/faultmap/gaincalc_oneset.py", line 211, in calc_weights_oneset
affectedvarindex,
File "/app/FaultMap/faultmap/gaincalculators.py", line 549, in calcweight
**self.parameters
File "/app/FaultMap/faultmap/transentropy.py", line 201, in calc_infodynamics_te
teCalc.addObservations(source, dest)
TypeError: No matching overloads found for infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(map,map), options are:
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[],int[],boolean[])
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[],int[],int,int)
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[],int[])
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[][][],int,int,int,int)
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[][],int,int)
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[][][],int,int)
public void infodynamics.measures.discrete.TransferEntropyCalculatorDiscrete.addObservations(int[][],int)

    at findOverload(native/common/jp_method.cpp:242)
    at findOverload(native/common/jp_method.cpp:245)
    at invoke(native/common/jp_method.cpp:253)
    at __call__(native/python/pyjp_method.cpp:142)

"""

Refactor write_csv*

The various incarnations of write_csv*() are all the same function - you can refactor to use only one function definition which has an optional header argument.

Error when running demo_all.py

INFO:root:Running demo_looprank.py
INFO:root:Running scenario simple5
INFO:root:Number of tags: 5
Traceback (most recent call last):
  File "demo_all.py", line 18, in 
    runall('demo_*.py', ['demo_all.py'])
  File "demo_all.py", line 15, in runall
    execfile(f)
  File "demo_looprank.py", line 24, in 
    m, alpha)
  File "/Users/alchemyst/Documents/Subjects/students/2014/streicher_s/LoopRank/ranking/noderank.py", line 475, in looprank_static
    calc_maingainrank(modgainmatrix, noderankdata, m)
TypeError: calc_maingainrank() takes exactly 5 arguments (3 given)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.