Code Monkey home page Code Monkey logo

oasislmf's Introduction

Oasis LMF logo

ktools version PyPI version FM Testing Tool

Oasislmf Testing PiWind output check PiWind MDK

OasisLMF

The oasislmf Python package, loosely called the model development kit (MDK) or the MDK package, provides a command line toolkit for developing, testing and running Oasis models end-to-end locally, or remotely via the Oasis API. It can generate ground-up losses (GUL), direct/insured losses (IL) and reinsurance losses (RIL). It can also generate deterministic losses at all these levels.

Versioning and Updates

Current Stable Versions

Release Schedule

Until end of 2023 Until the year 2023, we will be following a six-month release cycle for our stable versions. During each six-month period, we will release a new stable version with added features. These updates will adhere to the Semantic Versioning (semver) format and will increment the minor version number. That version of oaisislmf is then 'frozen' into a branch matching the new version number, so on release 1.28.0 the code base is copied to a branch stable/1.28.x where backported features and fixes are applied.

After 2023 Starting from 2023, we will transition to a yearly release cycle for our stable versions. Each year, we will release a new stable version with additional features.

Monthly Updates

Every month, we will provide updates to the latest stable version. These updates will include new compatible features and bug fixes, ensuring that our software remains up-to-date and reliable.

During the monthly update, if any bug fixes are required, they will also be applied to the older stable versions. This approach guarantees that all stable versions receive necessary bug fixes, while maintaining a consistent output numbers for that stable version.

Features

For running models locally the CLI provides a model subcommand with the following options:

  • model generate-exposure-pre-analysis: generate new Exposure input using user custom code (ex: geo-coding, exposure enhancement, or dis-aggregation...)
  • model generate-keys: generates Oasis keys files from model lookups; these are essentially line items of (location ID, peril ID, coverage type ID, area peril ID, vulnerability ID) where peril ID and coverage type ID span the full set of perils and coverage types that the model supports; if the lookup is for a complex/custom model the keys file will have the same format except that area peril ID and vulnerability ID are replaced by a model data JSON string
  • model generate-oasis-files: generates the Oasis input CSV files for losses (GUL, GUL + IL, or GUL + IL + RIL); it requires the provision of source exposure and optionally source accounts and reinsurance info. and scope files (in OED format), as well as assets for instantiating model lookups and generating keys files
  • model generate-losses: generates losses (GUL, or GUL + IL, or GUL + IL + RIL) from a set of pre-existing Oasis files
  • model run: runs the model from start to finish by generating losses (GUL, or GUL + IL, or GUL + IL + RIL) from the source exposure, and optionally source accounts and reinsurance info. and scope files (in OED or RMS format), as well as assets related to lookup instantiation and keys file generation

The optional --summarise-exposure flag can be issued with model generate-oasis-files and model run to generate a summary of Total Insured Values (TIVs) grouped by coverage type and peril. This produces the exposure_summary_report.json file.

For remote model execution the api subcommand provides the following main subcommand:

  • api run: runs the model remotely (same as model run) but via the Oasis API

For generating deterministic losses an exposure run subcommand is available:

  • exposure run: generates deterministic losses (GUL, or GUL + IL, or GUL + IL + RIL)

The reusable libraries are organised into several sub-packages, the most relevant of which from a model developer or user's perspective are:

  • api_client
  • model_preparation
  • model_execution
  • utils

Minimum Python Requirements

Starting from 1st January 2019, Pandas will no longer be supporting Python 2. As Pandas is a key dependency of the MDK we are dropping Python 2 (2.7) support as of this release (1.3.4). The last version which still supports Python 2.7 is version 1.3.3 (published 12/03/2019).

Also for this release (and all future releases) a minimum of Python 3.8 is required.

Installation

The latest released version of the package, or a specific package version, can be installed using pip:

pip install oasislmf[==<version string>]

Alternatively you can install the latest development version using:

pip install git+{https,ssh}://[email protected]/OasisLMF/OasisLMF

You can also install from a specific branch <branch name> using:

pip install [-v] git+{https,ssh}://[email protected]/OasisLMF/OasisLMF.git@<branch name>#egg=oasislmf

Enable Bash completion

Bash completion is a functionality which bash helps users type their commands by presenting possible options when users press the tab key while typing a command.

Once oasislmf is installed you'll need to be activate the feature by sourcing a bash file. (only needs to be run once)

Local

oasislmf admin enable-bash-complete

Global

echo 'complete -C completer_oasislmf oasislmf' | sudo tee /usr/share/bash-completion/completions/oasislmf

Dependencies

System

The package provides a built-in lookup framework (oasislmf.model_preparation.lookup.OasisLookup) which uses the Rtree Python package, which in turn requires the libspatialindex spatial indexing C library.

https://libspatialindex.github.io/index.html

Linux users can install the development version of libspatialindex from the command line using apt.

[sudo] apt install -y libspatialindex-dev

and OS X users can do the same via brew.

brew install spatialindex

The PiWind demonstration model uses the built-in lookup framework, therefore running PiWind or any model which uses the built-in lookup, requires that you install libspatialindex.

GNU/Linux

For GNU/Linux the following is a specific list of required system libraries

  • Debian: g++ compiler build-essential, libtool, zlib1g-dev autoconf on debian distros

    sudo apt install g++ build-essential libtool zlib1g-dev autoconf

  • Red Hat: 'Development Tools' and zlib-devel

Python

Package Python dependencies are controlled by pip-tools. To install the development dependencies first, install pip-tools using:

pip install pip-tools

and run:

pip-sync

To add new dependencies to the development requirements add the package name to requirements.in or to add a new dependency to the installed package add the package name to requirements-package.in. Version specifiers can be supplied to the packages but these should be kept as loose as possible so that all packages can be easily updated and there will be fewer conflict when installing.

After adding packages to either *.in file:

pip-compile && pip-sync

should be ran ensuring the development dependencies are kept up to date.

ods_tools

OasisLMF uses the ods_tools package to read exposure files and the setting files The version compatible with each OasisLMF is manage in the requirement files. below is the summary:

  • OasisLMF 1.23.x or before => no ods_tools
  • OasisLMF 1.26.x => use ods_tools 2.3.2
  • OasisLMF 1.27.0 => use ods_tools 3.0.0 or later
  • OasisLMF 1.27.1 => use ods_tools 3.0.0 or later
  • OasisLMF 1.27.2 => use ods_tools 3.0.4 or later

pandas

Pandas has released its major version number 2 breaking some of the compatibility with the 1st version Therefore, for all version of OasisLMF <= 1.27.2, the latest supported version for pandas is 1.5.3 Support for pandas 2, starts from version 1.27.3

Testing

To test the code style run:

flake8

To test against all supported python versions run:

tox

To test against your currently installed version of python run:

py.test

To run the full test suite run:

./runtests.sh

Publishing

Before publishing the latest version of the package make you sure increment the __version__ value in oasislmf/__init__.py, and commit the change. You'll also need to install the twine Python package which setuptools uses for publishing packages on PyPI. If publishing wheels then you'll also need to install the wheel Python package.

Using the publish subcommand in setup.py

The distribution format can be either a source distribution or a platform-specific wheel. To publish the source distribution package run:

python setup.py publish --sdist

or to publish the platform specific wheel run:

python setup.py publish --wheel

Creating a bdist for another platform

To create a distribution for a non-host platform use the --plat-name flag:

 python setup.py bdist_wheel --plat-name Linux_x86_64

 or

 python setup.py bdist_wheel --plat-name Darwin_x86_64

Manually publishing, with a GPG signature

The first step is to create the distribution package with the desired format: for the source distribution run:

python setup.py sdist

which will create a .tar.gz file in the dist subfolder, or for the platform specific wheel run:

python setup.py bdist_wheel

which will create .whl file in the dist subfolder. To attach a GPG signature using your default private key you can then run:

gpg --detach-sign -a dist/<package file name>.{tar.gz,whl}

This will create .asc signature file named <package file name>.{tar.gz,whl}.asc in dist. You can just publish the package with the signature using:

twine upload dist/<package file name>.{tar.gz,whl} dist/<package file name>.{tar.gz,whl}.asc

Documentation

License

The code in this project is licensed under BSD 3-clause license.

oasislmf's People

Contributors

awsbuild avatar bbetov-clgx avatar benhayes21 avatar carlfischerjba avatar danielfevans avatar fl-ndaq avatar hchagani-oasislmf avatar jbadougburns avatar jcrane512 avatar johcarter avatar kamalcharles avatar maxwellflitton avatar mtazzari avatar nastasi-oq avatar ncerutti avatar omegadroid avatar sambles avatar sr-murthy avatar sstruzik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oasislmf's Issues

No parameter for source accounts file

For oasislmf model sub-commands, you can set the source accounts file in the json using "source_accounts_file_path". This does not seem to be available or documented as a command line switch.

Arg parsing error in 1.2.1

From the Zurich workshop repo:

oasislmf model generate-oasis-files \
    -o "oasis_files" \
    -k "keys_data/MEEQ" \
    -v "keys_data/MEEQ/ModelVersion.csv" \
    -p "catrisks-meeq-canonical-loc-profile.json" \
    -e "source_data/MEEQ_loc.csv" \
    -b "transformation_files/MappingMapToCatrisks_CanLoc_A.xslt" \
    -d "transformation_files/MappingMapToCatrisks_ModelLoc.xslt"

Error trace:

Found canonical exposures profile: {'ROW_ID': {'ProfileElementName': 'ROW_ID', 'FieldName': 'LocationID'}, 'ACCNTNUM': {'ProfileElementName': 'ACCNTNUM', 'FieldName': 'AccountNumber'}, 'LOCNUM': {}, 'LATITUDE': {}, 'LONGITUDE': {}, 'POSTALCODE': {}, 'ADDRMATCH': {}, 'STATE': {}, 'STATECODE': {}, 'COUNTY': {}, 'COUNTYCODE': {}, 'CITY': {}, 'CITYCODE': {}, 'COUNTRY': {}, 'CNTRYSCHEME': {}, 'CNTRYCODE': {}, 'BLDGSCHEME': {}, 'BLDGCLASS': {}, 'OCCSCHEME': {}, 'OCCTYPE': {}, 'NUMBLDGS': {}, 'EQCV1VAL': {'ProfileElementName': 'EQCV1VAL', 'FieldName': 'TIV', 'PerilID': 3, 'CoverageTypeID': 1, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'TIV', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQCV1VCUR': {}, 'EQCV1DED': {'ProfileElementName': 'EQCV1DED', 'FieldName': 'CoverageDeductible', 'PerilID': 3, 'CoverageTypeID': 1, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'Deductible', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQCV1DCUR': {}, 'EQCV1LIMIT': {'ProfileElementName': 'EQCV1LIMIT', 'FieldName': 'CoverageLimit', 'PerilID': 3, 'CoverageTypeID': 1, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'Limit', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQCV1LCUR': {}, 'EQCV2VAL': {'ProfileElementName': 'EQCV2VAL', 'FieldName': 'TIV', 'PerilID': 3, 'CoverageTypeID': 3, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'TIV', 'FMTermGroupID': 2, 'ProfileType': 'Loc'}, 'EQCV2VCUR': {}, 'EQCV2DED': {'ProfileElementName': 'EQCV2DED', 'FieldName': 'CoverageDeductible', 'PerilID': 3, 'CoverageTypeID': 3, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'Deductible', 'FMTermGroupID': 2, 'ProfileType': 'Loc'}, 'EQCV2DCUR': {}, 'EQCV2LIMIT': {'ProfileElementName': 'EQCV2LIMIT', 'FieldName': 'CoverageLimit', 'PerilID': 3, 'CoverageTypeID': 3, 'FMLevelName': 'Coverage', 'FMLevel': 1, 'FMTermType': 'Limit', 'FMTermGroupID': 2, 'ProfileType': 'Loc'}, 'EQCV2LCUR': {}, 'EQSITELIM': {'ProfileElementName': 'EQSITELIM', 'FieldName': 'SiteLimit', 'FMLevelName': 'Site', 'FMLevel': 3, 'FMTermType': 'Limit', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQSITELCUR': {}, 'EQSITEDED': {'ProfileElementName': 'EQSITEDED', 'FieldName': 'SiteDeductible', 'FMLevelName': 'Site', 'FMLevel': 3, 'FMTermType': 'Deductible', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQSITEDCUR': {}, 'EQCOMBINEDLIM': {'ProfileElementName': 'EQCOMBINEDLIM', 'FieldName': 'CombinedLimit', 'FMLevelName': 'Combined', 'FMLevel': 2, 'FMTermType': 'Limit', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQCOMBINEDLCUR': {}, 'EQCOMBINEDDED': {'ProfileElementName': 'EQCOMBINEDDED', 'FieldName': 'CombinedDeductible', 'FMLevelName': 'Combined', 'FMLevel': 2, 'FMTermType': 'Deductible', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}, 'EQCOMBINEDDCUR': {}, 'COND1TYPE': {}, 'COND1NAME': {'ProfileElementName': 'COND1NAME', 'FieldName': 'SubLimitReference'}, 'COND1DEDUCTIBLE': {'ProfileElementName': 'COND1DEDUCTIBLE', 'FieldName': 'SubLimitDeductible'}, 'COND1LIMIT': {'ProfileElementName': 'COND1LIMIT', 'FieldName': 'SubLimitLimit', 'FMLevelName': 'Sublimit', 'FMLevel': 4, 'FMTermType': 'Limit', 'FMTermGroupID': 1, 'ProfileType': 'Loc'}}

Writing canonical exposures file /home/sam/repos/support/ZurichWorkshop2018/oasis_files/canexp-20181122121235.csv

Writing model exposures file /home/sam/repos/support/ZurichWorkshop2018/oasis_files/modexp-20181122121235.csv
Traceback (most recent call last):
  File "/home/sam/repos/core/env/py3/bin/oasislmf", line 7, in <module>
    sys.exit(RootCmd().run())
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/cmd/root.py", line 36, in run
    return super(OasisBaseCommand, self).run(args=args)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/argparsetree/cmd.py", line 159, in run
    return cmd_cls(sub_command_name).run(args)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/argparsetree/cmd.py", line 159, in run
    return cmd_cls(sub_command_name).run(args)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/argparsetree/cmd.py", line 161, in run
    return self.action(args) or 0
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/cmd/model.py", line 621, in action
    logger=self.logger
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/exposures/manager.py", line 1662, in start_oasis_files_pipeline
    self.transform_canonical_to_model(oasis_model=oasis_model, **kwargs)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/exposures/manager.py", line 534, in transform_canonical_to_model
    translator = Translator(input_file_path, output_file_path, transformation_file_path, xsd_path=validation_file_path, append_row_nums=False)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/exposures/csv_trans.py", line 51, in __init__
    self.xsd = (etree.parse(xsd_path) if xsd_path else None)
  File "src/lxml/etree.pyx", line 3444, in lxml.etree.parse (src/lxml/etree.c:83185)
  File "src/lxml/parser.pxi", line 1834, in lxml.etree._parseDocument (src/lxml/etree.c:120757)
  File "src/lxml/parser.pxi", line 1860, in lxml.etree._parseDocumentFromURL (src/lxml/etree.c:121104)
  File "src/lxml/parser.pxi", line 1764, in lxml.etree._parseDocFromFile (src/lxml/etree.c:120012)
  File "src/lxml/parser.pxi", line 1161, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/etree.c:114561)
  File "src/lxml/parser.pxi", line 598, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/etree.c:107738)
  File "src/lxml/parser.pxi", line 709, in lxml.etree._handleParseResult (src/lxml/etree.c:109447)
  File "src/lxml/parser.pxi", line 638, in lxml.etree._raiseParseError (src/lxml/etree.c:108301)
  File "/home/sam/repos/support/ZurichWorkshop2018/source_data/MEEQ_loc.csv", line 1
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

Installation failure in a Python 3 virtual env

Having created a local Python 3 virtual env. using

python3 -m venv oasislmf2-venv

and activating it, I had the following error in installing oasislmf (latest version 0.0.0.8):

(oasislmf2-venv) [srm@~/Documents/sandeep/cst/dev/oasis]$ pip3 install --no-cache-dir oasislmf
Collecting oasislmf
  Downloading oasislmf-1.0.8.tar.gz (42kB)
    100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 51kB 2.9MB/s 
Requirement already satisfied: six in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: pyodbc in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: requests in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: requests-toolbelt in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: shutilwhich in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: jsonpickle in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: pytz in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: pathlib2 in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: backports.tempfile in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: chainmap in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: argparsetree in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: python-interface in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: pandas in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: lxml in ./oasislmf2-venv/lib/python3.6/site-packages (from oasislmf)
Requirement already satisfied: idna<2.7,>=2.5 in ./oasislmf2-venv/lib/python3.6/site-packages (from requests->oasislmf)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./oasislmf2-venv/lib/python3.6/site-packages (from requests->oasislmf)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./oasislmf2-venv/lib/python3.6/site-packages (from requests->oasislmf)
Requirement already satisfied: certifi>=2017.4.17 in ./oasislmf2-venv/lib/python3.6/site-packages (from requests->oasislmf)
Requirement already satisfied: backports.weakref in ./oasislmf2-venv/lib/python3.6/site-packages (from backports.tempfile->oasislmf)
Requirement already satisfied: python-dateutil>=2 in ./oasislmf2-venv/lib/python3.6/site-packages (from pandas->oasislmf)
Requirement already satisfied: numpy>=1.9.0 in ./oasislmf2-venv/lib/python3.6/site-packages (from pandas->oasislmf)
Installing collected packages: oasislmf
  Running setup.py install for oasislmf ... error
    Complete output from command /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf2-venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-build-l8yxwkuj/oasislmf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-byih5aeb-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf2-venv/include/site/python3.6/oasislmf:
    running install
    Retrieving ktools 0_0_392_0
    Failed to get ktools tar (attempt 1)
    Failed to get ktools tar (attempt 2)
    Failed to get ktools tar (attempt 3)
    Failed to get ktools tar after 3 attempts
    error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)>
    
    ----------------------------------------
Command "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf2-venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-build-l8yxwkuj/oasislmf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-byih5aeb-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf2-venv/include/site/python3.6/oasislmf" failed with error code 1 in /private/var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-build-l8yxwkuj/oasislmf/
You are using pip version 9.0.1, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

OED update required fields

Matthew Jones [9:15 AM]
@pinkerton @joh Carter We are having some issues running reinsurance files through 0.396. Specifically: ReinsLayerNumber and TreatyShare seem to be mandatory fields (they shouldn't be) and, more importantly, in the reins_scope file it seems that blanks for PolNumber are not accepted - which makes filtering awkward if you have LocNumber (+AccNumber) filtering required where there are multiple policies in an account. Can you let me know which fields accept blanks and which do not, and whether there is a quick way to relax the mandatory requirement for some of the non-essential fields?

Bug (generate keys)

One particular bug identified in the keys generation subcommand (oasislmf model generate-keys)

(py27) [gdr@rdu-pydev01:~/oasis/omdk]$ oasislmf model generate-keys -k ../WPC_MEX_TC/keys_data/WPC_MEX_TC -v ../WPC_MEX_TC/keys_data/WPC_MEX_TC/ModelVersion.csv -l ../WPC_MEX_TC/src/keys_server/WPC_MEX_TC -e ../WPC_MEX_TC/keys_data/WPC_MEX_TC/exposures.csv -t 'list_keys' ./output_dail/test1.csv
Traceback (most recent call last):
  File "/anaconda3-440/envs/py27/bin/oasislmf", line 7, in <module>
    sys.exit(RootCmd().run())
  File "/anaconda3-440/envs/py27/lib/python2.7/site-packages/oasislmf/cmd/root.py", line 36, in run
    return super(OasisBaseCommand, self).run(args=args)
  File "/anaconda3-440/envs/py27/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
    return cmd_cls(sub_command_name).run(args)
  File "/anaconda3-440/envs/py27/lib/python2.7/site-packages/argparsetree/cmd.py", line 159, in run
    return cmd_cls(sub_command_name).run(args)
  File "/anaconda3-440/envs/py27/lib/python2.7/site-packages/argparsetree/cmd.py", line 161, in run
    return self.action(args) or 0
  File "/anaconda3-440/envs/py27/lib/python2.7/site-packages/oasislmf/cmd/model.py", line 121, in action
    output_file_path=args.output_file_path,
AttributeError: 'Namespace' object has no attribute 'output_file_path'

MDK - keys lookup factory does not generate keys error file

At the moment the MDK keys lookup factory does not generate a keys error file when generating keys files (oasislmf.keys.lookup.OasisKeysLookupFactory.save_keys). Hence the exposure manager doesn't generate one either for the generate-keys or model run subcommands.

Run analysis fails with Python 3.6

I tried to run PiWind test analysis using:
oasislmf model run -C mdk-oasislmf-piwind.json -r /tmp/analysis_test

Gave the following error:
Creating temporary folder /tmp/analysis_test/tmp for Oasis files
Getting model info and creating lookup service instance
Traceback (most recent call last):
File "/home/pinkerton/tutorial-env/bin/oasislmf", line 7, in
sys.exit(RootCmd().run())
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/oasislmf/cmd/root.py", line 36, in run
return super(OasisBaseCommand, self).run(args=args)
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/argparsetree/cmd.py", line 159, in run
return cmd_cls(sub_command_name).run(args)
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/argparsetree/cmd.py", line 161, in run
return self.action(args) or 0
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/oasislmf/cmd/model.py", line 473, in action
gen_oasis_files_cmd.action(args)
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/oasislmf/cmd/model.py", line 231, in action
lookup_package_path=lookup_package_file_path,
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/oasislmf/keys/lookup.py", line 260, in create
lookup_package = cls.get_lookup_package(lookup_package_path)
File "/home/pinkerton/tutorial-env/lib/python3.5/site-packages/oasislmf/keys/lookup.py", line 138, in get_lookup_package
lookup_package = importlib.import_module(package_name)
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/tmp/OasisPiWind/src/keys_server/init.py", line 6, in
from utils import *
ImportError: No module named 'utils'

Exposures manager - interface changes

@OmegaDroid I note that in the exposures manager interface (exposures.manager.OasisExposuresManagerInterface) and its implementation (exposures.manager.OasisExposuresManager) you've made class methods out of get_keys, load_canonical_profile, generate_items_file, generate_gulsummaryxref_file, generate_coverages_file, generate_oasis_files. Originally, in the corresponding classes in the MDK repo, these are just instance methods. Given that the implementation of this interface, i.e the manager, is not being used as a factory class, was there a reason for doing this?

My reasoning is that it makes sense to make a factory class, such as keys.lookup.OasisKeysLookupFactory, have class methods, because its purpose is to be used as a service class and you don't worry about needing multiple instances of the underlying class. But with the manager class it is possible that someone one could want to have multiple manager instances, for example, one to manage each set of models unique to a supplier (and we have multiple suppliers).

What's the advantage of making these methods class methods?

API Client - Error messages are shown as a list of chars

STARTED: oasislmf.api_client.client.health_check
COMPLETED: oasislmf.api_client.client.health_check in 0.0s
Loading analysis settings JSON file:
  OK: analysis_settings={u'analysis_settings': {u'gul_summaries': [{u'leccalc': {u'outputs': {u'full_uncertainty_aep': True, u'full_uncertainty_oep': True}, u'return_period_file': True}, u'lec_output': True, u'summarycalc': True, u'id': 1, u'aalcalc': True}], u'source_tag': u'db', u'il_summaries': [{u'leccalc': {u'out
puts': {u'full_uncertainty_aep': True, u'full_uncertainty_oep': True}, u'return_period_file': True}, u'lec_output': True, u'id': 1, u'aalcalc': True}, {u'leccalc': {u'outputs': {u'wheatsheaf_oep': True}, u'return_period_file': True}, u'id': 2, u'lec_output': True}], u'number_of_samples': 10, u'model_version_id': u'1', u'module_supplier_id': u'ARA', u'gul_threshold': 0, u'gul_output': True, u'il_output': True, u'exposure_location': u'L:', u'model_settings': {u'peril_wind': True, u'event_set': u'P', u'demand_surge': True, u'session_id': 350, u'event_occurrence_id': u'1', u'peril_surge': True}}}, do_il=True                           Running 1 analyses
STARTED: oasislmf.api_client.client.upload_inputs_from_directory
COMPLETED: oasislmf.api_client.client.upload_inputs_from_directory in 0.27s
STARTED: oasislmf.api_client.client.run_analysis_and_poll
STARTED: oasislmf.api_client.client.run_analysis
Analysis started
COMPLETED: oasislmf.api_client.client.run_analysis in 0.01s
Analysis started
Analysis failed: MissingInputsException(u'I', u'n', u'p', u'u', u't', u's', u' ', u'l', u'o', u'c', u'a', u't', u'i', u'o', u'n', u' ', u'n', u'o', u't', u' ', u'f', u'o', u'u', u'n', u'd', u':', u' ', u'/', u'v', u'a', u'r', u'/', u'w', u'w', u'w', u'/', u'o', u'a', u's', u'i', u's', u'/', u'u', u'p', u'l', u'o', u'a
', u'd', u'/', u'2', u'7', u'6', u'd', u'1', u'3', u'8', u'7', u'8', u'6', u'7', u'1', u'4', u'e', u'4', u'd', u'9', u'7', u'a', u'a', u'4', u'5', u'1', u'c', u'6', u'b', u'9', u'f', u'5', u'0', u'4', u'9', u'.', u't', u'a', u'r')                                                                                         Model API test failed: Analysis failed: MissingInputsException(u'I', u'n', u'p', u'u', u't', u's', u' ', u'l', u'o', u'c', u'a', u't', u'i', u'o', u'n', u' ', u'n', u'o', u't', u' ', u'f', u'o', u'u', u'n', u'd', u':', u' ', u'/', u'v', u'a', u'r', u'/', u'w', u'w', u'w', u'/', u'o', u'a', u's', u'i', u's', u'/', u'u'
, u'p', u'l', u'o', u'a', u'd', u'/', u'2', u'7', u'6', u'd', u'1', u'3', u'8', u'7', u'8', u'6', u'7', u'1', u'4', u'e', u'4', u'd', u'9', u'7', u'a', u'a', u'4', u'5', u'1', u'c', u'6', u'b', u'9', u'f', u'5', u'0', u'4', u'9', u'.', u't', u'a', u'r')                                                                  Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/cmd/test.py", line 101, in run_analysis
    client.run_analysis_and_poll(analysis_settings, input_location, output_directory)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/utils/log.py", line 82, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/api_client/client.py", line 215, in run_analysis_and_poll
    status, outputs_location = self.get_analysis_status(analysis_status_location)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/api_client/client.py", line 184, in get_analysis_status
    raise OasisException(error_message)
OasisException: Analysis failed: MissingInputsException(u'I', u'n', u'p', u'u', u't', u's', u' ', u'l', u'o', u'c', u'a', u't', u'i', u'o', u'n', u' ', u'n', u'o', u't', u' ', u'f', u'o', u'u', u'n', u'd', u':', u' ', u'/', u'v', u'a', u'r', u'/', u'w', u'w', u'w', u'/', u'o', u'a', u's', u'i', u's', u'/', u'u', u'p',
 u'l', u'o', u'a', u'd', u'/', u'2', u'7', u'6', u'd', u'1', u'3', u'8', u'7', u'8', u'6', u'7', u'1', u'4', u'e', u'4', u'd', u'9', u'7', u'a', u'a', u'4', u'5', u'1', u'c', u'6', u'b', u'9', u'f', u'5', u'0', u'4', u'9', u'.', u't', u'a', u'r')                                                                         
Finished: 0 completed, 1 failed
In [1]: ''.join([u'I', u'n', u'p', u'u', u't', u's', u' ', u'l', u'o', u'c', u'a
   ...: ', u't', u'i', u'o', u'n', u' ', u'n', u'o', u't', u' ', u'f', u'o', u'u
   ...: ', u'n', u'd', u':', u' ', u'/', u'v', u'a', u'r', u'/', u'w', u'w', u'w
   ...: ', u'/', u'o', u'a', u's', u'i', u's', u'/', u'u', u'p',
   ...:  u'l', u'o', u'a', u'd', u'/', u'2', u'7', u'6', u'd', u'1', u'3', u'8',
   ...:  u'7', u'8', u'6', u'7', u'1', u'4', u'e', u'4', u'd', u'9', u'7', u'a',
   ...:  u'a', u'4', u'5', u'1', u'c', u'6', u'b', u'9', u'f', u'5', u'0', u'4',
   ...:  u'9', u'.', u't', u'a', u'r'])
   ...:  
Out[1]: 'Inputs location not found: /var/www/oasis/upload/276d138786714e4d97aa451c6b9f5049.tar'

Add cookiecutter template

Add the cookiecutter template to the package and add commands to the cli to generate a new model.

We should also simplify the template so that the cli can take advantage of default paths.

PiWind example fails on Python 3.6 because of dictionary comparisons

When testing the PiWind example I ran into problems when a list of dictionaries gets sorted in oasislmf/exposures/manager.py, line 464. Python 2 is happy to sort such a list, but Python 3 is not.

I don't have a good intuition for why this sort is being performed but I'm guessing the goal is to impose an order on the output DataFrame?

The error I see currently is:

...
Generating Oasis files for model
Traceback (most recent call last):
*snip*
  File "/Users/user/oasis-lmf/OasisLMF/oasislmf/exposures/manager.py", line 593, in generate_oasis_files
    data_frame = self.load_master_data_frame(**kwargs)
  File "/Users/user/oasis-lmf/OasisLMF/oasislmf/exposures/manager.py", line 464, in load_master_data_frame
    filter(lambda v: v.get('FieldName') == 'TIV', six.itervalues(canonical_exposures_profile))
TypeError: '<' not supported between instances of 'dict' and 'dict'

I tried a fix in master...ElliotJH:python36-dict-comparison but I'm not clear why the sort is being performed so that key may be no good.

Generic model tests cases

Add generic unit test suite for the model steps that can be included in cookie cutter and hooked in with test data as it is generated, to provide model developers with a pre-built test suite. This could/should tie in with the test data used for integration tests.

Install problems using pip editable mode

pip install --editable git+{https,ssh}://[email protected]/OasisLMF/OasisLMF.git@<branch name>#egg=oasislmf

  • Ktools binaries missing
  • pip state for oasislmf package is broken
pip freeze | grep oasis
...
numpy==1.15.1
-e git+ssh://[email protected]/OasisLMF/OasisLMF.git@592c07a59cbf05d25a2e59de190233539c2c1504#egg=oasislmf
pandas==0.22.0
...
$ pip uninstall oasislmf
Can't uninstall 'oasislmf'. No files were found to uninstall.

PiWind model run issues

@OmegaDroid There's a new issue with the model run command for PiWind. I'm currently working on a branch tweaks which I created just now from the master after the merge of the fixes branch yesterday. The tests are all passing but running the PiWind model causes this error

(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model run -C ../OasisPiWind/mdk-oasislmf-piwind.json 
No handlers could be found for logger "root"
ERROR:root:Could not find periods data file: /Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/runs/ProgOasis-20180223140338/static/periods.bin

In preparing the model run inputs (oasislmf.model_execution.bin.prepare_model_run_inputs) I see that you're trying to copy a file called periods.bin into the input subfolder of the model run directory. No such file is assumed or exists for the PiWind model when I run it via the MDK script.

@benhayes21 Can you please verify that there is no such static file periods.bin for PiWind?

Unstable test - Py2

tests/model_execution/test_bin.py

    def test_check_gul_and_il_and_single_ri_directory_structure_missing_file_fail(self):
        with TemporaryDirectory() as d:
            for p in six.itervalues(INPUT_FILES):
                Path(os.path.join(d, p['name'] + '.csv')).touch()
            os.mkdir(os.path.join(d, "RI_1"))
            # Skip the first files
            first = True
            for p in six.itervalues(INPUT_FILES):
                if not first:
                    Path(os.path.join(d, "RI_1", p['name'] + '.csv')).touch()
                first = False

            with self.assertRaises(OasisException):
                check_inputs_directory(d, do_il=True, do_ri=True, check_binaries=True)

Fails intermittently in Python 2.7.x with:

=================================== FAILURES ===================================
 CheckInputDirectory.test_check_gul_and_il_and_single_ri_directory_structure_missing_file_fail 

self = <tests.model_execution.test_bin.CheckInputDirectory testMethod=test_check_gul_and_il_and_single_ri_directory_structure_missing_file_fail>

    def test_check_gul_and_il_and_single_ri_directory_structure_missing_file_fail(self):
        with TemporaryDirectory() as d:
            for p in six.itervalues(INPUT_FILES):
                Path(os.path.join(d, p['name'] + '.csv')).touch()
            os.mkdir(os.path.join(d, "RI_1"))
            # Skip the first files
            first = True
            for p in six.itervalues(INPUT_FILES):
                if not first:
                    Path(os.path.join(d, "RI_1", p['name'] + '.csv')).touch()
                first = False
    
            with self.assertRaises(OasisException):
>               check_inputs_directory(d, do_il=True, do_ri=True, check_binaries=True)
E               AssertionError: OasisException not raised

tests/model_execution/test_bin.py:436: AssertionError

works fine in Python 3.x

Package installation and MDK run issues

@OmegaDroid I built a fresh egg from the latest source and installed it. I got the following error when trying to run the MDK generate-oasis-files command against the PiWind config JSON

(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-oasis-files -C ../OasisPiWind/mdk-oasislmf-piwind.json
Traceback (most recent call last):
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/bin/oasislmf", line 4, in <module>
    __import__('pkg_resources').run_script('oasislmf==0.0.0', 'oasislmf')
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/EGG-INFO/scripts/oasislmf", line 7, in <module>
    sys.exit(RootCmd().run())
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/root.py", line 36, in run
    return super(OasisBaseCommand, self).run(args=args)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 151, in run
    return cmd_cls(sub_command_name).run(args)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 151, in run
    return cmd_cls(sub_command_name).run(args)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/argparsetree/cmd.py", line 153, in run
    return self.action(args) or 0
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/cmd/model.py", line 332, in action
    lookup_package_path=lookup_package_file_path,
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/keys/lookup.py", line 237, in create
    lookup_package = cls.get_lookup_package(lookup_package_path)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/oasislmf-venv/lib/python2.7/site-packages/oasislmf-0.0.0-py2.7.egg/oasislmf/keys/lookup.py", line 138, in get_lookup_package
    lookup_package = importlib.import_module(package_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/__init__.py", line 8, in <module>
    from .PiWind import *
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/__init__.py", line 3, in <module>
    from utils import *
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/utils/__init__.py", line 1, in <module>
    from .AreaPerilLookup import *
  File "/Users/srm/Documents/sandeep/cst/dev/oasis/OasisPiWind/src/keys_server/PiWind/utils/AreaPerilLookup.py", line 12, in <module>
    from shapely.geometry import (
ImportError: No module named shapely.geometry

So shapely is a requirement of the PiWind lookup service (OasisPiWind/src/keys_server/PiWind) and its requirements are contained in its own requirements file OasisPiWind/src/keys_server/PiWind/requirements.txt). Would it be possible for you to maybe add a command to oasislmf so that you can pre-install the custom requirements of any lookup service by specifying a path to the requirements file, or for this to be done automatically in some way.

The config JSON files for a given supplier's models live in the supplier repository, and you can assume that the requirements file for a specific model will exist as <supplier repo name>/src/keys_server/<model ID>/requirements.txt.

Command auto-completion feature

@OmegaDroid How difficult would it be add autocompletion to the CLI? Obviously this is not essential but working with CLIs we know it's nice to have, e.g. Git. Would this be difficult to do?

Performance tools

It would be very useful for model developers to have some performance tools for analyzing their model. The base line functionality would be a breakdown of run time by keys lookup and each of the run time steps. More advanced features could include:

  • Other instrumentation e.g. memory pressure
  • Parameter sweep of num simulations, locations
  • Automated exposure generation for various sized portfolios, based on area perils and vulnerabilities.

API-Worker stuck on error from CSV transforms

If the input generation fails at the transform function (Multithread call), then the CLI call will hang,
causing the worker to block on subsequent file generation requests.

Writing canonical exposures file /home/sam/repos/models/piwind/runs/OasisFiles-20181123135201/canexp-20181123135201.csv
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/utils/concurrency.py", line 128, in run
    task.result = task.func(*task.args) if task.args else task.func()
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/exposures/csv_trans.py", line 89, in process_chunk
    data
  File "/home/sam/repos/core/env/py3/lib/python3.6/site-packages/oasislmf/exposures/csv_trans.py", line 114, in csv_to_xml
    rec.set(csv_header[i], row[i])
  File "src/lxml/etree.pyx", line 826, in lxml.etree._Element.set (src/lxml/etree.c:52998)
  File "src/lxml/apihelpers.pxi", line 567, in lxml.etree._setAttributeValue (src/lxml/etree.c:24235)
  File "src/lxml/apihelpers.pxi", line 1643, in lxml.etree._attributeValidOrRaise (src/lxml/etree.c:35582)
ValueError: Invalid attribute name 'Unnamed: 0'

Visibility and Sphinx docs

Shouldn't the repository be public? It is currently private, and the license is public.

Also, the Sphinx docs can be built but aren't being published anywhere, and GitHub pages is disabled.

Broken test - Cannot assess lambda object

This fix:
cb713fd

Broke its corresponding test case:
https://github.com/OasisLMF/OasisLMF/blob/master/tests/cmd/test_test_testmoddelapi.py#L291-L311

 TestModelApiCmdRun.test_validation_is_successful___threadpool_is_started_for_each_analysis 

self = <tests.cmd.test_test_testmoddelapi.TestModelApiCmdRun testMethod=test_validation_is_successful___threadpool_is_started_for_each_analysis>

    @given(integers(min_value=1, max_value=5), booleans(), dictionaries(text(), text()))
>   def test_validation_is_successful___threadpool_is_started_for_each_analysis(self, num_analyses, do_il, settings):

tests/cmd/test_test_testmoddelapi.py:292: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py36/lib/python3.6/site-packages/hypothesis/core.py:578: in execute
    result = self.test_runner(data, run)
.tox/py36/lib/python3.6/site-packages/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
.tox/py36/lib/python3.6/site-packages/hypothesis/core.py:570: in run
    return test(*args, **kwargs)
tests/cmd/test_test_testmoddelapi.py:292: in test_validation_is_successful___threadpool_is_started_for_each_analysis
    def test_validation_is_successful___threadpool_is_started_for_each_analysis(self, num_analyses, do_il, settings):
.tox/py36/lib/python3.6/site-packages/hypothesis/core.py:517: in test
    result = self.test(*args, **kwargs)
tests/cmd/test_test_testmoddelapi.py:309: in test_validation_is_successful___threadpool_is_started_for_each_analysis
    self.assertEqual(fn.__name__, 'run_analysis')
E   AssertionError: '<lambda>' != 'run_analysis'
E   - <lambda>
E   + run_analysis

API version of the toolkit

At the moment, the toolkit is designed as a CLI that operates locally installed models. It would be useful to expand this to allow the CLI to also operate against deployed models via the Oasis API. This would allow users to test deployed models, or to script more complex work flows.

Package publishing issues on Ubuntu 16.04

Running the following on Ubuntu 16.04:
python setup.py publish

Hit the following snags:

  • Wheel package was not detected, even though it is installed. Doesn't seem to be listed by pip freeze.
  • For cleaning: oasis_lmf.egg-info should be oasislmf.egg-info

GUL items possibly generated for locations with failing or non-matching keys lookups

At present GUL may be generated for some locations, in a portfolio, even if those locations have failing or non-matching keys lookups - the only condition applied at present to generate a GUL item from a location and a keys item (which is basically a loc. ID, peril ID, coverage type ID) combination, is that the TIV for the coverage type must be positive in the can. loc. data file. This would produce a GUL item for such a location even if the keys lookup failed or did not match.

Bug (generate losses)

Bug in model generate-losses subcommand

(oasislmf-venv) [srm@~/Documents/sandeep/cst/dev/oasis/oasislmf]$ oasislmf model generate-losses -j ../OasisPiWind/analysis_settings.json -m ../OasisPiWind/model_data/ -x ./runs/OasisFiles-20180306122215/
 Could not find events data file: /Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/runs/ProgOasis-20180306122926/static/events_p.bin

Output summaries in MDK

Currently the MDK only outputs at portfolio level i.e. all losses grouped up. This need to replicate, at the minimum, the Flamingo functionality where a summary level can be picked from a predefined set for GUL, IL and RI losses. The current set supported in Flamingo is:

Prog, Policy, State, County, Location, LOB

We should also conider making this more generic i.e. any combination of OED exposure fields can define a summary set.

Install error 1.1.24

Installing collected packages: oasislmf
  Compiling /tmp/pip-install-MCQFe6/oasislmf/oasislmf-1.1.24.data/purelib/oasislmf/utils/keys_data.py ...
    File "/tmp/pip-install-MCQFe6/oasislmf/oasislmf-1.1.24.data/purelib/oasislmf/utils/keys_data.py", line 99
      return keys_records(from_peril_ids=just(peril_id), from_coverage_type_ids=just(coverage_type_id) min_size=min_size, max_size=max_size)
                                                                                                              ^
  SyntaxError: invalid syntax
  

Successfully installed oasislmf-1.1.24
Cleaning up...

Python API

The current version of the toolkit is designed as a command line interface. It would be useful to have a cleaner python interface exposing the same functionality. The following snippet illustrates how one of the subcommands can be invoked at present.

from argparse import Namespace
from oasislmf.cmd.model import GenerateLossesCmd
args = Namespace(
oasis_files_path='/path/to/Oasis/files',
analysis_settings_json_file_path='analysis/settings/json/file/path',
model_data_path='/model/data/path',
model_run_dir_path='/model/run/dir/path'
)
gen_losses = GenerateLossesCmd()
gen_losses.action(args)

This would be cleaner as:
import oasislmf
oasislmf.model.generate_losses(
oasis_files_path='/path/to/Oasis/files',
analysis_settings_json_file_path='analysis/settings/json/file/path',
model_data_path='/model/data/path',
model_run_dir_path='/model/run/dir/path')

On Install: Print warning or fail if ktool binaries are not built

This is caused by missing local package dependencies needed for a ktools build.

Example:

  1. Remove autoconf: sudo yum remove autoconf.noarch
  2. install oasislmf:
[ec2-user@ip-10-10-0-212 impactforecastingefs]$ sudo pip install oasislmf
Collecting oasislmf
  Using cached https://files.pythonhosted.org/packages/61/70/4146aa9efef9ba7af82f71e9a704859984962be1c6e478ab98bff25750cb/oasislmf-1.1.10.tar.gz
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): pyodbc in /usr/lib64/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): requests-toolbelt in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): shutilwhich in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): jsonpickle in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): pytz in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): pathlib2 in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): backports.tempfile in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): chainmap in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): argparsetree in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): python-interface in /usr/lib/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): pandas in /usr/lib64/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib64/python2.7/site-packages (from oasislmf)
Requirement already satisfied (use --upgrade to upgrade): scandir; python_version < "3.5" in /usr/lib64/python2.7/site-packages (from pathlib2->oasislmf)
Requirement already satisfied (use --upgrade to upgrade): backports.weakref in /usr/lib/python2.7/site-packages (from backports.tempfile->oasislmf)
Requirement already satisfied (use --upgrade to upgrade): typing>=3.5.2 in /usr/lib/python2.7/site-packages (from python-interface->oasislmf)
Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1.0.2 in /usr/lib/python2.7/site-packages (from python-interface->oasislmf)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in /usr/lib64/python2.7/site-packages (from pandas->oasislmf)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/lib/python2.7/site-packages (from pandas->oasislmf)
Installing collected packages: oasislmf
  Running setup.py install for oasislmf ... done
Successfully installed oasislmf-1.1.10

No ktools

Peril area lookup error

Dear all,
I tried executing the PiWind project by expanding the domain, which covers most portion of the UK and provided SourceLocPiWind10.csv with locations within the new domain.
After running the simulation, I got an error and found the following message to each location in the log file.
Please respond to resolve the issue.
Thank you..

Peril area lookup: location is 5.01184437588 units from the peril areas global boundary - the required minimum distance is 0 units

--

[test model-api] Download outputs url

Test is issuing call to http://oasis_api_server/outputs/ when it seems like it should be
http://oasis_api_server/outputs/{location}
Where the location is the outputs resource to download.

wget http://oasis_api_server/outputs/57bb6d2d863b44f3876b42e9f42c65d9

--2018-03-23 16:56:48--  http://oasis_api_server/outputs/57bb6d2d863b44f3876b42e9f42c65d9
Resolving oasis_api_server (oasis_api_server)... 192.168.16.5
Connecting to oasis_api_server (oasis_api_server)|192.168.16.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48296 (47K) [application/x-tar]
Saving to: '57bb6d2d863b44f3876b42e9f42c65d9'

57bb6d2d863b44f3876b42e9f42c65d9       100%[=========================================================================>]  47.16K  --.-KB/s    in 0s      

2018-03-23 16:56:48 (668 MB/s) - '57bb6d2d863b44f3876b42e9f42c65d9' saved [48296/48296]

root@db55bf67aa25:/var/www/oasis# file 57bb6d2d863b44f3876b42e9f42c65d9
57bb6d2d863b44f3876b42e9f42c65d9: gzip compressed data, was "57bb6d2d863b44f3876b42e9f42c65d9.tar", last modified: Fri Mar 23 16:29:34 2018, max compression
root@db55bf67aa25:/var/www/oasis# ./run_api_test_analysis.sh 
STARTED: oasislmf.api_client.client.health_check
COMPLETED: oasislmf.api_client.client.health_check in 0.01s
Loading analysis settings JSON file:
  OK: analysis_settings={u'analysis_settings': {u'source_tag': u'test_source', u'gul_summaries': [{u'aalcalc': False, u'leccalc': {u'return_period_file': False, u'outputs': {u'sample_mean_oep': False, u'sample_mean_aep': False, u'full_uncertainty_oep': False, u'wheatsheaf_aep': False, u'wheatsheaf_mean_aep': False, u'full_uncertainty_aep': False, u'wheatsheaf_mean_oep': False, u'wheatsheaf_oep': False}}, u'eltcalc': True, u'pltcalc': False, u'summarycalc': False, u'id': 1, u'lec_output': False}], u'il_summaries': [{u'aalcalc': False, u'leccalc': {u'return_period_file': False, u'outputs': {u'sample_mean_oep': False, u'sample_mean_aep': False, u'full_uncertainty_oep': False, u'wheatsheaf_aep': False, u'wheatsheaf_mean_aep': False, u'full_uncertainty_aep': False, u'wheatsheaf_mean_oep': False, u'wheatsheaf_oep': False}}, u'eltcalc': True, u'pltcalc': False, u'summarycalc': False, u'id': 1, u'lec_output': False}], u'analysis_tag': u'test_analysis', u'number_of_samples': 100, u'model_version_id': u'1', u'module_supplier_id': u'OasisIM', u'gul_threshold': 100, u'gul_output': True, u'il_output': False, u'prog_id': 1, u'model_settings': {u'event_occurrence_file_id': 1, u'use_random_number_file': True}}}, do_il=False
Running 1 analyses
STARTED: oasislmf.api_client.client.upload_inputs_from_directory
COMPLETED: oasislmf.api_client.client.upload_inputs_from_directory in 0.03s
STARTED: oasislmf.api_client.client.run_analysis_and_poll
STARTED: oasislmf.api_client.client.run_analysis
Analysis started
COMPLETED: oasislmf.api_client.client.run_analysis in 0.0s
Analysis started
STARTED: oasislmf.api_client.client.download_outputs
GET http://oasis_api_server/outputs/ failed: 404
Model API test failed: GET http://oasis_api_server/outputs/ failed: 404
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/cmd/test.py", line 101, in run_analysis
    client.run_analysis_and_poll(analysis_settings, input_location, output_directory)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/utils/log.py", line 82, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/api_client/client.py", line 224, in run_analysis_and_poll
    self.download_outputs(outputs_location, outputs_file)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/utils/log.py", line 82, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/api_client/client.py", line 318, in download_outputs
    self.download_resource('/outputs/' + outputs_location, localfile)
  File "/usr/local/lib/python2.7/dist-packages/oasislmf/api_client/client.py", line 286, in download_resource
    raise OasisException(exception_message)
OasisException: GET http://oasis_api_server/outputs/ failed: 404
Finished: 0 completed, 1 failed

Oasis model

@OmegaDroid I had some points about your refactoring of the Oasis model class

https://github.com/OasisLMF/OasisLMF/blob/master/oasislmf/models/model.py#L22

I note that you've added a wrapper method load_canonical_profile to the class definition

https://github.com/OasisLMF/OasisLMF/blob/master/oasislmf/models/model.py#L73

as well as a property for the oasis_files_pipeline object in the model resources dictionary.

https://github.com/OasisLMF/OasisLMF/blob/master/oasislmf/models/model.py#L155

From my personal viewpoint of design I think it's better than a model object only knows about supplier ID, model ID, model version and that it has a resources dictionary - when I wrote the framework I had the view that model objects should be agnostic about what resources are attached, and it is external entities like the exposure manager (which I based on Flamingo) or other clients that will attach whatever resources they need to model objects. So for example, a model object shouldn't know anything about the files pipeline object, as this is something that the exposures manager uses to generate the intermediate files and then Oasis files from these - the concept of a files pipeline is something that a model should be unaware of. Same for the canonical profile - this is again specific to the manager class.

I would say that, from this point of view, even the keys methods I added shouldn't be part of the model class definition, as they are just wrappers for methods in the lookup factory class.

PyTrans conversion errors/warnings

@sambles It would be good to supress the errors/warnings when PyTrans does the conversions of the source files, e.g.

True
True
End of input file

...

Input failed to Validate
<string>:15:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element 'rec', attribute 'CLASS_2': The attribute 'CLASS_2' is not allowed.
False
End of input file

...

Input failed to Validate
<string>:15:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element 'rec', attribute 'CLASS_2': The attribute 'CLASS_2' is not allowed.
False
Input failed to Validate
<string>:15:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element 'rec', attribute 'CLASS_2': The attribute 'CLASS_2' is not allowed.
False
End of input file

These messages might confuse users, and shouldn't be present during the logging that appears when running the MDK commands.

Overly generic exception message

The following exception is thrown in many circumstances when running "model run" or "model generate_oasis_files", for example when the specified config file does not exist. It would be useful to have more specific error messages:

Either the lookup config JSON file path or the keys data path + model version file path + lookup package path must be provided

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.