This repo includes code referenced in the paper A Rigorous Risk-aware Linear Approach to Extended Markov Ratio Decision Processes with Embedded Learning by Alexander Zadorojniy, Takayuki Osogami, and Orit Davidovich to appear in IJCAI 2023.

The code implements the paper's Grid World experiments, including an algorithm to find the theoretically optimal policy for the Grid World environment and Algorithm 3 from the paper.

We tested our code on a Python 3.8.0 environment.

Our scripts interact with storage, either local or S3.

Storage specs are provided in a configs.yaml. It includes the list of source and target storage folder or bucket names. If necessary, S3 credentials are added under their designated fields.

You'll find acceptable config formats under ./configs/. Here is the configs format for local storage

local:
    buckets:
        inputs: '<inputs-folder>'
        inputs_dest: '<inputs-dest-folder>'
        data: '<data-folder>'
        data_dest: '<data-dest-folder>'
        results: '<solutions-folder>'
        results_dest: '<solutions-dest-folder>'

and here it is for S3

s3:
    buckets:
        inputs: '<inputs-bucket>'
        inputs_dest: '<inputs-dest-bucket>'
        data: '<data-bucket>'
        data_dest: '<data-dest-bucket>'
        results: '<results-bucket>'
        results_dest: '<results-dest-bucket>'
    aws_secret_access_key: 'xxxx'
    aws_access_key_id: 'xxxx'
    endpoint_url: 'https://xxx.xxx.xxx'
    region: 'xx-xxxx'
    cloud_service_provider: 'aws'

Make a copy of one of the config example yamls under ./configs/ and store it under the root ./.

cd IBM-Extended-Markov-Ratio-Decision-Process
cp configs/ibm_configs_example.yaml configs.yaml

Then edit all field values. Note that bucket / folder names must be distinct.

Currently, two S3 providers are available under s3:cloud_service_provider: either aws or ibm. The endpoint_url is optional for AWS.

To run our scripts and notebooks, first clone our repo

git clone https://github.com/IBM/IBM-Extended-Markov-Ratio-Decision-Process.git

and then

cd IBM-Extended-Markov-Ratio-Decision-Process

Before you can run our scripts, you must first define your Grid World environment hyper-parameters. These hyper-parameters are defined in ./experiment.json.

Here are the hyper-parameters we used in the experiments reported in our IJCAI paper

{
	"algorithm": {
		"is_theory": [false],
		"is_max": [false]
	},
	"data": {
		"grid": [
			[5, 5]
		],
		"obstacles": [3],
		"noise": [0.1,0.3,1.0],
        "delta": [0.05],
        "epsilon": [0.01],
        "tolerance": [1e-8],
        "sensitivity": [1e-5],
		"discount": [0.95],
        "cost": [5.0],
		"init_state": [
			[0,0]
		]
	},
	"experiment": {
		"N": 50,
		"is_same": false,
		"experiment_id": "experiment_ijcai"
	}
}

Below you'll find the commands to run the scripts that

1. generate data from the Grid World environment,
2. apply the EMRDP algorithm with embedded learning (Algorithm 3 in the paper).

The scripts generate logs when a logs dir is provided under --path_log.

Here is how you run the script to generate data from the Grid World environment and store it locally

python scripts/grid_data.py --path_in ~/path/to/inputs --path_out ~/path/to/data --path_archive ~/path/to/inputs_dest/ --path_log ~/path/to/logs/dir

And here is how you run the script to generate data and upload it to S3

python scripts/grid_data.py --configs ~/path/to/configs.yaml --path_log ~/path/to/logs/dir

Here is how you run the script that applies the EMRDP algorithm against the Grid World datasets that were stored locally

python grid_emrdp.py --path_in ~/path/to/data --path_out ~/path/to/results --path_log ~/path/to/logs/dir

We implemented the cloud distribution of the EMRDP algorithm using Ray. You can run it, assuming you have a K8S cluster already set up. If not, follow the instructions for setting up an IBM OpenShift cluster here or an AWS K8S cluster here.

Before you run our scripts, make sure you're in the right K8S context. If you have the kubectl CLI installed, you can run

kubectl config current-context

Your context is also available to view on Docker Desktop -> kubernetes -> contexts.

The Ray cluster requires its own cluster yaml. We used the cluster yaml emrdp.yaml copied from another project. Note that emrdp.yaml assumes your configs.yaml is located under the dir root ./.

See here for instructions on how to define the Ray cluster yaml from scratch.

Start the Ray cluster with

ray up emrdp.yaml --no-config-cache --yes

Run the data generation script [with a log file]

ray submit emrdp.yaml scripts/grid_data.py -c configs.yaml --path_log . --no-config-cache

To check the log file, first log into the cluster head node

ray attach ~/Workspace/emrdp/emrdp.yaml

You can ls -lh to find the time-stamped name of the log file. For example, read the file with

(base) ray@emrdp-cluster-head-jvmbl:~$ cat grid_emrdp_data_2022-12-23_0347.log

Exit the cluster head node with

(base) ray@emrdp-cluster-head-jvmbl:~$ exit

Run the EMRDP algorithm script [without a log file]

ray submit emrdp.yaml scripts/grid_emrdp.py -c configs.yaml --no-config-cache

To update the Ray cluster yaml after a emrdp.yaml change

ray up emrdp.yaml --no-config-cache --yes

To shutdown the Ray cluster

ray down -y emrdp.yaml

To monitor Ray cluster autoscaling

ray exec emrdp.yaml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*'

To connect to a terminal on the Ray cluster head node

ray attach emrdp.yaml

Then exit the head node with

exit

See here for OpenShift-specific instructions on how to view the Ray dashboard.

--> On ray up you get ModuleNotFoundError related to ray.

pip install -U "ray[default,serve,k8s]"==1.13.0

--> When running grid_emrdp.py, you get alerts: Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings".

Follow this Ray core issue.

--> SSH still not available

Too many CPU / memory resources were requested as max values.