substra / substra Goto Github PK

View Code? Open in Web Editor NEW

266.0 9.0 34.0 5.5 MB

Low-level Python library used to interact with a Substra network

Home Page: https://docs.substra.org

License: Apache License 2.0

Python 99.88% Makefile 0.12%

federated-learning federated-learning-framework

substra's Introduction

Substra is an open source federated learning (FL) software. It enables the training and validation of machine learning models on distributed datasets. It provides a flexible Python interface and a web application to run federated learning training at scale. This specific repository is the low-level Python library used to interact with a Substra network.

Substra's main usage is in production environments. It has already been deployed and used by hospitals and biotech companies (see the MELLODDY project for instance). Substra can also be used on a single machine to perform FL simulations and debug code.

Substra was originally developed by Owkin and is now hosted by the Linux Foundation for AI and Data. Today Owkin is the main contributor to Substra.

Join the discussion on Slack and subscribe here to our newsletter.

To start using Substra

Have a look at our documentation.

Try out our MNIST example.

Support

If you need support, please either raise an issue on Github or ask on Slack.

Contributing

Substra warmly welcomes any contribution. Feel free to fork the repo and create a pull request.

Setup

To setup the project in development mode, run:

pip install -e ".[dev]"

To run all tests, use the following command:

make test

Some of the tests require Docker running on your machine before running them.

Code formatting

You can opt into auto-formatting of code on pre-commit using Black.

This relies on hooks managed by pre-commit, which you can set up as follows.

Install pre-commit, then run:

pre-commit install

Documentation generation

To generate the command line interface documentation, sdk and schemas documentation, the python version must be 3.8. Run the following command:

make doc

Documentation will be available in the references/ directory.

Changelog generation

The changelog is managed with towncrier. To add a new entry in the changelog, add a file in the changes folder. The file name should have the following structure: <unique_id>.<change_type>. The unique_id is a unique identifier, we currently use the PR number. The change_type can be of the following types: added, changed, removed, fixed.

To generate the changelog (for example during a release), use the following command (you must have the dev dependencies installed):

towncrier build --version=<x.y.z>

You can use the --draft option to see what would be generated without actually writing to the changelog (and without removing the fragments).

substra's People

Contributors

Stargazers

Watchers

substra's Issues

Local install using skaffold, on Linux

Following this guide, I tried to install Substra locally on a Linux machine (Ubuntu LTS 18.04)

The issues I encounted were:

Docker desktop is not available on Linux (only Mac and Windows). I did install docker engine but I guess I should then install kubernetes which was not part of the docker engine install ?
In part 3, to get the host IP adress from the container, the following command docker run -it --rm busybox ping host.docker.internal does not work on linux, cf. this thread.

SDK: a dataset can be linked to a single objective only

Currently the doc when creating a dataset is saying the opposite.
Backend must be updated too.

Local install using skaffold, on Windows

Following this guide, I tried to install Substra locally on a Windows machine.

The issues I encounted were:

helm must be installed before running the command helm init (I thought it might have been installed when installing Docker desktop)
helm init has been removed from helm 3.0. Should I use helm 2.* or just don't run helm init ?
when defining alias: sudo ifconfig lo0 alias 192.168.65.2. I am not sure how to translate this command on Windows. Following this link, I am not sure which network name to use among the eight that I have.

When SDK raises an InvalidRequest exception, the reason is not always displayed

At least when the request is failing due to a validation error from the backend, the reason is not displayed:

    raise exceptions.InvalidRequest.from_request_exception(e)
substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/

The reason causing the failure was in the above example was:

{"compute_plan_id":["Ensure this field has at least 64 characters."]}

Generated docs: describe outputs returned by SDK methods

We could use structure definition (dataclass, pydantic, marshmallow...) to parse the responses.
It would allow to have dynamic documentation.
~~It would allow also to return python object and to convert fields from camelCase to snake case.~~

EDIT:
As pointed out by @jmorel, the SDK should not modify the field names returned by the backend. It will have to be done by the backend.

Create a common option for the output_format

It could be good to replace the output_format options (--pretty, --json, --yaml) by a common one (for example --output / -o) with the format as an argument.
The --help option could be easier to read like this 🙂

Example:
For example, typing substra list dataset --output pretty instead of substra list dataset --pretty.

pip install substra fails

When running pip install substra, i have the following error:

Collecting cffi!=1.11.3,>=1.8 (from cryptography->SecretStorage>=3; sys_platform == "linux"->keyring->substra)
  Using cached https://files.pythonhosted.org/packages/f1/c7/72abda280893609e1ddfff90f8064568bd8bcb2c1770a9d5bb5edb2d1fea/cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->SecretStorage>=3; sys_platform == "linux"->keyring->substra)
Building wheels for collected packages: setproctitle
  Running setup.py bdist_wheel for setproctitle ... error
  Complete output from command /home/romain/substra_test/substra_venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-06hmudfk/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp9_mj366tpip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  building 'setproctitle' extension
  creating build
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DSPT_VERSION=1.1.10 -DHAVE_SYS_PRCTL_H=1 -I/home/romain/substra_test/substra_venv/include -I/usr/include/python3.6m -c src/setproctitle.c -o build/temp.linux-x86_64-3.6/src/setproctitle.o
  In file included from src/spt.h:15:0,
                   from src/setproctitle.c:14:
  src/spt_python.h:14:10: fatal error: Python.h: No such file or directory
   #include <Python.h>
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for setproctitle
  Running setup.py clean for setproctitle
Failed to build setproctitle
Installing collected packages: jeepney, zipp, importlib-metadata, six, pycparser, cffi, cryptography, SecretStorage, keyring, urllib3, idna, chardet, certifi, requests, setproctitle, consolemd, websocket-client, docker, substra
  Running setup.py install for setproctitle ... error
    Complete output from command /home/romain/substra_test/substra_venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-06hmudfk/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gpuzrirr-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/romain/substra_test/substra_venv/include/site/python3.6/setproctitle:
    running install
    running build
    running build_ext
    building 'setproctitle' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DSPT_VERSION=1.1.10 -DHAVE_SYS_PRCTL_H=1 -I/home/romain/substra_test/substra_venv/include -I/usr/include/python3.6m -c src/setproctitle.c -o build/temp.linux-x86_64-3.6/src/setproctitle.o
    In file included from src/spt.h:15:0,
                     from src/setproctitle.c:14:
    src/spt_python.h:14:10: fatal error: Python.h: No such file or directory
     #include <Python.h>
              ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/romain/substra_test/substra_venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-06hmudfk/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gpuzrirr-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/romain/substra_test/substra_venv/include/site/python3.6/setproctitle" failed with error code 1 in /tmp/pip-build-06hmudfk/setproctitle/

I am on Ubuntu 18.04 (inside a VM).

Issue with Get 130 - ibffi-dev amd64

Mac version : 10.15.3
helm : 2.16
tiller : 2.16

Hi guys,

I have an issue with a GET since one week:

Get:130 http://archive.ubuntu.com/ubuntu bionic/main amd64 libffi-dev amd64 3.2.1-8 [156 kB]
Fetched 113 MB in 53s (2140 kB/s)
Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libx/libxml2/libxml2_2.9.4+dfsg1-6.1ubuntu1.2_amd64.deb 404 Not Found [IP: 91.189.88.162 80]
Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_4.15.0-74.84_amd64.deb 404 Not Found [IP: 91.189.88.162 80]
Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3-stdlib-extensions/python3-distutils_3.6.9-1~18.04_all.deb Undetermined Error [IP: 91.189.88.162 80]
Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib1g-dev_1.2.11.dfsg-0ubuntu2_amd64.deb Undetermined Error [IP: 91.189.88.162 80]
Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libx/libxml2/libxml2-dev_2.9.4+dfsg1-6.1ubuntu1.2_amd64.deb 404 Not Found [IP: 91.189.88.162 80]
Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

FATA[0141] exiting dev mode because first build failed: build failed: build failed: building [substrafoundation/celeryworker]: build artifact: unable to stream build output: The command '/bin/sh -c apt-get install -y python3.6 python3-pip python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev' returned a non-zero code: 100

Any idea on how I can do?
I'm locked.

Drop support for python < 3

Python 2.6 and 2.7 are currently not tested and I think we should drop its support.

To do so, we should update the documentation and the setup.py.

CLI: documentation for field permission is outdated

It is now an object and not a string.

Docs: Missing a field in client doc

I think the field rank here is missing in fuction add_traintuple

link

Install - Download the code - Broken links

On:
https://github.com/SubstraFoundation/substra/blob/master/docs/local_install_skaffold.md
Section 1. Get the source code

The links to download the code are broken (404 - not found):

hlf-k8s
substra-frontend

Backend link works.

No mention of the need to specify public permissions

https://github.com/SubstraFoundation/substra/blob/c6bae9cb15388e186341dbb9f08be25e8ab762c1/substra/sdk/client.py#L726-L728

You could replace it by this code for better user information as the error thrown if you forget the public permissions is not very informative (400 Bad request).

"out_trunk_model_permissions": {
                    "public": bool,
                    "authorized_ids": list[str],
                },

Would be nice to have an example that use data submitted to two nodes

In the titanic example, data are submitted to one node only. It would be nice to have two scripts mimicking two actors that would add data in two nodes (two since in our local setup we use two nodes by default). This could be done in addition to the current example.
We could then in the compute plan example have an model that is created from data from two nodes.

CLI: add permissions option

When adding an asset through the CLI, there is currently no way to set permissions, they are always set to the default "public" value.

We need to design a way to add permissions through options and implement it.

Add anaconda package

It would be awesome to have the substra python package also available on anaconda & ready to be installed with conda install substra.
For now, it is returning a sad PackagesNotFoundError.
What do you think about this?

Error running hlf-k8s with kubernetes and skaffold on ubuntu server

I am currently trying to run the substra base example (with two members) using kubernetes and skaffold by following the local_install_skaffold.md instructions (https://github.com/SubstraFoundation/substra/blob/master/docs/local_install_skaffold.md) and adapting them to a ubuntu server 18.04.3 LTS environment. But I get an error when running the skaffold dev on the hlf_k8s repository:

WARN[0000] Could not get minikube docker env, falling back to local docker daemon: getting minikube env: Running [minikube docker-env --shell none]: stdout , stderr: * 'none' driver does not support 'minikube docker-env' command
, err: exit status 64: exit status 64
Listing files to watch...
 - substrafoundation/hlf-k8s
Generating tags...
 - substrafoundation/hlf-k8s -> substrafoundation/hlf-k8s:0.0.9
Checking cache...
 - substrafoundation/hlf-k8s: Found Locally
Tags used in deployment:
 - substrafoundation/hlf-k8s -> substrafoundation/hlf-k8s:28742731953db474dda3bb4780efdd32f2416144b3de7c037b4e45e48b991d07
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
Error: Chart.lock is out of sync with Chart.yaml
Cleaning up...
Error: unknown flag: --purge
Error: unknown flag: --purge
Error: unknown flag: --purge
FATA[0000] exiting dev mode because first deploy failed: deploying network-orderer: building helm dependencies: exit status 1

As I run the platform on a virtual machine I launched minikube using the --vm-driver='none' flag (and therefor with root permissions). Also I can't seem to use helm correctly, as for instance it doesn't have a helm init command in its man.
Minikube is running and working without errors on the vm, here is the result of the minikube status command:

host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Rename leaderboard variable in substra/cli/interface.py::leaderboard

In substra/cli/interface.py, the "leaderboard" variable should have another name in the leaderboard command, because a variable with the same name is already defined before.

I get this warning: Shadows name 'leaderboard' from outer scope

Doc: missing fields

compute_plan:
- aggregatuple: aggregatuple_id
- composite_traintuple: composite_traintuple_id and permissions.public

New cancel compute plan command

In link with the "fail fast" mission, we need to have in both the cli and sdk cancel compute plan commands / methods with the following signatures:

In the cli:

$ substra cancel compute_plan <compute_plan_id>

This command should support the same options as get compute plan.

In the sdk:

client.cancel_compute_plan(compute_plan_id)

Compute plans will get a new "status" property. It will have to be added to the printers in both list and single item displays.

And of course, all of this will need to be tested properly.

Tests: permissions definition in datastore assets are wrong

They should be defined in camelCase as they represent the response from the chaincode.

Output permissions definition in chaincode: link.

Doc: add asset unicity rules in documentation

It can be quite frustrating to know what must be modified in order to push a new version of an asset.

For instance to fix an objective metrics Dockerfile, the user must also update the associated description.

Incomplete JSON file schema in add compute_plan --help

When using the substra add compute_plan --help command, the JSON file schema provided is the following:

{
      "traintuples": list[{
          "algo_key": str,
          "data_manager_key": str,
          "train_data_sample_keys": list[str],
          "traintuple_id": str,
          "in_models_ids": list[str],
          "tag": str,
      }],
      "composite_traintuples": list[{
          "algo_key": str,
          "data_manager_key": str,
          "train_data_sample_keys": list[str],
          "in_head_model_id": str,
          "in_trunk_model_id": str,
          "out_trunk_model_permissions": {
              "authorized_ids": list[str],
          },
          "tag": str,
      }]
      "aggregatetuples": list[{
          "algo_key": str,
          "worker": str,
          "in_models_ids": list[str],
          "tag": str,
      }],
      "testtuples": list[{
          "objective_key": str,
          "data_manager_key": str,
          "test_data_sample_keys": list[str],
          "testtuple_id": str,
          "traintuple_id": str,
          "tag": str,
      }]
  }

But when I try to reproduce this schema and to add a compute_plan, I get this error:

Requests error status 400: {"composite_traintuples":[{"composite_traintuple_id":["This field is required."],"out_trunk_model_permissions":{"public":["This field is required."]}}],"aggregatetuples":[{"aggregatetuple_id":["This field is required."]}]}
Error: Request failed: InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/compute_plan/

Looks like some fields are missing in the example.

titanic: Missing keyrings.alt

When running the titanic examples with skaffold, I get an error related to keyring.alt.

Running the command below fixes it:

pip install keyrings.alt

Error when adding train algo in titanic (2)

I switched to the master branch, installed the keyrings.alt dependency, and am still running in a Ubuntu server LTS 18.04.2 environment.
Now when I run the add_dataset_objective.py script I do generate a assets_keys.json file in my titanic base folder.
But when I try to run the add_train_algo_random_forest.py script I get the following error:

Adding algo...
Requests error status 409: {"message":[{"pkhash":["algo with this pkhash already exists."]}],"pkhash":"5d5fca19c9d72195359f6b57c6f43846e5c0a41d20f3bd1f0444cc9312e6c1a2"}
algo already exists: key='5d5fca19c9d72195359f6b57c6f43846e5c0a41d20f3bd1f0444cc9312e6c1a2'
Registering traintuple...
Requests error status 400: {"objective_key":["This field may not be null."]}
Traceback (most recent call last):
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 107, in _request
    r.raise_for_status()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/add_train_algo_random_forest.py", line 80, in <module>
    }, exist_ok=True)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 420, in add_traintuple
    res = self._add(assets.TRAINTUPLE, data, timeout=timeout, exist_ok=exist_ok)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 154, in _add
    **requests_kwargs)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 207, in add
    **request_kwargs,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 155, in request
    **request_kwargs,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 119, in _request
    raise exceptions.InvalidRequest.from_request_exception(e)
substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/

I checked the content of the assets_keys.json file and the objective_key is specified, but for some reason it isn't passed on:

{
  "dataset_key": "c1d9b42d9538f825109c38c4f599d47391347d198a0770331ba790b7ebcfaa40",
  "objective_key": "1158d2f5c0cf9f80155704ca0faa28823b145b42ebdba2ca38bd726a1377e1cb",
  "train_data_sample_keys": [
    "b091986a07225b0d95e85b736a2856a717a8a0b98698b377e6048e4f861cc26d",
    "b61e5914eacf180d5db29af27e055ec2953fff2aa99347dbec4c49d34bff2270",
    "76bd8acd270d93b39d0a04fe15d8803b6041c0ca0c801fb1d96a60e572520455",
    "1c5e6be2d9341fef6288416b0edecdd0493ed13ed3ecb7361da75fc508bff019",
    "3350ec9bb4652daa74df08bc4868fe600f460f39773bce1597bfe1847b2dd0a2",
    "980a9f3da9defd7edd6fff4b728cd23762aac94931f6eb65432dc26cd9fd8997",
    "c20832fa897b6501f67ba9d93b4f77c50bf0126bdede54895058f287b13ca120",
    "df30631d848af93a7f18faef024556b0bf39b6c6a263532f7e5823035c2dc116",
    "d9f00baa90f70f8a2778970c93df49b1b6313f16ce609591b431685db584a523",
    "9542260897d66ba3d4b1479b6311ee13e7514d1091701c75186d276b54386aa9"
  ],
  "test_data_sample_keys": [
    "bb28f131a7caa5ef1b9d61f99160b7a06069ad5ce2765bdd213f2b32af2f5f81",
    "5483ea647daa49c6c6da9e2ed4a5ce6f9f7d4c029bdade549a6c8e8148c44555"
  ]
}

Local install using skaffold on MacOS

Following this guide and using minikube as suggested here, I have issues to run skaffold dev on hlf-k8s and backend. For frontend, I got the expected lines in the terminal.

For hlf-k8s, orderer, org 1 and org 2 are deployed and then I got these warnings:

WARN[0051] error adding label to runtime object: patching resource org-1/network-org-1-peer-1-hlf-k8s-hook-desinstall-chaincode: jobs.batch "network-org-1-peer-1-hlf-k8s-hook-desinstall-chaincode" not found 
WARN[0057] error adding label to runtime object: patching resource org-2/network-org-2-peer-1-hlf-k8s-hook-cleanup: jobs.batch "network-org-2-peer-1-hlf-k8s-hook-cleanup" not found 
WARN[0058] error adding label to runtime object: patching resource org-2/network-org-2-peer-1-hlf-k8s-hook-desinstall-chaincode: jobs.batch "network-org-2-peer-1-hlf-k8s-hook-desinstall-chaincode" not found 
Watching for changes...

followed by several lines [DEBUG] and [INFO] finishing by:

[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] tar: Removing leading `/' from member names
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-idcert created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-idkey created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-cacert created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-tls created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-client-tls created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-tlsrootcert created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-client-tlsrootcert created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-admincert created
[network-org-2-peer-1-hlf-k8s-bootstrap-peer-4qp64 fabric-tools] secret/hlf-adminkey created

and after stoping the process with ctrl+c:

Cleaning up...
Error: jobs.batch "network-orderer-hlf-k8s-hook-cleanup" already exists
Error: jobs.batch "network-org-1-peer-1-hlf-k8s-hook-cleanup" already exists
release "network-org-2-peer-1" deleted

For the backend, I got these kind of lines over and over:

[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] INFO - 2020-01-03 15:41:52,296 - events.apps - Retry to connect the event application to the ledger
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] ERROR - 2020-01-03 15:41:53,307 - events.apps - <_MultiThreadedRendezvous of RPC that terminated with:
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	status = StatusCode.UNAVAILABLE
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	details = "failed to connect to all addresses"
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	debug_error_string = "{"created":"@1578066113.307234644","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1578066113.307230733","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}"
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] >
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] Traceback (most recent call last):
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/src/app/events/apps.py", line 150, in ready
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     with get_hfc() as (loop, client):
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/local/lib/python3.6/contextlib.py", line 81, in __enter__
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     return next(self.gen)
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/src/app/substrapp/ledger_utils.py", line 122, in get_hfc
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     loop, client = LEDGER['hfc']()
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/src/app/backend/settings/deps/ledger.py", line 62, in get_hfc_client
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     decode=True
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     return future.result()
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/local/lib/python3.6/site-packages/hfc/fabric/client.py", line 1844, in query_channels
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     res = await asyncio.gather(*responses)
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]   File "/usr/local/lib/python3.6/site-packages/aiogrpc/channel.py", line 40, in __call__
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend]     return await fut
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	status = StatusCode.UNAVAILABLE
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	details = "failed to connect to all addresses"
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] 	debug_error_string = "{"created":"@1578066113.307234644","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1578066113.307230733","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}"
[backend-org-1-substra-backend-server-cdc97b8db-d2l79 substra-backend] >

I tried with minikube v1.5.1 and v1.6.0 and helm v2.16.1.

Enable Client.login() to use information from ~/.substra and passwords from keyring

In order to login in a script, we currently have to proceed as follows:

client = substra.Client()
client.add_profile('node-1', 'node-1', 'p@$swr0d44',  'http://substra-backend.node-1.com')
client.login()

And we have to do so even if the node-1 profile already exists in the .substra file and the associated password is in the keyring.

It'd be much easier if we could just do (assuming node-1 is an existing profile):

client = substra.Client(profile_name='node-1')
client.login()

Not only is it shorter but it also removes the need to write down password in the scripts.

Support skaffold setup in examples

Our examples currently rely on the docker-compose setup and have to be manually modified in order to work with the skaffold setup.

We need a way of handling both setups as long as they are both officially supported.

Auto-batching of tuples creation in add_compute_plan

There is an existing limit to how big a compute plan can be at creation. This limit depends on a lot of factors, but mainly on the size of messages exchanged between the backend and the chaincode. In order to create very large compute plan we add an update_compute_plan method that allows us to add tuples to a compute plan batch by batch.

We could implement an auto-batching feature, (add_compute_plan(data, batch=True)) that would automatically split a compute plan into well-sized tuple batches.

This would require the SDK to analyse the shape of the compute plan so that a batch only depends on tuples created in previous batches.

Error when trying to register testtuple in titanic

As I try tu run the titanic example on my Ubuntu server LTS 18.04.2 environment I run into the following issue:
I've successfully generated my data samples, then added my dataset objective to substra. But when I run the add_train_algo_random_forest.py I get an error message as the script tries to add the test tuple to substra. Here's the error message I get:

Adding algo...
Registering traintuple...
Registering testtuple...
Requests error status 400: {"message":"could not register this testtuple, the traintuple 8438e2f00b5c86d2c0fe1ad6d6f3f854f5afc71e0e393e6b238d708752802536 has a status failed","pkhash":"359adcfe5128bf9b89502586f3d4d2f222160f0fe7b8e6f6f19a217dc4cef912"}
Traceback (most recent call last):
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 107, in _request
    r.raise_for_status()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/testtuple/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/add_train_algo_random_forest.py", line 92, in <module>
    }, exist_ok=True)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 498, in add_testtuple
    res = self._add(assets.TESTTUPLE, data, timeout=timeout, exist_ok=exist_ok)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 154, in _add
    **requests_kwargs)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 207, in add
    **request_kwargs,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 155, in request
    **request_kwargs,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 119, in _request
    raise exceptions.InvalidRequest.from_request_exception(e)
substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/testtuple/: could not register this testtuple, the traintuple 8438e2f00b5c86d2c0fe1ad6d6f3f854f5afc71e0e393e6b238d708752802536 has a status failed

If I inspect the logs of my worker pod here's what I get:

ERROR - 2020-01-31 09:57:48,543 - substrapp.tasks.tasks - [00-01-0000-72d1733]                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                        
  File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task                                                                                                                                              
    R = retval = fun(*args, **kwargs)                                                                                                                                                                                                     
  File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 650, in __protected_call__                                                                                                                                      
    return self.run(*args, **kwargs)                                                                                                                                                                                                      
  File "/usr/src/app/substrapp/tasks/tasks.py", line 530, in compute_task                                                                                                                                                                 
    max_retries=int(getattr(settings, 'CELERY_TASK_MAX_RETRIES')))                                                                                                                                                                        
  File "/usr/local/lib/python3.6/dist-packages/celery/app/task.py", line 704, in retry                                                                                                                                                    
    raise_with_context(exc)                                                                                                                                                                                                               
  File "/usr/src/app/substrapp/tasks/tasks.py", line 523, in compute_task                                                                                                                                                                 
    prepare_materials(subtuple, tuple_type)                                                                                                                                                                                               
  File "/usr/src/app/substrapp/tasks/tasks.py", line 555, in prepare_materials                                                                                                                                                            
    prepare_algo(directory, traintuple_type, subtuple)                                                                                                                                                                                    
  File "/usr/src/app/substrapp/tasks/tasks.py", line 105, in prepare_algo                                                                                                                                                                 
    content = get_algo(tuple_type, tuple_)                                                                                                                                                                                                
  File "/usr/src/app/substrapp/tasks/tasks.py", line 98, in get_algo                                                                                                                                                                      
    metadata['content']['hash'],                                                                                                                                                                                                          
  File "/usr/src/app/substrapp/tasks/utils.py", line 36, in get_asset_content                                                                                                                                                             
    return get_remote_file_content(url, authenticate_worker(node_id), content_hash, salt=salt)                                                                                                                                            
  File "/usr/src/app/substrapp/utils.py", line 187, in get_remote_file_content                                                                                                                                                            
    raise NodeError(f"url {url}: hash doesn't match {content_hash} vs {computed_hash}")                                                                                                                                                   
substrapp.utils.NodeError: url http://substra-backend.node-1.com/algo/5d5fca19c9d72195359f6b57c6f43846e5c0a41d20f3bd1f0444cc9312e6c1a2/file/: hash doesn't match 5d5fca19c9d72195359f6b57c6f43846e5c0a41d20f3bd1f0444cc9312e6c1a2 vs 3163d
INFO - 2020-01-31 09:57:48,679 - substrapp.ledger_utils - smartcontract invoke:logFailTrain; elaps=136.07ms; error=None

Improve configuration errors

Here are a list of situations in which we should get ConfigException or a ProfileNotFoundException but get a random error instead.

I think that all values explicitly passed to the Client should be checked at init.

Without a `.substra` file

>>> c = Client()
>>> client.set_profile('foo')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jeremy/.substra'

>>> c = Client(profile_name='foo')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jeremy/.substra'

With or without a `.substra` file

>>> c = Client()
>>> c.login()
MissingSchema: Invalid URL 'None/api-token-auth/': No schema supplied. Perhaps you meant http://None/api-token-auth/?

>>> c = Client(config_path='foo')
>>> c.login()
MissingSchema: Invalid URL 'None/api-token-auth/': No schema supplied. Perhaps you meant http://None/api-token-auth/?

>>> c = Client(config_path='foo', profile_name='bar')
FileNotFoundError: [Errno 2] No such file or directory: 'foo'

>>> c = Client(config_path='foo')
>>> c.set_profile('bar')
FileNotFoundError: [Errno 2] No such file or directory: 'foo'

Docs: update local install with skaffold setup

Should we:

keep the instructions for docker-compose and add skaffold instructions
or, replace docker-compose instructions with skaffold instructions

For skaffold, we should pay attention to the settings required to mount correctly the volume shared between the localhost and the backend pods.

Add dataset with empty file

When trying to add a dataset with an empty description or data_opener, the error is always the same.
It should be good to precise which file is empty in the error message.

EDIT: The reason is finally displayed, but in JSON.

Error: Request failed: InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/data_manager/: [{'description': ['The submitted file is empty.']}]

Docs: add guide to ingest data in substra

This step will often be complicated and more documentation will help our users.

Questions to be answered:

What are the modes available to add data to substra ?
How to choose between the different modes ?
How to add "a lot of" data samples (>100-1000) ?
How to add data samples depending on their size (>10-100mo) ?
...

It would be good to:

enrich the current documentation for the add_data_sample(s) command / method.
add a dedicated guide: "How to ingest data in Substra"

CLI: add profile in commands suggested by printers

A few of the asset printers provide commands to the user:

a command to display the description of an asset after a get
a command to download an asset after a get
a command to display a leaderboard

These suggested commands currently do not include the profile that was used to run the previous command. This prevent users from simply copy/pasting the suggested commands in their terminal.

Example

Current ouput

1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1
KEY                         1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3
NAME                        Skin Lesion Classification Objective
METRICS                     macro-average recall
TEST DATASET KEY            None
TEST DATA SAMPLE KEYS       None
OWNER                       MyPeer1MSP
PERMISSIONS                 Processable by anyone

Download this objective's metric:
	substra download objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Display this objective's description:
	substra describe objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Display this objective's leaderboard:
	substra leaderboard 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Desired output

$ substra get objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1
KEY                         1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3
NAME                        Skin Lesion Classification Objective
METRICS                     macro-average recall
TEST DATASET KEY            None
TEST DATA SAMPLE KEYS       None
OWNER                       MyPeer1MSP
PERMISSIONS                 Processable by anyone

Download this objective's metric:
	substra download objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Display this objective's description:
	substra describe objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Display this objective's leaderboard:
	substra leaderboard 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Error when trying to add an objective during titanic example

I am trying to run the add_dataset_objective.py on my ubuntu server 18.04.3 LTS environment. Here is the error I get, even though the passwords are correctly specified in the config.json files :

Requests error status 400: {"password":["This field is required."]}
Traceback (most recent call last):
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 44, in login
    r.raise_for_status()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com:80/api-token-auth/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/add_dataset_objective.py", line 59, in <module>
    client.login()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 86, in login
    res = self.client.login()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 53, in login
    raise exceptions.BadLoginException.from_request_exception(e)
substra.sdk.exceptions.BadLoginException: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com:80/api-token-auth/

The error gets fixed if I install the keyrings.alt dependency.

Make it clearer that we should use the compute plan method to create tuples

Ideas:

Documentation / SDK:

add a warning, or recommended section in the add_testtuple and add_traintuple methods docstrings
or, deprecate add_testtuple / add_traintuple methods (make it private? add a deprecation warning at runtime...)

Examples should only use compute plans?

Keyring might not return the store password

Keyring may not be working on some OS (seen on Debian).

One encountered error is that it is failing silently to fetch the password. It leads then to an authentication error, this is misleading for the user.

To solve this, we could:

display a proper error message when keyring fails to fetch the password (Fetching password error: please check your keyring installation)
update the documentation for Debian (requires https://packages.debian.org/fr/stretch/debian-keyring and https://pypi.org/project/keyrings.alt/) (optional)

SDK: when adding an asset, don't always add permissions field

Currently when adding an asset a permissions is added by default. This should not be the case for the traintuple and testtuple assets for instance.

Adding a permissions field should be done explicitly when it's required.

Error when adding train algo in titanic

I'm trying to run the add_train_algo_random_forest.py in the Titanic example on my ubuntu server 18.04.3 LTS environment and get the following error:

Adding algo...
Requests error status 409: {"message":[{"pkhash":["algo with this pkhash already exists."]}],"pkhash":"45df2bbabc90ab1ea2581609e77a2f6d07c1d9d27d734a24357b84545504ec2a"}
algo already exists: key='45df2bbabc90ab1ea2581609e77a2f6d07c1d9d27d734a24357b84545504ec2a'
Registering traintuple...
Requests error status 400: {"message":"url http://substra-backend.node-1.com/data_manager/470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5/opener/: hash doesn't match 470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5 vs 3163d83bbafed39314abe0d22419087b1ca5be2fc243c5b6b99e0ce5a344e9f0"}
Traceback (most recent call last):
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 107, in _request
    r.raise_for_status()
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-2.com:80/data_manager/470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/add_train_algo_random_forest.py", line 79, in <module>
    data_keys = client.get_dataset(data_manager_key)['trainDataSampleKeys']
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 572, in get_dataset
    return self.client.get(assets.DATASET, dataset_key)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 172, in get
    path=f"{key}",
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 155, in request
    **request_kwargs,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 119, in _request
    raise exceptions.InvalidRequest.from_request_exception(e)
substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-2.com:80/data_manager/470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5/: url http://substra-backend.node-1.com/data_manager/470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5/opener/: hash doesn't match 470077b542617692152f9da3b9a24202c76032baf9d0dfd05e4ee412c0eb2ef5 vs 3163d83bbafed39314abe0d22419087b1ca5be2fc243c5b6b99e0ce5a344e9f0

I'm running it on the multinodes-examples branch of the substra repository.

titanic: examples don't work with skaffold

The config file points at the docker-compose backend URLs, so the titanic script don't work.

run-local: remove evaluation from training step

KeyError with full traceback

When I tried to add a traintuple without the "keys" field in the --data-samples-path field, I got a full traceback instead of only an error message:

Traceback (most recent call last):
  File "/Users/mael/.venv/substra/bin/substra", line 11, in <module>
    load_entry_point('substra', 'console_scripts', 'substra')()
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/mael/.venv/substra/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/mael/substra/substra/substra/cli/interface.py", line 172, in wrapper
    return fn(*args, **kwargs)
  File "/Users/mael/substra/substra/substra/cli/interface.py", line 611, in add_traintuple
    data['train_data_sample_keys'] = data_samples['keys']
KeyError: 'keys'

Error when trying to run substra config

I am trying to run the substra config http://substra-backend.node-1.com:80 -v 0.0 -u node-1 -p 'p@$swr0d44' command on my ubuntu server 18.04.3 LTS environment. Here is the error I get :

Traceback (most recent call last):
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/bin/substra", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/cli/interface.py", line 256, in add_profile_to_config
    insecure=insecure,
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/config.py", line 125, in add_profile
    keyring.set_password(name, username, password)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/keyring/core.py", line 62, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/home/ubuntu/dev/substra/examples/titanic/.venv/lib/python3.6/site-packages/keyring/backends/fail.py", line 24, in get_password
    raise RuntimeError(msg)
RuntimeError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.

The error gets fixed if I install the keyrings.alt dependency.

Add debug for the CLI's tests

Currently, this is really hard to debug the CLI.
It could be good to add some information when a test crashes.
For instance, the full traceback, the exc_code, etc.

SDK improve add asset method

The add_asset method is always doing a get_asset before returning. This is not required and it should be avoided.

This is done in order to ensure that the get_asset and the add_asset methods return the same objects (or similar objects).

For instance an add_method could return a object with a pkhash field and a get_asset method could return an object with a key field. This kind of behaviour makes it very difficult to handle Timeout exceptions (if the add_asset method timeout, we retry on till the object exist using the get_asset method). That's one of the reason why a get_asset has been added at the end of the add_asset method. It is also good to have consistent responses for get/add methods.

We should investigate more to identify the assets that are really different and fix the problem at the source: in the backend or in the chaincode and update the required responses.

mock_requests returns a mock that is always called

In lots of places, we use mock_requests and check that the returned mock has been called. These assert m.is_called() are present everywhere. However it seems they don't work as we think they do.

Here is a test that shouldn't pass:

def test_foo(mocker):
    m = mock_requests(mocker, "post", response='foo')
    assert m.is_called()

But it does!

We need to investigate why it passes and update the mock_requests so that it works as we think it should.

CLI: add in model keys for add_traintuple command

It will be great to add this missing optional parameter.
A TODO already exists in the code and should be removed.

[load testing] Missing traintuples

I'm currently writing load tests for the platform : Substra/substra-tests#50

While running the command pytest tests/test_load.py -rs -v --durations=0 -k test_load_multi_node_aggregates with compute_plan_size=100 against the backend's master branch (b72edc65) and the chaincode's index-error branch (a2dc0e4, just adds logs to what master already does), my test fails.

All logs are below, but the important part is that the list compute plan command returns:

substra list compute_plan
COMPUTE PLAN ID                                                     TRAINTUPLES COUNT       COMPOSITE TRAINTUPLES COUNT     AGGREGATETUPLES COUNT       TESTTUPLES COUNT    STATUS
e7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890    101                     0                               100                         0                   todo

First thing is that the compute_plan is stuck in todo.

But there should be 201 traintuples, not 101.

And the list traintuples command fails as well:

substra list traintuple
Requests error status 400: {"message":"could not retrieve parent traintuple with key 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf - traintuple 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf not found"}
Error: Request failed: InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/: could not retrieve parent traintuple with key 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf - traintuple 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf not found

Command log:

pytest tests/test_load.py -rs -v --durations=0 -k test_load_multi_node_aggregates
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.7.3, pytest-5.3.0, py-1.8.0, pluggy-0.13.0 -- /Users/jeremy/.virtualenvs/substra/bin/python3.7
cachedir: .pytest_cache
tests run uuid: 5390c4c731ce40bb9d8ecc638e402b9d'
substra network configuration loaded from: '/Users/jeremy/substra/substra-tests/tests/../values.yaml'
substra network setup:
  - node: name=node-1 msp_id=MyOrg1MSP address=http://substra-backend.node-1.com
  - node: name=node-2 msp_id=MyOrg2MSP address=http://substra-backend.node-2.com
rootdir: /Users/jeremy/substra/substra-tests
plugins: celery-4.2.1, mock-1.12.1, cov-2.8.1
collected 3 items / 2 deselected / 1 selected

tests/test_load.py::test_load_multi_node_aggregates[100] FAILED                                                                                       [100%]

========================================================================= FAILURES ==========================================================================
___________________________________________________________ test_load_multi_node_aggregates[100] ____________________________________________________________

self = <substra.sdk.rest_client.Client object at 0x10ecb3668>, request_name = 'get', url = 'http://substra-backend.node-1.com/traintuple/'
request_kwargs = {'params': 'search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890'}
fn = <function get at 0x10da28c80>
kwargs = {'params': 'search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890'}, r = <Response [400]>

    def _request(self, request_name, url, **request_kwargs):
        """Base request helper."""

        if request_name == 'get':
            fn = requests.get
        elif request_name == 'post':
            fn = requests.post
        else:
            raise NotImplementedError

        # override default request arguments with input arguments
        kwargs = dict(self._default_kwargs)
        kwargs.update(request_kwargs)

        # do HTTP request and catch generic exceptions
        try:
            r = fn(url, headers=self._headers, **kwargs)
>           r.raise_for_status()

../substra/substra/sdk/rest_client.py:107:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Response [400]>

    def raise_for_status(self):
        """Raises stored :class:`HTTPError`, if one occurred."""

        http_error_msg = ''
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode('utf-8')
            except UnicodeDecodeError:
                reason = self.reason.decode('iso-8859-1')
        else:
            reason = self.reason

        if 400 <= self.status_code < 500:
            http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)

        elif 500 <= self.status_code < 600:
            http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)

        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/?search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890

../../.virtualenvs/substra/lib/python3.7/site-packages/requests/models.py:940: HTTPError

During handling of the above exception, another exception occurred:

compute_plan_size = 100
global_execution_env = (<substratest.factory.AssetsFactory object at 0x10db53e48>, Network(sessions=[<substratest.client.Session object at 0x10db53d68>, <substratest.client.Session object at 0x10db24be0>]))

    @pytest.mark.parametrize('compute_plan_size', COMPUTE_PLAN_SIZES)
    def test_load_multi_node_aggregates(compute_plan_size, global_execution_env):
        """
        Shape of the compute plan:

        Node A: traintuple 0 --> traintuple --> aggregate --> traintuple --> aggregate ...
        Node B:              \-> traintuple /             \-> traintuple /
        """

        factory, network = global_execution_env
        session_1 = network.sessions[0].copy()
        session_2 = network.sessions[0].copy()

        dataset_1 = session_1.state.datasets[0]
        dataset_2 = session_2.state.datasets[0]

        spec = factory.create_algo()
        algo = session_1.add_algo(spec)

        spec = factory.create_aggregate_algo()
        aggregate_algo = session_1.add_aggregate_algo(spec)

        cp_spec = factory.create_compute_plan()
        first_tuple = cp_spec.add_traintuple(
            algo=algo,
            dataset=dataset_1,
            data_samples=[dataset_1.train_data_sample_keys[0]],
        )

        previous_aggregatetuple = first_tuple
        for _ in range(compute_plan_size):
            tuples = [
                cp_spec.add_traintuple(
                    algo=algo,
                    dataset=dataset,
                    data_samples=[dataset.train_data_sample_keys[0]],
                    in_models=[previous_aggregatetuple],
                ) for dataset in [dataset_1, dataset_2]
            ]
            previous_aggregatetuple = cp_spec.add_aggregatetuple(
                aggregate_algo=aggregate_algo,
                worker=session_1.node_id,
                in_models=tuples,
            )

        cp = session_1.add_compute_plan(cp_spec)
>       first_tuple = cp.list_traintuple()[0]

tests/test_load.py:150:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
substratest/assets.py:422: in list_traintuple
    tuples = self._session.list_traintuple(filters=filters)
substratest/client.py:216: in list_traintuple
    res = self._client.list_traintuple(*args, **kwargs)
../substra/substra/sdk/client.py:628: in list_traintuple
    return self.client.list(assets.TRAINTUPLE, filters=filters)
../substra/substra/sdk/rest_client.py:184: in list
    **request_kwargs,
../substra/substra/sdk/rest_client.py:155: in request
    **request_kwargs,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <substra.sdk.rest_client.Client object at 0x10ecb3668>, request_name = 'get', url = 'http://substra-backend.node-1.com/traintuple/'
request_kwargs = {'params': 'search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890'}
fn = <function get at 0x10da28c80>
kwargs = {'params': 'search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890'}, r = <Response [400]>

    def _request(self, request_name, url, **request_kwargs):
        """Base request helper."""

        if request_name == 'get':
            fn = requests.get
        elif request_name == 'post':
            fn = requests.post
        else:
            raise NotImplementedError

        # override default request arguments with input arguments
        kwargs = dict(self._default_kwargs)
        kwargs.update(request_kwargs)

        # do HTTP request and catch generic exceptions
        try:
            r = fn(url, headers=self._headers, **kwargs)
            r.raise_for_status()

        except requests.exceptions.ConnectionError as e:
            raise exceptions.ConnectionError.from_request_exception(e)

        except requests.exceptions.Timeout as e:
            raise exceptions.Timeout.from_request_exception(e)

        except requests.exceptions.HTTPError as e:
            logger.error(f"Requests error status {e.response.status_code}: {e.response.text}")

            if e.response.status_code == 400:
>               raise exceptions.InvalidRequest.from_request_exception(e)
E               substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/traintuple/?search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890: could not retrieve parent traintuple with key 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf - traintuple 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf not found

../substra/substra/sdk/rest_client.py:119: InvalidRequest
--------------------------------------------------------------------- Captured log call ---------------------------------------------------------------------
ERROR    substra.sdk.rest_client:rest_client.py:116 Requests error status 400: {"message":"could not retrieve parent traintuple with key 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf - traintuple 40a30edeb7313f0e9acf7e04a86d6af5a3e13123623b70014d49eebe3a11beaf not found"}
================================================================== slowest test durations ===================================================================
8.45s call     tests/test_load.py::test_load_multi_node_aggregates[100]
7.48s setup    tests/test_load.py::test_load_multi_node_aggregates[100]
0.01s teardown tests/test_load.py::test_load_multi_node_aggregates[100]
============================================================= 1 failed, 2 deselected in 16.27s ==============================================================

In the worker logs for org1, I have:

│ substra-backend   [SQL Queries for] /compute_plan/                                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.003] SELECT authtoken_token.key,  authtoken_token.user_id,  authtoken_token.created,  auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.e │
│ mail,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FROM authtoken_token INNER JOIN auth_user ON (authtoken_token.user_id = auth_user.id) WHERE authtoken_token.key = '0b5fc165082e18cb851571373cb127b611b2be58'                                         │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.003 seconds (1 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:22] "POST /compute_plan/ HTTP/1.1" 201 20359                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [SQL Queries for] /traintuple/                                                                                                                                                                                                                             │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.003] SELECT authtoken_token.key,  authtoken_token.user_id,  authtoken_token.created,  auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.e │
│ mail,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FROM authtoken_token INNER JOIN auth_user ON (authtoken_token.user_id = auth_user.id) WHERE authtoken_token.key = '0b5fc165082e18cb851571373cb127b611b2be58'                                         │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.003 seconds (1 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:22] "GET /traintuple/?search=traintuple%3AcomputePlanID%3Ae7f07657336f3b0841b8cf03369dab3b86a9e0edbbf803970cc7e0806e9ba890 HTTP/1.1" 400 212                                                                                              │
│                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [SQL Queries for] /algo/aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff/file/                                                                                                                                                             │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.019] SELECT substrapp_algo.pkhash,  substrapp_algo.file,  substrapp_algo.description,  substrapp_algo.validated FROM substrapp_algo WHERE substrapp_algo.pkhash = 'aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff'                    │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.023 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:23] "GET /algo/aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff/file/ HTTP/1.1" 200 548                                                                                                                                   │
│ substra-backend [19/Dec/2019 15:25:31] "GET /readiness HTTP/1.1" 200 2                                                                                                                                                                                                       │
│ substra-backend INFO - 2019-12-19 15:25:34,062 - events.apps - Processing task da0f0318fe6262e5911058f69c754ab617831ea3004281aa95708e94456a8392: type=traintuple status=todo                                                                                                 │
│                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [SQL Queries for] /algo/aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff/file/                                                                                                                                                             │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.003] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT substrapp_algo.pkhash,  substrapp_algo.file,  substrapp_algo.description,  substrapp_algo.validated FROM substrapp_algo WHERE substrapp_algo.pkhash = 'aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff'                    │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.005 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:34] "GET /algo/aeb2d551a1fb776e61d1d79e61ba72b1ae39a521e579416437445ff2743a53ff/file/ HTTP/1.1" 200 548                                                                                                                                   │
│
substra-backend   [SQL Queries for] /model/e41b28842f09293ea616a056cfde1f9da1ac550135802efa346f15b576b179ca/file/                                                                                                                                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT substrapp_model.pkhash,  substrapp_model.validated,  substrapp_model.file FROM substrapp_model WHERE substrapp_model.pkhash = 'e41b28842f09293ea616a056cfde1f9da1ac550135802efa346f15b576b179ca'                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.004 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:35] "GET /model/e41b28842f09293ea616a056cfde1f9da1ac550135802efa346f15b576b179ca/file/ HTTP/1.1" 200 6                                                                                                                                    │
│ substra-backend INFO - 2019-12-19 15:25:39,369 - events.apps - Processing task 5384f339553b3f594235fe51acfa80e8475aeb97a93dee05ea9a44103939b238: type=aggregatetuple status=todo                                                                                             │
│                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [SQL Queries for] /aggregate_algo/e93f5354ad4470d15ee7418af5a1263b54acc3a05460ad583c051cb2b235dc4c/file/                                                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT substrapp_aggregatealgo.pkhash,  substrapp_aggregatealgo.file,  substrapp_aggregatealgo.description,  substrapp_aggregatealgo.validated FROM substrapp_aggregatealgo WHERE substrapp_aggregatealgo.pkhash = 'e93f5354ad4470d15ee7418af5a126 │
│ 3b54acc3a05460ad583c051cb2b235dc4c'                                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.004 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:39] "GET /aggregate_algo/e93f5354ad4470d15ee7418af5a1263b54acc3a05460ad583c051cb2b235dc4c/file/ HTTP/1.1" 200 529                                                                                                                         │
│                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [SQL Queries for] /model/2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b/file/                                                                                                                                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT substrapp_model.pkhash,  substrapp_model.validated,  substrapp_model.file FROM substrapp_model WHERE substrapp_model.pkhash = '2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b'                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.004 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:40] "GET /model/2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b/file/ HTTP/1.1" 200 6                                                                                                                                    │
substra-backend   [SQL Queries for] /model/2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b/file/                                                                                                                                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.002] SELECT auth_user.id,  auth_user.password,  auth_user.last_login,  auth_user.is_superuser,  auth_user.username,  auth_user.first_name,  auth_user.last_name,  auth_user.email,  auth_user.is_staff,  auth_user.is_active,  auth_user.date_joined FR │
│ OM auth_user WHERE auth_user.username = 'MyOrg1MSP'                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT node_incomingnode.node_id,  node_incomingnode.secret FROM node_incomingnode WHERE node_incomingnode.node_id = 'MyOrg1MSP'                                                                                                                   │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [0.001] SELECT substrapp_model.pkhash,  substrapp_model.validated,  substrapp_model.file FROM substrapp_model WHERE substrapp_model.pkhash = '2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b'                                            │
│                                                                                                                                                                                                                                                                              │
│ substra-backend   [TOTAL TIME: 0.004 seconds (3 queries)]                                                                                                                                                                                                                    │
│ substra-backend [19/Dec/2019 15:25:40] "GET /model/2302f37e01ff8fb57a22e7c5f3346b551997bd6dc6e69e2f6e5725bff263cf6b/file/ HTTP/1.1" 200 6                                                                                                                                    │
│ substra-backend INFO - 2019-12-19 15:25:45,051 - events.apps - Processing task f170eb58c5b5932e3e9c56c0004d3c1f0755b46b96f58b68e3a9a87ec69e04cc: type=traintuple status=todo                                                                                                 │
│ substra-backend Process Process-2:                                                                                                                                                                                                                                           │
│ substra-backend Traceback (most recent call last):                                                                                                                                                                                                                           │
│ substra-backend   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap                                                                                                                                                                        │
│ substra-backend     self.run()                                                                                                                                                                                                                                               │
│ substra-backend   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run                                                                                                                                                                                │
│ substra-backend     self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                │
│ substra-backend   File "/usr/src/app/events/apps.py", line 138, in wait                                                                                                                                                                                                      │
│ substra-backend     loop.run_until_complete(stream)                                                                                                                                                                                                                          │
│ substra-backend   File "/usr/local/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete                                                                                                                                                                    │
│ substra-backend     return future.result()                                                                                                                                                                                                                                   │
│ substra-backend   File "/usr/local/lib/python3.6/site-packages/hfc/fabric/channel/channel_eventhub.py", line 560, in handle_stream                                                                                                                                           │
│ substra-backend     self._processChaincodeEvents(block)                                                                                                                                                                                                                      │
│ substra-backend   File "/usr/local/lib/python3.6/site-packages/hfc/fabric/channel/channel_eventhub.py", line 465, in _processChaincodeEvents                                                                                                                                 │
│ substra-backend     e['tx_status'])                                                                                                                                                                                                                                          │
│ substra-backend   File "/usr/src/app/events/apps.py", line 72, in on_tuples                                                                                                                                                                                                  │
│ substra-backend     tuple_owner = tuple_get_worker(tuple_type, _tuple)                                                                                                                                                                                                       │
│ substra-backend   File "/usr/src/app/events/apps.py", line 43, in tuple_get_worker                                                                                                                                                                                           │
│ substra-backend     return _tuple['dataset']['worker']                                                                                                                                                                                                                       │
│ substra-backend TypeError: 'NoneType' object is not subscriptable                                                                                                                                                                                                            │

[docker-compose] nexus.hyperledger.org port 443: No route to host

Hi, I'm trying to install Substra with docker-compose (following this guide) and I am getting a connection error on the domain nexus.hyperledger.org since yesterday while running the bootstrap.sh script in the hlf-k8s repo.

This script stops like this:

===> Downloading version 1.4.3 platform specific fabric-ca-client binary
===> Downloading:  https://nexus.hyperledger.org/content/repositories/releases/org/hyperledger/fabric-ca/hyperledger-fabric-ca/linux-amd64-1.4.3/hyperledger-fabric-ca-linux-amd64-1.4.3.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to nexus.hyperledger.org port 443: No route to host

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
==> There was an error downloading the binary file. Switching to incremental download.
==> Downloading file...
~/s/hlf-k8s ❯❯❯ ./bootstrap.sh
===> List out hyperledger docker images
===> Create custom docker images
Sending build context to Docker daemon  1.343MB
Step 1/6 : FROM hyperledger/fabric-tools:1.4.3
 ---> 18ed4db0cd57
Step 2/6 : RUN apt-get update &&   apt-get install -y curl netcat apt-transport-https vim &&   curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - &&   echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee -a /etc/apt/sources.list.d/kubernetes.list &&   apt-get update &&   apt-get install -y kubectl
 ---> Using cache
 ---> 1434bb649c77
Step 3/6 : RUN wget https://github.com/fullstorydev/grpcurl/releases/download/v1.3.0/grpcurl_1.3.0_linux_x86_64.tar.gz &&   tar xvzf grpcurl_1.3.0_linux_x86_64.tar.gz &&   mv grpcurl /bin
 ---> Using cache
 ---> 0bf1fd03de40
Step 4/6 : COPY ./setup-hfc-binaries.sh .
COPY failed: stat /var/lib/docker/tmp/docker-builder802122375/setup-hfc-binaries.sh: no such file or directory
Sending build context to Docker daemon  1.343MB
Step 1/15 : FROM hyperledger/fabric-tools:1.4.3
 ---> 18ed4db0cd57
Step 2/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 82f3432f6b7a
Step 3/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-updates main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 6d26f5948e6c
Step 4/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-security main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 4e8e0c0d5c66
Step 5/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-proposed main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 87377d707977
Step 6/15 : RUN apt-get update
 ---> Using cache
 ---> b26191ba73bf
Step 7/15 : RUN apt-get install -y curl netcat python3-pip python3.6-dev libssl-dev python3-setuptools python3-pip libffi-dev docker.io
 ---> Using cache
 ---> c0eabed1816f
Step 8/15 : COPY images/substra-ca-tools/bootstrap-binaries.sh /tmp
 ---> Using cache
 ---> 54676ad896b7
Step 9/15 : RUN cd /tmp; sh bootstrap-binaries.sh;
 ---> Running in f540dfe7f072
===> Downloading version 1.4.3 platform specific fabric-ca-client binary
===> Downloading:  https://nexus.hyperledger.org/content/repositories/releases/org/hyperledger/fabric-ca/hyperledger-fabric-ca/linux-amd64-1.4.3/hyperledger-fabric-ca-linux-amd64-1.4.3.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to nexus.hyperledger.org port 443: No route to host

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
==> There was an error downloading the binary file. Switching to incremental download.
==> Downloading file...
Failure downloading binaries (curl RC=7). Please try again and the download will resume from where it stopped.
The command '/bin/sh -c cd /tmp; sh bootstrap-binaries.sh;' returned a non-zero code: 1
Sending build context to Docker daemon  1.343MB
Step 1/15 : FROM hyperledger/fabric-tools:1.4.3
 ---> 18ed4db0cd57
Step 2/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 82f3432f6b7a
Step 3/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-updates main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 6d26f5948e6c
Step 4/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-security main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 4e8e0c0d5c66
Step 5/15 : RUN echo "deb http://cn.archive.ubuntu.com/ubuntu bionic-proposed main multiverse restricted universe" >> /etc/apt/sources.list
 ---> Using cache
 ---> 87377d707977
Step 6/15 : RUN apt-get update
 ---> Using cache
 ---> b26191ba73bf
Step 7/15 : RUN apt-get install -y curl netcat python3-pip python3.6-dev libssl-dev python3-setuptools python3-pip libffi-dev docker.io
 ---> Using cache
 ---> c0eabed1816f
Step 8/15 : COPY images/substra-ca-tools-debug/bootstrap-binaries.sh /tmp
 ---> Using cache
 ---> 98fbf7c0826f
Step 9/15 : RUN cd /tmp; sh bootstrap-binaries.sh;
 ---> Running in 05e8a22cd730
===> Downloading version 1.4.3 platform specific fabric-ca-client binary
===> Downloading:  https://nexus.hyperledger.org/content/repositories/releases/org/hyperledger/fabric-ca/hyperledger-fabric-ca/linux-amd64-1.4.3/hyperledger-fabric-ca-linux-amd64-1.4.3.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to nexus.hyperledger.org port 443: No route to host

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
==> There was an error downloading the binary file. Switching to incremental download.
==> Downloading file...
Failure downloading binaries (curl RC=7). Please try again and the download will resume from where it stopped.
The command '/bin/sh -c cd /tmp; sh bootstrap-binaries.sh;' returned a non-zero code: 1

Any idea on how to work this out, please?