cytomine / cytomine-software-router Goto Github PK

Problem : Jobs was not ran without output in the job logs.

Logs of software_router component : "Batch job submission failed: Unable to contact slurm controller (connect failure)".
How to check : run squeue into the slurm component sudo docker exec -it slurm squeue you should get the same error message.

When : A long singularity image building on a Kubuntu VM (8 proc, 8Gb RAM)

Fixed by a sh restart.sh of the Cytomine bootstrap

I met
"log4j:WARN No appenders could be found for logger (be.cytomine.software.consumer.Main).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info."
when I was checking the logs of software_router, can you help fix this? Thanks.

When we are behind a corporate proxy, we cannot natively reach github API.

Solution : java -jar -Dhttp.proxyHost=... -Dhttp.proxyPort= ... (idem with https) Cytomine-software-router.jar

As you are often running in Docker containers, allow an auto set of these parameters if environment variables used by Docker are set.

When we add a software user repository and fetch all the targeted repositories, sometimes I get "Error: unable to build: conveyor failed to get: no descriptor found for reference".

It seems that, as our external softwares are often based on the same image base (cytomine/python-software for example), we got a multithread issue with concurrent fetch.

There is example in the related issue
https://github.com/sylabs/singularity/issues/3634 and https://github.com/sylabs/singularity/issues/5020

Short solution : implement a pullingQueue to assure that there is no concurrency pulling.
Later solution : Should be resolved with singularity 3.6

If some parameters are not specified in the config.groovy file (see here), it leads to an error.

In particular, one can observe here that if the parameter connectionRetries is not specified, it is considered as a Map and the conversion to an int is impossible.

If Core is not found, it is useless to continue.
+
with docker restart container policy, the exit 1 of the software_router will kill the container then restart it until core is found.

Currently, it is possible to add software by Github and DockerHub repositories and uploading a zip containing a Dockerfile.

Adding a "manual upload" with a Singularity recipe file instead of Dockerfile in a zip could be supported.
https://github.com/cytomine/Cytomine-software-router/blob/master/src/main/groovy/be/cytomine/software/management/SingleSoftwareManager.groovy#L60

Improve backward communication from SR to the Core for error handling (not existing repositories or other potential errors).

Currently, we have a dependency between the name and the repository to fetch.

So, I cannot run a software with name "Statistic_Computation" on a repo S_Stat_Comput

The problem occurs during the launching step because softwareTable is filled with the software name that will be used for later fetching.

We should not use the name to determine a repository url
OR
We should have two different fields : name (user friendly name) and repositoryName

See https://stackoverflow.com/questions/53134212/invalid-privatekey-when-using-jsch
It seems to be mandatory to have -----BEGIN RSA PRIVATE KEY----- header

See how to support OpenSSH keys