lren-chuv / algorithm-repository Goto Github PK
View Code? Open in Web Editor NEWRepository of functions or algorithms packaged (with Docker) for Algorithm Factory/Woken
Repository of functions or algorithms packaged (with Docker) for Algorithm Factory/Woken
Seen using latest web-analytics-starter:
0bdc026ca55c hbpmip/python-histograms:0.3.6 "python /histograms.…" 55 seconds ago Exited (1) 53 seconds ago
docker logs 0bdc026ca55c 2 changed files master
INFO:root:Using default number of bins: 20
Traceback (most recent call last):
File "/histograms.py", line 149, in <module>
main()
File "/histograms.py", line 37, in main
histograms_results = compute_histograms(dep_var, indep_vars, nb_bins)
File "/histograms.py", line 46, in compute_histograms
histograms.append(compute_histogram(dep_var, nb_bins=nb_bins))
File "/histograms.py", line 59, in compute_histogram
categories, categories_labels = compute_categories(dep_var, nb_bins)
File "/histograms.py", line 91, in compute_categories
minimum = min(values)
ValueError: min() arg is an empty sequence
"PARAM_covariables=",
"PARAM_grouping=dataset,gender,agegroup,alzheimerbroadcategory",
"PARAM_meta={\"tiv\":{\"description\":\"Total intra-cranial volume\",\"methodology\":\"lren-nmm-volumes\",\"label\":\"TIV\",\"code\":\"tiv\",\"units\":\"cm3\",\"type\":\"real\"},\"agegroup\":{\"enumerations\":[{\"code\":\"-50y\",\"label\":\"-50y\"},{\"code\":\"50-59y\",\"label\":\"50-59y\"},{\"code\":\"60-69y\",\"label\":\"60-69y\"},{\"code\":\"70-79y\",\"label\":\"70-79y\"},{\"code\":\"+80y\",\"label\":\"+80y\"}],\"description\":\"Age Group\",\"methodology\":\"mip-cde\",\"label\":\"Age Group\",\"code\":\"agegroup\",\"type\":\"polynominal\"},\"alzheimerbroadcategory\":{\"enumerations\":[{\"code\":\"AD\",\"label\":\"Alzheimer's disease\"},{\"code\":\"CN\",\"label\":\"Cognitively Normal\"},{\"code\":\"Other\",\"label\":\"Other\"}],\"description\":\"There will be two broad categories taken into account. Alzheimer's disease (AD) in which the diagnostic is 100% certain and \\\"Other\\\" comprising the rest of Alzheimer's related categories. The \\\"Other\\\" category refers to Alzheime's related diagnosis which origin can be traced to other pathology eg. vascular. In this category MCI diagnosis can also be found. In summary, all Alzheimer's related diagnosis that are not pure.\",\"methodology\":\"mip-cde\",\"label\":\"Alzheimer Broad Category\",\"code\":\"alzheimerbroadcategory\",\"type\":\"polynominal\"},\"dataset\":{\"enumerations\":[{\"code\":\"cde_features_a\",\"label\":\"CHUV\"},{\"code\":\"cde_features_b\",\"label\":\"Brescia\"},{\"code\":\"cde_features_c\",\"label\":\"Lille\"}],\"description\":\"Variable used to differentiate datasets.\",\"methodology\":\"mip-cde\",\"label\":\"Dataset\",\"code\":\"dataset\",\"type\":\"polynominal\"},\"gender\":{\"enumerations\":[{\"code\":\"M\",\"label\":\"Male\"},{\"code\":\"F\",\"label\":\"Female\"}],\"description\":\"Gender of the patient - Sex assigned at birth\",\"methodology\":\"mip-cde\",\"label\":\"Gender\",\"code\":\"gender\",\"length\":1,\"type\":\"binominal\"}}",
"PARAM_query=SELECT \"tiv\",\"dataset\",\"gender\",\"agegroup\",\"alzheimerbroadcategory\" FROM cde_features_a WHERE \"tiv\" IS NOT NULL",
"PARAM_variables=tiv"
While running python-knn/tests/test.sh:
Run the distributed-knn...
WARNING: Dependency conflict: an older version of the 'docker-py' package may be polluting the namespace. If you're experiencing crashes, run the following command to remedy the issue:
pip uninstall docker-py; pip uninstall docker; pip install docker
Starting tests_db_1 ... done
Traceback (most recent call last):
File "/knn.py", line 28, in <module>
from sklearn_to_pfa.sklearn_to_pfa import sklearn_to_pfa
File "/usr/local/lib/python3.6/site-packages/sklearn_to_pfa/sklearn_to_pfa.py", line 23, in <module>
import titus.prettypfa
File "/usr/local/lib/python3.6/site-packages/titus/prettypfa.py", line 26, in <module>
from titus.pfaast import Subs
File "/usr/local/lib/python3.6/site-packages/titus/pfaast.py", line 25, in <module>
import titus.lib.core
File "/usr/local/lib/python3.6/site-packages/titus/lib/core.py", line 25, in <module>
from titus.signature import Sig
File "/usr/local/lib/python3.6/site-packages/titus/signature.py", line 23, in <module>
import titus.P as P
File "/usr/local/lib/python3.6/site-packages/titus/P.py", line 20, in <module>
from titus.datatype import Type
File "/usr/local/lib/python3.6/site-packages/titus/datatype.py", line 23, in <module>
import avro.io
File "/usr/local/lib/python3.6/site-packages/avro/io.py", line 200
bits = (((ord(self.read(1)) & 0xffL)) |
^
Seen on CLM Vertex, with research + CLM datasets:
"PARAM_variables=rs610932_a",
"CHRONOS_JOB_NAME=python_histograms_bab479f1_1b1b_4e03_b381_dfa4f6893111",
"PARAM_grouping=dataset,gender,agegroup,alzheimerbroadcategory",
"OUT_JDBC_JAR_PATH=/usr/lib/R/libraries/postgresql-9.4-1201.jdbc41.jar",
"OUT_JDBC_PASSWORD=aDB/neuroinfo",
"PARAM_covariables=",
"mesos_task_id=ct:1507726483217:2:python_histograms_bab479f1_1b1b_4e03_b381_dfa4f6893111:",
"CHRONOS_RESOURCE_DISK=256.0",
"DOCKER_IMAGE=hbpmip/python-histograms:4cb93ea",
"MESOS_SANDBOX=/mnt/mesos/sandbox",
"OUT_JDBC_DRIVER=org.postgresql.Driver",
"PARAM_query=select rs610932_a,dataset,gender,agegroup,alzheimerbroadcategory from mip_local_features where rs610932_a is not null and dataset is not null and gender is not null and agegroup is not null and alzheimerbroadcategory is not null ",
"CHRONOS_RESOURCE_CPU=0.5",
"IN_JDBC_URL=jdbc:postgresql://hos49130.intranet.chuv:31433/research",
"OUT_JDBC_URL=jdbc:postgresql://hos49130.intranet.chuv:31433/woken",
"IN_JDBC_USER=research",
"NODE=hos49130.intranet.chuv",
"JOB_ID=bab479f1-1b1b-4e03-b381-dfa4f6893111",
"OUT_JDBC_USER=woken",
"PARAM_meta={\"agegroup\":{\"enumerations\":[{\"code\":\"-50y\",\"label\":\"-50y\"},{\"code\":\"50-59y\",\"label\":\"50-59y\"},{\"code\":\"60-69y\",\"label\":\"60-69y\"},{\"code\":\"70-79y\",\"label\":\"70-79y\"},{\"code\":\"+80y\",\"label\":\"+80y\"}],\"description\":\"Age Group\",\"methodology\":\"mip-cde\",\"label\":\"Age Group\",\"code\":\"agegroup\",\"type\":\"polynominal\"},\"alzheimerbroadcategory\":{\"enumerations\":[{\"code\":\"AD\",\"label\":\"Alzheimer's disease\"},{\"code\":\"CN\",\"label\":\"Cognitively Normal\"},{\"code\":\"Other\",\"label\":\"Other\"}],\"description\":\"There will be two broad categories taken into account. Alzheimer's disease (AD) in which the diagnostic is 100% certain and \\\"Other\\\" comprising the rest of Alzheimer's related categories. The \\\"Other\\\" category refers to Alzheime's related diagnosis which origin can be traced to other pathology eg. vascular. In this category MCI diagnosis can also be found. In summary, all Alzheimer's related diagnosis that are not pure.\",\"methodology\":\"mip-cde\",\"label\":\"Alzheimer Broad Category\",\"code\":\"alzheimerbroadcategory\",\"type\":\"polynominal\"},\"dataset\":{\"enumerations\":[{\"code\":\"edsd\",\"label\":\"EDSD\"},{\"code\":\"adni\",\"label\":\"ADNI\"},{\"code\":\"ppmi\",\"label\":\"PPMI\"}],\"description\":\"Variable used to differentiate datasets.\",\"label\":\"Dataset\",\"code\":\"dataset\",\"type\":\"polynominal\"},\"rs610932_a\":{\"sql_type\":\"int\",\"enumerations\":[{\"code\":0,\"label\":0},{\"code\":1,\"label\":1},{\"code\":2,\"label\":2}],\"description\":\"\",\"methodology\":\"lren-nmm-volumes\",\"label\":\"rs610932_A\",\"code\":\"rs610932_a\",\"type\":\"polynominal\"},\"gender\":{\"enumerations\":[{\"code\":\"M\",\"label\":\"Male\"},{\"code\":\"F\",\"label\":\"Female\"}],\"description\":\"Gender of the patient - Sex assigned at birth\",\"methodology\":\"mip-cde\",\"label\":\"Gender\",\"code\":\"gender\",\"length\":1,\"type\":\"binominal\"}}",
"HOST=hos49130.intranet.chuv",
"IN_JDBC_JAR_PATH=/usr/lib/R/libraries/postgresql-9.4-1201.jdbc41.jar",
"CHRONOS_RESOURCE_MEM=512.0",
"MESOS_CONTAINER_NAME=mesos-3987976c-26a3-415d-8263-f8db921fb1b4",
"IN_JDBC_PASSWORD=r5s3a6c9d8p2v",
"[email protected]",
"IN_JDBC_DRIVER=org.postgresql.Driver",
"PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"LC_ALL=C.UTF-8",
"COMPUTE_IN=/data/in",
"COMPUTE_OUT=/data/out",
"MODEL=histograms",
"FUNCTION=python-histograms",
"CODE=histo",
"NAME=Histograms"
woken version: 9750bdc
woken-validation version: 9750bdc
knn version: 0.2.1 or naive-bayes version: 0.2.0
When running RapidMiner algorithms on some co-variables with no sufficient data i guess (e.g. : leftcerebralwhitematter), RapidMiner complains that: does not have sufficient capabilities for handling an example set with only one label
http://frontend/experiment/?variables=alzheimerbroadcategory&coVariables=minimentalstate
An error has occurred while running your experiment K-nearest neighbors with k=5. Here's the message:
Invalid JSON:
java.lang.Exception: com.opendatagroup.hadrian.errors.PFASemanticException: PFA semantic error at JSON line:col 1:12319 (PFA field "fcns -> toArray -> do -> 0"): array constructed with "new" has wrong type for item 0: {"type":"enum","symbols":["_28","_27","_21","_30","_29","_18","_26","_9","_24","_13","_22","_25","_16","_12","_20","_23","_19","_17","_4","_15","_11","_8"],"name":"Enum_minimentalstate"} rather than "double"
INFO:root:variable: rs17125944_c
INFO:root:groups: ['dataset', 'gender', 'agegroup', 'alzheimerbroadcategory']
INFO:root:columns: ['rs17125944_c', 'dataset', 'gender', 'agegroup', 'alzheimerbroadcategory']
Traceback (most recent call last):
File "/main.py", line 273, in
main()
File "/main.py", line 31, in main
json.dumps(generate_descriptive_stats(var, groups, data, data_columns),
File "/main.py", line 45, in generate_descriptive_stats
output.append(generate_histogram(data, data_columns, var))
File "/main.py", line 78, in generate_histogram
var_categories)
File "/main.py", line 105, in histo_nominal
sums[v.rstrip()] += 1
KeyError: '0'
docker inspect 8ca8c5b6026a
[
{
"Id": "8ca8c5b6026a719d2c1f7b7677905c5d487d4b574ee58854367001d7ae28d4ed",
"Created": "2017-10-16T10:45:54.46321336Z",
"Path": "/docker-entrypoint.sh",
"Args": [
"compute"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2017-10-16T10:45:54.795594797Z",
"FinishedAt": "2017-10-16T10:45:55.313828253Z"
},
"Image": "sha256:5ad4879b87429baf105d219aee70c96575a7c4a084593574a4a48562c789747f",
"ResolvConfPath": "/var/lib/docker/containers/8ca8c5b6026a719d2c1f7b7677905c5d487d4b574ee58854367001d7ae28d4ed/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/8ca8c5b6026a719d2c1f7b7677905c5d487d4b574ee58854367001d7ae28d4ed/hostname",
"HostsPath": "/var/lib/docker/containers/8ca8c5b6026a719d2c1f7b7677905c5d487d4b574ee58854367001d7ae28d4ed/hosts",
"LogPath": "",
"Name": "/mesos-3b920c1c-0295-46f7-913f-29efaf61a17f",
"RestartCount": 0,
"Driver": "overlay",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/mesos/slaves/dcf68ead-69c7-47cf-a4e5-5c062974548c-S0/docker/links/3b920c1c-0295-46f7-913f-29efaf61a17f:/mnt/mesos/sandbox"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": {}
},
"NetworkMode": "host",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 512,
"Memory": 536870912,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay/8b0cedaad7e59864d3157a855d0c8829c6827ca7f2878492ad5630f08e095058/root",
"MergedDir": "/var/lib/docker/overlay/aa995cca2fd566f1a64511665d92fe9ca5a5a34ec03f1713a00e00c91667787e/merged",
"UpperDir": "/var/lib/docker/overlay/aa995cca2fd566f1a64511665d92fe9ca5a5a34ec03f1713a00e00c91667787e/upper",
"WorkDir": "/var/lib/docker/overlay/aa995cca2fd566f1a64511665d92fe9ca5a5a34ec03f1713a00e00c91667787e/work"
},
"Name": "overlay"
},
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/mesos/slaves/dcf68ead-69c7-47cf-a4e5-5c062974548c-S0/docker/links/3b920c1c-0295-46f7-913f-29efaf61a17f",
"Destination": "/mnt/mesos/sandbox",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "hos49130",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"IN_JDBC_PASSWORD=r5s3a6c9d8p2v",
"IN_JDBC_URL=jdbc:postgresql://hos49130.intranet.chuv:31433/research",
"OUT_JDBC_USER=woken",
"PARAM_covariables=",
"PARAM_meta={"rs17125944_c":{"sql_type":"int","enumerations":[{"code":0,"label":0},{"code":1,"label":1},{"code":2,"label":2}],"description":"","methodology":"lren-nmm-volumes","label":"rs17125944_C","code":"rs17125944_c","type":"polynominal"},"agegroup":{"enumerations":[{"code":"-50y","label":"-50y"},{"code":"50-59y","label":"50-59y"},{"code":"60-69y","label":"60-69y"},{"code":"70-79y","label":"70-79y"},{"code":"+80y","label":"+80y"}],"description":"Age Group","methodology":"mip-cde","label":"Age Group","code":"agegroup","type":"polynominal"},"alzheimerbroadcategory":{"enumerations":[{"code":"AD","label":"Alzheimer's disease"},{"code":"CN","label":"Cognitively Normal"},{"code":"Other","label":"Other"}],"description":"There will be two broad categories taken into account. Alzheimer's disease (AD) in which the diagnostic is 100% certain and \"Other\" comprising the rest of Alzheimer's related categories. The \"Other\" category refers to Alzheime's related diagnosis which origin can be traced to other pathology eg. vascular. In this category MCI diagnosis can also be found. In summary, all Alzheimer's related diagnosis that are not pure.","methodology":"mip-cde","label":"Alzheimer Broad Category","code":"alzheimerbroadcategory","type":"polynominal"},"dataset":{"enumerations":[{"code":"edsd","label":"EDSD"},{"code":"adni","label":"ADNI"},{"code":"ppmi","label":"PPMI"}],"description":"Variable used to differentiate datasets.","label":"Dataset","code":"dataset","type":"polynominal"},"gender":{"enumerations":[{"code":"M","label":"Male"},{"code":"F","label":"Female"}],"description":"Gender of the patient - Sex assigned at birth","methodology":"mip-cde","label":"Gender","code":"gender","length":1,"type":"binominal"}}",
"PARAM_query=select rs17125944_c,dataset,gender,agegroup,alzheimerbroadcategory from mip_local_features where rs17125944_c is not null and dataset is not null and gender is not null and agegroup is not null and alzheimerbroadcategory is not null ",
"IN_JDBC_DRIVER=org.postgresql.Driver",
"JOB_ID=40c37a3b-c0f2-4c87-ae1f-360d490a91a3",
"OUT_JDBC_PASSWORD=aDB/neuroinfo",
"OUT_JDBC_URL=jdbc:postgresql://hos49130.intranet.chuv:31433/woken",
"PARAM_variables=rs17125944_c",
"IN_JDBC_JAR_PATH=/usr/lib/R/libraries/postgresql-9.4-1201.jdbc41.jar",
"DOCKER_IMAGE=hbpmip/python-histograms:4cb93ea",
"NODE=hos49130.intranet.chuv",
"CHRONOS_RESOURCE_CPU=0.5",
"OUT_JDBC_DRIVER=org.postgresql.Driver",
"HOST=hos49130.intranet.chuv",
"OUT_JDBC_JAR_PATH=/usr/lib/R/libraries/postgresql-9.4-1201.jdbc41.jar",
"PARAM_grouping=dataset,gender,agegroup,alzheimerbroadcategory",
"CHRONOS_RESOURCE_MEM=512.0",
"IN_JDBC_USER=research",
"CHRONOS_RESOURCE_DISK=256.0",
"MESOS_SANDBOX=/mnt/mesos/sandbox",
"CHRONOS_JOB_NAME=python_histograms_40c37a3b_c0f2_4c87_ae1f_360d490a91a3",
"MESOS_CONTAINER_NAME=mesos-3b920c1c-0295-46f7-913f-29efaf61a17f",
"mesos_task_id=ct:1508150752607:2:python_histograms_40c37a3b_c0f2_4c87_ae1f_360d490a91a3:",
"CHRONOS_JOB_OWNER=[email protected]",
"PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"LC_ALL=C.UTF-8",
"COMPUTE_IN=/data/in",
"COMPUTE_OUT=/data/out",
"MODEL=histograms",
"FUNCTION=python-histograms",
"CODE=histo",
"NAME=Histograms"
],
"Cmd": [
"compute"
],
"Image": "hbpmip/python-histograms:4cb93ea",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"eu.humanbrainproject.category": "Python"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "b32fbbaa1aa9b729443855d9c090aa67e149387d338ebf4a858fa74e357295ed",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/default",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"host": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "dd644a5262cb021644570c9257a95e808a3d4fca5e813fd3513752dfd3a26e8b",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
}
]
woken version: 9750bdc
woken-validation version: 9750bdc
knn version: 0.2.1 or naive-bayes version: 0.2.0
When using nominal variables with "integers" as categories (e.g. apoe4), RapidMiner complains (see woken logs) that : "numerical label not supported"
Use the following template to list all Docker images created in this project, add also for each algorithm a description and a list of its capabilities (predictive, distributed)
https://github.com/LREN-CHUV/java-base-docker-images/blob/master/README.md
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.