Exploratory work looking at using Kubernetes Federation to run bioinformatics workflows on multiple distant clusters. This could be utilized to send a containerized workflow to different sites with immovable datasets.
This will create three clusters alpha, beta, and mgmt.
git clone https://github.com/slateci/minifed
cd minifed/
./init.sh
git clone https://github.com/KevinSayers/federatedpoc
kubectl config use-context alpha
kubectl create -f pv.yaml
kubectl create -f claim.yaml
kubectl config use-context beta
kubectl create -f pv.yaml
kubectl create -f claim.yaml
This will execute the workflow on each of the clusters. Currently the context must be switched manually. Ideally the Job API could be used to submit jobs to the federated cluster as a whole.
nextflow kuberun KevinSayers/rnaseq-nf -v task-pv-claim:/newtest -profile alpha
nextflow kuberun KevinSayers/rnaseq-nf -v task-pv-claim:/newtest -profile beta
Federate clusters created on different cloud providers.