Repository for the HPC Cluster LSF implementation files. Learn more

Initial configuration:

$ cp sample/configs/hpc_workspace_config.json config.json
$ ibmcloud iam api-key-create trl-tyos-api-key --file ~/.ibm-api-key.json -d "my api key"
$ cat ~/.ibm-api-key.json | jq -r ."apikey"
# copy your apikey
$ vim config.json
# paste your apikey and set entitlements for LSF

You also need to generate github token if you use private Github repository.

Deployment:

$ ibmcloud schematics workspace new -f config.json --github-token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ibmcloud schematics workspace list
Name               ID                                            Description   Status     Frozen
hpcc-lsf-test       us-east.workspace.hpcc-lsf-test.7cbc3f6b                     INACTIVE   False

OK
$ ibmcloud schematics apply --id us-east.workspace.hpcc-lsf-test.7cbc3f6b
Do you really want to perform this action? [y/N]> y

Activity ID b0a909030f071f51d6ceb48b62ee1671

OK
$ ibmcloud schematics logs --id us-east.workspace.hpcc-lsf-test.7cbc3f6b
...
 2021/04/05 09:44:54 Terraform apply | Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
 2021/04/05 09:44:54 Terraform apply |
 2021/04/05 09:44:54 Terraform apply | Outputs:
 2021/04/05 09:44:54 Terraform apply |
 2021/04/05 09:44:54 Terraform apply | sshcommand = ssh -J [email protected]  [email protected]
 2021/04/05 09:44:54 Command finished successfully.
 2021/04/05 09:45:00 Done with the workspace action

OK
$ ssh -J [email protected]  [email protected]

$ ibmcloud schematics destroy --id us-east.workspace.hpcc-lsf-test.7cbc3f6b

• Login to controller node using ssh command
• Check the existing machines part of cluster using $bhosts or $bhosts -w command, this should show all instances
• Submit a job that would spin 1 VM and sleep for 10 seconds -> $bsub -n 1 sleep 10, once submitted command line will show a jobID
• Check the status for job using -> $bjobs -l
• Check the log file under /opt/ibm/lsf/log/ibmgen2... for any messages around provisioning of the new machine
• Continue to check status of nodes using lshosts, bjobs or bhosts commands
• To test multiple VMs you can run multiple sleep jobs -> $bsub -n 10 sleep 10 -> This will create 10 VMs and each job will sleep for 10 seconds

The storage node is configured as an NFS server and the data volume is mounted to the /data directory which is exported to share with LSF cluster nodes.

• Login to the storage node using SSH (ssh -J [email protected] [email protected])
• The below command shows that the data volume, /dev/vdd, is mounted to /data on the storage node.

# df -k | grep data
/dev/vdd       104806400 1828916 102977484   2% /data`

• The command below shows that /data is exported as a NFS shared directory.

# exportfs -v
/data         	10.242.66.0/23(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

• At the NFS client end, the LSF cluster nodes in this case, we mount the /data directory in NFS server to the local directory, /mnt/data.

# df -k | grep data
10.242.66.4:/data 104806400  1828864 102977536   2% /mnt/data

The command above shows that the local directory, /mnt/data, is mounted to the remote /data directory on the NFS server, 10.242.66.4.

For ease of use, we create a soft link, /home/lsfadmin/shared, pointed to /mnt/data. The data that needs to be shared across the cluster can be placed in /home/lsfadmin/shared.

/home/lsfadmin>ls -l
total 0
lrwxrwxrwx. 1 root root 9 May 27 14:52 shared -> /mnt/data

• Login to the controller as shown in the ssh_command output

$ ssh -J [email protected] [email protected]

• Submit a job to write the host name to the /home/lsfadmin/shared directory on the NFS server

$ bsub sh -c 'echo $HOSTNAME > /home/lsfadmin/shared/hello.txt'

• Wait until the job is finished and then run the command to confirm the hostname is written to the file on the NFS share

$ bjobs
$ cat /home/lsfadmin/shared/hello.txt
ibm-gen2host-10-241-0-21  # worker hostname

• Login to scale storage node using SSH. (ssh -J [email protected] [email protected], details will be available in the logs output with key spectrum_scale_storage_ssh_command)
• The below command shows the gpfs cluster setup on scale storage node.

# /usr/lpp/mmfs/bin/mmlscluster

• The below command shows file system mounted on number of nodes

# /usr/lpp/mmfs/bin/mmlsmount all

• The below command shows the fileserver details. This command can be used to validate file block size(Inode size in bytes).

#   /usr/lpp/mmfs/bin/mmlsfs all -i

• Login to controller node using SSH. (ssh -J [email protected] [email protected])
• The below command shows the gpfs cluster setup on computes node. This should contain the controller,controller-candidate, and worker nodes.

# /usr/lpp/mmfs/bin/mmlscluster

• Create a file on mountpoint path(e.g /gpfs/fs1) and verify on other nodes that the file can be accessed.

• Open a new command line terminal.
• Run the following command to access the storage cluster:

#ssh -L 22443:localhost:443 -J root@{FLOATING_IP_ADDRESS} root@{STORAGE_NODE_IP_ADDRESS}

• where STORAGE_NODE_IP_ADDRESS needs to be replaced with the storage IP address associated with hpc-pc-scale-storage-0, which you gathered earlier, and FLOATING_IP_ADDRESS needs to be replaced with the floating IP address that you identified.

• Open a browser on the local machine, and run https://localhost:22443. You will get an SSL self-assigned certificate warning with your browser the first time that you access this URL.

• Enter your login credentials that you set up when you created your workspace to access the Spectrum Scale GUI. Accessing the compute cluster

• Open a new command line terminal.

• Run the following command to access the compute cluster:

 #ssh -L 21443:localhost:443 -J root@{FLOATING_IP_ADDRESS} root@{COMPUTE_NODE_IP_ADDRESS}

• where COMPUTE_NODE_IP_ADDRESS needs to be replaced with the storage IP address associated with hpc-pc-primary-0, which you gathered earlier, and FLOATING_IP_ADDRESS needs to be replaced with the floating IP address that you identified.

• Open a browser on the local machine, and run https://localhost:21443. You will get an SSL self-assigned certificate warning with your browser the first time that you access this URL.

• Enter your login credentials that you set up when you created your workspace to access the Spectrum Scale GUI.