Microsoft Linux Cluster User Notes

From MRC Centre for Outbreak Analysis and Modelling
Jump to navigation Jump to search

Using the Linux Cluster

Get in touch

This is still a bit experimental, so mail Wes to get started.

Sort out your shares

  • On linux, the ~ symbol refers to something that is probably not what you want - it's your local linux home directory, on the particular node that your job happens to get run on. So different jobs might have a different local linux home directory, because they might be run on different nodes.
  • Instead, use ~/dide/home - which will be mapped to your DIDE home directory.
  • If you want to have other shares mapped, then, edit or create a file in your DIDE home directory, called .pam_mount.conf.xml which contains something like this.
<?xml version="1.0" encoding="utf-8" ?>
<pam_mount>
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/tmp" path="Tmp" server="fi--didef2.dide.ic.ac.uk" fstype="cifs" />
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/f2gsim" path="GlobalSim" server="fi--didef2.dide.ic.ac.uk" fstype="cifs" />
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/nas1gsim" path="Test" server="fi--didenas1.dide.ic.ac.uk" fstype="cifs" />
</pam_mount>
  • Obviously, replace <dide-username> with your username. So note here that /homes/<dide-username> is what Linux refers to when you use ~.
  • The full path to your dide home directory, from a linux job, will therefore be: /homes/<dide-username>/dide/home.

Running Something

Hello World in C

  • We'll probably want the code to be compiled on the cluster, so here's a really simple example that compiles and runs something....
  • Here's my complicated simulation code, which I'm going to save as \\fi--san03\homes\wrh1\testlinux\hw.c.
#include <stdio.h>
int main() {
  printf("Hello World!\n");
  return 0;
}
  • And now I'm going to write a script that compiles this code on Linux, and runs it collecting some results.
  • IMPORTANT: - this is going to be a .sh files, and must have Linux line-endings.
cd ~/dide/home/testlinux
gcc -o hw hw.c
chmod 755 ./hw
./hw > output.txt

Launching it from Windows

  • I'm going to write \\fi--san03\homes\wrh1\testlinux\launch.bat - which is a Windows batch file to launch the job.
job submit /scheduler:fi--didelxhn /jobtemplate:LinuxNodes ~/dide/home/testlinux/run.sh
  • This launch.bat can be anywhere on your local machine, but I've kept everything in the same place for simplicity.
  • Run the launch.bat on your Windows machine, and when the job runs, you'll see the hw executable file, and the output.txt file appear.

Launching it from the Web Portal

  • If you're not on Windows (which may be one reason you want to run on a linux cluster...), then use the HPC Web Portal to launch the job.
  • Log into the portal - if you've got rights, you'll see fi--didelxn listed. (Otherwise, see the section at the top...!)
  • Submit Job on the fi--didelxn.
  • Paste ~/dide/home/testlinux/run.sh in the main job window. Set other things if you like. And submit it.
  • Look for hw and output.txt to appear in the test folder.