Microsoft Linux Cluster User Notes: Difference between revisions

From MRC Centre for Outbreak Analysis and Modelling
Jump to navigation Jump to search
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Mounting shares for cluster use==


The only thing you really need to do before using the linux nodes on the fi--didelxhn cluster, is sort out any windows shares you're going to want access to. When you log in (or when the cluster logs in as you), your DIDE home directory will be automatically mounted as ~/dide/home, (where ~ is automatically set to /homes/<dide-username> - which is cluster-node specific, hence different on every node. Don't save stuff there!). If you want access to other windows shares, then create a file in your DIDE home directory, called <code>.pam_mount.conf.xml</code> containing something like this, replacing <dide-username> with your username (mine would be wrh1):-
== Using the Linux Cluster ==
 
=== Get in touch ===
This is still a bit experimental, so mail Wes to get started.
 
=== Sort out your shares ===
 
* On linux, the <code>~</code> symbol refers to something that is '''probably not what you want''' - it's your local linux home directory, on the particular node that your job happens to get run on. So different jobs might have a different local linux home directory, because they might be run on different nodes.
* Instead, use <code>~/dide/home</code> - which will be mapped to your DIDE home directory.
* If you want to have other shares mapped, then, edit or create a file in your '''DIDE home directory''', called <code>.pam_mount.conf.xml</code> which contains something like this.


<pre>
<pre>
Line 11: Line 19:
</pam_mount>
</pam_mount>
</pre>
</pre>
* Obviously, replace <code><dide-username></code> with your username. So note here that <code>/homes/<dide-username></code> is what Linux refers to when you use <code>~</code>.
* The full path to your dide home directory, from a linux job, will therefore be: <code>/homes/<dide-username>/dide/home</code>.
== Running Something ==
=== Hello World in C ===
* We'll probably want the code to be compiled on the cluster, so here's a really simple example that compiles and runs something....
* Here's my complicated simulation code, which I'm going to save as <code>\\fi--san03\homes\wrh1\testlinux\hw.c</code>.
<pre>
#include <stdio.h>
int main() {
  printf("Hello World!\n");
  return 0;
}
</pre>
* And now I'm going to write a script that compiles this code on Linux, and runs it collecting some results.
* '''IMPORTANT:''' - this is going to be a <code>.sh</code> files, and must have Linux line-endings.
<pre>
cd ~/dide/home/testlinux
gcc -o hw hw.c
chmod 755 ./hw
./hw > output.txt
</pre>
=== Launching it from Windows ===
* I'm going to write <code>\\fi--san03\homes\wrh1\testlinux\launch.bat</code> - which is a '''Windows''' batch file to launch the job.
<pre>
job submit /scheduler:fi--didelxhn /jobtemplate:LinuxNodes ~/dide/home/testlinux/run.sh
</pre>
* This launch.bat can be anywhere on your local machine, but I've kept everything in the same place for simplicity.
* Run the launch.bat on your Windows machine, and when the job runs, you'll see the <code>hw</code> executable file, and the <code>output.txt</code> file appear.
=== Launching it from the Web Portal ===
* If you're not on Windows (which may be one reason you want to run on a linux cluster...), then use the HPC Web Portal to launch the job.
* Log into the portal - if you've got rights, you'll see <code>fi--didelxn</code> listed. (Otherwise, see the section at the top...!)
* Submit Job on the <code>fi--didelxn</code>.
* Paste <code>~/dide/home/testlinux/run.sh</code> in the main job window. Set other things if you like. And submit it.
* Look for <code>hw</code> and <code>output.txt</code> to appear in the test folder.

Latest revision as of 12:06, 10 October 2017

Using the Linux Cluster

Get in touch

This is still a bit experimental, so mail Wes to get started.

Sort out your shares

  • On linux, the ~ symbol refers to something that is probably not what you want - it's your local linux home directory, on the particular node that your job happens to get run on. So different jobs might have a different local linux home directory, because they might be run on different nodes.
  • Instead, use ~/dide/home - which will be mapped to your DIDE home directory.
  • If you want to have other shares mapped, then, edit or create a file in your DIDE home directory, called .pam_mount.conf.xml which contains something like this.
<?xml version="1.0" encoding="utf-8" ?>
<pam_mount>
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/tmp" path="Tmp" server="fi--didef2.dide.ic.ac.uk" fstype="cifs" />
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/f2gsim" path="GlobalSim" server="fi--didef2.dide.ic.ac.uk" fstype="cifs" />
  <volume options="vers=2.1,nodev,nosuid" user="*" mountpoint="/homes/<dide-username>/dide/nas1gsim" path="Test" server="fi--didenas1.dide.ic.ac.uk" fstype="cifs" />
</pam_mount>
  • Obviously, replace <dide-username> with your username. So note here that /homes/<dide-username> is what Linux refers to when you use ~.
  • The full path to your dide home directory, from a linux job, will therefore be: /homes/<dide-username>/dide/home.

Running Something

Hello World in C

  • We'll probably want the code to be compiled on the cluster, so here's a really simple example that compiles and runs something....
  • Here's my complicated simulation code, which I'm going to save as \\fi--san03\homes\wrh1\testlinux\hw.c.
#include <stdio.h>
int main() {
  printf("Hello World!\n");
  return 0;
}
  • And now I'm going to write a script that compiles this code on Linux, and runs it collecting some results.
  • IMPORTANT: - this is going to be a .sh files, and must have Linux line-endings.
cd ~/dide/home/testlinux
gcc -o hw hw.c
chmod 755 ./hw
./hw > output.txt

Launching it from Windows

  • I'm going to write \\fi--san03\homes\wrh1\testlinux\launch.bat - which is a Windows batch file to launch the job.
job submit /scheduler:fi--didelxhn /jobtemplate:LinuxNodes ~/dide/home/testlinux/run.sh
  • This launch.bat can be anywhere on your local machine, but I've kept everything in the same place for simplicity.
  • Run the launch.bat on your Windows machine, and when the job runs, you'll see the hw executable file, and the output.txt file appear.

Launching it from the Web Portal

  • If you're not on Windows (which may be one reason you want to run on a linux cluster...), then use the HPC Web Portal to launch the job.
  • Log into the portal - if you've got rights, you'll see fi--didelxn listed. (Otherwise, see the section at the top...!)
  • Submit Job on the fi--didelxn.
  • Paste ~/dide/home/testlinux/run.sh in the main job window. Set other things if you like. And submit it.
  • Look for hw and output.txt to appear in the test folder.
Retrieved from "http://mrcdata.dide.ic.ac.uk/wiki/index.php?title=Microsoft_Linux_Cluster_User_Notes&oldid=428"