Node Installation Documentation

This document is my log of installing the Microsoft Linux Cluster...

HeadNode

• Install Windows 2012 R2, and HPC Pack 2012 R2 U3 Head Node onto a domain server - I called it fi--didelxhn.
• Create a folder C:\HPCLinux, and create a network share called hpclinux that allow everyone access to it.
• copy "%CCP_DATA%InstallShare\LinuxNodeAgent\*.* in that folder. (setup.py and hpcnodeagent.tar.gz arrive)
• Run powershell as admin.
• Export-HpcLinuxCertificate –FilePath C:\HPCLinux\cert.pfx and give it a magic password.
• (To make a certificate manually, a script something like the below might do it, but I couldn't make it work...

New-SelfsignedCertificateEx -Subject "CN=Microsoft HPC Linux Communication" -EKU "Server Authentication","Client Authentication" -KeySpec "Signature" -KeyUsage "DigitalSignature,DataEncipherment,KeyEn
cipherment,NonRepudiation,KeyCertSign" -SAN "fi--didemrchnb","fi--didemrchnb.dide.local","fi--didemrchnb.dide.ic.ac.uk" -NotAfter 2039/01/01 -StoreLocation "LocalMachine" -exportable

Nodes

Install linux and enable SSH

• I'm now using the normal Ubuntu 16.04.02 server, on USB with Rufus.
• Use entire disk with LVM
• Select OpenSSH-server when offered.
• sudo apt-get install gcc g++ openjdk-9-jdk-headless subversion
• sudo apt-get update
• sudo apt-get upgrade

Sort out infiniband support

• The cards I used were the old Voltaire ones, so a bit of hacking was needed:-
• sudo nano /etc/modules - and add ib_mthca rdma_ucm ib_umad ib_uverbs ib_ipoib ib_srp ib_sdp
• sudo modprobe ib_ipoib
• sudo nano /etc/network/interfaces and add the below, where x is the node number+1. (eg, fi--didelx15 should be 12.0.0.16).

auto eth0
  iface eth0 inet dhcp
  metric 100

auto eth1
  iface eth1 inet dhcp
  metric 101

auto ib0
iface ib0 inet static
    address 12.0.0.x
    netmask 255.255.255.0
    broadcast 12.0.0.255
    metric 102

• This assumes that eth0 is the enterprise network (129.31.26.x) and eth1 is the private (11.0.0.x) networks.
• We may need to disable IPv6.
• sudo nano /etc/sysctl.conf, and add the following somewhere:

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

Add the HPC mount for some useful bits

• sudo mkdir -p /hpclinux
• sudo apt-get install cifs-utils
• sudo mount -t cifs //fi--didelxhn/HPCLinux /hpclinux -o user=adminuser,dom=dide.local

Adding to the domain

Install NTP support

• sudo apt-get install ntp
• sudo cp /hpclinux/linux_inst/ntp.conf /etc/ntp.conf (This essentially removed the pools as the main source of time, and replaces with `server time.imperial.ac.uk`
• sudo /etc/init.d/ntp stop
• sudo apt install ntpdate
• sudo ntpdate time.imperial.ac.uk
• sudo /etc/init.d/ntp start

Domain things

• sudo apt-get install winbind libpam-winbind libnss-winbind krb5-user krb5-config libpam-krb5
• The domain, when asked, is DIDE.local - case sensitive.
• sudo cp /hpclinux/linux_inst/nsswitch.conf /etc/nsswitch.conf - adds winbind to passwd group, and removes [NOTFOUND=return] from hosts.
• sudo cp /hpclinux/linux_inst/smb.conf /etc/samba/smb.conf - lots of config for DIDE.
• sudo cp /hpclinux/linux_inst/krb5.conf /etc/krb5.conf - lots more config for DIDE.
• ifconfig -a and make note of the IP address if you haven't already.
• sudo nano /etc/hosts and replace with:-

127.0.0.1     localhost
129.31.x.y    fi--didelx99.dide.local fi--didelx99.dide.ic.ac.uk fi--didelx99
129.31.26.137 fi--didelxhn.dide.local fi--didelxhn.dide.ic.ac.uk fi--didelxhn
129.31.26.21  fi--didedc1.dide.local fi--didedc1.dide.ic.ac.uk fi--didedc1
129.31.26.171 fi--didedc6.dide.local fi--didedc6.dide.ic.ac.uk fi--didedc6
129.31.26.172 fi--didedc7.dide.local fi--didedc7.dide.ic.ac.uk fi--didedc7

I'm not sure strictly why some of these (eg, fi--didelxhn) needs adding, since Ubuntu can already ping fi--didelxhn in all of those forms. However, without adding that, an HttpException occurs when adding the node to the cluster, so this is the non-entirely understood workaround.

• sudo net cache flush
• sudo service smbd restart
• sudo service nmbd restart
• sudo service winbind restart
• sudo kinit adminuser@DIDE.LOCAL
• sudo net ads join -U adminuser

Preparing drive mounting

• sudo apt-get install libpam-mount
• sudo cp /hpclinux/linux_inst/pam_mount.conf.xml /etc/security/pam_mount.conf.xml - this enables looking for .pam_mount.conf.xml in the home folder, and automatically sets up a mount point (on fi--san02) to that folder beforehand.
• sudo cp /hpclinux/linux_inst/.pam_mount.conf.xml /etc/skel - for convenience really. Suggest that users copy all the "." files from /etc/skel to their home folder, to get a nice experience when ssh-ing.

Installing HPC

• cd /hpclinux
• sudo ./install_filters.sh
• sudo python setup.py -install -clusname:fi--didelxhn -certfile:cert.pfx (you'll need the magic password).
• If you need to reinstall/readd, then sudo python setup.py -uninstall and redo the line above.

Securing SSH

• sudo usermod -aG sudo user if you need to add any sudo-ers.
• sudo nano /etc/ssh/sshd_config if you need to set ssh users.
  • Add a line AllowGroups ssh
  • Also, be good and add DenyUsers root and DenyGroups root when you've setup sudo-ers.
  • sudo usermod -aG ssh user to add each user to ssh.
  • sudo service ssh restart to apply changes. Don't lock yourself out muppet-brain.