Using the Xeon Phi

The Xeon Phi is essentially linux on a chip, on a special card, inside some of our HPC nodes. It's extremely multi-core, so offers itself for algorithms that have reasonably simple but very parallel sections. Here is the experience I've collected so far in using it.

Using Visual Studio with Intel C++ Parallel Compiler

Setup the Project

• Create a new 64-bit project.
• Choose to use the Intel C++ Compiler from the Projects menu.
• Get into Release, x64 mode with the menus at the top.
• In Project Preferences:-
  • Linker, General [Intel C++], Additional Options for MIC Offload Linker, add -no-fortlib
  • (The above is a workaround - the compiler assumes everyone has fortran installed and cries if we don't.)
  • C/C++, Code Generation [Intel C++], Enable OpenMP Offloading Compilation... - choose Intel MIC Architecture
  • And check the Target Device below is also set to Intel MIC Architecture.
  • C/C++, Language [Intel C++], OpenMP Support: Generate Parallel Code
  • C/C++, Language, Runtime Library: Multi-threaded (/MT) - we'll copy more DLLs across later.

Some Code

In this example. I've setup a project called PhiTest, and a single file, main.cpp.

#include <omp.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  
  int thread_id;

  // Here's an OpenMP loop using main processor  

  #pragma omp parallel private (thread_id)
  {
    thread_id = omp_get_thread_num();
    printf("Local thread %d\n", thread_id);
    #pragma omp barrier
    #pragma omp single
    printf("There are %d local threads\n", omp_get_num_threads());
  }

  // Insert an extra offload pragma to do OpenMP on the Phi.

  #pragma offload target(mic)
  #pragma omp parallel private (thread_id)
  {
    thread_id = omp_get_thread_num();
    printf("Offload thread %d\n", thread_id);
    #pragma omp barrier
    #pragma omp single
    printf("There are %d offload threads\n", omp_get_num_threads());
  }
  return EXIT_SUCCESS;
}

And compile it. You might get some "warning #3335: *MIC* offload features on this platform currently require that RTTI be disabled" - don't worry about them.

Prepare a cluster job

• Decide where to run the job as normal. I'm going for T:\Wes\Phi (which is \\fi--didef2\Tmp\Wes\Phi) in this example.
• Copy the executable there, which will be in Release\x64 in your project folder. Mine is called PhiTest.exe.
• Find a folder something like: C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.1.051\compilers_and_libraries_2016\windows\redist\intel64_win\compiler
  • Copy cilkrts20.dll, libiomp5md.dll and liboffload.dll to your run folder (T:\Wes\Phi for me)
• We also need to copy the library for the Phi itself to have. Find a folder something like: C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2016.1.051\compilers_and_libraries_2016\windows\compiler\lib\mic
  • Make a folder called lib in your test folder (ie, T:\Wes\Phi\Lib), and copy all the files you just found into it, including the "locale" folder. If you like command-line copying, then something like

    xcopy *.* /e T:\Wes\Phi\Lib

A batch file to run the job

I'll call this run.bat, and put it in T:\Wes\Phi. I'll assume we'll have the working directory set, so...

set MIC_LD_LIBRARY=\\fi--didef2\Tmp\Wes\Phi\lib
PhiTest.exe

And my launch file

job submit /scheduler:fi--didemrchnb /numnodes:1 /singlenode:false /jobtemplate:Phi /workdir:\\fi--didef2\Tmp\Wes\Phi /stdout:out.txt /stderr:err.txt run.bat

So remember the /singlenode:false is the silly hack we have to do when we ask for a single, whole node.

And the result

In my out.txt, I have...

\\fi--didef2\Tmp\Wes\Phi>set MIC_LD_LIBRARY_PATH=\\fi--didef2\Tmp\Wes\Phi\lib 

\\fi--didef2\Tmp\Wes\Phi>\\fi--didef2\Tmp\Wes\Phi\PhiTest.exe
Offload thread 112
Offload thread 43
Offload thread 117
....
Offload thread 45
There are 240 offload threads

Local thread 12
Local thread 5
Local thread 9
....
There are 16 local threads

That's a lot of threads. Note that my code did the local bit first, but the output has come out in reverse. There may be interleaving issues with stdout, so in real code, do it better!