Home › Articles

GPU v1.0 Linear Solver Library for OpenFOAM

Submitted by symscape on January 21, 2013 - 15:20

For the latest version of ofgpu for OpenFOAM visit "GPU v1.1 Linear Solver Library for OpenFOAM".

ofgpu v1.0 is the latest version of our free GPL library that provides GPU (sometimes referred to as GPGPU) linear solvers for OpenFOAM® v2.1.x. The library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. GPU acceleration holds the promise of providing significant speed up at relatively low cost and with low power consumption compared to other alternatives. If you want to try our ofgpu library with OpenFOAM then we recommend that you use either a dedicated (i.e., not displaying graphics) high-performance NVIDIA graphics card or a TESLA card.

ofgpuOpenFOAM on Windows using ofgpu

OpenFOAM GUI

Are you looking for a GUI driven, integrated simulation environment with GPU support for Windows and Linux with NVIDIA CUDA devices incorporating a selection of the RANS OpenFOAM solvers and various import/export filters?

Get Started With Caedium Now

Background

In our ofgpu library we use the Cusp library's Krylov solvers to produce equivalent GPU (CUDA) versions of the standard OpenFOAM linear solvers:

We also have support in ofgpu for the Cusp preconditioners:

  • no
  • diagonal
  • smoothed_aggregation - equivalent to Algebraic Multi-Grid (AMG)
  • scaled_bridson_ainv
  • bridson_ainv
  • nonsym_bridson_ainv

Prerequisites

Preparation

Install and test:

Build Procedure

This build procedure assumes single precision.

  1. Download our free GPL licensed ofgpu library [last update Jan. 21, 2013 to v1.0].
  2. Configure and generate the build system for ofgpu with CMake, Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install
  3. Build ofgpu
    • On *nix systems make install and create the symbolic link: ln -s <ofgpu-root>/install/release/lib <ofgpu-root>/install/lib
    • For Windows you will need to build ofgpu natively using Visual Studio and then transfer ofgpu.dll and ofgpu.lib to your Linux system for cross compiling. Place your natively compiled Windows files in <ofgpu-root>/install/win/lib on your Linux system.
  4. Patch your OpenFOAM source with our CUDA modifications. Note that this is the same source code patch that adds Windows support for OpenFOAM 2.1.x.
  5. chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake
  6. Build OpenFOAM in the usual manner, but with the addition of the environment variables:
    • WM_GPU='CUDA'
    • CUDA_ARCH_PATH=<ofgpu-root>
    • WM_PRECISION_OPTION=SP

    Note that single precision is required. Double precision is untested.

The build process will produce the following 2 new OpenFOAM libraries:

  • $(WM_PROJECT_DIR)/platforms/<platform>/lib/cuda/[lib]gpu.[so|dll]
  • $(WM_PROJECT_DIR)/platforms/<platform>/lib/gpuless/[lib]gpu.[so|dll]

Under Windows you will need to have the $(WM_PROJECT_DIR)/platforms/<platform>/lib/cuda directory in your executable path to use the library.

Example Case

For an example of a GPU-enabled case try pitzDailyGpu.

The GPU library is loaded in <case>/system/controlDict with:

functions
{
   cudaGpu
   {
      type cudaGpu;
      functionObjectLibs ("gpu");
      cudaDevice 0;
   }
}

As an example the p and U|k|epsilon linear solvers in the <case>/system/fvSolution file are modified as follows:

p
{
   solver PCGgpu;
   preconditioner smoothed_aggregation;
   tolerance 1e-06;
   relTol 0.01;
}

"(U|k|epsilon)"
{
   solver PBiCGgpu;
   preconditioner diagonal;
   tolerance 1e-05;
   relTol 0.1;
}

Notes

OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape.

Comments

why case with GPU-enable is far slow than the one of GPU-disable

Submitted by LIQNEU (not verified) on February 13, 2013 - 11:10.

I have tested the "double precision" and "single-precision" OpenFOAM with OFGPU-v1.0 under windows 7 platform.

For an example of a GPU-enabled and -diable case of pitzDailyGpu under the same computer with non-parallel cpu conditions. And clock time is 188s and 67s, respectively.

I wonder why case with GPU-enable is far slow than the one of GPU-disable. Any suggestions?

my GPU is:

Starting time loop

::LoadLibrary (LiQ) = "gpuCUDA"
ofgpu CUDA info begin
Available CUDA devices
Device: 0
Name: "GeForce GTS 450"
Driver Version: 5.0
Runtime Version: 5.0
Capability: 2.1
Processors: 4
Cores: 192
Memory (MBytes): 511.688
GPU Clock rate (GHz): 1.566
Selected CUDA device: 0
ofgpu CUDA info end

LiQ

I have further tested the

Submitted by LIQNEU (not verified) on February 13, 2013 - 22:51.

I have further tested the different cases considering grid number using OF-2.1.x and ofgpu-v.1 at double precision mode.

results:
grid number 12225 GPU-disable 67s; GPU-enable 188s for timestep number 753.
grid number 135400 GPU-disable 1777s; GPU-enable 957s for timestep number 1000

Advantage of GPU-enbale solver can be achieved through increasing grid number.

Thanks Symscape's excellent work.

LiQ

New problem: when grid number

Submitted by LiQNEU (not verified) on February 20, 2013 - 04:33.

New problem:
when grid number is 1024000, compressibleInterFoam's GPU is 610.805s and CPU with GPU-disable is 579.499s, why?
I confusion!!!

a problem in the build procedure

Submitted by JACK (not verified) on March 11, 2013 - 08:51.

Hi,everyone
I have a problem in the Build Procedure

the second step:Configure and generate the build system for ofgpu with CMake, Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install
the third step:On *nix systems make install and create the symbolic link: ln -s <ofgpu-root>/install/release/lib <ofgpu-root>/install/lib

I have build the system of ofgpu, but I can't find the file of <ofgpu-root>/install, or what's mean of <ofgpu-root>/install.
I can't continue the next steps because of the problem. any suggestions? please help me.

Thank you.

CMake help

Submitted by symscape on March 11, 2013 - 09:40.

For CMake configuration help see the comment from a previous ofgpu release at "Versions and CMake".

Thank you for you attention.

Submitted by JACK (not verified) on March 18, 2013 - 20:18.

Thank you for you attention. I have solve the problems with your help.

But at the fourth step: Patch your OpenFOAM source with our CUDA modifications. I am very confused.
Can you provide me some details procedures or some tips about how to Install MinGW-w64 cross-compiler and zlib on Linux because I'm a new programmer.

the error is:

checking _mingw_mac.h usability... no
checking _mingw_mac.h presence... no
checking for _mingw_mac.h... no
configure: error: Please check if the mingw-w64 header set and the build/host option are set properly.
configure: error: ./configure failed for mingw-w64-crt

Please help me. Thank you very much.

Good luck!!

Use a pre-compiled MinGW-w64

Submitted by symscape on March 18, 2013 - 20:22.

I suggest you pick up a pre-compiled MinGW-w64 compiler rather than compile your own.

Hi,everyone Now I have

Submitted by Auggie (not verified) on March 22, 2013 - 07:52.

Hi,everyone

Now I have finished the third step,but I am confused in the fourth step in the build procedure.
The MinGW-w64 is a cross-compiler, but I am only in the linux not windows. I was wondering if it is necessary to install the MinGW and are there some adjustment for installing steps in linux?

Thank you for all of your helps.

Still have to patch OpenFOAM

Submitted by symscape on March 22, 2013 - 10:17.

If you only need a Linux build then you don't need to install the MinGW-w64 compiler, but you still have to patch OpenFOAM.

skip the first four steps

Submitted by Auggie (not verified) on March 23, 2013 - 22:00.

Thank you for you help.

I skip all the procedures about MinGW-w64 because I only need a Linux build.

So in the fourth step: Patch your OpenFOAM source with our CUDA modifications, I adjusted and simplied the steps.

For example, I skip the first four steps in the patch build process and go the the fifth step immediately (Install the OpenFOAM 2.1.x source distribution on Linux).

I am going to skip all the adjustments about MinGW-w64 such as the environment variables, I was wondering if it is right and workable.

Is there any suggestion? Thank you .

the error

Submitted by Auggie (not verified) on March 23, 2013 - 23:27.

I do what I said above.

1.git checkout 68d39c199c99cda16108ddcf019743d4e6f18854

Note: checking out '68d39c199c99cda16108ddcf019743d4e6f18854'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b new_branch_name

HEAD is now at 68d39c1... fvcReconstruct: Corrected name of new version of fvc::reconstruct

2. patch -p0 -i v8-mingw-openfoam-2-1-x.patch

but the error is :

patching file extra/scotch/src/Make.inc/Makefile.inc.linux
patching file extra/scotch/src/Make.inc/Makefile.inc.mingw-w64
patching file extra/scotch/src/Make.inc/Makefile.inc.darwin
patching file extra/scotch/src/libscotch/common.c
patching file extra/scotch/src/libscotch/scotch.h
patching file extra/scotch/src/libscotch/Makefile
patching file extra/scotch/src/Makefile
patching file extra/tecio/tecio.patch
patching file wmake/rules/linux64Gcc/gpulibCUDA
can't find file to patch at input line 3352
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|Index: wmake/rules/linux64Gcc/c
|===================================================================
|--- wmake/rules/linux64Gcc/c (.../tags/2.1.x-6) (revision 779)
|+++ wmake/rules/linux64Gcc/c (.../branches/2.1.x-6) (revision 779)
--------------------------
File to patch:

I was wondering what I should do next. Thank you very much.

patch successfully

Submitted by Auggie (not verified) on March 25, 2013 - 08:53.

Dear Symscape

Thank you for all of your help.

I have tried it again and again. Now I have patched the v8-mingw-openfoam-2-1-x.patch successfully.
I skip the 7th step about seting OpenFOAM MinGW-w64 environment variables and source etc/bashrc because I don't install the MinGW and the 8th step about Installing the ThirdParty-2.1.0 source.
Then I go to build OpenFOAM. When I run ./Allwmake in the $(WM_PROJECT_DIR). The errors appears.

I hope I'm not cluttering the comments, and here is the part of the output:

wclean mpi
wmake libso mpi
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UOPwrite.C
could not open file omp.h for source file UOPwrite.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UOPwrite.C
Making dependency list for source file UIPread.C
could not open file omp.h for source file UIPread.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UIPread.C
Making dependency list for source file UPstream.C
could not open file omp.h for source file UPstream.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UPstream.C
Making dependency list for source file PstreamGlobals.C
could not open file omp.h for source file PstreamGlobals.C
......
solidEquilibriumDisplacementFoam.C:38:19: fatal error: fvCFD.H: No such file or directory
compilation terminated.
make[2]: *** [Make/linux64GccDPOpt/solidEquilibriumDisplacementFoam.o] error 1
......

Do you have any idea what could be the problem? Thank you.

Error

Submitted by Anonymous (not verified) on August 22, 2013 - 14:55.

I get this error when I do: make install

25 errors detected in the compilation of "/tmp/tmpxft_0001d17e_00000000-6_sparsematrixsystem.cpp1.ii".
CMake Error at CMakeFiles/ofgpu_generated_sparsematrixsystem.cu.o.cmake:256 (message):
Error generating file
/home/shashank/ofgpu/ofgpu/obj/release/ofgpu/./ofgpu_generated_sparsematrixsystem.cu.o

make[2]: *** [ofgpu/./ofgpu_generated_sparsematrixsystem.cu.o] Error 1
make[1]: *** [ofgpu/CMakeFiles/ofgpu.dir/all] Error 2
make: *** [all] Error 2

How did you managed to get it

Submitted by Roberto Ribeiro (not verified) on September 9, 2013 - 10:22.

How did you managed to get it run in double precision mode? what changes did you made?

Thanks.

parallel computations

Submitted by diablo80 (not verified) on September 19, 2013 - 07:16.

Hi,

does your lib also work in parallel, if I use more than one CPU? I'm getting strange continuity errors by calculating your test case using more than one CPU.

ofgpu does not support MPI

Submitted by symscape on September 19, 2013 - 07:19.

If by parallel you mean in combination with MPI, then no ofgpu does not support MPI.

yes I meant in combination

Submitted by diablo80 (not verified) on September 19, 2013 - 07:22.

yes I meant in combination with MPI.

BTW, does your lib support more than one nvidia card? (topic: multiple gpu)

ofgpu does not support multiple GPU cards

Submitted by symscape on September 19, 2013 - 07:28.

No, ofgpu does not support multiple GPU cards.

Hi, can i have some quick

Submitted by ROberto RIbeiro (not verified) on September 19, 2013 - 09:46.

Hi,

can i have some quick and brief information in what the methods SparseMatrixSystem::initialize(SparseMatrixArgs const & args) and SparseMatrixSystem::assignMatrix(SparseMatrixArgs const & args) do? Like in which memory system they operate host or device?

Thanks

Hi, I've got some problems

Submitted by Elia (not verified) on September 20, 2013 - 11:40.

Hi, I've got some problems regarding OpenFOAM compiling.

I'm on ubuntu 12.04.
I've successfully installed CUDA and cusp (I've run a lot of benchmark cases)
I've successfully installed ofgpu-1-0 libraries (I'm pretty sure...) in /opt/OpenFOAM/ofgpu/

Then I have to compile OpenFOAM.
My command-line input are:

cd /opt/OpenFOAM/
git clone https://github.com/OpenFOAM/OpenFOAM-2.1.x.git
cd OpenFOAM-2.1.x
git checkout 68d39c199c99cda16108ddcf019743d4e6f18854
cp ~/Downloads/v9-mingw-openfoam-2-1-x.patch .
patch -p0 -i v9-mingw-openfoam-2-1-x.patch --------> OK! (NO ERROR MESSAGES)
vim ./etc/bashrc
-----> modified this voices into bashrc file

  • foamInstall=/opt/$WM_PROJECT
  • export WM_PRECISION_OPTION=SP
  • export WM_GPU='CUDA'
  • export CUDA_ARCH_PATH=/opt/OpenFOAM/ofgpu

chmod ug+x /opt/OpenFOAM/OpenFOAM-2.1.x/src/gpu/Allwmake
source ./etc/bashrc
./Allwmake

It begins to compile and after few seconds of beautiful outputs an "error cascade" occurs on my terminal.
The first ERROR message I get is this:

wclean mpi
wmake libso mpi
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UOPwrite.C
could not open file mpi.h for source file UOPwrite.C
Making dependency list for source file UIPread.C
could not open file mpi.h for source file UIPread.C
Making dependency list for source file UPstream.C
could not open file mpi.h for source file UPstream.C
Making dependency list for source file PstreamGlobals.C
could not open file mpi.h for source file PstreamGlobals.C
SOURCE=UOPwrite.C ; g++ -m64 -Dlinux64 -DWM_SP -Wall -Wextra -Wno-unused-parameter -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM/ThirdParty-2.1.x/platforms/linux64Gcc/openmpi-1.5.3/include -IlnInclude -I. -I/opt/OpenFOAM/OpenFOAM-2.1.x/src/OpenFOAM/lnInclude -I/opt/OpenFOAM/OpenFOAM-2.1.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccSPOptOPENMPI/UOPwrite.o
In file included from UOPwrite.C:29:0:
/usr/include/mpi.h:253:39: fatal error: mpi_portable_platform.h: No such file or directory
compilation terminated.
make: *** [Make/linux64GccSPOptOPENMPI/UOPwrite.o] Error 1

It seeks mpi.h but I thought this patched version of OpenFOAM-gpu should skip mpi compiling.

Any suggestions?

Thank you in advance,

Elia

Patch does not remove MPI support

Submitted by symscape on September 20, 2013 - 11:57.

The patch adds GPU support, it does not remove MPI support. You can run in either MPI or GPU modes, just not at the same time. If you don't want to compile MPI then set WM_MPLIB=NONE

You should set your environment variables similar to:

export FOAM_INST_DIR="$HOME/projects/of/2.1"; source $FOAM_INST_DIR/OpenFOAM-2.1.x/etc/bashrc WM_OSTYPE=POSIX WM_ARCH_OPTION=64 WM_PRECISION_OPTION=SP WM_CC=gcc WM_CXX=g++ foamCompiler=system WM_MPLIB=OPENMPI MPI_ARCH_PATH="$HOME/projects/openmpi/install/linux" WM_GPU=CUDA

Note you only export FOAM_INST_DIR, then pass the other variables as arguments to OpenFOAM-2.1.x/etc/bashrc

Dynamically select cards

Submitted by S. Paul (not verified) on October 10, 2013 - 02:26.

Hi,

I have 2 gpu cards on my system. ofgpu detects both but always select device-0 for executation. Is is possible to dynamically select any one of the device during runtime?

Set device in controlDict

Submitted by symscape on October 10, 2013 - 09:04.

Assuming you have multiple NVIDIA cards then you can change the device in:
<case>/system/controlDict
See the original description above for the syntax.

Recognized devices and device numbers are listed in the std output during startup, as shown in the screenshot.

Feedback

Questions? Ideas? Problems?

Forums

Newsletter

more