GPU v1.1 Linear Solver Library for OpenFOAM

ofgpu v1.1 is the latest version of our free GPL library that provides GPU (sometimes referred to as GPGPU) linear solvers for OpenFOAM® v2.2.x. The library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. GPU acceleration holds the promise of providing significant speed up at relatively low cost and with low power consumption compared to other alternatives. If you want to try our ofgpu library with OpenFOAM then we recommend that you use either a dedicated (i.e., not displaying graphics) high-performance NVIDIA graphics card or a Tesla card.

OpenFOAM on Windows using ofgpuOpenFOAM on Windows using ofgpuHosted by Caedium Professional

OpenFOAM GUI

Are you looking for a GUI driven, integrated simulation environment with GPU support for Windows and Linux with NVIDIA CUDA devices incorporating a selection of the RANS OpenFOAM solvers and various import/export filters?

Get Started With Caedium Now

Background

In our ofgpu library we use the Cusp library's Krylov solvers to produce equivalent GPU (CUDA) versions of the standard OpenFOAM linear solvers:

We also have support in ofgpu for the Cusp preconditioners:

  • no
  • diagonal
  • smoothed_aggregation - equivalent to Algebraic Multi-Grid (AMG)
  • scaled_bridson_ainv
  • bridson_ainv
  • nonsym_bridson_ainv

Prerequisites

Preparation

Install and test:

Build Procedure

This build procedure assumes single precision.

  1. Download our free GPL licensed ofgpu library [last update Jan. 17, 2014 to v1.1].
  2. Configure and generate the build system for ofgpu with CMake, Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install
  3. Build ofgpu
    • On *nix systems make install and create the symbolic link: ln -s <ofgpu-root>/install/release/lib <ofgpu-root>/install/lib
    • For Windows you will need to build ofgpu natively using Visual Studio and then transfer ofgpu.dll and ofgpu.lib to your Linux system for cross compiling. Place your natively compiled Windows files in <ofgpu-root>/install/win/lib on your Linux system.
  4. Patch your OpenFOAM source with our CUDA modifications. Note that this is the same source code patch that adds Windows support for OpenFOAM 2.2.x.
  5. chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake
  6. Build OpenFOAM in the usual manner, but with the addition of the environment variables:
    • WM_GPU='CUDA'
    • CUDA_ARCH_PATH=<ofgpu-root>
    • WM_PRECISION_OPTION=SP

    Note that single precision is required. Double precision is untested.

The build process will produce the following 2 new OpenFOAM libraries:

  • $(WM_PROJECT_DIR)/platforms/<platform>/lib/cuda/[lib]gpu.[so|dll]
  • $(WM_PROJECT_DIR)/platforms/<platform>/lib/gpuless/[lib]gpu.[so|dll]

Under Windows you will need to have the $(WM_PROJECT_DIR)/platforms/<platform>/lib/cuda directory in your executable path to use the library.

Example Case

For an example of a GPU-enabled case try pitzDailyGpu.

The GPU library is loaded in <case>/system/controlDict with:

functions
{
   cudaGpu
   {
      type cudaGpu;
      functionObjectLibs ("gpu");
      cudaDevice 0;
   }
}

As an example the p and U|k|epsilon linear solvers in the <case>/system/fvSolution file are modified as follows:

p
{
   solver PCGgpu;
   preconditioner smoothed_aggregation;
   tolerance 1e-06;
   relTol 0.01;
}

"(U|k|epsilon)"
{
   solver PBiCGgpu;
   preconditioner diagonal;
   tolerance 1e-05;
   relTol 0.1;
}

Notes

OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape.

Comments

Error Compiling

HI,

thanks for your gpu code. I tried to compiling it but I got some errors. I follow all your steps but it seems there are some error with matching function in sparsematrixargs.h.

PBiCGgpu.C: In member function ‘virtual Foam::solverPerformance Foam::PBiCGgpu::solve(Foam::scalarField&, const scalarField&, Foam::direction) const’:
PBiCGgpu.C:104:1: error: no matching function for call to ‘ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, Foam::label, Foam::label, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::const_iterator, Foam::UList::iterator, Foam::UList::const_iterator, const label&, const scalar&, const scalar&, Foam::scalar&, Foam::scalar&, Foam::label&, bool&)’
)
^
PBiCGgpu.C:104:1: note: candidates are:
In file included from PBiCGgpu.C:24:0:
ofgpu/sparsematrixargs.h:46:5: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, ofgpu::index, ofgpu::index, const index*, const index*, const index*, const real*, const real*, const index*, const index*, const real*, ofgpu::real*, const real*, ofgpu::index, ofgpu::real, ofgpu::real, ofgpu::real&, ofgpu::real&, ofgpu::index&, bool&)
SparseMatrixArgs(char const * const preconditionerName,
^
ofgpu/sparsematrixargs.h:46:5: note: no known conversion for argument 7 from ‘Foam::UList::const_iterator {aka const double*}’ to ‘const real* {aka const float*}’
ofgpu/sparsematrixargs.h:24:10: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const ofgpu::SparseMatrixArgs&)
struct SparseMatrixArgs {
^
ofgpu/sparsematrixargs.h:24:10: note: candidate expects 1 argument, 20 provided

Best Regards
Flora

Compile OpenFOAM in single precision

It appears you are trying to compile OpenFOAM in double precision (DP) with ofgpu in single precision (SP)

Confusion and compiling errors.

Hi,
Thanks for the GPU code. I followed the building instruction to step 3. The ofgpu was successfully built how ever in the /install folder, there is only one folder named lib and there is only one file inside it: libofgpu.so. In this case, does it mean there is no need to perform the step as stated in instruction: ln -s /install/release/lib /install/lib ?
With the assumption that there is no need to create the soft link, I went on to step step 4. I'm not sure about the meaning of "CUDA modification". Is it simply apply the "MinGW Windows Modifications" patch at the root directory of OpenFOAM 2.2.x or I need to start from step 1 on webpage(http://www.symscape.com/openfoam-2-2-x-on-windows) to Install MinGW-w64 cross-compiler and zlib" first?

Any suggestion is appreciated.
Best wishes,
Alex

Set build type for cmake

Sounds like you didn't set the build type (release) for cmake, that is why you are missing the release folder.

You only need the MinGW-w64 compiler if you are going to cross compile for Windows.

Hi, Thanks for your

Hi,

Thanks for your reply. After I set the build type to release. The problem solved! Then I followed step 4 and 5 without any problem. (I also did "git checkout 8a983dba63b246772c69ed0fa9cc3b3e33a10f92" to make sure the version is right)
However, at step 6, after sourcing the provided .bashrc and exporting WM_GPU='CUDA' CUDA_ARCH_PATH=/home/chenhui_epcc/ofgpu-1-1-0 and WM_PRECISION_OPTION=SP, I started to execute the Allwmake in the OpenFOAM root directory. Then I got several errors. The first one is(the error message is in the end of the code):
---------------------------------------------------------------------------------------------------------------------------
SOURCE=signals/sigFpe.C ; g++ -m64 -Dlinux64 -DWM_SP -Wall -Wextra -Wno-unused-parameter -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-100 -DFOAM_USE_INOTIFY -IlnInclude -I. -I/home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude -fPIC -c $SOURCE -o Make/linux64GccDPOpt/sigFpe.o
In file included from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/FixedList.H:43:0,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/List.C:29,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/List.H:259,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/HashTable.C:30,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/Istream.H:184,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/token.H:455,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/UILListIO.C:28,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/UILList.C:92,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/UILList.H:330,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/ILList.H:39,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/IDLList.H:35,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/entry.H:45,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/dictionary.H:53,
from /home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/JobInfo.H:42,
from signals/sigFpe.C:28:
/home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/Hash.H:212:12: error: ‘intptr_t’ does not name a type
typedef intptr_t HashType;
----------------------------------------------------------------------------------------------------------------------------

And following that I got similar errors like:
----------------------------------------------------
/home/chenhui_epcc/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude/Hash.H:219:21: error: ‘HashType’ was not declared in this scope
return Hash()(HashType(p), seed);
------------------------------------------------------------------

Any help is appreciated!

Compiler issue

Check your compiler supports 'intptr_t'

I fixed the intptr_t error by

I fixed the intptr_t error by adding:

#include <stdint.h>

to $FOAM_SRC/OpenFOAM/primitives/hashes/Hash/Hash.H

$FOAM_GPU_LIBBIN

I don't remember reading where $FOAM_GPU_LIBBIN had to be defined?

Its used in $FOAM_SRC/gpu/cuda/Make/files and breaks my build if not set. I've set it to $FOAM_LIBBIN.

$FOAM_GPU_LIBBIN

I think this is because you've made the assumption everyone uses bashrc. You haven't modified cshrc in your patch.

bashrc only

There are only so many hours in a day, so I focused my efforts on bashrc. I don't see how not supporting cshrc is a show stopper. Definitely it would be better if I mentioned no cshrc support in the documentation though.

bashrc

I completely agree symscape. I appreciate how much time you must have put into this already, thanks. I've got it working now in Suse Linux 12.3, with tcsh (not that I like csh, its just we're stuck with it here for historic reasons). The only other headaches with the install were getting 'scotch' to build (path issues) and finally putting the compiled libraries in the right folders. It seems to be working now.

If only I could get OpenFOAM to do this implicit coupled conjugate-heat-transfer problem with radiation and turbulence on a high-aspect-ratio mesh I'd finally be happy.

Trending Now