An Update on HPC's Missing Middle

The report "Council on Competitiveness and USC-ISI Broad Study of Desktop Technical Computing End Users and HPC" [pdf] by the Council on Competitiveness and sponsored by DARPA in 2008 found that a large number of companies deployed virtual prototyping and data modeling software (e.g., Computational Fluid Dynamics [CFD]) on entry level High Performance Computing (HPC) or desktop computers as opposed to realizing the performance and fidelity gains available with true HPC systems. In a subsequent follow up report [pdf] the authors coined the phrase 'the missing middle' to describe the gap in HPC usage.


The report noted that HPC adoption did not strongly correlate with large companies (though in another report it did), so the missing middle here refers to HPC users not company size. The report concluded that "...the studies also revealed some major gaps in the rapidly growing HPC market - and each gap represents an important opportunity for bolstering U.S. economic and national security strength." I contend that this conclusion extends to many other countries too.


The report suggests that the lack of HPC adoption is primarily due to:

  • Lack of easy-to-use application software
  • Lack of sufficient human expertise
  • Costs (hardware, software, and personnel)

The report was published in 2008 and was based on studies performed during 2005-2008, I would argue that there has been major progress made in most areas.


I can speak for the CFD field which has seen a marked shift in recent times toward unified simulation environments, as with our own Caedium system, where the entire CFD workflow is better supported. No longer is there a need to switch between multiple disparate applications to perform certain tasks. This has made CFD much easier to learn (a single GUI) and use. I think the 'lack of easy-to-use application software' and the 'lack of sufficient human expertise' at the application software level is being addressed.

An added bonus is that the performance/cost ratio for HPC hardware continues to improve, especially if you factor in the emergence of GPUs - thank you Moore's Law.

Another game changer is the availability of cloud-based HPC services as an alternative to the hassle of hosting and maintaining your own HPC cluster, which reduces both upfront hardware costs and human expertise requirements.

Application Software Costs

The Council on Competitiveness report cited cost as a major constraint preventing further adoption of HPC. It's clear that application costs are still a stubborn barrier preventing further HPC adoption. As an example, consider that many CFD vendors are still using legacy (and lucrative) per-core pricing - in an era when even a phone can have a dual-core processor. In somewhat of a compromise these same vendors have begun to offer unlimited parallel licenses, but at eye-watering premiums. It is worth noting that the software cost to develop CFD to run on 2-processes compared to a 100-processes is the same.

It is time to abandon legacy pricing models to promote HPC applications for the missing middle. It is time to offer reasonable parallel pricing to let HPC fulfill its promise.


Hi Rich, in general I am all

Hi Rich, in general I am all for HPC systems. However, very often they are much less useful in practice than one would expect. This basically comes down to the fact that whenever you get a good system, you will end up with few power users who totally bog down the queue. I've often been in a situation where my job ran much faster on a cluster - once it started running. But it first had to sit in the queue for few days. I could have gotten results in the meantime on my local workstation. And since most desktop computers now come with multiple cores (my 800 dollar laptop has 8), it's very easy to deploy parallel / multi-threaded applications on your local computer.

Workstations are good enough

Yes, like you say today's multi-core workstations are very capable. Maybe the report reflects that workstations are good enough to satisfy 90% of HPC applications.