Parallelism Goes Mainstream
July 29, 2011 - 7:27pm
A clear sign that parallel computing is starting to go mainstream is that the general press starts to cover the topic. Recently The Economist ran an article on the need for parallelism for applications that run in multi-core environments – http://www.economist.com/node/18750706
The summary point is that few have reaped the benefits of the transition to multi-core technology. While virtualization has allowed multiple applications to run simultaneously on a single multi-core processor, few applications have fully leveraged the technology of multiple cores working together to make software run faster. Ultimately this translates into lost opportunity to being competitive.
The world of parallelism, that is using multiple compute cores, has long been treated as the domain of the mad scientist with little view given to how it can help the broader market accelerate research and innovation. HPC is finally becoming entrenched in very large manufacturing organizations such as aerospace and automobile design where the sheer size and the complexity of the modeling have forced the application developers to move to a parallel architecture.
But ultimately the system is only as fast as the slowest component. Along with parallelism in the compute clusters, these applications have also benefited from the move to a parallel file system to manage their storage. This is where Panasas comes in. With an ability to move big data sets at speeds that are only dreamed of in traditional storage environments, Panasas allows very large HPC clusters to take full advantage of multicore capabilities and the resulting application parallelism means huge advances in science.
Competitors who offer a traditional NAS architecture serving up NFS have often argued that Panasas is proprietary. But up until this year every parallel file system was proprietary because without an official standard it was up to visionary companies like IBM, Sun and Panasas to move the industry forward. The standardization process comes later when the industry at large believes that it is the direction of the future, and with HPC workloads now widely using a parallel file system, the momentum has finally pushed a standard across the finish line and into the Linux Kernel. Just to show that things do not move fast in the standards world, the IETF approved the first Internet Draft for the pNFS standard in 2008 and finally by June 2011, pNFS for files and objects was only then included in latest kernel (version 3.0).
The release of Linux Kernel 3.0 was a huge industry milestone. Finally another hurdle to take full advantage of Parallelism has been removed with the standardization of pNFS as the long term file system of choice for HPC applications. However it will likely be at least one more year before pNFS is in the hands of users as it still requires Linux software vendors to include it in commercial releases.