10. An extreme scale implicit solver for highly-heterogeneous PDEs (UT-Austin, NYU)

Research thrusts: Fast & reliable solution of multiphysics/multiscale problems
Research sub-thrusts: Accurate, robust discretizations; Error estimation; Fast solvers

Many PDE modeling problems within the DOE portfolio are characterized by a wide range of spatial scales, extreme heterogeneity/anisotropy in material properties, and severely nonlinear constitutive laws, often requiring high order discretizations, adaptive mesh refinement, and implicit solution. We have developed an extreme-scale parallel solver that is capable of solving such problems with optimal complexity and high parallel efficiency out to the limits of the largest available supercomputers.

Figure 14: Slices through mantle convection model of earth, depicting aggressive mesh adaptivity at subducting plate boundaries and 8 orders of magnitude variation in viscosity (color). Solver achieves 96% parallel efficiency in scaling from 1 rack (16,384 cores) to 96 racks (1,572,864 cores) of Sequoia. Largest problem has 602 billion degrees of freedom.

To maximize accuracy and minimize runtime, the solver incorporates a number of advances, including aggressive multi-octree adaptive mesh refinement, mixed continuous-discontinuous discretization, arbitrarily-high-order accuracy, and hybrid spectral/geometric/algebraic multigrid. For saddle point problems, the solver employs a novel Schur complement preconditioner, which is provably spectrally equivalent to the exact Schur complement [124] yet can be applied in O(N) time. Despite the significant challenges to extreme scalability presented by all of these features, we demonstrate that with careful attention to design of algorithms, data structures, and implementation, scalability of this algorithmically-optimal implicit solver to the full 1.6 million cores of Sequoia (the LLNL IBM BG/Q system) can be achieved while maintaining 96% parallel efficiency for severely nonlinear, ill-conditioned, heterogeneous, and anisotropic PDEs [122]. We have applied variants of the solver to very high resolution simulations of global mantle convection [122, 123, 124] (Fig 14) and Antarctic ice sheet flow [62, 63]. This work received the 2015 Gordon Bell Prize [122], and an earlier version received the SC14 Best Poster Award (from 193 submissions). The paper describing the underlying numerical algorithms won Best Student Paper at the 2016 Copper Mountain Conference on Iterative Methods [123]. Finally, Johann Rudi, the student who was first author on this work, will receive the ACM/IEEE George Michael Memorial HPC Fellowship at SC16.