Advanced Iterative Methods as Elementary Iterations on Larger Spaces – Jongho Park

Abstract:

A central goal of scientific computing is to develop accurate and efficient solvers for scientific problems, and this goal is often pursued through sophisticated numerical methods. In modern machine learning, by contrast, the basic optimization procedure is often comparatively simple, typically gradient descent and its variants, while much of the complexity is shifted to larger models. This talk examines this contrast from the viewpoint of scientific computing.
We show that many advanced iterative methods, including domain decomposition and multigrid methods, can be interpreted as elementary iterations applied to equivalent problems posed on larger spaces. For example, a classical multigrid method can be viewed as a Gauss–Seidel iteration for a suitable expanded system associated with a multilevel frame. To make this interpretation rigorous, we introduce an auxiliary-space framework that recasts an iterative method for the original system as an equivalent, but more elementary, method for a lifted auxiliary system.
The framework applies to a broad range of advanced methods. We illustrate its utility through applications to various modern iterative methods. Finally, we discuss how this viewpoint can inform the design of numerical methods for problems arising in machine learning.