A Dynamical Systems Perspective on Optimization with Momentum (Talk)
My talk will be divided into two parts. In the first part, I will analyze Nesterov's accelerated gradient method from a dynamical systems point of view. More precisely, I will derive the accelerated gradient method by discretizing an ordinary differential equation with a semi-implicit Euler integration scheme. I will analyze both the ordinary differential equation and the discretization for obtaining insights into the phenomenon of acceleration. In particular, geometric properties of the dynamics, such as asymptotic stability, time-reversibility, and phase-space volume contraction are shown to be preserved through the discretization. In the second part, I will show that these geometric properties are enough for characterizing the convergence rate. The results therefore provide criteria that are easily verifiable for the accelerated convergence of any momentum-based optimization algorithm. The results also yield guidance for the design of new optimization algorithms. The talk will focus on unconstrained optimization problems with smooth and strongly-convex objective functions, even though the analysis potentially generalizes to non-convex or non-Euclidean settings, or when the decision variables are constrained to a smooth manifold.
Biography: Michael Muehlebach studied mechanical engineering at ETH Zurich and specialized in robotics, systems, and control during his Master's degree. He received the B.Sc. and the M.Sc. in 2010 and 2013, respectively, before joining the Institute for Dynamic Systems and Control for his Ph.D. He graduated under the supervision of Prof. R. D'Andrea in 2018 and joined the group of Prof. Michael I. Jordan at the Unversity of California, Berkeley as a postdoctoral researcher. He is interested in a wide variety of subjects, including dynamics, control, machine learning, and optimization. During his Ph.D. he developped approximations of the constrained linear quadratic regulator problem and applied them to model predictive control. He also designed control and estimation algorithms for a balancing robot and a flying machine. His postdoctoral research is concerned with analyzing momentum-based optimization algorithms from a dynamical systems point of view. He recieved the Outstanding D-MAVT Bachelor Award for his Bachelor's degree and the Willi-Studer prize for the best Master's degree. His Ph.D. thesis was awarded with the ETH Medal and the HILTI prize for innovative research. His postdoctoral research is funded by the Branco Weiss Fellowship.
Details
- 19 September 2019 • 14:00 - 15:00
- MPI-IS Stuttgart, Heisenbergstr. 3, seminar room 2P4
- Intelligent Control Systems
