How bad are the BFGS and DFP methods when the objective function is quadratic?
Mathematical Programming: Series A and B
Sizing and least-change secant methods
SIAM Journal on Numerical Analysis
Analysis of a self-scaling quasi-Newton method
Mathematical Programming: Series A and B
CUTEr and SifDec: A constrained and unconstrained testing environment, revisited
ACM Transactions on Mathematical Software (TOMS)
Information geometry of U-Boost and Bregman divergence
Neural Computation
Clustering with Bregman Divergences
The Journal of Machine Learning Research
Matrix Nearness Problems with Bregman Divergences
SIAM Journal on Matrix Analysis and Applications
Sparse quasi-Newton updates with positive definite matrix completion
Mathematical Programming: Series A and B
A Bregman extension of quasi-Newton updates I: an information geometrical framework
Optimization Methods & Software
| Hi-index | 7.29 |
In Part I of this series of articles, we introduced the information geometric framework of quasi-Newton methods and gave an extension of Hessian update formulas based on the Bregman divergence. The purpose of this article is to investigate the convergence and robustness properties of extended Hessian update formulas. Fletcher has studied a variational problem which derives the approximate Hessian update formula of the quasi-Newton methods. We point out that the variational problem is identical to optimization of the Kullback-Leibler divergence, which is a discrepancy measure between two probability distributions. Then, we introduce the Bregman divergence as an extension of the Kullback-Leibler divergence, and derive extended quasi-Newton update formulas based on the variational problem with the Bregman divergence. The proposed update formulas belong to a class of self-scaling quasi-Newton methods. We study the convergence property of the proposed quasi-Newton method. Moreover, we apply tools in the robust statistics to analyze the robustness properties of Hessian update formulas against numerical rounding errors or a shift of tuning parameters included in line search methods of the step length. As the main contribution of this paper, we present that the influence of perturbations in the line search is bounded only for the standard BFGS formula for the Hessian approximation. Numerical studies are conducted to verify the usefulness of the tools borrowed from the robust statistics.