Smooth minimization of non-smooth functions

Authors:
Yu Nesterov
Affiliations:
Center for Operations Research and Econometrics (CORE), Catholic University of Louvain (UCL), 34 voie du Roman Pays, 1348, Louvain-la-Neuve, Belgium
Venue:
Mathematical Programming: Series A and B
Year:
2005

Citing 0
Cited 122

Convex optimization techniques for fitting sparse Gaussian graphical models

ICML '06 Proceedings of the 23rd international conference on Machine learning
Enhancing Lagrangian Dual Optimization for Linear Programs by Obviating Nondifferentiability

INFORMS Journal on Computing
Efficient solutions to relaxations of combinatorial problems with submodular penalties via the Lovász extension and non-smooth convex optimization

SODA '07 Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms
Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data

The Journal of Machine Learning Research
Rounding of convex sets and efficient gradient methods for linear programming problems

Optimization Methods & Software
Non-smoothness in classification problems

Optimization Methods & Software - THE JOINT EUROPT-OMS CONFERENCE ON OPTIMIZATION, 4-7 JULY, 2007, PRAGUE, CZECH REPUBLIC, PART I
An algebraic characterization of the optimum of regularized kernel methods

Machine Learning
Unconstrained Convex Minimization in Relative Scale

Mathematics of Operations Research
Penalty and Smoothing Methods for Convex Semi-Infinite Programming

Mathematics of Operations Research
An accelerated gradient method for trace norm minimization

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Some First-Order Algorithms for Total Variation Based Image Restoration

Journal of Mathematical Imaging and Vision
Mining brain region connectivity for alzheimer's disease study via sparse inverse covariance estimation

Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
A Nonlinear Probabilistic Curvature Motion Filter for Positron Emission Tomography Images

SSVM '09 Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision
Enhancement of Blurred and Noisy Images Based on an Original Variant of the Total Variation

SSVM '09 Proceedings of the Second International Conference on Scale Space and Variational Methods in Computer Vision
Compressive Algorithms--Adaptive Solutions of PDEs and Variational Problems

Proceedings of the 13th IMA International Conference on Mathematics of Surfaces XIII
First-order algorithm with O(ln(1/ε )) convergence for ε -equilibrium in two-person zero-sum games

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 1
Removing Multiplicative Noise by Douglas-Rachford Splitting Methods

Journal of Mathematical Imaging and Vision
Accuracy Certificates for Computational Problems with Convex Structure

Mathematics of Operations Research
An entropy regularization technique for minimizing a sum of Tchebycheff norms

Applied Numerical Mathematics
Deblurring Poissonian images by split Bregman techniques

Journal of Visual Communication and Image Representation
A Quasi-Newton Approach to Nonsmooth Convex Optimization Problems in Machine Learning

The Journal of Machine Learning Research
Tetrolet shrinkage with anisotropic total variation minimization for image approximation

Signal Processing
Faster approximation schemes for fractional multicommodity flow problems via dynamic graph algorithms

Proceedings of the forty-second ACM symposium on Theory of computing
Total variation projection with first order schemes

ICIP'09 Proceedings of the 16th IEEE international conference on Image processing
A fast and accurate first-order algorithm for compressed sensing

ICIP'09 Proceedings of the 16th IEEE international conference on Image processing
Mathematical Modeling of Textures: Application to Color Image Decomposition with a Projected Gradient Algorithm

Journal of Mathematical Imaging and Vision
Semi-supervised sparse metric learning using alternating linearization optimization

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Smoothing Techniques for Computing Nash Equilibria of Sequential Games

Mathematics of Operations Research
Convex optimization

Algorithms and theory of computation handbook
Convex relaxation for multilabel problems with product label spaces

ECCV'10 Proceedings of the 11th European conference on Computer vision: Part V
Solving structured sparsity regularization with proximal methods

ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part II
Improved dual decomposition based optimization for DSL dynamic spectrum management

IEEE Transactions on Signal Processing
Interior-Point Method for Nuclear Norm Approximation with Application to System Identification

SIAM Journal on Matrix Analysis and Applications
Exploiting redundancy for aerial image fusion using convex optimization

Proceedings of the 32nd DAGM conference on Pattern recognition
m-SNE: multiview stochastic neighbor embedding

ICONIP'10 Proceedings of the 17th international conference on Neural information processing: theory and algorithms - Volume Part I
An effective approach to pose invariant 3D face recognition

MMM'11 Proceedings of the 17th international conference on Advances in multimedia modeling - Volume Part I
Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization

The Journal of Machine Learning Research
Topology Selection in Graphical Models of Autoregressive Processes

The Journal of Machine Learning Research
Adaptive First-Order Methods for General Sparse Inverse Covariance Selection

SIAM Journal on Matrix Analysis and Applications
A Duality-Based Splitting Method for $\ell^1$-$TV$ Image Restoration with Automatic Regularization Parameter Choice

SIAM Journal on Scientific Computing
Learning the Morphological Diversity

SIAM Journal on Imaging Sciences
Randomized algorithm to determine the eigenvector of a stochastic matrix with application to the PageRank problem

Automation and Remote Control
Model-Based Multiple Rigid Object Detection and Registration in Unstructured Range Data

International Journal of Computer Vision
Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing

International Journal of Computer Vision
Efficient clustering earth mover's distance

ACCV'10 Proceedings of the 10th Asian conference on Computer vision - Volume Part II
A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging

Journal of Mathematical Imaging and Vision
Robust principal component analysis?

Journal of the ACM (JACM)
Distributed algorithms via gradient descent for fisher markets

Proceedings of the 12th ACM conference on Electronic commerce
The minimum-rank gram matrix completion via modified fixed point continuation method

Proceedings of the 36th international symposium on Symbolic and algebraic computation
A Family of Simple Non-Parametric Kernel Learning Algorithms

The Journal of Machine Learning Research
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

The Journal of Machine Learning Research
Smoothing a program soundly and robustly

CAV'11 Proceedings of the 23rd international conference on Computer aided verification
Evaluation of a first-order primal-dual algorithm for MRF energy minimization

EMMCVPR'11 Proceedings of the 8th international conference on Energy minimization methods in computer vision and pattern recognition
Linear discriminant dimensionality reduction

ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part I
An alternating direction method for dual MAP LP relaxation

ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part II
Domain adaptation in regression

ALT'11 Proceedings of the 22nd international conference on Algorithmic learning theory
Towards multi-semantic image annotation with graph regularized exclusive group lasso

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Convex and Network Flow Optimization for Structured Sparsity

The Journal of Machine Learning Research
A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science

SIAM Journal on Imaging Sciences
On the Complexity of the Hybrid Proximal Extragradient Method for the Iterates and the Ergodic Mean

SIAM Journal on Optimization
Solving Log-Determinant Optimization Problems by a Newton-CG Primal Proximal Point Algorithm

SIAM Journal on Optimization
Trace Norm Regularization: Reformulations, Algorithms, and Multi-Task Learning

SIAM Journal on Optimization
Applying Metric Regularity to Compute a Condition Measure of a Smoothing Algorithm for Matrix Games

SIAM Journal on Optimization
Alternating Direction Algorithms for $\ell_1$-Problems in Compressive Sensing

SIAM Journal on Scientific Computing
NESTA: A Fast and Accurate First-Order Method for Sparse Recovery

SIAM Journal on Imaging Sciences
Locally Parallel Texture Modeling

SIAM Journal on Imaging Sciences
Minimizing the Condition Number of a Gram Matrix

SIAM Journal on Optimization
Approximating Semidefinite Packing Programs

SIAM Journal on Optimization
A First-Order Smoothed Penalty Method for Compressed Sensing

SIAM Journal on Optimization
An Accelerated Proximal Gradient Algorithm for Frame-Based Image Restoration via the Balanced Approach

SIAM Journal on Imaging Sciences
Improved approximation schemes for linear programming relaxations of combinatorial optimization problems

IPCO'05 Proceedings of the 11th international conference on Integer Programming and Combinatorial Optimization
Near-optimal no-regret algorithms for zero-sum games

Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms
New approximation algorithms for minimum enclosing convex shapes

Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms
Feasible and accurate algorithms for covering semidefinite programs

SWAT'10 Proceedings of the 12th Scandinavian conference on Algorithm Theory
Efficient minimization of the non-local potts model

SSVM'11 Proceedings of the Third international conference on Scale Space and Variational Methods in Computer Vision
Optimization with Sparsity-Inducing Penalties

Foundations and Trends® in Machine Learning
Structured Learning and Prediction in Computer Vision

Foundations and Trends® in Computer Graphics and Vision
Distance metric learning with eigenvalue optimization

The Journal of Machine Learning Research
Optimal distributed online prediction using mini-batches

The Journal of Machine Learning Research
Exact covariance thresholding into connected components for large-scale graphical lasso

The Journal of Machine Learning Research
Sparse Occlusion Detection with Optical Flow

International Journal of Computer Vision
Alternating Direction Method for Covariance Selection Models

Journal of Scientific Computing
A Fast Fixed Point Algorithm for Total Variation Deblurring and Segmentation

Journal of Mathematical Imaging and Vision
A* orthogonal matching pursuit: Best-first search for compressed sensing signal recovery

Digital Signal Processing
Convex approximations to sparse PCA via Lagrangian duality

Operations Research Letters
Efficient Euclidean projections via Piecewise Root Finding and its application in gradient projection

Neurocomputing
Continuous Multiclass Labeling Approaches and Algorithms

SIAM Journal on Imaging Sciences
Structured sparsity via alternating direction methods

The Journal of Machine Learning Research
Lagrangian multipliers and split Bregman methods for minimization problems constrained on Sn-1

Journal of Visual Communication and Image Representation
Hybrid social media network

Proceedings of the 20th ACM international conference on Multimedia
Auto-grouped sparse representation for visual analysis

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part I
Order-Preserving sparse coding for sequence classification

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part II
Segmentation over detection by coupled global and local sparse representations

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part V
Fully sparse topic models

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Stochastic coordinate descent methods for regularized smooth and nonsmooth losses

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Multiscale Texture Extraction with Hierarchical (BV,Gp,L2) Decomposition

Journal of Mathematical Imaging and Vision
Accelerated Linearized Bregman Method

Journal of Scientific Computing
Bregmanized Domain Decomposition for Image Restoration

Journal of Scientific Computing
A cyclic projected gradient method

Computational Optimization and Applications
A double smoothing technique for solving unconstrained nondifferentiable convex optimization problems

Computational Optimization and Applications
Computing real solutions of polynomial systems via low-rank moment matrix completion

Proceedings of the 37th International Symposium on Symbolic and Algebraic Computation
A tutorial on dual decomposition and lagrangian relaxation for inference in natural language processing

Journal of Artificial Intelligence Research
Accelerated gradient methods and dual decomposition in distributed model predictive control

Automatica (Journal of IFAC)
Content placement via the exponential potential function method

IPCO'13 Proceedings of the 16th international conference on Integer Programming and Combinatorial Optimization
Combining Lagrangian decomposition and excessive gap smoothing technique for solving large-scale separable convex optimization problems

Computational Optimization and Applications
Sparse non Gaussian component analysis by semidefinite programming

Machine Learning
A new approach to computing maximum flows using electrical flows

Proceedings of the forty-fifth annual ACM symposium on Theory of computing
Smoothing multivariate performance measures

The Journal of Machine Learning Research
Exact Histogram Specification for Digital Images Using a Variational Approach

Journal of Mathematical Imaging and Vision
High-quality computational imaging through simple lenses

ACM Transactions on Graphics (TOG)
Audio classification with low-rank matrix representation features

ACM Transactions on Intelligent Systems and Technology (TIST) - Special Section on Intelligent Mobile Knowledge Discovery and Management Systems and Special Issue on Social Web Mining
ParNes: a rapidly convergent algorithm for accurate recovery of sparse and approximately sparse signals

Numerical Algorithms
Smooth approximation method for non-smooth empirical risk minimization based distance metric learning

Neurocomputing
BILGO: Bilateral greedy optimization for large scale semidefinite programming

Neurocomputing
Domain adaptation and sample bias correction theory and algorithm for regression

Theoretical Computer Science
Accelerated training of max-margin Markov networks with kernels

Theoretical Computer Science
A linearly convergent first-order algorithm for total variation minimisation in image processing

International Journal of Bioinformatics Research and Applications
An infeasible-point subgradient method using adaptive approximate projections

Computational Optimization and Applications
Homogeneous Penalizers and Constraints in Convex Image Restoration

Journal of Mathematical Imaging and Vision
Kernel regression with sparse metric learning

Journal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology
A class of distributed optimization methods with event-triggered communication

Computational Optimization and Applications
A joint convex penalty for inverse covariance matrix estimation

Computational Statistics & Data Analysis

Quantified Score

Hi-index	0.01

Visualization

Abstract

In this paper we propose a new approach for constructing efficient schemes for non-smooth convex optimization. It is based on a special smoothing technique, which can be applied to functions with explicit max-structure. Our approach can be considered as an alternative to black-box minimization. From the viewpoint of efficiency estimates, we manage to improve the traditional bounds on the number of iterations of the gradient schemes from ** keeping basically the complexity of each iteration unchanged.