Machine learning and data mining via mathematical programming-based support vector machines

• Authors: Glenn Martin Fung;Olvi L. Mangasarian
• Venue: Machine learning and data mining via mathematical programming-based support vector machines
• Year: 2003

Several issues that arise in machine learning and data mining are addressed using mathematical programming based support vector machines (SVMs). We address the following important problems. Instead of a standard SVM that classifies points by assigning them to one of two disjoint halfspaces, points are classified by assigning them to the closest of two parallel planes (in input or feature space) that are pushed apart as far as possible. This formulation leads to an extremely fast and simple algorithm for generating a linear or nonlinear classifier that merely requires the solution of a single system of nonsingular linear equations. Multiclass and incremental extensions of this proximal formulation are also presented. Prior knowledge, in the form of multiple polyhedral sets each belonging to one of two categories, is introduced into a reformulation of a linear SVM classifier. The resulting formulation is solved efficiently by a linear program and results in enhanced testing set correctness. A finite concave minimization algorithm is proposed for constructing classifiers that use a minimal number of data points both in generating and characterizing classifiers. The algorithm is theoretically justified by linear programming perturbation theory and a leave-one-out error bound, as well as by effective computational results on several real world datasets. Another very fast Newton based stand-alone algorithm to solve this problem is also presented. The problem of incorporating unlabeled data into a support vector machine is formulated as a concave minimization problem on a polyhedral set for which a stationary point is quickly obtained by solving a few (5 to 7) linear programs. We also propose an implicit Lagrangian formulation of a support vector machine classifier that results in a highly effective iterative scheme and that is solved here by a finite Newton method. The proposed method, which is extremely fast and terminates in 6 or 7 iterations, can handle classification problems in very high dimensional spaces, e.g. over 28,000, in a few seconds on a 400 MHz Pentium II machine. The method can also handle problems with large datasets and requires no specialized software other than a commonly available solver for a system of linear equations. Finite termination of the proposed method is established. To sum up, we present several mathematical programming based algorithms that address various important SVM related issues such as: speed, scalability, data dependence and sparse representation, use of unlabeled data and knowledge incorporation.