Support Vector Machines (SVMs)
Overview
Support Vector Machines (SVMs) are a family of supervised learning models that construct decision boundaries by maximizing the margin between data points of different classes. By focusing on a subset of critical training examples, SVMs aim to achieve strong generalization, particularly in high-dimensional feature spaces.
SVMs can be applied to both classification and regression tasks and are notable for their use of kernel methods, which allow linear decision boundaries in transformed feature spaces without explicitly computing those transformations.
Model Structure
- Decision function defined by a separating hyperplane
- Margin maximization objective
- Dependence on a subset of training points (support vectors)
- Kernel functions to enable non-linear decision boundaries
- Explicit regularization through margin and slack variables
Design Rationale
SVMs were designed to balance model complexity and generalization by maximizing the margin between classes, a principle grounded in statistical learning theory. This approach provides robustness to overfitting, especially in settings where the number of features is large relative to the number of samples.
Kernel methods extend this framework to non-linear problems while preserving convex optimization properties, allowing complex decision boundaries to be learned with well-defined theoretical guarantees.
Training Paradigm
- Optimization of a convex objective function
- Hinge loss (classification) or ε-insensitive loss (regression)
- Quadratic programming or specialized solvers
- Kernel selection and regularization as primary tuning mechanisms
- Training complexity grows with dataset size and number of support vectors
Notable Variants
- Linear SVM
- Kernel SVM (e.g., RBF, polynomial)
- Support Vector Regression (SVR)
- One-Class SVM