Tree-Based Models
Overview
Tree-based models are a family of machine learning methods that represent decision logic as a hierarchy of conditional rules organized in a tree structure. These models recursively partition the feature space into regions with similar target values, enabling them to capture non-linear relationships and feature interactions without requiring explicit feature engineering.
Tree-based models are widely used in classical machine learning, particularly for structured and tabular data, and serve as the foundation for many of the most effective ensemble methods.
Model Structure
- Hierarchical tree structure composed of internal decision nodes and terminal leaf nodes
- Axis-aligned splits based on feature thresholds
- Leaves store predictions or class distributions
- Can model non-linear relationships and interactions
- Ensembles combine multiple trees to improve generalization
Design Rationale
Decision trees were designed to provide an interpretable, rule-based approach to prediction that mirrors human decision-making processes. However, individual trees are prone to overfitting and instability.
Ensemble extensions such as bagging and boosting were introduced to mitigate these issues by aggregating multiple trees, reducing variance, improving robustness, and increasing predictive performance while retaining the core tree-based representation.
Training Paradigm
- Greedy, recursive partitioning of the feature space
- Split selection based on impurity or loss reduction criteria
- Trees grown using top-down induction
- Ensemble methods train multiple trees using resampling or sequential error correction
- Regularization achieved through depth limits, pruning, or ensemble constraints
Notable Variants
- Decision Trees
- Random Forests
- Gradient Boosting Machines
- XGBoost
- LightGBM
- CatBoost