Feature Selection and Variable Selection

Selection objectives

  • Predictive selection - Chooses a small or inexpensive input subset for a specified learner and prediction task. Its risk-cost trade-off need not identify a unique set of generating variables. Guyon and Elisseeff 2003.
  • Minimal-optimal versus all-relevant - A small subset sufficient for prediction and the full set of relevant variables are different targets; redundant variables can matter to the latter. Nilsson et al. 2007.
  • Conditional discovery - Model-X knockoffs test whether the target is independent of a variable given all remaining inputs. This target differs from keeping every useful substitute. Candes et al. 2018.
  • Unsupervised selection - Usefulness is defined by a chosen surrogate, such as reconstruction, neighborhoods or cluster structure. Preserving it does not establish utility for an unspecified target. Li et al. 2017.
  • Acquisition - Static selection uses one subset; an adaptive policy chooses the next measurement from values already observed. This differs from selecting an explanation after seeing all inputs. Covert et al. 2023.

Interactions and interchangeable variables

  • XOR example - For independent fair bits, Y = X1 XOR X2 is independent of either input alone, yet determined by their pair. Even nonlinear univariate dependence scores can miss joint information. Brown et al. 2012.
  • Duplicate example - If X1 = X2 = Y, either input predicts perfectly, and each is conditionally uninformative given the other. Changing between these subsets need not change predictions. Statnikov et al. 2013.

Comparative evidence

  • Tabular feature-selection benchmark - On 12 datasets with MLP/FT-Transformer predictors, Deep Lasso improves prediction with corrupted or second-order added features. Products of original variables can be useful; rejecting every added feature is not the objective. Cherepanova et al. 2023.
  • SelectArena - Across 71 datasets and five learners, tree importance and F-tests offer strong prediction/runtime trade-offs. Selection uses fixed subset sizes and a one-hour cap, with random filling/subsampling for interrupted searches; validity measures rejection of added noise. Schäfer et al. 2026.

Filter criteria and screening

Marginal screening

  • Variance threshold - Removes features with low training-set variance without using labels; scale affects the result. docs.
  • Univariate tests - Chi-squared for non-negative counts/frequencies; ANOVA F for class differences; linear-regression F for marginal linear association. Individual scores can miss interactions. docs, docs, docs.
  • Sure independence screening (SIS/ISIS) - Screens marginal correlations before subset fitting; ISIS repeats screening. Signal/design assumptions give high-probability retention of all signals, while extra variables may remain. Fan and Lv 2008.

Information, redundancy, and interactions

  • Relief - Updates feature weights from nearby same-class and different-class examples. Relief-family review covers interaction sensitivity and extensions. Kira and Rendell 1992, Urbanowicz et al. 2018.
  • MI estimation - Discrete and continuous features require appropriate treatment; mislabeling their type can give incorrect estimates. Nonlinear dependence estimation does not make marginal scoring interaction-aware. docs.
  • Joint mutual information (JMI) - Scores candidates jointly with already selected variables. Brown et al. derive JMI and related criteria as approximations to conditional-likelihood selection. Yang and Moody 1999, Brown et al. 2012.
  • FCBF (fast correlation-based filter) - Uses symmetrical uncertainty to screen relevance and remove redundant features. Yu and Liu 2003.
  • Consistency-based selection - Penalizes identical selected-feature patterns with conflicting labels. Dash and Liu compare search strategies using this criterion. Dash and Liu 2003.
  • Mutual information - A dependence quantity; Kraskov et al. provide an estimator. Selection criteria include mRMR/JMI, while forward search constructs a subset. These are separate choices. Kraskov et al. 2004, Brown et al. 2012.
  • mRMR - Balances feature-target mutual information against redundancy among selected features. Hanchuan Peng et al. 2005.
  • INTERACT - Removes features using changes in class inconsistency, targeting interactions that marginal ranking can miss. Zhao and Liu 2007.

Conditional independence and Markov boundaries

  • Koller-Sahami - Eliminates features using approximate Markov blankets and information-theoretic redundancy criteria. Koller and Sahami 1996.
  • Grow-Shrink - Grows a candidate blanket and then removes conditionally irrelevant variables. Margaritis and Thrun 1999.
  • IAMB - Incremental association-based blanket discovery, with forward inclusion and backward pruning. Tsamardinos et al. 2003.
  • Max-Min blanket discovery - Identifies parents/children and extends the search to a target’s Markov blanket. A predictive blanket alone does not identify causal direction. Tsamardinos et al. 2003.
  • Markov blankets - Seek variables that render the target conditionally independent of the remaining variables. Discovery depends on conditional-independence tests and distributional assumptions. Fu and Desmarais 2010.
  • Markov boundary - An inclusion-minimal blanket. Multiple boundaries can exist, yielding alternative sufficient subsets; disagreement need not identify a uniquely wrong selection. Statnikov et al. 2013.
  • FOCI - Greedily adds variables using a conditional-dependence statistic. Its sufficient-set recovery result needs detectable improvement at every insufficient subset; purely joint signals can defeat this forward-search condition. Azadkia and Chatterjee 2021.

Subset search and cardinality optimization

  • Floating search - Alternates inclusion and exclusion to revisit earlier choices. Adaptive floating search varies the search depth. Pudil et al. 1994, Somol et al. 1999.
  • Sequential selection - Greedily adds or removes features according to cross-validation scores. SequentialFeatureSelector supports forward and backward search. docs.
  • Population-based subset search - Genetic search and particle swarms explore subsets scored by a chosen criterion; that criterion defines the selection objective. Miao and Niu 2016, Kennedy and Eberhart 1995, Tu et al. 2007.
  • Wrappers - Evaluate candidate subsets using a particular learner’s predictive performance. Search and validation are part of the method. Kohavi and John 1997.
  • Recursive feature elimination (RFE) - Repeatedly fits a model and removes low-ranked inputs. RFE and RFECV implement elimination and cross-validated subset-size selection. Guyon et al. 2002; docs, docs.
  • Best-subset optimization - Mixed-integer formulations constrain feature count directly. Solver bounds can certify global optimality or quantify a remaining gap when stopped early. Bertsimas et al. 2016.
  • Fast L0 selection - Coordinate descent and local combinatorial search optimize L0-penalized objectives. Their local optimality conditions differ from a global MIO certificate. Hazimeh and Mazumder 2020.

Embedded sparsity and model-based selection

Penalized and Bayesian models

  • Spike-and-slab - Mixture priors represent exclusion and inclusion of regression coefficients. George and McCulloch develop computational approaches. Mitchell and Beauchamp 1988, George and McCulloch 1997.
  • Lasso - An L1 penalty yields sparse regression coefficients, coupling variable selection with fitting. Tibshirani 1996.
    • Adaptive lasso - Weights each coefficient’s L1 penalty using an initial estimate. Zou establishes selection consistency and oracle asymptotics under design, initialization and tuning conditions; this is not a finite-sample guarantee of correct support. Zou 2006.
    • Lasso support recovery - Sparse prediction does not establish correct variable recovery. Selection consistency requires additional assumptions, notably restrictions on predictor dependence. Zhao and Yu 2006.
  • Elastic net - Combines L1 and L2 penalties, encouraging groups of correlated predictors to enter together. Zou and Hastie 2005.
  • Group lasso - Selects predefined groups of coefficients, such as all encoded levels of a categorical variable. This differs from elastic net’s grouping of correlated predictors. Yuan and Lin 2005.
  • HSIC Lasso - Uses feature-wise kernels to select variables with strong nonlinear dependence on the response and limited redundancy with one another; a sparse alternative to linear Lasso screening. Yamada et al. 2014.

Trees, importance, and subset rules

  • Decision trees - Split-based importance sums impurity reductions attributed to each input; thresholding can select a subset. These scores can favor high-cardinality inputs. docs, docs.
  • Gradient boosting - Sequential function fitting can supply feature-importance rankings. Importance thresholding produces a subset. Friedman 2001, Mason et al. 1999; docs.
  • Random forests - Ensembles provide importance scores used for ranking and selection. Boulesteix et al. review practical pitfalls. A ranking still needs a subset-selection rule. Breiman 2001, Boulesteix et al. 2012.
  • Boruta - Targets all-relevant selection by comparing random-forest importances with randomized shadow features. Kursa et al. 2010, Kursa and Rudnicki 2010; code.
  • VSURF - Forest importance and stepwise selection produce subsets for interpretation and prediction. Genuer et al. 2010, Genuer et al. 2015.

Differentiable selectors

  • Concrete autoencoders - Learn which original inputs to retain through a relaxed discrete selector and reconstruction decoder. Balın et al. 2019; code.
  • Stochastic gates (STG) - Learn input gates jointly with a predictor using a continuous relaxation of an L0 feature-count penalty. Yamada et al. 2020; code.
  • LassoNet - Ties nonlinear feature use to sparse linear skip connections through a hierarchical constraint; produces a regularization path of global subsets. Lemhadri et al. 2021; code.
  • Deep Lasso - Applies a group penalty to loss gradients with respect to input features, extending linear sparsity to input sensitivity. Gradient norms also rank features. Cherepanova et al. 2023.

Unsupervised and graph-based selection

  • FSSEM - Searches feature subsets alongside expectation-maximization clustering; examines likelihood and scatter criteria and their dimensionality biases. Dy and Brodley 2004.
  • Laplacian Score - Ranks features by preservation of local graph structure; applicable without class labels. He et al. 2005.
  • Principal feature analysis - Uses PCA-derived structure to choose representative original variables. Its output is an input subset rather than principal-component coordinates. Lu et al. 2007.
  • Spectral feature selection (SPEC) - Evaluates features through a similarity graph’s spectrum, covering supervised and unsupervised criteria. Zhao and Liu 2007.
  • Multi-cluster feature selection (MCFS) - Combines a spectral embedding with sparse regression to select features preserving cluster structure. Cai et al. 2010.

Resampling, error control and post-selection inference

  • Ensemble selection - Aggregates feature preferences across perturbed datasets or selectors. Aggregation alone supplies no generic FDR guarantee. Saeys et al. 2008, Shen et al. 2019.
  • Stability selection - Thresholds selection frequencies across subsamples. Its original bound controls the expected number of false selections under exchangeability and other assumptions. Meinshausen and Buehlmann 2010.
    • Complementary-pairs stability selection - Pairs disjoint half-samples and bounds selection of variables with low selection probabilities, without the original exchangeability requirement. Shah and Samworth 2012.
  • Artificial controls, different targets - Boruta compares forest importances with shuffled shadows for all-relevant selection; model-X uses exchangeable controls for conditional discovery with FDR control under its assumptions. Kursa and Rudnicki 2010, Candes et al. 2018.
  • Knockoff+ - Selects against synthetic control variables with finite-sample false discovery rate control in the paper’s Gaussian linear-model setting. Barber and Candès 2015.
  • Post-selection inference - Conditions on lasso selection to form intervals for selected-model coefficients. Exact results use a fixed design and Gaussian errors with known variance. Lee et al. 2016.
  • Model-X knockoffs - Permits arbitrary response models; FDR control requires exchangeability and knockoffs independent of the response conditional on the inputs. Candes et al. 2018; docs.
    • Deep knockoffs - Learns approximate model-X knockoff generators by encouraging pairwise exchangeability. Generator diagnostics matter: approximate sampling alone does not establish exact false-discovery control. Romano et al. 2019.
    • Knockoff robustness - Flexibility in the response model relies on modeling the input distribution. Errors in estimated feature conditionals can inflate FDR; robustness bounds quantify this dependence. Barber et al. 2020.

Instancewise selection

  • Instancewise selection - L2X learns a feature subset for each example to explain a fixed predictor; this differs from one global subset used for all observations. Chen et al. 2018.
  • INVASE - A selector, predictor and baseline network learn instance-specific feature subsets through actor-critic training; the selected variables and subset size can change between observations. Yoon et al. 2019.

Cost-sensitive and adaptive acquisition

  • Dynamic feature selection - Learns a policy approximating greedy conditional-mutual-information acquisition from observed inputs. Recovery of the greedy policy assumes optimal training; it is not a globally optimal acquisition guarantee. Covert et al. 2023.

Evaluation and comparative evidence

Validation and selection stability

  • Selection leakage - Choosing features before resampling can bias estimated prediction error. Ambroise and McLachlan demonstrate the problem in microarray classification; pipeline guidance keeps selection inside training folds. Ambroise and McLachlan 2002; docs.
  • Stability measurement - Compares scores, ranks, or subsets under data perturbations. Nogueira, Sechidis, Brown provide a chance-corrected measure, intervals, and tests. Kalousis et al. 2006, Nogueira et al. 2018.
  • Search overfitting - A noisy validation criterion can overfit selector, subset-size and predictor choices. Nested resampling or an untouched test set must evaluate the complete search, including learned preprocessing. Cawley and Talbot 2010.
  • MUVR - Combines variable selection with repeated double cross-validation in multivariate modeling. Shi et al. 2018; code.

Correlated features and importance bias

  • Forest importance bias - Strobl et al. show preference for variables with more categories or different measurement scales in studied forest importance measures. Importance is not an unbiased relevance test. Strobl et al. 2007.
  • Conditional permutation importance - Permutes within a conditioning scheme to account for predictor dependence. Gregorutti et al. analyze correlation effects and recursive elimination. Strobl et al. 2008, Gregorutti et al. 2016.
  • Held-out permutation importance - Measures a fitted model’s score loss after permuting a feature. Correlated substitutes can mask importance; the result depends on the model and score. docs.

Task-specific selection and method recommendation

  • Selector recommendation - Learns from dataset metafeatures and previous selector evaluations. Wang et al. 2013, Parmezan et al. 2017, Shen et al. 2020.
  • Uplift modeling - Selects inputs for heterogeneous treatment-effect prediction rather than ordinary outcome prediction. Zhao et al. develop task-specific statistical and information-theoretic criteria. Zhao et al. 2022.

Software

  • Python (classical selection) - scikit-learn provides filters, model thresholds, RFE/RFECV, and sequential selection; scikit-feature collects research implementations. docs, code.
  • Python (learned selection and error control) - Concrete autoencoders, STG, LassoNet, knockpy. code, code, code, docs.
  • Supporting tools - feature-selector removes missingness/constant/collinearity-based columns; these are data-quality rules. BoostARoota implements XGBoost-based shadow comparisons. code, code.
  • R (search and filters) - FSinR and praznik; copent supplies supporting copula/transfer-entropy estimation. Aragón-Royón et al. 2020, Ma 2020; code, code, code.
  • R (model-based selection) - Boruta, VSURF, glmnet, spikeSlabGAM, knockoff. Scheipl 2011; code, code, code, code, code.
  • L0Learn - Approximate L0-regularized regression and classification through coordinate descent and local combinatorial search; C++ implementation with R/Python interfaces. Hazimeh et al. 2023.
  • Archived software - fscaret was archived by CRAN in August 2025; retained for historical reference. code.

Surveys and further reading

  • An Introduction to Variable and Feature Selection - Relevance, redundancy, ranking, subset search, and validation. Guyon and Elisseeff 2003.
  • Feature-selection survey - Reviews filter, wrapper and embedded approaches. Miao and Niu 2016.
  • Feature Selection: A Data Perspective - Conventional, structured, heterogeneous, and streaming data. Li et al. 2017.
  • Feature Selection and Feature Extraction in Pattern Analysis - Selection of existing variables and construction of new representations. Ghojogh et al. 2019.
  • Attribution rankings - Man-Chan compare permutation, LIME and SHAP rankings for random forests; reported stability differences do not establish predictive gains from selecting those features. Man and Chan 2020.

Connections