Feature selection

Looking for usage documentation? Check out Interpretability.

`feature_selection.feature_selection`

Sequential feature selection wrapper around scikit-learn’s SFS. Picks a subset of features that work well for estimator by repeatedly fitting estimator on candidate subsets and keeping the one that maximizes a cross-validated score. Forwards the relevant SequentialFeatureSelector hyperparameters; always computes the baseline (all-features) and selected (subset-only) CV scores so they are available on the returned object regardless of verbose. Note that while we expose feature selection here, TabPFN is very robust to noisy / uninformative features in its native in-context-learning regime, so the accuracy gain from running this selector is often marginal — on real datasets the all-features baseline and the selected subset typically score within cross-validation noise of each other. The value of running selection on TabPFN is usually more in terms of interpretability. Note that other interpretability methods, such as SHAP, are also supported and are generally much faster because they can use the KV cache. Sequential feature selection is expensive: forward selection with n_features_to_select=k on d features uses on the order of cv * sum_{i=0..k-1} (d - i) model fits, plus 2 more for the baseline / selected CV scores. n_jobs parallelizes candidate-feature evaluation within each round; pass -1 for all cores. Note that v3’s KV cache does not help here — every candidate has a different X_train so the cache invalidates between fits.

feature_selection.feature_selection(
    estimator: BaseEstimator,
    X: np.ndarray,
    y: np.ndarray,
    n_features_to_select: int | float | str,
    feature_names: list[str] | None = None,
    *,
    cv: int | BaseCrossValidator | Iterable = 5,
    scoring: str | Callable | None = None,
    direction: str = "forward",
    n_jobs: int | None = None,
    tol: float | None = None,
    verbose: bool = True,
    **kwargs: Any,
) -> FeatureSelectionResult

Parameters

Parameter	Type	Default	Description
`estimator`	`BaseEstimator`	Required	The model to use for feature selection.
`X`	`np.ndarray`	Required	Input features, shape `(n_samples, n_features)`.
`y`	`np.ndarray`	Required	Target values, shape `(n_samples,)`.
`n_features_to_select`	`int \| float \| str`	Required	Number of features to keep. `int` for an absolute count, `float` for a fraction of the total, or `"auto"` to let `tol` decide (requires `tol`).
`feature_names`	`list[str] \| None`	`None`	Optional list of feature names. When provided, the returned `FeatureSelectionResult` carries the selected names under `selected_names`.
`cv`	`int \| BaseCrossValidator \| Iterable`	`5`	Cross-validation folds — `int` (k-fold), CV generator, or iterable of splits. Default `5`.
`scoring`	`str \| Callable \| None`	`None`	Metric to maximize. `str` (e.g. `"roc_auc"`, `"neg_log_loss"`, `"r2"`) or a callable. Default `None` uses sklearn’s per-estimator default (`accuracy` for classifiers, `r2` for regressors).
`direction`	`str`	`"forward"`	`"forward"` (start empty, add features) or `"backward"` (start full, remove features). Backward is much more expensive but sometimes preferred when features are redundant.
`n_jobs`	`int \| None`	`None`	Parallelism over candidate features in each round. `-1` uses all cores. Default `None` is single-threaded.
`tol`	`float \| None`	`None`	Stop condition for auto selection — only used when `n_features_to_select="auto"`. Forward selection stops when adding a feature improves the CV score by less than `tol`.
`verbose`	`bool`	`True`	When `True` (default), print the pre- and post-selection CV scores and the names of the selected features. The scores are computed and returned regardless.
`**kwargs`	`Any`	—	Forwarded to `SequentialFeatureSelector` for forward compatibility with future sklearn options.

Returns

Type	Description
`FeatureSelectionResult`	Selection results containing the fitted selector, support mask, selected feature indices and names, and baseline and selected-feature cross-validation scores.

`feature_selection.FeatureSelectionResult`

View source

Result of running feature_selection. Attributes

Attribute	Type	Description
`selector`	`SequentialFeatureSelector`	The underlying fitted `SequentialFeatureSelector`. Use it for `.transform(X)` to project to the selected columns, or for any sklearn-style downstream work.
`support_mask`	`np.ndarray`	Boolean array of shape `(n_features,)` — `True` for the columns SFS picked.
`selected_indices`	`list[int]`	Integer indices of the selected columns, in ascending order.
`selected_names`	`list[str] \| None`	Selected feature names, in the same order as `selected_indices`. `None` iff `feature_names` wasn’t passed.
`baseline_score_mean`	`float`	Mean cross-validated score of `estimator` on all features, using the same `cv` and `scoring` as the selection step.
`baseline_score_std`	`float`	Standard deviation across CV folds for the baseline score.
`selected_score_mean`	`float`	Mean cross-validated score of `estimator` on the selected subset of features.
`selected_score_std`	`float`	Standard deviation across CV folds for the selected-subset score.

Reference

tabpfn

tabpfn-client

tabpfn-extensions

REST API

`feature_selection.feature_selection`

`feature_selection.FeatureSelectionResult`

​feature_selection.feature_selection

​feature_selection.FeatureSelectionResult

`feature_selection.feature_selection`

`feature_selection.FeatureSelectionResult`