Preprocessing pipelines

`PreprocessingPipeline`

Modality-aware preprocessing pipeline that handles column slicing. This pipeline applies a sequence of preprocessing steps to data, where each step can be registered to target specific feature modalities. The pipeline handles slicing columns based on registered modalities, passing only relevant columns to each step, reassembling data after each step, and tracking feature schema updates. For backwards compatibility, steps can be registered as (step, modalities) tuples where the step receives only columns matching the specified modalities, or as bare steps that receive all columns. In the latter case, the column modalities need to be tracked by the step itself. Going forward, only the former format will be supported. Initialize the pipeline with preprocessing steps.

PreprocessingPipeline(
    steps: list[PreprocessingStep | StepWithModalities],
)

Parameters

Parameter	Type	Default	Description
`steps`	`list[PreprocessingStep \| StepWithModalities]`	Required	List of preprocessing steps. Each can be a `PreprocessingStep` (receives all columns) or a tuple of (`PreprocessingStep`, set[FeatureModality]) where the step receives only columns matching the specified modalities.

`PreprocessingPipeline.fit_transform`

View source

Fit and transform the data using the pipeline.

PreprocessingPipeline.fit_transform(
    X: np.ndarray | torch.Tensor,
    feature_schema: FeatureSchema,
) -> PreprocessingPipelineResult

Parameters

Parameter	Type	Default	Description
`X`	`np.ndarray \| torch.Tensor`	Required	2d array of shape (n_samples, n_features).
`feature_schema`	`FeatureSchema`	Required	feature schema.

Returns

Type	Description
`PreprocessingPipelineResult`	`PreprocessingPipelineResult` with transformed data and updated schema.

`PreprocessingPipeline.transform`

View source

Transform the data using the fitted pipeline.

PreprocessingPipeline.transform(
    X: np.ndarray | torch.Tensor,
) -> PreprocessingPipelineResult

Parameters

Parameter	Type	Default	Description
`X`	`np.ndarray \| torch.Tensor`	Required	2d array of shape (n_samples, n_features).

Returns

Type	Description
`PreprocessingPipelineResult`	`PreprocessingPipelineResult` with transformed data and feature schema.

`PreprocessingPipeline.num_added_features`

View source

Return the number of added features. Threads an evolving feature schema through the steps so that each step sees the feature count that includes columns added by prior steps.

PreprocessingPipeline.num_added_features(
    n_samples: int,
    feature_schema: FeatureSchema,
) -> int

Parameters

Parameter	Type	Default	Description
`n_samples`	`int`	Required	—
`feature_schema`	`FeatureSchema`	Required	—

Returns

Type	Description
`int`	—

`PreprocessingPipeline.has_data_dependent_feature_expansion`

View source

Return True if any step has data-dependent feature expansion.

PreprocessingPipeline.has_data_dependent_feature_expansion() -> bool

Returns

Type	Description
`bool`	—

`clean_data`

View source

Clean the data by converting dtypes and ordinally encoding categorical columns.

clean_data(
    X: np.ndarray,
    feature_schema: FeatureSchema,
    *,
    passthrough_inf: bool = False,
) -> tuple[np.ndarray, OrderPreservingColumnTransformer, FeatureSchema]

Parameters

Parameter	Type	Default	Description
`X`	`np.ndarray`	Required	The data to clean.
`feature_schema`	`FeatureSchema`	Required	The feature schema corresponding to the data.
`passthrough_inf`	`bool`	`False`	If `True`, +/-inf values are carried through the ordinal encoding stage unchanged instead of crashing it (see `process_text_na_dataframe`).

Returns

Type	Description
`tuple[np.ndarray, OrderPreservingColumnTransformer, FeatureSchema]`	A tuple containing the cleaned data, the ordinal encoder, and the inferred feature modalities.

`fit_preprocessing`

View source

Fit preprocessing pipelines in parallel.

fit_preprocessing(
    configs: Sequence[EnsembleConfig],
    X_train: np.ndarray,
    y_train: np.ndarray,
    *,
    feature_schema: FeatureSchema,
    n_preprocessing_jobs: int,
    parallel_mode: Literal["block", "as-ready", "in-order"],
    pipelines: Sequence[PreprocessingPipeline],
    subsample_feature_indices: list[np.ndarray | None] | None = None,
    subsample_row_indices: list[np.ndarray] | None = None,
) -> Iterator[tuple[int, EnsembleConfig, PreprocessingPipeline, np.ndarray, np.ndarray, FeatureSchema]]

Parameters

Parameter	Type	Default	Description
`configs`	`Sequence[EnsembleConfig]`	Required	List of ensemble configurations.
`X_train`	`np.ndarray`	Required	Training data.
`y_train`	`np.ndarray`	Required	Training target.
`feature_schema`	`FeatureSchema`	Required	feature schema.
`n_preprocessing_jobs`	`int`	Required	Number of worker processes to use. If `1`, then the preprocessing is performed in the current process. This avoids multiprocessing overheads, but may not be able to full saturate the CPU. Note that the preprocessing itself will parallelise over multiple cores, so one job is often enough. If `>1`, then different estimators are dispatched to different proceses, which allows more parallelism but incurs some overhead. If `-1`, then creates as many workers as CPU cores. As each worker itself uses multiple cores, this is likely too many. It is best to select this value by benchmarking.
`parallel_mode`	`Literal[“block”, “as-ready”, “in-order”]`	Required	Parallel mode to use. * `"block"`: Blocks until all workers are done. Returns in order. * `"as-ready"`: Returns results as they are ready. Any order. * `"in-order"`: Returns results in order, blocking only in the order that needs to be returned in.
`pipelines`	`Sequence[PreprocessingPipeline]`	Required	Preprocessing pipelines, one per configuration.
`subsample_feature_indices`	`list[np.ndarray \| None] \| None`	`None`	Indices of features to subsample. If not provided, no features are subsampled.
`subsample_row_indices`	`list[np.ndarray] \| None`	`None`	Indices of rows to subsample per estimator. If not provided, no row subsampling is done.

Returns

Type	Description
`Iterator[tuple[int, EnsembleConfig, PreprocessingPipeline, np.ndarray, np.ndarray, FeatureSchema]]`	Iterator of tuples containing the config index, ensemble configuration, the fitted preprocessing pipeline, the transformed training data, the transformed target, and the feature schema.

`generate_classification_ensemble_configs`

View source

Generate ensemble configurations for classification.

generate_classification_ensemble_configs(
    *,
    num_estimators: int,
    add_fingerprint_feature: bool,
    polynomial_features: Literal["no", "all"] | int,
    feature_shift_decoder: Literal["shuffle", "rotate"] | None,
    preprocessor_configs: Sequence[PreprocessorConfig],
    class_shift_method: Literal["rotate", "shuffle"] | None,
    n_classes: int,
    random_state: int | np.random.Generator | None,
    num_models: int,
    outlier_removal_std: float | None,
    passthrough_inf: bool = False,
) -> list[ClassifierEnsembleConfig]

Parameters

Parameter	Type	Default	Description
`num_estimators`	`int`	Required	Number of ensemble configurations to generate.
`add_fingerprint_feature`	`bool`	Required	Whether to add fingerprint features.
`polynomial_features`	`Literal[“no”, “all”] \| int`	Required	Maximum number of polynomial features to add, if any.
`feature_shift_decoder`	`Literal[“shuffle”, “rotate”] \| None`	Required	How shift features
`preprocessor_configs`	`Sequence[PreprocessorConfig]`	Required	Preprocessor configurations to use on the data.
`class_shift_method`	`Literal[“rotate”, “shuffle”] \| None`	Required	How to shift classes for classpermutation.
`n_classes`	`int`	Required	Number of classes.
`random_state`	`int \| np.random.Generator \| None`	Required	Random number generator.
`num_models`	`int`	Required	Number of models to use.
`outlier_removal_std`	`float \| None`	Required	The standard deviation to remove outliers.
`passthrough_inf`	`bool`	`False`	Whether to pass infinite values through to the model.

Returns

Type	Description
`list[ClassifierEnsembleConfig]`	List of ensemble configurations.

`generate_regression_ensemble_configs`

View source

Generate ensemble configurations for regression.

generate_regression_ensemble_configs(
    *,
    num_estimators: int,
    add_fingerprint_feature: bool,
    polynomial_features: Literal["no", "all"] | int,
    feature_shift_decoder: Literal["shuffle", "rotate"] | None,
    preprocessor_configs: Sequence[PreprocessorConfig],
    target_transforms: Sequence[TransformerMixin | Pipeline | None],
    random_state: int | np.random.Generator | None,
    num_models: int,
    outlier_removal_std: float | None,
    passthrough_inf: bool = False,
) -> list[RegressorEnsembleConfig]

Parameters

Parameter	Type	Default	Description
`num_estimators`	`int`	Required	Number of ensemble configurations to generate.
`add_fingerprint_feature`	`bool`	Required	Whether to add fingerprint features.
`polynomial_features`	`Literal[“no”, “all”] \| int`	Required	Maximum number of polynomial features to add, if any.
`feature_shift_decoder`	`Literal[“shuffle”, “rotate”] \| None`	Required	How shift features
`preprocessor_configs`	`Sequence[PreprocessorConfig]`	Required	Preprocessor configurations to use on the data.
`target_transforms`	`Sequence[TransformerMixin \| Pipeline \| None]`	Required	Target transformations to apply.
`random_state`	`int \| np.random.Generator \| None`	Required	Random number generator.
`num_models`	`int`	Required	Number of models to use.
`outlier_removal_std`	`float \| None`	Required	The standard deviation to remove outliers.
`passthrough_inf`	`bool`	`False`	Whether to pass infinite values through to the model.

Returns

Type	Description
`list[RegressorEnsembleConfig]`	List of ensemble configurations.

`PreprocessingPipelineResult`

View source

Result from the preprocessing pipeline.

PreprocessingPipelineResult(
    X: np.ndarray | torch.Tensor,
    feature_schema: FeatureSchema,
)

Fields

Name	Type	Default	Description
`X`	`np.ndarray \| torch.Tensor`	Required	The transformed array.
`feature_schema`	`FeatureSchema`	Required	Updated feature schema (may have new columns added).

`PreprocessingStepResult`

View source

Result of a feature preprocessing step.

PreprocessingStepResult(
    X: np.ndarray | torch.Tensor,
    feature_schema: FeatureSchema,
    X_added: np.ndarray | torch.Tensor | None = None,
    modality_added: FeatureModality | None = None,
)

Fields

Name	Type	Default	Description
`X`	`np.ndarray \| torch.Tensor`	Required	Transformed array. For steps registered with specific modalities, this is only the transformed columns (not the full array). The shape should match the input shape unless columns are removed.
`feature_schema`	`FeatureSchema`	Required	Feature schema for the columns this step processed. Contains 0-based indices relative to the step’s input. Should NOT include added_columns - the pipeline handles that.
`X_added`	`np.ndarray \| torch.Tensor \| None`	`None`	Optional new features to append (e.g., fingerprint features). These are handled by the pipeline, which concatenates them and updates the schema accordingly. Steps should NOT concatenate these internally.
`modality_added`	`FeatureModality \| None`	`None`	Modality for the added features. Required if `X_added` is provided.

Reference

tabpfn

tabpfn-client

tabpfn-extensions

REST API

Preprocessing pipelines