Fine-tuned regressor

Looking for usage documentation? Check out Fine tuning.

`FinetunedTabPFNRegressor`

A scikit-learn compatible wrapper for fine-tuning the TabPFNRegressor. This class encapsulates the fine-tuning loop, allowing you to fine-tune TabPFN on a specific dataset using the familiar .fit() and .predict() API.

FinetunedTabPFNRegressor(
    *,
    device: str = "cuda",
    epochs: int = 30,
    time_limit: int | None = None,
    learning_rate: float = 1e-05,
    weight_decay: float = 0.01,
    validation_split_ratio: float | None = 0.1,
    n_finetune_ctx_plus_query_samples: int = 50000,
    finetune_ctx_query_split_ratio: float = 0.2,
    n_inference_subsample_samples: int | None = None,
    random_state: int = 0,
    early_stopping: bool = True,
    early_stopping_patience: int = 8,
    validation_frequency: int = 1,
    min_delta: float = 0.0001,
    grad_clip_value: float | None = 1.0,
    use_lr_scheduler: bool = True,
    lr_warmup_only: bool = False,
    n_estimators_finetune: int = 2,
    n_estimators_validation: int = 2,
    n_estimators_final_inference: int = 8,
    use_activation_checkpointing: bool = True,
    shard_estimators_across_gpus: bool = False,
    save_checkpoint_interval: int | None = 10,
    use_fixed_preprocessing_seed: bool = True,
    experiment_logger: FinetuningLogger | None = None,
    extra_regressor_kwargs: dict[str, Any] | None = None,
    ce_loss_weight: float = 0.0,
    crps_loss_weight: float = 1.0,
    crls_loss_weight: float = 0.0,
    mse_loss_weight: float = 1.0,
    mse_loss_clip: float | None = None,
    mae_loss_weight: float = 0.0,
    mae_loss_clip: float | None = None,
    eval_metric: Literal["mse"] | None = None,
    model_version: ModelVersion | None = None,
)

Parameters

Parameter	Type	Default	Description
`device`	`str`	`"cuda"`	The device to run the model on. Defaults to “cuda”.
`epochs`	`int`	`30`	The total number of passes through the fine-tuning data. Defaults to `30`.
`time_limit`	`int \| None`	`None`	Time limit in seconds for fine-tuning. If `None`, no time limit is applied. Defaults to `None`.
`learning_rate`	`float`	`1e-05`	The learning rate for the AdamW optimizer. A small value is crucial for stable fine-tuning. Defaults to 1e-5.
`weight_decay`	`float`	`0.01`	The weight decay for the AdamW optimizer. Defaults to `0.01`.
`validation_split_ratio`	`float \| None`	`0.1`	Fraction of the original training data reserved as a validation set for early stopping and monitoring. Set to 0 or `None` to disable validation: all data is then used for fine-tuning, per-epoch evaluation is skipped, and early stopping is disabled. Ignored when explicit validation data is passed to `fit`. Defaults to `0.1`.
`n_finetune_ctx_plus_query_samples`	`int`	`50000`	The total number of samples per meta-dataset during fine-tuning (context plus query) before applying the `finetune_ctx_query_split_ratio`. Defaults to 50_000.
`finetune_ctx_query_split_ratio`	`float`	`0.2`	The proportion of each fine-tuning meta-dataset to use as query samples for calculating the loss. The remainder is used as context. Defaults to `0.2`.
`n_inference_subsample_samples`	`int \| None`	`None`	The total number of subsampled training samples per estimator during validation and final inference. If `None`, no subsampling is applied and the full training set is used as context. Defaults to `None`.
`random_state`	`int`	`0`	Seed for reproducibility of data splitting and model initialization. Defaults to `0`.
`early_stopping`	`bool`	`True`	Whether to use early stopping based on validation performance. Defaults to `True`.
`early_stopping_patience`	`int`	`8`	Number of validation checks to wait for improvement before early stopping. Defaults to `8`.
`validation_frequency`	`int`	`1`	Number of epochs between validation checks. A value of 1 (default) validates after every epoch. The initial evaluation of the unfine-tuned model still runs whenever validation data is available. Must be a positive integer.
`min_delta`	`float`	`0.0001`	Minimum change in metric to be considered as an improvement. Defaults to 1e-4.
`grad_clip_value`	`float \| None`	`1.0`	Maximum norm for gradient clipping. If `None`, gradient clipping is disabled. Gradient clipping helps stabilize training by preventing exploding gradients. Defaults to `1.0`.
`use_lr_scheduler`	`bool`	`True`	Whether to use a learning rate scheduler (linear warmup with optional cosine decay) during fine-tuning. Defaults to `True`.
`lr_warmup_only`	`bool`	`False`	If `True`, only performs linear warmup to the base learning rate and then keeps it constant. If `False`, applies cosine decay after warmup. Defaults to `False`.
`n_estimators_finetune`	`int`	`2`	If set, overrides `n_estimators` of the underlying estimator only during fine-tuning to control the number of estimators (ensemble size) used in the training loop. If `None`, the value from `kwargs` or the estimator default is used. Defaults to `2`.
`n_estimators_validation`	`int`	`2`	If set, overrides `n_estimators` only for validation-time evaluation during fine-tuning (early-stopping / monitoring). If `None`, the value from `kwargs` or the estimator default is used. Defaults to `2`.
`n_estimators_final_inference`	`int`	`8`	If set, overrides `n_estimators` only for the final fitted inference model that is used after fine-tuning. If `None`, the value from `kwargs` or the estimator default is used. Defaults to `8`.
`use_activation_checkpointing`	`bool`	`True`	Whether to use activation checkpointing to reduce memory usage. Defaults to `True`.
`shard_estimators_across_gpus`	`bool`	`False`	When `True` under DDP, shard the fine-tuning estimators across ranks to reduce per-rank activation memory. Defaults to `False`.
`save_checkpoint_interval`	`int \| None`	`10`	Number of epochs between checkpoint saves. This only has an effect if `output_dir` is provided during the `fit()` call. If `None`, no intermediate checkpoints are saved. The best model checkpoint is always saved regardless of this setting. Defaults to `10`.
`use_fixed_preprocessing_seed`	`bool`	`True`	Whether to use a fixed preprocessing seed. If `True`, the preprocessing will always use the same random seed throughout data batches. This is helpful in most cases because, e.g., the column order will stay the same across batches. If `False`, the preprocessing will use a different random seed for each batch.
`experiment_logger`	`FinetuningLogger \| None`	`None`	—
`extra_regressor_kwargs`	`dict[str, Any] \| None` `TabPFNRegressor` options	`None`	Additional keyword arguments to pass to the underlying `TabPFNRegressor`, such as `n_estimators`.
`ce_loss_weight`	`float`	`0.0`	Weight for the bar distribution negative log-likelihood term (cross-entropy-like). Defaults to `0.0`.
`crps_loss_weight`	`float`	`1.0`	Weight for the continuous ranked probability score (CRPS) term computed on ordered bar probabilities (squared CDF error). Defaults to `1.0`.
`crls_loss_weight`	`float`	`0.0`	Weight for the continuous ranked logarithmic score (CRLS) term computed on ordered bar probabilities (log score on cumulative probabilities). Defaults to `0.0`.
`mse_loss_weight`	`float`	`1.0`	Weight for an auxiliary MSE loss term added to the bar distribution loss. Set to 0.0 to disable. Defaults to `1.0`.
`mse_loss_clip`	`float \| None`	`None`	Optional upper bound for the auxiliary MSE loss term. If `None`, no clipping is applied. Defaults to `None`.
`mae_loss_weight`	`float`	`0.0`	Weight for an auxiliary MAE term computed on the mean decoded prediction. Defaults to `0.0`.
`mae_loss_clip`	`float \| None`	`None`	Optional upper bound for the auxiliary MAE loss term. If `None`, no clipping is applied. Defaults to `None`.
`eval_metric`	`Literal[“mse”] \| None`	`None`	The evaluation metric to use for early stopping and monitoring. For regression, the default is “mse”. Other metrics are not currently supported.
`model_version`	`ModelVersion \| None`	`None`	—

`FinetunedTabPFNRegressor.finetune_model_version`

View source

Model version to fine-tune; falls back to the package default version. When model_version is not set explicitly, this resolves to settings.tabpfn.model_version (the same default a plain TabPFNClassifier/TabPFNRegressor uses), so fine-tuning tracks the current default model version instead of a hardcoded one. Returns

Type	Description
`ModelVersion`	—

`FinetunedTabPFNRegressor.predict`

View source

Predict target values for X.

FinetunedTabPFNRegressor.predict(
    X: XType,
    **kwargs,
) -> RegressionResultType

Type aliases

XType = Any
RegressionResultType = np.ndarray | list[np.ndarray] | MainOutputDict | FullOutputDict

Parameters

Parameter	Type	Default	Description
`X`	`XType`	Required	The input samples of shape (n_samples, n_features).
`**kwargs`	— `predict` options	—	Additional keyword arguments to pass to the underlying inference regressor (e.g., output_type, quantiles).

Returns

Type	Description
`RegressionResultType`	The predicted target values with shape (n_samples,).

`FinetunedTabPFNRegressor.fit`

View source

Fine-tune the TabPFN model on the provided training data.

FinetunedTabPFNRegressor.fit(
    X: XType,
    y: YType,
    X_val: XType | None = None,
    y_val: YType | None = None,
    output_dir: Path | None = None,
) -> FinetunedTabPFNRegressor

Type aliases

XType = Any
YType = Any

Parameters

Parameter	Type	Default	Description
`X`	`XType`	Required	The training input samples of shape (n_samples, n_features).
`y`	`YType`	Required	The target values of shape (n_samples,).
`X_val`	`XType \| None`	`None`	Optional validation input samples.
`y_val`	`YType \| None`	`None`	Optional validation target values.
`output_dir`	`Path \| None`	`None`	Directory path for saving checkpoints. If `None`, no checkpointing is performed and progress will be lost if training is interrupted.

Returns

Type	Description
`FinetunedTabPFNRegressor`	The fitted instance itself.

Reference

tabpfn

tabpfn-client

tabpfn-extensions

REST API

Fine-tuned regressor

`FinetunedTabPFNRegressor`

`FinetunedTabPFNRegressor.finetune_model_version`

`FinetunedTabPFNRegressor.predict`

`FinetunedTabPFNRegressor.fit`

​FinetunedTabPFNRegressor

​FinetunedTabPFNRegressor.finetune_model_version

​FinetunedTabPFNRegressor.predict

​FinetunedTabPFNRegressor.fit

`FinetunedTabPFNRegressor`

`FinetunedTabPFNRegressor.finetune_model_version`

`FinetunedTabPFNRegressor.predict`

`FinetunedTabPFNRegressor.fit`