PFNs as Learned Nonparametric Regression Algorithms

Kernel smoothers, trees, forests, boosting, and amortized Bayesian prediction

Published

August 29, 2026

1 The common statistical object

Suppose a training sample is \(D_n=\{(X_i,Y_i)\}_{i=1}^n\) and the regression target is

\[ m(x)=\mathbb E[Y\mid X=x]. \]

Kernel regression, trees, random forests, gradient-boosted trees, and regression PFNs all turn \(D_n\) into a prediction rule. Their main difference is how they define similarity, how that similarity changes with the sample, and where the inductive bias comes from.

A useful reference form is a linear smoother,

\[ \widehat m(x)=\sum_{i=1}^n \alpha_i(x;X_{1:n})Y_i, \qquad \sum_i\alpha_i(x;X_{1:n})=1. \]

Kernel regression has this form exactly. Trees and forests also admit versions of it. Boosting and PFNs generally do not, but each can be studied through a local equivalent kernel or label influence,

\[ a_i(x;D_n)=\frac{\partial \widehat m(x;D_n)}{\partial Y_i}. \]

This derivative asks: if one training label moved slightly, how much would the query prediction move? For a fixed Nadaraya–Watson smoother, \(a_i=\alpha_i\). For boosting and PFNs, the influences can depend on all features and labels, can be signed, and need not sum to one.

ImportantThe central distinction

Traditional methods choose an algorithmic bias directly: Euclidean locality, recursive partitions, bagged partitions, or stagewise loss minimization. A PFN learns the prediction algorithm itself from a distribution over synthetic tasks. At inference time, the labeled table is its context and the network performs the learned algorithm in one forward pass.

2 Kernel regression: locality is specified in advance

The Nadaraya–Watson estimator is

\[ \widehat m_h(x) =\frac{\sum_i K\!\left((x-X_i)/h\right)Y_i} {\sum_j K\!\left((x-X_j)/h\right)}. \]

It is a convex average of nearby outcomes. The kernel \(K\), bandwidth \(h\), distance metric, and preprocessing determine the neighborhood before labels are inspected. The method is nonparametric because the effective number of local degrees of freedom can grow with \(n\); it is not because the algorithm has literally zero parameters.

The strengths are transparent locality and classical asymptotics. The weakness is geometry: in moderate dimension, generic distance becomes uninformative unless the regression surface has additional low-dimensional structure. Bandwidth selection then becomes the central regularization problem.

Self-attention superficially resembles kernel regression. One attention head computes

\[ \operatorname{Attn}(q,K,V) =\sum_i \frac{\exp(q^\top k_i/\sqrt d)}{\sum_j\exp(q^\top k_j/\sqrt d)}v_i. \]

This is a normalized exponential kernel in a learned representation. But an attention head is only one layer inside a PFN. Keys, queries, and values are transformed; heads interact through residual blocks and MLPs; labels enter the representations; and the final output is a predictive distribution. Calling a PFN “kernel regression” is therefore a useful analogy but an incorrect identity.

3 Trees: locality is a learned partition

A regression tree recursively partitions feature space. If \(L(x)\) denotes the terminal leaf containing \(x\), then

\[ \widehat m_{\text{tree}}(x) =\frac{1}{|L(x)|}\sum_{i:X_i\in L(x)}Y_i. \]

Thus a tree is also a local average, but its kernel is a hard, data-adaptive co-membership indicator:

\[ \alpha_i^{\text{tree}}(x) =\frac{1\{X_i\in L(x)\}}{|L(x)|}. \]

The tree learns where discontinuities and interactions are useful. Unlike an isotropic kernel, it can ignore irrelevant coordinates and divide the space asymmetrically. A single tree is interpretable but unstable: small changes to \(D_n\) can alter high-level splits and move many observations between neighborhoods.

4 Random forests: an adaptive kernel made from many partitions

A regression forest averages randomized trees. If \(L_b(x)\) is the leaf of tree \(b\), then a standard leaf-average forest can be written

\[ \widehat m_{\text{RF}}(x)=\sum_i \alpha_i^{\text{RF}}(x)Y_i, \]

\[ \alpha_i^{\text{RF}}(x) =\frac{1}{B}\sum_{b=1}^B \frac{1\{X_i\in L_b(x)\}}{|L_b(x)|}. \]

The forest kernel is the frequency with which training observation \(i\) shares a small leaf with \(x\), adjusted for leaf size. This is why Lin and Jeon describe random forests as adaptive nearest-neighbor methods. Bagging stabilizes the discontinuous neighborhoods of individual trees; feature subsampling decorrelates trees and lets different interaction structures compete.

The resulting notion of proximity is supervised and anisotropic. Two points can be close because many outcome-relevant splits fail to separate them, even if their raw Euclidean distance is not small.

5 Boosting: a stagewise basis expansion

Gradient boosting constructs an additive predictor

\[ F_M(x)=F_0(x)+\nu\sum_{m=1}^M f_m(x), \]

where each tree \(f_m\) approximately follows the negative gradient of the empirical loss evaluated at \(F_{m-1}\). Under squared error, each new tree fits current residuals. Boosting therefore refines difficult regions sequentially rather than averaging independently randomized partitions.

Because split selection and residuals depend on \(Y\), a boosted tree ensemble is not generally a fixed positive-weight average of the original outcomes. Its local label influences may be signed. The nearest kernel analogy is a learned, stagewise, signed equivalent kernel, but the additive-basis view is cleaner.

5.1 XGBoost and LightGBM

Both libraries implement gradient-boosted decision trees; neither changes the basic statistical object. Their distinctions are chiefly optimization, regularization, and systems design.

  • XGBoost uses first- and second-order loss derivatives, an explicit regularized tree objective, shrinkage, row/column subsampling, and exact or histogram-based split algorithms.
  • LightGBM is also histogram-based and is especially associated with leaf-wise tree growth, Gradient-based One-Side Sampling (GOSS), and Exclusive Feature Bundling (EFB). Leaf-wise growth can reduce loss rapidly but needs depth/leaf constraints to prevent overfitting on small samples.

For both, learning rate, tree complexity, number of rounds, subsampling, and regularization collectively replace the single bandwidth of kernel regression. Early stopping is often the effective complexity selector.

6 A one-dimensional visual comparison

The simulation is not a benchmark. It exposes the geometry: smooth distance weights, hard partitions, averaged randomized partitions, and sequential residual correction. The random-forest panel shows how averaging randomized partitions stabilizes a single tree; feature subsampling becomes more meaningful with multiple predictors.

Show simulation code 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

rng = np.random.default_rng(260829)
n = 90
x = np.sort(rng.uniform(-2.7, 2.7, size=n))
truth = lambda z: np.sin(1.7 * z) + 0.22 * z**2 - 0.15 * z
y = truth(x) + rng.normal(scale=0.32, size=n)
grid = np.linspace(-2.8, 2.8, 400)

# Nadaraya--Watson with a Gaussian kernel.
h = 0.34
weights = np.exp(-0.5 * ((grid[:, None] - x[None, :]) / h) ** 2)
nw = (weights @ y) / weights.sum(axis=1)

X = x[:, None]
X_grid = grid[:, None]
tree = DecisionTreeRegressor(
    max_depth=5, min_samples_split=12, min_samples_leaf=6, random_state=260829
).fit(X, y)
forest = RandomForestRegressor(
    n_estimators=300,
    max_depth=7,
    min_samples_leaf=4,
    max_features=1.0,
    bootstrap=True,
    random_state=260829,
    n_jobs=-1,
).fit(X, y)
boost = GradientBoostingRegressor(
    n_estimators=180,
    learning_rate=0.055,
    max_depth=2,
    min_samples_leaf=5,
    loss="squared_error",
    random_state=260829,
).fit(X, y)

fits = {
    "Gaussian kernel": nw,
    "Single tree": tree.predict(X_grid),
    "Random forest": forest.predict(X_grid),
    "Gradient-boosted shallow trees": boost.predict(X_grid),
}

plt.style.use("dark_background")
fig, axes = plt.subplots(2, 2, figsize=(11, 8), constrained_layout=True)
fig.patch.set_facecolor("#202124")
for ax, (name, fitted) in zip(axes.flat, fits.items()):
    ax.set_facecolor("#202124")
    ax.scatter(x, y, s=20, color="#8ab4f8", alpha=0.9)
    ax.plot(grid, truth(grid), color="#fbbc04", lw=2, ls="--", label="truth")
    ax.plot(grid, fitted, color="#81c995", lw=2.6, label="estimate")
    ax.set(title=name, xlabel="x", ylabel="y")
    ax.spines[["top", "right"]].set_visible(False)
plt.show()
Figure 1: Four ways to turn a sample into a regression function.

7 PFNs: amortized inference over datasets

A Prior-Data Fitted Network starts with a task prior \(\pi\) over data-generating processes. Pretraining repeatedly samples a task, a labeled context, and held-out queries, then minimizes expected predictive loss:

\[ \theta^*=\arg\min_\theta \mathbb E_{P\sim\pi} \mathbb E_{D,Q\sim P} \left[-\log q_\theta(Y_Q\mid X_Q,D)\right]. \]

If training succeeds and the model class and simulated prior are rich enough, \(q_{\theta^*}\) approximates the posterior predictive under \(\pi\). The expensive integration over latent functions and hyperparameters has been amortized into the network weights. On a new table, there is normally no gradient descent: conditioning on the context plays the role that fitting plays in a conventional estimator.

For regression, the PFN can output a discretized or quantile representation of the entire conditional distribution, not merely a conditional mean. A point estimate is a functional of that distribution. This makes the direct comparison with a mean-regression smoother incomplete: the PFN is closer to a learned conditional-distribution estimator.

7.1 In what sense is a PFN nonparametric?

There are two different parameter counts:

  1. The pretrained network has a large but fixed finite parameter vector \(\theta\). In that ordinary neural-network sense, it is parametric.
  2. The in-context prediction rule uses a variable-size empirical dataset as an input. Its effective local neighborhoods and complexity can change with the whole table. In that algorithmic sense, it can emulate nonparametric procedures.

It is best described as a fixed-capacity amortized inference machine that can implement a broad family of sample-adaptive, nonparametric-like estimators. It is not literally an infinite-dimensional estimator, and its context window imposes a hard capacity constraint.

8 The NanoTabICLv2 implementation makes the connection concrete

The accompanying NanoTabICL repository, locally inspected at commit 4a7f9c7, separates the learned algorithm from its synthetic task prior.

8.1 Architecture: a learned, table-conditional similarity rule

In model.py:

  • lines 35–40 standardize features using training rows, group features, embed covariates, and inject training labels;
  • lines 42–44 perform column-wise attention across rows, with test rows attending only to training rows;
  • lines 46–51 perform row-wise attention across feature groups;
  • lines 53–60 inject labels again, perform in-context attention, and map query representations to output logits or regression quantiles.

The final in-context attention layer has a kernel-smoother flavor because each query aggregates value vectors from labeled training rows. But the representation supplied to that layer has already been transformed using the entire table. The induced “distance” can therefore adapt to scale, feature interactions, missingness patterns, empirical distributions, and labels. It is closer to a learned adaptive kernel over datasets than to a fixed kernel over raw \(x\).

8.2 Prior: classical estimators become motifs in a mixture over functions

The connection is even more explicit in prior.py. The random function generator at lines 144–149 mixes linear, quadratic, Gaussian-process, tree, discretization, MLP, expectation-maximization-like, and product mechanisms. In particular:

  • lines 168–179 generate random tree ensembles;
  • lines 181–187 generate nearest-center discretizations;
  • lines 189–205 generate random Fourier-feature Gaussian processes;
  • lines 207–214 generate softmax distance mixtures.

Thus pretraining does not commit to “the world is a kernel” or “the world is a tree.” It presents the network with tasks exhibiting smoothness, partitions, nearest-neighbor structure, interactions, and compositions. The learned PFN is rewarded for recognizing which motif is plausible from the observed table and producing an appropriate predictive distribution.

This is the strongest relationship to traditional nonparametrics: the prior is a distribution over many classical inductive biases, and the transformer amortizes adaptation among them.

9 Side-by-side comparison

Method Basic prediction mechanism Where similarity comes from Role of labels in geometry Main regularization Predictive distribution
Kernel regression Positive local average Chosen kernel, metric, bandwidth Usually none after bandwidth selection Bandwidth Requires extra machinery
Single tree Average within one leaf Greedy supervised partition Determines splits Depth, pruning, leaf size Usually extra machinery
Random forest Average across leaf co-memberships Ensemble of randomized supervised partitions Determines each tree’s splits Leaf size, depth, feature/row sampling, number of trees Quantile forests or conformal variants
XGBoost / LightGBM Additive sequence of trees Stagewise residual/gradient correction Determines every new basis function Shrinkage, rounds, tree penalties, sampling, early stopping Usually separate objectives or wrappers
Regression PFN Learned map from an entire labeled table to query distribution Deep representations and table-conditioned attention Enters context representations directly Pretraining prior, architecture, context size, ensembling Native design target

10 Bias, variance, and out-of-distribution behavior

Classical tuning parameters expose the bias–variance tradeoff directly. A large kernel bandwidth, large tree leaves, shallow forests, or strong boosting shrinkage increase bias and reduce variance. PFNs bury the analogous tradeoffs in the task prior, architecture, checkpoint, preprocessing ensemble, and context construction. This is convenient at inference but makes diagnosis less transparent.

The PFN prior is both its strength and its failure mode. If a new table resembles the synthetic task distribution, amortized model selection can be remarkably effective. If the task lies outside that support—unusual causal structure, extreme dimensionality, dependence across rows, heavy extrapolation, or a novel loss—the network may confidently apply the wrong learned algorithm. A kernel’s bandwidth can be inspected; a forest’s leaves can be audited; a PFN’s implicit neighborhood requires influence, attention, perturbation, or surrogate analyses.

11 Fine-tuning revisited

This comparison clarifies what PFN fine-tuning means. Fine-tuning is not the analogue of fitting one random forest to one dataset. It changes the amortized algorithm or prior represented by \(\theta\). Training episodes still split data into labeled contexts and hidden-label queries:

\[ \min_\theta \;\mathbb E_{(C,Q)} \left[-\log q_\theta(y_Q\mid x_Q,C)\right]. \]

Fine-tuning on domain-specific synthetic tasks says which kernels, partitions, interactions, noise models, and extrapolation patterns should be expected. Fine-tuning on one observed table must manufacture many leakage-safe pseudo-tasks and is easy to overfit. LoRA or adapters can constrain that update, but the statistical object remains meta-learning across context/query episodes.

12 Practical stopping rule

  • Use kernel or local-polynomial regression when dimension is low, smoothness is scientifically plausible, and interpretable locality or classical inference matters.
  • Use a single tree when a compact rule list and visible interactions matter more than predictive stability.
  • Use a random forest for a stable, low-tuning nonlinear baseline and when adaptive neighborhoods are appealing.
  • Use XGBoost or LightGBM when supervised tabular prediction, scalable training, and careful validation justify tuning a high-performance tree ensemble.
  • Use a PFN when the table fits the checkpoint’s supported regime, training time must be near zero, probabilistic predictions are useful, and broad amortized adaptation is worth less transparent inductive bias.

The methods are not arranged on a ladder from primitive to modern. They trade explicit structure for learned structure. PFNs push furthest toward learning the estimator itself; classical nonparametrics make more of that estimator visible and controllable.

13 References