The TabICLv2 regression head does not directly emit a single point estimate. In a dedicated regression checkpoint trained with pinball loss, it predicts 999 conditional quantiles at probability levels \(\mathcal{A}=\{0.001,0.002,\ldots,0.999\}\), forming a dense grid that approximates the conditional distribution of the target.

Like classification, regression keeps the same overall backbone structure: repeated feature grouping, target-aware embedding, and the column/row/ICL transformer blocks \(\text{TF}_\text{col}\), \(\text{TF}_\text{row}\), and \(\text{TF}_\text{icl}\); observed targets still enter twice. Classification embeds those targets as class IDs and emits class logits. Regression swaps the task-specific interfaces and loss: scalar linear target embedders replace class lookup tables, and the output MLP emits 999 raw quantiles per test row instead of logits. Its checkpoint also uses bias-free LayerNorm, whereas the classification checkpoint uses LayerNorm with bias.

This post explains what those quantiles represent, how pinball loss trains them, how inference turns the raw grid into a monotone predictive distribution, and how the same outputs support both fast point estimates and probabilistic predictions. It also maps the regression path in NanoTabICL through max_classes=0 and out_dim=999, and notes what the compact model leaves outside the forward pass. The following figure shows the architecture of TabICLv2.

A short quiz at the end lets you check your understanding.

Subscribe now

Quantile predictions for regression

Tabular foundation models adopt different strategies for regression. TabPFNv2 and TabPFN-2.5 model the predictive distribution by discretizing the target space into bins and applying cross-entropy loss. TabICLv2 instead uses a dedicated regression checkpoint that directly predicts quantiles. It retains the same overall backbone structure while changing the task-specific interfaces, loss, and LayerNorm configuration.

The subsections below build from quantile definitions and pinball loss to training and inference.

What quantiles are

To see what this regression head is learning, first recall what a quantile represents: the smallest value at which the cumulative probability reaches at least \(\alpha\). Start with the unconditional case: one target \(Y\), no features yet. Let \(Y\) be a real-valued target random variable with cumulative distribution function (CDF)

For a probability level \(\alpha\in(0,1)\), define its quantile function as the generalized inverse

In words, \(Q(\alpha)\) is the smallest target value whose cumulative probability reaches at least \(\alpha\). This definition remains valid when the CDF has jumps. When the CDF is continuous and strictly increasing, it reduces to the familiar inverse relation \(F_Y(Q(\alpha))=\alpha\). For example, \(Q(0.5)\) is the median and \(Q(0.9)\) is the 90th percentile.

Now make the distribution depend on the row. In supervised regression, the target distribution depends on the input row. To avoid overloading the table symbol \(X\), write \(Z\) for the random feature vector of a single row and \(x\) for a particular observed row. The conditional quantile function (CQF) is

Here \(F_{Y\mid Z=x}(q)=P(Y\leq q\mid Z=x)\) is the conditional CDF at row \(x\). The probability level \(\alpha\) selects a location on that row’s conditional distribution. The two notations denote the same quantity, written two ways: \(Q_x(\alpha)\) treats the quantile as a function of \(\alpha\) (the inverse-CDF view, as with \(Q(\alpha)\) above), while \(q_\alpha(x)\) treats \(\alpha\) as a label on the \(\alpha\)-quantile as a function of row \(x\) (the prediction view, as with \(\hat{q}_\alpha(x)\) below). This post uses whichever notation reads more naturally in context.

Instead of asking the model for one conditional summary, TabICLv2 asks it for many summaries spread across the distribution. Specifically, it predicts 999 such quantiles at probability levels

These 999 outputs form a dense grid of estimated points on \(Q_x(\alpha)\). They are not, by themselves, a full predictive distribution; the inference-time distribution wrapper constructs one by making the grid monotone, interpolating between its points, and extrapolating beyond its endpoints.

Pinball loss

Each output is trained with pinball loss, also called quantile loss or check loss. If the model predicts \(\hat{q}_\alpha(x)\) for level \(\alpha\) and the observed target is \(y\), define the residual

The pinball loss is

Equivalently,

Here \(\hat{q}\) is shorthand for \(\hat{q}_\alpha(x)\), and \(\mathbf{1}\{y<\hat{q}\}\) is an indicator. Underprediction means \(y>\hat{q}\), so the loss grows at rate \(\alpha\) as the miss increases. Overprediction means \(y<\hat{q}\), so it grows at rate \(1-\alpha\). The result is a tilted absolute-value penalty.

This asymmetry makes the loss target a specific quantile. For \(\alpha=0.5\),

so minimizing the expected loss recovers a median. For \(\alpha=0.9\), an equally sized underprediction costs nine times as much as an overprediction. The optimum is therefore pushed upward until it represents the conditional 90th percentile.

Training and constructing a distribution

With pinball loss defined, the regression head trains one raw scalar \(\hat{q}_\alpha(x)\) per level in \(\mathcal{A}\). Each level has a separate output coordinate, but all levels share the backbone and output MLP hidden representation. For each example \((x,y)\), training averages pinball loss equally across all 999 levels:

Predicting each level separately raises one practical issue: the architecture imposes no explicit monotonicity or cross-quantile constraint, and pretraining adds no auxiliary penalty for crossing quantiles. The raw outputs can therefore violate the ordering required of a valid quantile function:

TabICLv2 resolves this issue while turning the grid points into a full predictive distribution at inference. First, it enforces monotonicity (default: sort; alternative: isotonic regression). Second, it piecewise-linearly interpolates between the corrected points. Third, because the grid stops at \(0.001\) and \(0.999\), it extrapolates parametric tails—exponential by default, GPD optional. The reconstructed distribution then exposes a PDF, CDF, inverse CDF (ICDF), and analytical moments across \(\mathbb{R}\).

Prediction intervals and point estimates

Two standard uses of the corrected quantile function are interval construction and point estimation.

Once the raw outputs have been turned into a monotone quantile function, prediction intervals are a direct use of it. Continuing to write \(\hat{q}_\alpha(x)\) for the corrected quantile value, a central \((1-\gamma)\) prediction interval—where \(\gamma\in(0,1)\) is the total tail probability outside the interval—is

For example, a 90% interval uses \(\gamma=0.1\):

If the interval has calibrated marginal coverage—that is, empirically, the fraction of held-out targets falling inside it is close to the nominal level (e.g. ~90%)—it should contain the true target approximately 90% of the time over repeated samples from the same data-generating process. This marginal coverage does not by itself guarantee conditional coverage for every input row or calibration of every individual quantile. It is an empirical calibration property of the predictions, not something guaranteed merely by using pinball loss or by sorting the quantiles.

Prediction intervals use specific quantile levels; point estimation uses the full grid. For point estimation, TabICLv2’s fast mean path averages the 999 predicted quantiles. This is motivated by the quantile-function identity

when the conditional expectation exists. With a dense, evenly spaced grid of quantiles, this integral can be approximated by a simple average. Because \(\mathcal{A}\) is an evenly spaced grid on \((0,1)\), the sum is a Riemann-sum approximation of the integral:

Here \(\hat{\mu}(x)\) is the fast point prediction and \(|\mathcal{A}|=999\). The grid omits \(\alpha=0\) and \(\alpha=1\); the fast mean therefore uses only the \(0.001\)–\(0.999\) grid and ignores the parametric tails extrapolated beyond those endpoints (see the distribution-construction steps above).

The current implementation constructs the monotone distribution before taking this average. Default sorting only reorders values, so it preserves the average of the raw outputs. The current unweighted isotonic-regression alternative can change individual values, but its pooled averages preserve the total sum and therefore the overall average. Monotonicity correction matters for interpreting the outputs as a quantile function and for distribution operations, but neither current correction method changes the simple average.

The same 999 raw outputs therefore support two inference paths: a fast point estimate through averaging, and richer probabilistic predictions through a reconstructed monotone distribution.

Implementation in NanoTabICL

The subsections above describe full TabICLv2 regression inference. NanoTabICL exposes only the regression forward path through max_classes=0 and out_dim=999. The following sections trace the target embedders and output head, explain target scaling, and identify the full TabICLv2 inference steps left outside the compact model.

These two NanoTabICL constructor arguments make the regression path visible:

def __init__(self, max_classes: int, out_dim: int, ...):
    # classification: max_classes = out_dim (= 10 typically)
    # regression: max_classes = 0, out_dim = n_quantiles

The README combines these regression settings in its example:

model = NanoTabICLv2(
    max_classes=0,
    out_dim=999,
    embed_dim=96,
    col_num_blocks=2,
    row_num_blocks=2,
    icl_num_blocks=4,
    col_nhead=4,
    row_nhead=4,
    icl_nhead=4,
)
y_train = torch.randn(batch_size, n_train)
y_test_pred_quantiles = model(X_train_and_test, y_train)

This example instantiates a randomly initialized model. Its 999 outputs acquire quantile meaning only after compatible pinball-loss pretraining or after loading compatible trained regression weights; NanoTabICL provides neither.

These arguments control different sides of the model:

NanoTabICL exposes out_dim directly. The full TabICLv2 constructor instead exposes num_quantiles and internally sets out_dim=num_quantiles for regression.

One more checkpoint-compatibility difference matters: NanoTabICL uses LayerNorm with bias, matching the TabICLv2 classification checkpoint, while the full regression checkpoint uses LayerNorm without bias. The compact model therefore explains the regression architecture, but it is not a drop-in reimplementation of every regression-checkpoint detail.

Regression target embeddings

The task switch appears first in the two target embedders:

self.y_embed_in = (
    ClassEmbedding(max_classes, embed_dim)
    if max_classes > 0
    else nn.Linear(1, embed_dim)
)
self.y_embed_icl = (
    ClassEmbedding(max_classes, icl_dim)
    if max_classes > 0
    else nn.Linear(1, icl_dim)
)

For classification, ClassEmbedding treats each target as an integer class id. For regression, nn.Linear(1, ...) treats each target as one continuous scalar and projects it into the required token space:

y_train scalar
    -> nn.Linear(1, embed_dim)  for feature-token target-aware embedding
    -> nn.Linear(1, icl_dim)    for row-token ICL embedding

The first projection is added before column-wise processing:

emb = self.x_embed(x)
emb[:, :n_train] += self.y_embed_in(y[:, :, None, None])

Here y has shape (batch, n_train). Adding two singleton dimensions gives (batch, n_train, 1, 1). The linear layer transforms the final size-one dimension, producing (batch, n_train, 1, embed_dim), which broadcasts across all grouped feature positions in each training row:

feature-level target embedding:
(batch, n_train)
    -> (batch, n_train, 1, 1)
    -> (batch, n_train, 1, embed_dim)
    -> broadcast across cols

After row compression, the second projection is added before dataset-wise ICL:

emb[:, :n_train] += self.y_embed_icl(y[:, :, None])

At this point emb has shape (batch, rows, icl_dim). The added singleton dimension lets nn.Linear(1, icl_dim) produce one row-level target vector for each labeled training row:

row-level target embedding:
(batch, n_train)
    -> (batch, n_train, 1)
    -> (batch, n_train, icl_dim)

Both additions select emb[:, :n_train], so no target value is injected into test rows.

Because both embedders consume raw scalar targets, y_train must be standardized before the forward pass and predictions back-transformed afterward. NanoTabICL scales X_train_and_test internally using training rows, but it does not transform y_train; its feature scaling is also asymmetric because it divides by the training-row standard deviation without subtracting the training mean. The README warns:

# warning: for regression, you need to standardize y yourself
# (and backtransform the output)

This matters because the same standardized target values are passed into both regression target embedders, and the 999 outputs are produced on that standardized scale. Quantiles are equivariant under positive affine maps: if \(Y’ = aY + b\) with \(a>0\), then \(Q_{Y’}(\alpha) = a\,Q_Y(\alpha) + b\). Standardizing and back-transforming therefore preserves each output’s probability-level meaning.

A minimal per-table target transformation would look like:

y_mean = y_train.mean(dim=1, keepdim=True)
y_std = y_train.std(dim=1, unbiased=False, keepdim=True).clamp_min(1e-8)

y_train_scaled = (y_train - y_mean) / y_std
q_scaled = model(X_train_and_test, y_train_scaled)
q_original = q_scaled * y_std[:, :, None] + y_mean[:, :, None]

Every predicted quantile uses the same inverse affine transformation shown above. Target standardization is entirely your responsibility.

From test-row states to raw quantiles

The final ICL block uses labeled training rows as keys and values, while computing outputs only for test-row queries:

emb = self.icl_blocks[-1](emb[:, n_train:], emb[:, :n_train])

Its output has shape (batch, n_test, icl_dim). The output head then maps each test-row state to out_dim values:

self.out_mlp = get_mlp(icl_dim, icl_dim * 2, out_dim)
return self.out_mlp(self.out_ln(emb))

after final ICL block: (batch, n_test, icl_dim)
after output MLP:      (batch, n_test, out_dim)
with out_dim=999:      (batch, n_test, 999)

There is no softmax, sorting operation, or monotonicity constraint in this output path. The architecture itself also does not attach \(\alpha\) values to the 999 output positions. Their interpretation as the indexed grid

comes from training each position against its corresponding pinball-loss level. NanoTabICL provides the architecture and raw forward-pass outputs, but it does not provide that pretraining loop.

What NanoTabICL leaves outside the model

NanoTabICL returns the raw tensor (batch, n_test, 999) directly. To make those outputs meaningful quantile predictions, the model must first be trained with the corresponding pinball-loss levels or supplied compatible trained weights. Distribution construction and prediction intervals are then implemented downstream of the tensor. Beyond the forward pass described above, NanoTabICL does not include:

• pinball-loss pretraining;

• the mapping from output positions to quantile levels as a model object;

• monotonicity correction for crossing quantiles;

• interpolation, tail extrapolation, or distribution statistics;

• a scikit-learn prediction interface.

Share

Summary

TabICLv2 handles regression through a dedicated regression checkpoint that predicts 999 conditional quantiles rather than a single point or a discretized target distribution. Each output level is trained with pinball loss, which targets the corresponding conditional quantile through its asymmetric penalty on under- and over-prediction. The raw 999 scalars are not, by themselves, a valid predictive distribution; at inference time, TabICLv2 sorts or otherwise corrects crossing quantiles, interpolates between grid points, and extrapolates the tails to build a full distribution wrapper.

The same quantile grid supports two inference paths. A fast point estimate averages the 999 predicted levels, approximating the conditional mean through the quantile-function identity. Richer probabilistic outputs come from the reconstructed distribution: central \((1-\gamma)\) prediction intervals read directly from symmetric quantile pairs, where \(\gamma\) is the total tail probability outside the interval, while PDF, CDF, and moment calculations use the interpolated body and parametric tails. Nominal interval coverage is an empirical calibration property, not something guaranteed by pinball loss alone.

Regression reuses the same overall compression-then-ICL backbone structure as classification, while changing the task-specific interfaces, loss, and LayerNorm configuration. Observed training targets are embedded as continuous scalars through linear maps at both the feature-token and row-token stages, and the output head emits out_dim=999 raw values per test row with no softmax or built-in monotonicity constraint. NanoTabICL makes this path visible through max_classes=0 and out_dim=999, but users must standardize y themselves, train the model with the corresponding pinball-loss levels or supply compatible trained weights, and apply the full TabICLv2 inference pipeline downstream of its raw (batch, n_test, 999) output; it leaves pinball-loss pretraining, monotonicity correction, distribution construction, and scikit-learn wrappers outside the model.

This regression path completes the six-part walkthrough of TabICLv2’s architecture: repeated feature grouping, target-aware embedding, compression-then-ICL, QASSMax, many-class classification, and quantile regression. Together, the posts trace one table from grouped feature tokens through row-level in-context learning to either class probabilities or a full predictive distribution over a continuous target.

Quiz

Take the quiz below to test your understanding, and share your answers and doubts in the comments. The questions get progressively harder from 1 to 10.

1. What does the quantile grid \(\mathcal{A}=\{0.001,0.002,\ldots,0.999\}\) represent?

2. What is pinball loss, and how does its asymmetry target a specific quantile?

3. Why are the raw 999 model outputs not, by themselves, a full predictive distribution?

4. How is pinball loss applied during training for the 999 quantile outputs?

5. Why can the raw 999 quantile outputs cross, and how does TabICLv2 correct them at inference time?

6. After monotonicity correction, how does TabICLv2 construct a full predictive distribution from the corrected quantile grid?

7. How is a central \((1-\gamma)\) prediction interval read from the corrected quantile function, and does pinball loss alone guarantee that a nominal 90% interval covers 90% of held-out targets?

8. How does TabICLv2’s fast point-estimate path approximate the conditional mean, and why do the current monotonicity-correction methods preserve that mean?

9. How does TabICLv2’s regression strategy differ from TabPFNv2 and TabPFN-2.5, and which NanoTabICL arguments activate its regression path?

10. Suppose you only need a fast point estimate and never use prediction intervals, PDF, CDF, or moments. Which inference steps could you skip, and what would you lose?

Supporting more than \(10\) classes is not simply a matter of widening the output head. Observed training labels enter TabICLv2 twice: first through target-aware embedding before \(\text{TF}_\text{col}\), and again at the ICL stage before \(\text{TF}_\text{icl}\). The native model expects small class ids at both points and produces only \(10\) logits per prediction.

TabICLv2 preserves that pretrained interface by decomposing the large label space on both sides of the architecture. Mixed-radix ensembling converts each training label into several small-label views for target-aware column embedding. Hierarchical classification composes several node-local predictions, each with at most \(10\) choices, into probabilities over the original classes. This post explains both mechanisms, how they work together, and the boundary between the full TabICLv2 implementation and NanoTabICL’s native small-class model.

Subscribe now

Many-class classification

The two mechanisms introduced above operate only on observed training labels; test labels remain the unknown values to be predicted. Throughout this post, class labels are assumed to have been encoded as contiguous integers from \(0\) to \(C-1\). Both many-class mechanisms rely on this assumption: the mixed-radix implementation infers the class count as y_train.max() + 1, and hierarchical decoding uses original class labels as output-column indices.

The discussion starts with hierarchical classification because the output bottleneck is the easiest one to see. Mixed-radix ensembling then solves the analogous problem on the input-label side.

The output bottleneck: more than 10 classes

TabICLv2 is pretrained on classification tasks with at most 10 classes. Let \(C\) be the number of downstream classes, \(x\) be the test row representation being classified, and \(y\in\{0,\ldots,C-1\}\) be the true class label. A direct \(C\)-class classifier would assign probabilities with a \(C\)-way softmax:

where \(s_c(x)\) is the score, or logit, for class \(c\). This is natural when \(C\leq 10\), but it no longer matches the interface the model was trained to use when \(C\) is much larger.

TabICLv2’s solution is to avoid that direct \(C\)-way decision. Instead of training a new large head, it repeatedly asks the native classifier to solve decisions with at most \(10\) choices.

Hierarchical classification

Let the full class set be

A hierarchy starts by partitioning \(\mathcal{Y}\) into disjoint groups:

where \(K\leq 10\). The root classifier predicts which group contains the true class. If a group still contains more than \(10\) original classes, that group is partitioned again. Repeating this process creates a tree whose internal nodes each have at most \(10\) children. When a node contains at most \(10\) original classes, the native classifier can predict directly among those classes.

Each original class is then identified by a path through the tree: group choices at internal nodes, followed by a final class choice inside a small leaf node. For class \(c\), write \(v_t(c)\) for the node visited at depth \(t\), \(b_t(c)\) for the local branch or local class choice made there, and \(L(c)\) for the number of local decisions needed to identify \(c\). The decision path is

Each local decision stays inside the pretrained class budget:

where \(K_t(c)\) is the number of available choices at the node reached by class \(c\) at step \(t\). The model never has to solve a \(C\)-way decision directly. It solves several decisions with at most \(10\) choices whose combination identifies one original class.

Let \(\mathcal{D}\) denote the complete labeled context dataset and \(\mathcal{D}_v\) the subset assigned to node \(v\). At depth \(t\), write the node-local probability as

Multiplying the node-local probabilities along the path for class \(c\) gives

So the probability of an original class is the product of the local probabilities along its path. TabICLv2 therefore replaces a single \(C\)-way softmax with a composition of native predictions, each using at most \(10\) labels and logits.

Building the tree in TabICLv2

TabICLv2 builds the hierarchy from the sorted observed class labels. For the usual max_classes=10 classifier checkpoint, if a node contains \(N\) classes and \(N>10\), the number of child groups is

where \(N\) is the number of classes at the current node. The \(N\) classes are split into \(K\) nearly equal contiguous groups. Any child group that still contains more than \(10\) classes is split again. This keeps every local classifier within the model’s native class capacity while keeping the tree reasonably balanced.

This is a computational hierarchy over contiguous ranges of encoded class ids, not a learned or domain-defined taxonomy. Nearby encoded ids need not be semantically related. When a broader prediction ensemble includes class-id shuffling, different members can also group the original labels differently.

For example, with \(C=57\), the root node uses

groups, with sizes close to \(57/6\). The first three groups contain 10 classes each, and the last three groups contain 9 classes each. Because every group already has at most \(10\) classes, the tree has two prediction levels: the root predicts among six groups, and the child classifiers predict directly among their 9 or 10 original classes. For a larger \(C\), some root groups would still contain more than \(10\) classes, so those groups would be split recursively.

Inference with the native ICL predictor

At inference time, TabICLv2 applies the hierarchy by recursively calling the native small-class ICL predictor. The hierarchy is not a new \(C\)-class output head; it is a wrapper around the pretrained predictor. The implementation does not pass previous branch choices into one autoregressive decoder. Instead, each tree node \(v\) makes a fresh native ICL prediction using its assigned training subset \(\mathcal{D}_v\). For each candidate class, its predefined path identifies the node-specific contexts whose probabilities contribute to its score, and the implementation recursively evaluates every child rather than selecting only one branch at runtime.

Operationally, the wrapper performs the following steps:

1. Partition the class set at each node into at most \(10\) groups.

2. At an internal node, select that node’s training-row subset, relabel those rows by their child-group index, and run the native ICL classifier on the test row to obtain group probabilities.

3. At a leaf node, select its training-row subset, relabel its original classes to contiguous local ids, and run the native classifier directly among those classes.

4. Score every valid original class by multiplying the probabilities along its path.

5. Take the argmax if a hard class prediction is needed.

The key detail is that this is not greedy decoding. TabICLv2 does not choose one group at the root and discard the rest. It recursively scores child nodes and combines probabilities, so every valid class receives a probability.

Picking the predicted class

For every valid class \(c<C\), the path score is the product of the internal group probabilities and the final local class probability along \(\pi(c)\). Mathematically, the same argmax can be computed in log space:

The current implementation performs the recursive probability multiplications directly. The composed probabilities are sufficient for prediction, but callers may still request a logits-shaped output. In that case, the implementation returns derived logits, rather than raw decoder logits, by converting each final composed probability \(p\) as

where \(\tau\) is the softmax temperature and the implementation uses \(\epsilon=10^{-6}\).

That completes the output-side story: after row representations are built, the model can score every original class by composing native small-class predictions. The other bottleneck happens earlier in the pipeline, before \(\text{TF}_\text{col}\), where labeled context rows still need target-aware embeddings.

Mixed-radix ensembling

Hierarchical classification fixes prediction, but it cannot by itself explain how the model processes labeled context rows with many-class targets. Before the model reaches \(\text{TF}_\text{icl}\), those rows have already passed through target-aware embedding and \(\text{TF}_\text{col}\). If \(C>10\), the raw class id is too large for the native target-aware embedding interface. Mixed-radix ensembling (MRE) fixes this input-side problem by turning each large label into several small-label views, running \(\text{TF}_\text{col}\) once per view, and averaging the resulting representations. Hierarchical relabeling then keeps the later ICL-side label embedding within max_classes during recursive local predictions.

The mixed-radix construction begins by representing one large class id as several small digits. The implementation chooses the smallest possible number of views,

then computes a balanced initial base

Starting from the balanced list \([k,\ldots,k]\), the implementation returns \(D\) positive-integer bases, also called radices,

such that

The product condition ensures that there are enough digit combinations to represent all \(C\) classes, while the per-base upper bound keeps every digit within the native class capacity. A base of 1 would add no information, so the useful selected bases are nontrivial. For example, with \(C=25\), the implementation selects the balanced bases \([5,5]\), rather than another valid but less balanced choice such as \([10,3]\).

For a class label \(y\in\{0,\ldots,C-1\}\), define positional weights

with the convention that an empty product is \(1\), so \(w_{D-1}=1\). Here \(w_i\) is the place value of digit \(i\). The mixed-radix digits are

Each digit stays within a small class range:

so every digit is compatible with the 10-class pretraining regime. For represented labels, the original class id can be reconstructed from its digits:

where \(y<C\). If \(\prod_i k_i>C\), some digit combinations do not correspond to real downstream classes. Those combinations are simply unused.

For example, suppose \(C=57\), the same class count used in the hierarchy example. The two mechanisms decompose those labels differently: hierarchy splits the labels into contiguous ranges for output prediction, while mixed radix splits each class id into digits for input embedding.

For MRE, the implementation first minimizes \(D\). Two views are sufficient, and \(\lceil\sqrt{57}\rceil=8\), so the balanced bases selected are \([8,8]\), with \(8\cdot8=64\geq57\). The bases \([10,6]\) would also satisfy the capacity constraints because \(10\cdot6=60\geq57\), but they are less balanced. With \([8,8]\), digit \(y^{(0)}\) is the high place and \(y^{(1)}\) is the low place:

Class \(y=42\) becomes \((5,2)\), because \(42=5\cdot8+2\). Class \(y=56\) becomes \((7,0)\). The combinations \((7,1)\) through \((7,7)\) would represent 57 through 63, so they are unused when the true class set has only 57 classes.

In TabICLv2, these digits provide several small-label views of the original class. Instead of embedding the large class id \(y\) directly, the model embeds one digit \(y^{(i)}\) at a time. Operationally, TabICLv2 creates one labeled-context view per digit, runs \(\text{TF}_\text{col}\) once per view, and averages the resulting representations.

Let \(E_1[r,j]\in\mathbb{R}^d\) denote the \(d\)-dimensional feature-group representation for row \(r\) and grouped feature position \(j\) before target-aware embedding. For mixed-radix digit view \(i\), define the masked target vector

The same learned target-aware encoder is reused for every digit view. When writing \(E_1+u^{(i)}\) below, \(u_r^{(i)}\) is broadcast across every grouped feature position \(j\) in row \(r\), so the addition means \(E_1[r,j]+u_r^{(i)}\). A simplified view of the averaged representation is

Here \(O_\text{avg}\) is the averaged column-transformer representation across the \(D\) digit views. This is MRE. It exposes information about a large class label through several small-label views, each compatible with the pretrained target-aware embedding interface.

The important boundary is that MRE is not the final many-class decoder. It prepares representations by making context labels embeddable. Hierarchical classification then handles the final prediction over the original \(C\) classes.

Operational implication

Both mechanisms require repeated use of the native model: MRE uses multiple label views, and hierarchical classification uses node-specific training subsets. Consequently, the current full implementation does not support KV caching for many-class classification because these changing inputs are incompatible with the available caching path.

With the full many-class orchestration established, NanoTabICL provides a concrete view of the native small-class interface that both mechanisms reuse.

Implementation in NanoTabICL

NanoTabICL is also a useful boundary marker: it exposes that native interface, but it stops before the many-class orchestration. In particular, the compact repository does not include:

• the recursive hierarchical classification wrapper;

• mixed-radix digit construction;

• multiple \(\text{TF}_\text{col}\) passes over digit views;

• path-probability decoding for \(C>10\).

The standard README classification example shows this native interface directly:

model = NanoTabICLv2(max_classes=10, out_dim=10)
X_train_and_test = torch.randn(batch_size, n_train+n_test, n_cols)
y_train = torch.randint(10, size=(batch_size, n_train)).float()
y_test_pred_logits = model(X_train_and_test, y_train)

The two NanoTabICL constructor arguments that matter for this post are max_classes and out_dim:

def __init__(self, max_classes: int, out_dim: int, ...):
    # classification: max_classes = out_dim (= 10 typically)

NanoTabICL exposes these as independent constructor arguments, and they control different sides of the compact model:

In the full TabICLv2 classification constructor, out_dim is not independently exposed: it is set internally to max_classes. NanoTabICL’s explicit separation still helps show the two sides of the native interface.

The two target embedding tables are initialized in __init__:

self.y_embed_in = (
    ClassEmbedding(max_classes, embed_dim)
    if max_classes > 0
    else nn.Linear(1, embed_dim)
)
self.y_embed_icl = (
    ClassEmbedding(max_classes, icl_dim)
    if max_classes > 0
    else nn.Linear(1, icl_dim)
)

For classification, max_classes > 0, so both are ClassEmbedding layers. The first table injects labels into feature tokens before \(\text{TF}_\text{col}\). This is the native interface that MRE repeatedly uses with different digit views:

emb = self.x_embed(x)
emb[:, :n_train] += self.y_embed_in(y[:, :, None, None])

At this point emb has shape (batch, rows, cols, embed_dim). The slice emb[:, :n_train] selects only labeled training rows, and y[:, :, None, None] gives the target embedder singleton axes so the resulting label vector can broadcast across all feature positions in each training row. Test rows are not touched.

The second table injects labels again after row compression, just before dataset-wise ICL. Hierarchical classification reuses this embedding and the later output head for each node-local prediction:

emb[:, :n_train] += self.y_embed_icl(y[:, :, None])
for block in self.icl_blocks[:-1]:
    emb = block(emb, kv_max_idx=n_train)
emb = self.icl_blocks[-1](emb[:, n_train:], emb[:, :n_train])

Now emb has shape (batch, rows, icl_dim), so self.y_embed_icl(...) returns one row-level label vector per training row. In the loop, kv_max_idx=n_train restricts keys and values to training rows, preventing test rows from being used as labeled context. The ICL blocks use the training rows as labeled context, and the final block computes outputs only for test rows:

queries:      emb[:, n_train:]   -> test rows
keys/values:  emb[:, :n_train]   -> training rows

Implementation note: the README stores y_train as a float tensor, but the classification embedder casts labels to integer indices before lookup:

class ClassEmbedding(nn.Embedding):
    def forward(self, y: torch.Tensor) -> torch.Tensor:
        return super().forward(y.squeeze(-1).long())

The classification path is therefore still a lookup-table path. Labels can be carried as floats in the example tensor, but their values must be valid class indices from 0 to 9.

The output head is a separate MLP:

self.out_mlp = get_mlp(icl_dim, icl_dim * 2, out_dim)

return self.out_mlp(self.out_ln(emb))

With out_dim=10, the returned tensor has shape:

(batch, n_test, 10)

Those 10 values are the native small-class logits. This gives the concrete interface that the many-class wrappers depend on:

training labels 0..9
    -> class embeddings for feature-level and row-level target injection
    -> compression-then-ICL over labeled training rows
    -> 10 logits per test row

For \(C>10\), full TabICLv2 adds orchestration around this interface. Mixed-radix ensembling repeatedly feeds small digit labels into the target-aware embedding side, keeping context-label embeddings compatible with \(\text{TF}_\text{col}\). Hierarchical classification repeatedly asks the native small-class predictor to solve node-local branch decisions during the \(\text{TF}_\text{icl}\) stage. NanoTabICL does not include those wrappers; it makes the reuse point visible.

Share

Summary

TabICLv2 supports many-class classification without changing the small-class interface learned during pretraining. Because observed labels enter the model before both \(\text{TF}_\text{col}\) and \(\text{TF}_\text{icl}\), it must decompose the problem at both stages rather than merely replace the output head.

Mixed-radix ensembling handles the target-aware embedding side. It expresses each large class id as several digits with at most \(10\) values, runs the column transformer for each digit view, and averages the resulting representations. Hierarchical classification handles the ICL and output side. It organizes the original classes into a balanced tree, makes fresh native predictions among at most \(10\) choices at each node, and multiplies probabilities along each path to score every original class.

Together, these mechanisms turn one unsupported \(C\)-class task into multiple predictions that remain within the model’s native label and logit capacity. That reuse requires multiple forward passes over different label views and node-specific contexts, which is also why the current many-class path does not support KV caching. NanoTabICL exposes the small-class interface being reused through max_classes=10, its two class-embedding stages, and out_dim=10, but leaves out the mixed-radix and hierarchical orchestration.

The next post covers quantile predictions for regression, the regression strategy TabICLv2 uses to model predictive uncertainty without discretizing the target into classification bins.

Quiz

Take the quiz below to test your understanding, and share your answers and doubts in the comments. The questions get progressively harder from 1 to 10.

1. What are the two many-class bottlenecks TabICLv2 has to handle when \(C>10\)?

2. Which mechanism handles the output side of many-class classification, and which mechanism handles the input-label embedding side?

3. Why does TabICLv2 avoid adding a new direct \(C\)-class output head for many-class classification?

4. In hierarchical classification, what does the path \(\pi(c)\) represent for an original class \(c\)?

5. How is the probability of an original class computed from local hierarchical predictions?

6. For \(C=57\) and max_classes=10, why does the root node use six groups in the example hierarchy?

7. Why is hierarchical decoding in TabICLv2 not greedy decoding?

8. What conditions must the mixed-radix bases \([k_0,k_1,\ldots,k_{D-1}]\) satisfy, and how does TabICLv2 choose among valid bases?

9. With mixed-radix bases \([8,8]\) for \(C=57\), what digits represent class \(y=42\), and which digit combinations are unused?

10. What many-class machinery does NanoTabICL expose, and what does it deliberately leave out?

But the row-level ICL transformer is not the only place where attention must scale with training-set size. TabICLv2 also uses attention over rows inside the column-wise induced-attention stage. As the number of training rows grows, ordinary softmax attention can become too diffuse. Even when one training row is the best match for a query row, or the most useful row for an inducing vector to summarize, the softmax denominator grows with all the other rows. The best row can then receive only a small fraction of the attention mass. This is the attention-fading problem.

This post covers Query-Aware Scalable Softmax, or QASSMax, the mechanism TabICLv2 uses to make attention more robust as context size changes. We will start with ordinary softmax as a temperature-controlled normalization, show why attention fades when the number of keys grows, then build up from Scalable Softmax (SSMax) to QASSMax.

The figure shows the architectural destination; the next sections explain why those two attention sites need length-aware softmax in the first place. The post ends with a quiz that you can take to test your understanding.

Subscribe now

Query-aware scalable softmax

Query-aware scalable softmax, or QASSMax, modifies attention by rescaling the query before the query-key dot products are computed. Because softmax is applied to those dot products, changing the query changes the logits. With scalar scaling, this changes how sharp or broad the attention distribution becomes; with element-wise scaling, it can also change the ranking of keys.

The purpose is to keep attention selective when the number of context samples becomes much larger than the sequence lengths seen during pretraining. To avoid overloading notation, I will use \(N\) for the number of keys in a generic attention calculation. When we specialize the discussion to TabICLv2, I will use \(n\) for the number of training rows.

Softmax as temperature

Start with standard scaled dot-product attention for one attention head. Let

be the query vector, and let

be the \(N\) key vectors. Here \(d_\text{head}\) is the dimension of one attention head. The unnormalized attention logit for key \(j\) is

where \(j\in\{1,\ldots,N\}\). The attention weight assigned to key \(j\) is

Here \(z=(z_1,\ldots,z_N)\), \(a_j\) is the normalized attention weight, and \(\ell\) is only the summation index over keys. The attention output is the weighted average

where \(v_j\) is the value vector associated with key \(j\).

Now rescale the query by a positive scalar \(\lambda\):

The new logit is

So scalar query scaling is the same as scalar logit scaling. It is also the same as changing the softmax temperature. For temperature \(\tau>0\),

Writing \(\lambda=1/\tau\), we get

Lower temperature, or larger \(\lambda\), makes the distribution sharper. Higher temperature, or smaller \(\lambda\), makes it broader. The relative odds between keys \(j\) and \(\ell\) become

The key point is that scaling does not change the ranking of logits. If \(z_j>z_\ell\), then \(\lambda z_j>\lambda z_\ell\). Scaling changes how decisively softmax turns that ranking into probability mass.

Why softmax fades as context grows

This matters in long contexts because the softmax denominator grows with the number of keys. Consider a simplified attention calculation with one relevant key and many distractor keys:

• the relevant key has logit \(z_\star\);

• each of the \(N-1\) distractors has lower logit \(z_0\); and

• the logit gap is \(\Delta=z_\star-z_0>0\).

Under ordinary softmax, the attention weight on the relevant key is

For a fixed gap \(\Delta\), the denominator grows as \(N\) grows, so

The problem is not that softmax forgets which key has the highest logit. The relevant key still ranks first. The problem is that many weaker keys can collectively absorb most of the probability mass.

In TabICLv2, this matters at the attention sites whose key sequence grows with the number of training rows. In \(\text{TF}_\text{icl}\), a test row may attend over many labeled training rows; in the first induced-attention step of \(\text{TF}_\text{col}\), learned inducing vectors summarize many training-row tokens for a fixed grouped feature position. In both cases, as the number of training rows grows, the ordinary softmax denominator grows with all the other rows too. This is attention fading.

SSMax: scale logits with log n

Scalable Softmax, or SSMax, addresses this failure mode by making the logit scale grow with context length. It keeps softmax as the normalization function, but rescales the query before computing attention logits.

Now specialize from \(N\) generic keys to \(n\) training rows. In the TabICLv2 paper’s notation, let \(q_h=(q_{hi})\) be a query vector at attention head \(h\), where \(i\) indexes the coordinates inside the head. SSMax rescales the query with a learnable per-head scalar \(s_h\):

This makes each logit

Returning to the one-relevant-key example, the attention weight becomes

The approximation uses \(n-1\approx n\) and

This shows why \(\log n\) appears: the softmax denominator grows with the number of keys, so the effective relevant-vs-distractor gap must also grow with context length. In this simplified example, assuming \(s_h>0\), \(s_h\Delta>1\) is where the relevant key does not fade away as \(n\) becomes large.

That condition is only intuition from the toy setup. Real attention has many different logit values, not one repeated distractor logit. The useful lesson is the scaling law: increasing the context length changes the softmax denominator, so the model benefits from a length-aware logit scale.

QASSMax: length base + query gate

SSMax gives every query in the same attention head the same length-dependent scale. QASSMax keeps the length-aware idea but makes it more flexible in two ways:

• the base scale is a learned vector function of \(\log n\), not one scalar \(s_h\); and

• the scale is modulated by a bounded gate that depends on the current query.

For head \(h\), let \(q_h=(q_{hi})\) be the query vector, with \(i\) indexing head dimensions. QASSMax rescales each query element as

Equivalently, in vector notation,

where

and

Here \(\odot\) denotes element-wise multiplication. \(B_h(n)\) is the length-dependent base vector for head \(h\), and \(G_h(q_h)\) is the query-dependent gate for that same head.

For \(H\) attention heads, the base network has type

It takes \(\log n\) as input and returns one base scale for every head dimension. The gate network has type

It maps the current query to an element-wise gate. This is the element-wise QASSMax variant used by the official default configuration.

The two terms have different roles. The base term \(B_h(n)\) handles the predictable effect of context length. As \(n\) changes, the model can learn how much the logits should be rescaled before softmax.

The gate \(G_h(q_h)\) handles query-specific sharpness. Some queries should retrieve one highly relevant row. Other queries should aggregate signal from many rows. Because

the gate satisfies

So the query-dependent part can reduce or increase the magnitude of the base scaling contribution, but the gate itself is bounded. The full scale is still not guaranteed to be positive, because the base MLP output is unconstrained.

There is one important difference from ordinary temperature scaling. A positive scalar temperature changes attention sharpness while preserving the ranking of logits. QASSMax uses element-wise scaling, and \(\text{MLP}_\text{base}(\log n)\) is not constrained to be a positive scalar, so it can change the direction of the query vector and therefore can change logit rankings as well as sharpness.

At initialization, QASSMax behaves like length-dependent scaling without extra query modulation. The query-aware part can be learned gradually.

Why put the gate on the query, not the output?

One design choice remains: the gate is applied to the query before attention, rather than to the output after attention.

Selective attention needs per-query sharpness. A scalar temperature can express this idea in a simple form. For query row or query token \(r\), let \(z_{rj}\) be the attention logit between query \(r\) and key \(j\). With a query-specific temperature \(\tau_r>0\), the attention weight would be

QASSMax is more expressive than this scalar-temperature view because it scales different head dimensions differently and can change more than sharpness alone. The scalar-temperature equation is still useful as intuition: the query participates in controlling how selective its attention distribution should be.

Some gated attention mechanisms apply the gate after attention, for example

Here \(K\) is the matrix of keys, \(V\) is the matrix of values, \(\text{Attn}(q,K,V)\) is the ordinary attention output for query \(q\), and \(\tilde{o}\) is the gated output. This kind of gate changes the output after attention weights have already been computed.

QASSMax applies the gate earlier. Because the gate changes the query \(\tilde{q}_h\), it changes the logits before softmax:

So the gate affects the attention weights themselves, not only the post-attention output. This is why QASSMax is a softmax-logit modification rather than an output gating mechanism.

In short, QASSMax fights fading in two places: the base term adapts to context length, and the query gate lets each query adjust its attention geometry and selectivity.

Where TabICLv2 uses QASSMax

TabICLv2 uses QASSMax where the number of training rows directly affects the number of keys. There are two such places in the architecture:

• the first induced-attention stage of \(\text{TF}_\text{col}\), where inducing tokens summarize row tokens for a fixed grouped feature position; and

• \(\text{TF}_\text{icl}\), where row representations attend across the dataset.

It is not used in the row-wise transformer that compresses feature positions within one row. That stage attends across columns, not across a large set of training examples.

The paper’s stress test makes the failure mode concrete.

This placement matches the failure mode. In the paper’s needle-in-haystack classification task, the model must focus on one anchor sample among many negative samples. Without scalable softmax, normalized attention entropy rises and accuracy drops as the number of negatives grows. The paper reports entropy divided by \(\log n\) and averaged across heads and layers in \(\text{TF}_\text{icl}\). QASSMax keeps that entropy low and maintains 100% accuracy even with 15K negatives, outperforming SSMax at extreme scales.

The implementation section below shows how this placement appears in NanoTabICL.

Implementation in NanoTabICL

NanoTabICL wires QASSMax through the ssmax=True flag. The flag name is broad, but in this implementation it constructs a QASSMax layer. The flag is enabled only in the attention stages where the key sequence can grow with the number of training rows:

self.col_blocks = nn.ModuleList([
    InducedTransformerBlock(
        embed_dim=embed_dim,
        num_heads=col_nhead,
        n_inducing=n_cls_rows,
        ssmax=True,
    )
    for _ in range(col_num_blocks)
])

self.icl_blocks = nn.ModuleList([
    TransformerBlock(embed_dim=icl_dim, num_heads=icl_nhead, ssmax=True)
    for _ in range(icl_num_blocks)
])

The row-wise transformer blocks do not enable QASSMax:

self.row_blocks = nn.ModuleList([
    TransformerBlock(embed_dim=embed_dim, num_heads=row_nhead, use_rope=True)
    for _ in range(row_num_blocks)
])

So the placement is:

This matches the motivation from the previous sections. QASSMax is used when the number of keys is tied to training-set size. It is not used merely because a transformer block exists.

Column-wise induced attention

The column-wise stage has one important detail. InducedTransformerBlock contains two transformer calls:

self.tfm1 = TransformerBlock(embed_dim=embed_dim, num_heads=num_heads, ssmax=ssmax)
self.tfm2 = TransformerBlock(embed_dim=embed_dim, num_heads=num_heads)

kv = self.tfm1(self.inducing_vectors.expand(q.shape[0], -1, -1), q if kv is None else kv, kv_max_idx=kv_max_idx)
return self.tfm2(q, kv, q_max_idx=q_max_idx)

Only the first transformer, tfm1, receives ssmax=ssmax. The second transformer, tfm2, is constructed without QASSMax because it attends over the fixed number of inducing vectors, not directly over all training rows.

for block in self.col_blocks:
    emb = block.col_attn(emb, kv_max_idx=n_train)

The col_attn wrapper transposes the table so each grouped feature position is processed as a separate sequence over rows. Inside the attention module, the effective shape is:

(batch * cols, rows, embed_dim)

The argument kv_max_idx=n_train slices the key/value sequence to the labeled training rows before attention runs. Therefore, when QASSMax is called in tfm1, its length argument is the number of training rows:

n = k.size(-2) = n_train

This is exactly the long-context setting QASSMax is meant for. The learned inducing vectors query a potentially large set of training-row tokens for a fixed grouped feature position.

Dataset-wise ICL attention

The second QASSMax placement is the dataset-wise ICL stack. After row compression, emb has shape:

(batch, rows, icl_dim)

NanoTabICL adds target information to training row tokens, then applies the ICL blocks:

emb[:, :n_train] += self.y_embed_icl(y[:, :, None])
for block in self.icl_blocks[:-1]:
    emb = block(emb, kv_max_idx=n_train)
emb = self.icl_blocks[-1](emb[:, n_train:], emb[:, :n_train])

For the intermediate ICL blocks, kv_max_idx=n_train means every row can query the sequence, but keys and values are restricted to training rows. For the final ICL block, the query sequence is explicitly test rows and the key/value sequence is explicitly training rows:

queries:      emb[:, n_train:]   -> test rows
keys/values:  emb[:, :n_train]   -> training rows

In both cases, the key length passed to QASSMax is the training context size. So the ICL transformer receives the same length-aware query scaling discussed above.

Query scaling inside TransformerBlock

Inside TransformerBlock, the ssmax=True flag creates a QASSMax layer:

self.ssmax_layer = (
    QASSMax(num_heads=num_heads, head_dim=embed_dim // num_heads)
    if ssmax
    else None
)

The attention method first projects query, key, and value vectors, then reshapes them into multi-head form:

(batch, seq_len, embed_dim)
    -> (batch, heads, seq_len, head_dim)

Then QASSMax is applied directly to the projected query tensor:

q = q if self.ssmax_layer is None else self.ssmax_layer(q=q, n=k.size(-2))
q, k = (t if self.rope is None else self.rope(t) for t in [q, k])
attn_output = nn.functional.scaled_dot_product_attention(...)

This placement matters. In NanoTabICL, the QASSMax-enabled blocks do not use RoPE; RoPE is enabled in the row-wise transformer, where QASSMax is not used. In this implementation, QASSMax changes the projected query before scaled_dot_product_attention computes the logits \(q^\top k\), so it changes the attention weights themselves, not only the post-attention output.

The QASSMax module

The module has two learned pieces: base_mlp for length-dependent scaling and query_mlp for query-dependent modulation.

Both pieces are two-layer MLPs with 64 hidden neurons and GELU activation. GELU stands for Gaussian Error Linear Unit; one common definition is

where \(\Phi(x)\) is the standard normal CDF.

class QASSMax(nn.Module):
    def __init__(self, num_heads: int, head_dim: int, n_hidden: int = 64):
        super().__init__()
        self.base_mlp = get_mlp(1, n_hidden, num_heads * head_dim)
        self.query_mlp = get_mlp(head_dim, n_hidden, head_dim)
        nn.init.zeros_(self.query_mlp[-1].weight)
        nn.init.zeros_(self.query_mlp[-1].bias)

    def forward(self, q: torch.Tensor, n: int) -> torch.Tensor:
        batch_size, num_heads, seq_len, head_dim = q.shape
        logn = q.new_tensor(math.log(max(1, n))).view(1, 1)
        return (
            self.base_mlp(logn).view(1, num_heads, 1, head_dim)
            * (1 + torch.tanh(self.query_mlp(q)))
            * q
        )

The forward pass starts from the projected query tensor:

(batch, heads, query_len, head_dim)

The length input is converted to a one-element tensor:

logn: (1, 1)

After base_mlp(logn) and .view(1, num_heads, 1, head_dim), the base scale has shape:

(1, heads, 1, head_dim)

This is the implementation of \(B_h(n)\). The singleton batch and query dimensions make the same length-dependent scale broadcast across all examples and all query positions.

The query-dependent gate is:

1 + torch.tanh(self.query_mlp(q))

It has the same shape as q:

(batch, heads, query_len, head_dim)

This is the implementation of \(G_h(q_h)\). Because tanh lies in \((-1,1)\), the multiplicative gate lies in \((0,2)\). The final return statement multiplies the original query by both terms:

The zero initialization of the last query_mlp layer controls the starting behavior:

nn.init.zeros_(self.query_mlp[-1].weight)

nn.init.zeros_(self.query_mlp[-1].bias)

At initialization, query_mlp(q) is zero, so the gate starts as:

So QASSMax initially behaves like length-dependent scaling without extra query modulation. The query-aware part is learned gradually rather than perturbing attention sharply at initialization.

The implementation therefore mirrors the mathematical decomposition from the previous section:

That is the whole implementation idea: QASSMax is not a separate attention kernel. It is a learned query rescaling step inserted immediately before ordinary scaled dot-product attention.

Share

Summary

Ordinary softmax attention can fade as the number of keys grows: even a clearly relevant row can lose attention mass to many individually weaker distractors. SSMax addresses this by scaling logits with a learned factor proportional to \(\log n\), matching the way the softmax denominator grows with context length.

QASSMax makes that idea more flexible. It rescales the query with a learned length-dependent base term \(B_h(n)\) and a bounded query-dependent gate \(G_h(q_h)\), so attention can adapt both to the number of training rows and to the specific query being processed. In NanoTabICL, this appears exactly where long row contexts matter: the first induced-attention step of the column-wise transformer and the dataset-wise ICL transformer.

The next post covers many-class classification, where TabICLv2 extends a model pretrained with at most 10 classes to settings with many more labels.

Quiz

Take the quiz below to test your understanding, and share your answers and doubts in the comments. The difficulty increases from question 1 to question 10 in increasing order of the question number.

1. What practical attention problem does QASSMax address in TabICLv2?

2. Why is multiplying the query vector by a positive scalar equivalent to changing softmax temperature?

3. In the one-relevant-key example, why does the relevant key's attention weight go to zero as \(N\) grows under ordinary softmax?

4. What is the basic scaling idea behind SSMax?

5. Why does \(\log n\) appear in scalable softmax rather than just \(n\)?

6. What are the two multiplicative components QASSMax adds to the query?

7. What does the bounded query gate \(1+\tanh(\text{MLP}_\text{gate}(q_h))\) allow the model to do?

8. Why can QASSMax change more than just the sharpness of the attention distribution?

9. Where is QASSMax used in NanoTabICL, and where is it not used?

10. In the NanoTabICL QASSMax.forward method, why does base_mlp(logn) have shape (1, heads, 1, head_dim) after reshaping, while the query gate has the same shape as q?

This post covers the next architectural step: compression-then-ICL. TabICLv2 does not run dataset-level in-context learning directly over all \(n\times m\) row-feature tokens. It first contextualizes grouped feature positions across rows, compresses each row into a smaller row representation, and then runs in-context learning over those row tokens.

In this post, we start from the target-aware tensor \(E_2\), pass it through column-wise and row-wise transformer blocks, and end with dataset-wise ICL, where test row representations attend to labeled training row representations. A short quiz at the end lets you check whether the architecture and label-flow details are clear.

Subscribe now

Compression-then-ICL

Starting point: target-aware tokens \(E_2\)

Before compression begins, TabICLv2 has already constructed a target-aware token tensor

Here \(d\) is the token embedding dimension. Each token \(E_2[i,j]\in\mathbb{R}^d\) represents row \(i\) and grouped feature position \(j\). After repeated feature grouping, \(m\) denotes the number of grouped feature positions being processed by the transformer stack.

The previous post derived target-aware embedding in full. For training rows, the update is

where \(E_1\) is the feature-only tensor from repeated feature grouping and \(u_i\) is the row-level target vector from target-aware embedding (see the previous post for \(\operatorname{Embed}_\text{TAE}\) and the test-row masking rule). For a labeled row, the same target vector is broadcast across all grouped feature positions.

The important point is that compression does not operate on raw feature tokens. It operates on feature tokens that already contain training-label information. For training rows, each grouped feature token has access to the observed target through \(u_i\). For test rows, the target component is absent, because the target is what the model must predict.

The compression-then-ICL pipeline explains what happens next. TabICLv2 must convert the \(n\times m\) grid of row-feature tokens into row-level representations that can be used for prediction. It does this in three stages:

1. column-wise embedding applies a Set Transformer \(\text{TF}_\text{col}\) to each grouped feature position;

2. row-wise interaction uses \(\text{TF}_\text{row}\) with learned \([\text{CLS}]\) tokens to compress grouped feature embeddings within each row;

3. dataset-wise ICL uses \(\text{TF}_\text{icl}\) so test row representations can attend to labeled training row representations.

The diagram below shows the same three-stage path, with tensor shapes at each step:

Stage 1: Column-wise embedding (\(\text{TF}_\text{col}\))

The first stage processes one grouped feature position at a time, across rows. For a fixed grouped feature index \(j\), the model sees the row sequence

This is “column-wise” in the table sense: the grouped feature position is fixed, and the row index varies.

The column-wise block uses a Set Transformer-style variant of attention. Here, Set Transformer means attention through learned inducing tokens rather than direct all-pairs row attention.

Because the training-row tokens already contain target information, this stage can learn how a grouped feature behaves across labeled examples. For example, through the induced summaries, \(\text{TF}_\text{col}\) can relate a test row’s token at position \(j\) to training-row tokens at the same position \(j\), where the training tokens encode both feature values and observed targets.

TabICLv2 does this with induced attention rather than full row-by-row attention. In the default no-leakage setting, a small set of learned inducing tokens first summarizes the training rows for each grouped feature position. Then all rows, including test rows, attend back to those summaries. Informally, for each \(j\),

The train-only summary step matters in this default setting. Test rows can receive context built from labeled examples, but they do not contribute to the summaries themselves. This keeps the direction of information aligned with supervised prediction: labeled rows inform unlabeled rows, not the other way around. Let

denote the output of the column-wise stage.

Stage 2: Row-wise compression (\(\text{TF}_\text{row}\))

The second stage changes the axis of attention. After column-wise embedding, each token \(\tilde{E}[i,j]\) has context from other rows at the same grouped feature position. Now the model must combine the \(m\) grouped feature positions inside each row.

For a fixed row \(i\), the row-wise transformer receives

along with learned \([\text{CLS}]\) tokens. The grouped feature tokens provide the content of the row. The \([\text{CLS}]\) tokens provide learned readout positions whose outputs are kept after attention.

If there are \(c\) learned \([\text{CLS}]\) tokens, the row-wise transformer produces \(c\) summary vectors for row \(i\). These are flattened into a single row representation

where \(d_\text{row}=c\cdot d\) in the simple flattened-CLS view. This is the main compression step:

The model has moved from one token per row-feature position to one representation per row.

Stage 3: Dataset-wise ICL (\(\text{TF}_\text{icl}\))

The row representations

become the tokens over which \(\text{TF}_\text{icl}\) operates. This is the stage that most directly resembles in-context learning: each row is now an example token, and test examples can use labeled training examples as context.

Before \(\text{TF}_\text{icl}\), TabICLv2 injects target information a second time. For training rows, the model adds another target embedding to the row representation. For test rows, no true target is supplied. This second embedding has a different role from the target-aware embedding in the previous post:

• the first target embedding entered before column-wise and row-wise processing, so labels could shape feature and row representations;

• the second target embedding enters after row compression, so the ICL transformer receives label values on context rows while test/query rows remain unlabeled.

To keep the same row-ordering convention as the previous post, let \(n_\text{train}\) be the number of training rows. In NanoTabICL, training rows are placed first, so

Define

where \(z_i\in\mathbb{R}^{d_\text{row}}\) is the row token passed to \(\text{TF}_\text{icl}\), and \(\operatorname{Embed}_\text{ICL}\) maps an observed target into \(\mathbb{R}^{d_\text{row}}\). The ICL transformer lets test rows attend to labeled training rows and outputs contextualized test-row states; the prediction head then maps those states to \(\hat{y}_i\), the predicted target for test row \(i\).

Why this design?

Now that the data path is defined, the design choice becomes clearer: TabICLv2 separates representation building from dataset-level prediction.

Column-wise and row-wise processing build representations. The column-wise stage relates the same grouped feature position across rows through induced summaries. The row-wise stage combines grouped feature positions into one row representation. Dataset-wise ICL then performs the final train-test interaction over row tokens.

The computational reason is that full attention over every row-feature token would make the dataset-level interaction much larger than necessary. The model would be reasoning over \(n\times m\) tokens when the final prediction problem is naturally row-level. Compression changes the final ICL input from a grid of feature tokens to a sequence of row tokens.

The representational reason is that the row tokens are not plain feature summaries. They have already passed through target-aware column-wise processing and row-wise aggregation. By the time \(\text{TF}_\text{icl}\) runs, each test row token can attend to labeled training row tokens that carry both feature context and target information.

At a glance

Implementation in NanoTabICL

NanoTabICL implements a compact educational version of the compression-then-ICL stack in the same order as the architecture above. Production TabICLv2 adds engineering details such as inference managers, caching/offloading, and reserved CLS slots, but the same three-stage structure is visible in NanoTabICLv2.forward.

After repeated feature grouping and the first target-aware embedding, the working tensor is called emb:

 (batch, rows, cols, embed_dim)

Here cols is the number of grouped feature positions in NanoTabICL. At this point, emb[:, :n_train] has already received the first target embedding, so the compression stack starts from the implementation counterpart of \(E_2\).

Row and column attention helper

The implementation snippets below rely on two thin wrappers around the same helper. The helper that makes row-wise and column-wise attention compact is TableAttnBase:

def row_attn(self, q, kv=None, **kwargs):
    n_batch, n_rows, n_cols, embed_dim = q.shape
    q, kv = (None if t is None else t.flatten(0, 1) for t in [q, kv])
    return self(q, kv, **kwargs).reshape(n_batch, n_rows, -1, embed_dim)

def col_attn(self, q, kv=None, **kwargs):
    return self.row_attn(
        q.transpose(1, 2),
        None if kv is None else kv.transpose(1, 2),
        **kwargs,
    ).transpose(1, 2)

row_attn flattens (batch, rows) into one larger batch dimension, so standard sequence attention can run across columns inside each row. col_attn swaps the row and column axes, reuses row_attn, and swaps the axes back. This is the implementation trick that lets the same transformer block operate over either axis of a 2D table without writing separate attention kernels for row-wise and column-wise attention.

Column-wise block

First, \(\text{TF}_\text{col}\) applies induced attention within each grouped feature position:

for block in self.col_blocks:
    emb = block.col_attn(emb, kv_max_idx=n_train)

The call to col_attn means: keep the grouped feature position fixed and attend across rows. Internally, it transposes the table so each grouped feature position becomes its own row sequence, applies the same sequence module on that axis, and then transposes back. For the attention module, one fixed grouped feature position is seen as:

(batch * cols, rows, embed_dim)

The argument kv_max_idx=n_train is the leakage-control detail. Inside TransformerBlock.forward, it slices the key/value sequence to the first n_train rows:

if kv_max_idx is not None: kv = kv[..., :kv_max_idx, :]

In the column-wise induced block, this means the learned inducing vectors summarize only training rows in the default no-leakage path. The original row tokens then attend back to those train-derived summaries. Test rows can receive context from labeled training examples, but they do not contribute to the column-wise context unless an implementation path explicitly enables embedding with test rows.

The block itself is InducedTransformerBlock, which implements induced attention:

self.tfm1 = TransformerBlock(embed_dim=embed_dim, num_heads=num_heads, ssmax=ssmax)
self.tfm2 = TransformerBlock(embed_dim=embed_dim, num_heads=num_heads)
self.inducing_vectors = nn.Parameter(0.02 * torch.randn(1, n_inducing, embed_dim))

kv = self.tfm1(self.inducing_vectors.expand(q.shape[0], -1, -1), q if kv is None else kv, kv_max_idx=kv_max_idx)
return self.tfm2(q, kv, q_max_idx=q_max_idx)

The first transformer call lets the learned inducing vectors query the training rows and produce a compact set of latent summaries in the default path:

training-row tokens -> n_inducing latent summaries

The second transformer call lets the original row tokens attend back to those summaries. The output still has shape (batch, rows, cols, embed_dim). So this block is not the final \(n\times m\) to \(n\) compression step; it is the column-wise contextualization step that prepares better tokens for row compression.

The full TabICLv2 implementation exposes an embed_with_test option for cases where one deliberately allows the column embedder’s inducing points to attend to train and test rows together. NanoTabICL follows the default no-leakage path shown above.

Row-wise block

Second, \(\text{TF}_\text{row}\) adds learned row-level \([CLS]\) tokens and attends across feature positions within each row:

emb = torch.cat([self.row_cls_tokens.expand(n_batch, n_rows, -1, -1), emb], dim=2)
for block in self.row_blocks[:-1]:
    emb = block.row_attn(emb)
emb = self.row_blocks[-1].row_attn(emb, q_max_idx=self.row_cls_tokens.size(-2))
emb = self.row_ln(emb).flatten(-2, -1)

Here row_attn keeps the row fixed and attends across the column axis. The prepended row_cls_tokens are learned summary positions. They are not labels and they are not additional input features; they are readout slots that can attend to the grouped feature tokens in the same row.

NanoTabICL prepends these CLS tokens with torch.cat, which gives the following shape transition:

The final row block is called with q_max_idx=self.row_cls_tokens.size(-2), which means the model only computes outputs for the CLS positions in that last row-attention pass. The grouped feature-token outputs are no longer needed after they have contributed to the CLS summaries.

That flattened size is icl_dim:

icl_dim = embed_dim * n_cls_cols

So the row-wise block is the implementation's main compression point:

(batch, rows, cols, embed_dim)
    -> (batch, rows, icl_dim)

The full tabicl implementation avoids the explicit torch.cat by reserving CLS slots during column embedding and then replacing those reserved slots with learned CLS tokens before row attention. In that implementation, the shape before row interaction is closer to (batch, rows, grouped_feature_positions + reserved_cls_slots, embed_dim). The mathematical role is the same: these positions are learned readout slots whose final outputs are concatenated into the row representation.

ICL block

Third, \(\text{TF}_\text{icl}\) performs in-context learning over row tokens. This is where NanoTabICL injects targets a second time:

emb[:, :n_train] += self.y_embed_icl(y[:, :, None])
for block in self.icl_blocks[:-1]:
    emb = block(emb, kv_max_idx=n_train)
emb = self.icl_blocks[-1](emb[:, n_train:], emb[:, :n_train])
return self.out_mlp(self.out_ln(emb))

At this point, before the second target injection, emb has shape:

(batch, rows, icl_dim)

The call self.y_embed_icl(y[:, :, None]) returns a tensor with shape (batch, n_train, icl_dim). Adding it to emb[:, :n_train] marks the training row tokens as labeled context examples. Test rows are left without a target embedding.

The intermediate ICL blocks again use kv_max_idx=n_train, so every row is updated by attending only to training rows as keys and values. In the final ICL block, NanoTabICL avoids computing outputs for training rows because only test predictions are needed:

queries:      emb[:, n_train:]   -> test rows
keys/values:  emb[:, :n_train]   -> training rows

So the final output contains predictions only for the test rows:

(batch, n_test, out_dim)

That last-block shortcut is a NanoTabICL simplification for the hands-on implementation. The architectural statement is broader: \(\text{TF}_\text{icl}\) uses labeled training row representations as context to produce test-row states, and the output head decodes those states into predictions.

Share

Summary

Compression-then-ICL is the part of TabICLv2 that turns a grid of target-aware feature tokens into row-level predictions. The column-wise transformer first lets each grouped feature position learn from labeled examples across rows. The row-wise transformer then uses learned \([\text{CLS}]\) tokens to compress each row from \(m\) feature-position tokens into a fixed-dimensional row representation. Finally, the ICL transformer operates over row tokens, adding target information to training rows and letting test rows attend to that labeled context.

The main idea is separation of roles: feature and row processing happen before dataset-wise prediction, so the final ICL stage does not need to attend over every cell-level token.

One detail this post has mostly treated as a black box is query-aware scalable softmax (QASSMax). The actual TabICLv2 architecture uses QASSMax in the induced column attention and the ICL transformer for long-context scaling; the label-flow story above is unchanged. The next post covers QASSMax, the attention-scaling mechanism TabICLv2 uses to preserve selective attention as the number of context samples grows.

Quiz

Take the quiz below to test your understanding, and share your answers and doubts in the comments. The difficulty increases from question 1 to question 10 in increasing order of the question number.

1. What are the three main stages in TabICLv2’s compression-then-ICL pipeline?

2. What does the tensor \(E_2\in\mathbb{R}^{n\times m\times d}\) represent before compression begins?

3. Why are target embeddings added to training-row feature tokens but not to test-row feature tokens?

4. In \(\text{TF}_\text{col}\), which axis is attended over for a fixed grouped feature position \(j\)?

5. What role do learned inducing tokens play in the column-wise Set Transformer block?

6. Why does the default no-leakage setting build column-wise summaries from training rows only?

7. How do the learned \([\text{CLS}]\) tokens in \(\text{TF}_\text{row}\) help convert an \(n\times m\times d\) tensor into row-level representations?

8. Explain why NanoTabICL injects target information twice: once before column-wise processing and once before the ICL block.

9. In the final ICL block, why can the implementation use test rows as queries and training rows as keys and values instead of computing outputs for every row?

10. Suppose \(\text{TF}_\text{icl}\) attended over every cell-level token instead of compressed row tokens. What computational and modeling tradeoffs would this create compared with the compression-then-ICL design?

But repeated feature grouping is still feature-only. It tells the model more about how columns sit next to other columns, but it does not yet tell the model which rows produced which outcomes. The next architectural step, target-aware embedding, adds that supervised signal for training rows.

The following figure shows the full TabICLv2 architecture. This post focuses on the target-aware embedding block, immediately after repeated feature grouping.

In this post, we start from the feature-only tensor \(E_1\), add observed targets only to training-row tokens, and leave test rows unlabeled.

Subscribe now

Target-aware embedding

Starting point: feature-only tokens \(E_1\)

Start with the feature matrix after preprocessing/normalization:

where \(n\) is the number of rows, \(m\) is the number of feature columns, and \(x_{ij}\) is the value of feature \(j\) in row \(i\). In the previous post, repeated feature grouping produced a feature-token tensor

containing one \(d\)-dimensional embedding for each row \(i\in\{1,\ldots,n\}\) and each group position \(j\in\{1,\ldots,m\}\). The token

therefore represents row \(i\) at grouped feature position \(j\). At this point the representation is still feature-only. It encodes the input values \(X\), including the local feature context introduced by repeated feature grouping, but it does not yet include the observed targets of the training rows.

The operation: add a target embedding to every training-row token

Target-aware embedding changes \(E_1\) into a target-aware tensor

by adding an embedding of the observed target \(y_i\) to each feature token in training row \(i\). This is an elementwise vector addition in the same \(d\)-dimensional token space.

To write the operation cleanly, let \(n_\text{train}\) be the number of training rows. In the code implementation, training rows are placed first, so

is the set of row indices whose targets are observed. The test rows are

whose targets must be predicted. For \(i\in\mathcal{I}_\text{train}\), \(y_i\in\mathcal{Y}\) denotes the observed target for row \(i\), where \(\mathcal{Y}\) is the target space. In classification, \(\mathcal{Y}\) is a finite set of class labels; in regression, \(\mathcal{Y}\subseteq\mathbb{R}\).

Define the target-aware embedding map

Then define a row-level vector

where \(\mathbf{0}_d\in\mathbb{R}^d\) is the zero vector. The target-aware representation is

where for a training row,

while for a test row,

For each training row \(i\), TabICLv2 computes one target vector and broadcasts it across all \(m\) feature tokens in that row. Test rows receive the zero vector instead. This boundary is essential: for test rows, \(y_i\) is the value the model must infer, so adding \(\operatorname{Embed}_\text{TAE}(y_i)\) would leak the answer.

Classification vs regression implementations

The embedding map depends on the prediction task. For classification, it maps discrete class labels to label vectors; for regression, it maps a scalar target to the token space.

For classification with \(K\leq K_{\max}\) classes, assume labels have been encoded as

In TabICLv2, \(K_{\max}=10\) for the pretrained small-class target encoder. The embedding map can be viewed as a learned lookup table

Here \(W_\text{cls}[k]\in\mathbb{R}^d\) is the target vector associated with class \(k\). The full TabICLv2 implementation realizes this with a one-hot-plus-linear layer, which is equivalent to selecting a learned vector for each supported class. NanoTabICL uses an nn.Embedding wrapper for the same lookup-table idea.

Tasks with more than \(K_{\max}=10\) classes need extra handling. The full TabICLv2 implementation uses mixed-radix ensembling for the target-aware column-embedding stage and hierarchical classification later in the ICL stage. That many-class machinery is outside this post’s small-class view; here we focus on the standard \(K\leq 10\) case.

For regression, where \(y_i\in\mathbb{R}\), the target embedding is a learned linear layer, which can be written as an affine map from the scalar target to the \(d\)-dimensional token space:

where \(a\) and \(b\) are learned vectors. In both classification and regression, the output of \(\operatorname{Embed}_\text{TAE}\) lives in \(\mathbb{R}^d\), the same space as the feature token \(E_1[i,j]\). That is why direct addition is well-defined.

Why add to tokens instead of appending a target column?

If the target were appended as a separate token, the feature-token count would change from \(m\) to \(m+1\). Target-aware addition keeps the shape fixed:

The label information is therefore available at every grouped feature token before the column-wise and row-wise transformer stages, but the architecture still processes \(m\) feature positions rather than \(m+1\). This is different from the TabPFNv2-style target-column appending, where the target is represented as an additional column alongside the feature columns.

Why this helps before the transformers run

Keeping the shape fixed is the computational benefit. The representational benefit connects back to representation collapse, but from a different angle than repeated feature grouping.

The next stage is column-wise processing: for each grouped feature position \(j\), the transformer processes that position across many rows. Because training rows now carry embedded targets, the column-wise transformer can see feature patterns together with observed outcomes.

Let \(Y\) denote the target random variable. In the previous post, we considered two feature random variables \(X_a\) and \(X_b\) with similar marginal distributions,

where \(P_{X_j}\) denotes the marginal distribution of feature \(X_j\). Similar marginals do not imply similar predictive roles. The two features can still have different target relationships:

for some values \(x\). Repeated feature grouping helps with the feature-feature part of this problem: a token is not built from one isolated scalar. Target-aware embedding helps with the feature-target part: during column-wise processing, training-row tokens carry both a feature representation and the observed outcome for that row.

The important nuance is that target-aware embedding does not distinguish feature positions within the same row by itself. The same vector \(\operatorname{Embed}_\text{TAE}(y_i)\) is added to every grouped feature token in row \(i\). The feature-position information still comes from \(E_1[i,j]\); the target-aware term supplies row-level supervised context.

This row-level supervised context becomes useful across examples. For training rows \(i\) and \(r\) with different targets,

So, for each grouped feature position \(j\), the column-wise transformer receives examples of the form “this feature pattern occurred in a row with this target.” Across many rows, that makes feature-target association available earlier than it would be in a purely feature-only embedding.

Informally, for a training row with feature vector \(x_i=(x_{i1},\ldots,x_{im})\), the representation changes from a feature-only encoding to a feature-target encoding:

Here \(E_1[i,\cdot]\) and \(E_2[i,\cdot]\) denote all grouped feature tokens for row \(i\), while \(\phi\) and \(\psi\) are informal names for feature-only and feature-target representation functions. The feature-target encoding applies only to training rows; test rows still carry \(E_1\) at this stage. This is the first target injection in TabICLv2. A second target embedding is added later, after row aggregation, before dataset-wise ICL.

Implementation in NanoTabICL

NanoTabICL implements target-aware embedding with two pieces: a task-dependent target embedder and one masked in-place addition to the training rows.

During initialization, the target embedder depends on whether the model is configured for classification or regression:

self.y_embed_in = (
    ClassEmbedding(max_classes, embed_dim)
    if max_classes > 0
    else nn.Linear(1, embed_dim)
)

For classification, ClassEmbedding is a learnable lookup table:

class ClassEmbedding(nn.Embedding):
    def reset_parameters(self) -> None:
        nn.init.uniform_(self.weight, -1/math.sqrt(self.num_embeddings), 1/math.sqrt(self.num_embeddings))

    def forward(self, y: torch.Tensor) -> torch.Tensor:
        return super().forward(y.squeeze(-1).long())

The call to long() turns labels such as 0, 1, or 2 into embedding-table indices. The custom initialization matches the nn.Linear weight initialization scale. For regression, nn.Linear(1, embed_dim) implements the affine scalar-to-vector map \(a y_i+b\).

The actual target-aware update is one line in forward:

emb[:, :n_train] += self.y_embed_in(y[:, :, None, None])

The shape logic is the key to reading this line:

PyTorch broadcasting adds that target vector across all cols grouped feature positions in the corresponding training row. This is the code version of:

The masking boundary is the important safety rule. NanoTabICL never indexes emb[:, n_train:] in this addition, so test rows remain feature-only at this stage:

training rows: feature token + target embedding
test rows:     feature token only

That single slice, :n_train, is what prevents target leakage in the implementation.

Share

Summary

After target-aware embedding, training-row feature tokens carry supervised context, while test-row feature tokens remain unlabeled. TabICLv2 gets this effect by adding one target embedding to every grouped feature token in each labeled row, without changing the \(n\times m\times d\) tensor shape produced by repeated feature grouping.

The result is \(E_2\): a target-aware feature-token tensor ready for the compression-then-ICL pipeline. The next post covers how TabICLv2 compresses these tokens into row representations, adds target information again at the row-token level for labeled rows, and then performs dataset-wise in-context learning.

The following figure illustrates the architecture of TabICLv2. This post covers repeated feature grouping. Later posts will cover target-aware embedding, column/row transformers, QASSMax, and the prediction heads.

Subscribe now

Repeated feature grouping

Let’s assume a dataset with the following properties and notation:

• \(n\) rows and \(m\) columns,

• feature random variables denoted as \((X_1,\ldots,X_m),\)

• a target random variable denoted as \(Y\),

• the value of feature \(j\in\{1,\ldots,m\}\) in row \(i\in\{1,\ldots,n\}\) denoted as \(x_{ij}\), and

• the full \(j\)-th column denoted as \(x_{\cdot j}=(x_{1j},\ldots,x_{nj})\).

TabICLv2 first asks how to represent the features in a table.

The problem: similar columns, different roles

In tabular data, two features can have similar marginal distributions,

where \(P_{X_j}\) denotes the marginal distribution of feature \(X_j\). But similar marginals do not imply similar predictive roles. For example, days_since_signup and days_since_last_purchase may both be positive, right-skewed variables, but their relationships to churn can be very different. This creates a representation problem: before the model can reason over feature interactions, its initial feature embeddings must preserve enough information to tell features apart.

Independent column embedding collapses

TabICL-style feature embedding can process each feature with the same encoder. A simplified way to write such an independent column encoder is

where \(d\) is the embedding dimension. The column vector \(x_{\cdot j}\) is mapped to a feature representation

Each column is embedded on its own, without looking at neighboring columns. But if \(\phi\) mostly sees each feature through its own values, then two columns with similar empirical distributions may be mapped to similar embeddings:

Here \(\|\cdot\|_2\) is Euclidean distance and \(\cos(e_a,e_b)\) is cosine similarity. This is the representation-collapse problem: distinct features become nearly indistinguishable in representation space even though their semantics, correlations, or target relationships differ.

The issue here is that the marginal distribution is not the entire identity of a feature. In reality, a feature is also characterized by its joint behavior with other features and with the target. Individual feature embedding can underuse this context. Repeated feature grouping addresses the feature-feature part of that context; the next post covers how target information enters the representation.

Grouping neighboring columns can lead to information loss

TabPFNv2 and TabPFN-2.5 mitigate this collapse by grouping multiple columns into single tokens. But, although grouping gives each feature token some neighboring-feature context, it also reduces the number of effective feature tokens, which can discard fine-grained feature information. The tradeoff is that one token now represents several original columns, so the model has fewer distinct feature positions to work with.

Downstream attention layers then operate on coarser tokens. If fine-grained feature identity is weakened early, later layers have less direct information with which to distinguish individual columns. TabICLv2 uses repeated feature grouping to keep the contextualization benefit while preserving \(m\) effective feature positions.

TabICLv2’s fix: grouping with circular shifts

For a table with \(m\) feature columns, TabICLv2 creates \(m\) groups with \((0,1,3)\) offset pattern relative to every feature with circular shifts. This is why the method is called repeated feature grouping.

To understand this mathematically, let us define the mapper

so \(\rho(t)\) maps any integer \(t\) back into the column index set \(\{1,\ldots,m\}\). Then, the group anchored at feature \(j\) contains columns

For example, for \(m=5\) columns:

Here, feature 1 appears as anchor in group 1, as offset \(+1\) in group 5, and as offset \(+3\) in group 3.

For row \(i\), the grouped input for position \(j\) is

where the vector \(g_j(i)\in\mathbb{R}^3\) contains three scalar feature values from the same row. For example, \(g_1(i)=(x_{i,1},x_{i,2},x_{i,4})\) takes three values from column \(j\), \(j+1\), and \(j+3\) of row \(i\). Each group is encoded by a shared linear map

producing

The resulting tensor \(E_1\in\mathbb{R}^{n\times m\times d}\) contains one \(d\)-dimensional embedding for each row \(i\) and each group position \(j\). Put simply, one shared linear map turns each 3-value group into a \(d\)-dimensional token.

With the shift pattern \((0,1,3)\), for \(m \geq 7\) columns, no pair of columns appears together in more than one group. This gives each feature several contextual views without repeatedly coupling the same feature pairs. The result is a representation that helps prevent feature symmetries between neighboring columns while preserving \(m\) effective feature positions.

Implementation in NanoTabICL

In NanoTabICL, repeated feature grouping happens at the start of NanoTabICLv2.forward. The relevant model parameters are set during initialization:

 self.feature_group_size = feature_group_size
 self.x_embed = nn.Linear(feature_group_size, embed_dim)

The default feature_group_size is 3, so self.x_embed is a shared linear map from a 3-value feature group into the token dimension. This corresponds to the mathematical map

The grouping itself is implemented by indexing shifted versions of the column axis:

 idxs = torch.arange(n_cols, dtype=torch.long, device=x.device)
 x = torch.stack(
     [x[:, :, (idxs + (2 ** i - 1)) % n_cols]
      for i in range(self.feature_group_size)],
     dim=-1,
 )
 emb = self.x_embed(x)

The expression (2 ** i - 1) is the code version of the offset pattern. With feature_group_size=3, the loop uses:

The modulo operation % n_cols is the circular wraparound. If idxs = [0, 1, 2, 3, 4], then the offset 3 gives [3, 4, 0, 1, 2], so the last columns wrap back to the beginning.

The shape transition is the main thing to notice:

So NanoTabICL keeps the same number of column positions, cols, but each position has already looked at a small circular group of neighboring columns. That is the implementation counterpart of preserving \(m\) effective feature slots while giving each slot local feature context.

Share

Summary

Repeated feature grouping addresses a core weakness of independently embedding tabular features: columns with similar value distributions can become hard to distinguish. TabICLv2 groups each feature with shifted companion features, giving the model multiple contextual views while keeping the number of effective feature positions unchanged. The next post covers target-aware embedding, the step where TabICLv2 injects observed targets into the feature representations of training rows.

Subscribe now

Yesterday, I came across the ML for Trading book by Stefan Jansen through a note posted by him on Substack. It caught my attention because I had been looking for something like it to learn the field and was also trying to find coding resources to build SAM upon. I found both in this one.

The following picture shows a screenshot from the official website of the book. It covers much of the classical ML material that I was planning to discuss in this newsletter already in addition to tabular foundation models. Moreover, it covers many of the production and workflow pieces I wanted to build into SAM to make it a quant research and engineering platform for personal and educational use.

The book also comes with several more add-ons that are highly relevant to today's AI-dominant workflows.

The part I found immediately useful was the following set of libraries. The core libraries I looked at are MIT licensed and already contain a lot of wiring that I would otherwise have to do on my own, and I would certainly either get them wrong due to lack of domain knowledge or make a lot of mistakes.

It’s likely to be a good idea to not reinvent the wheel. So, today, I rebuilt SAM around ML4T libraries.

The implementation details will change as SAM develops further, but the architectural decision is the durable part. For now, it’s enough to know that SAM is a thin orchestration layer on top of ML4T: SAM orchestrates; ML4T computes, validates, backtests, and executes. SAM owns the repeatable workflow layer: configuration, CLI routing, strategy selection, pipeline sequencing, manifests, promotion gates, and operator controls. ML4T owns the specialized trading machinery: market data access, feed specifications, feature engineering, signal diagnostics, model training surfaces, backtesting, live execution, broker wrappers, and risk guards.

If you are interested in looking at the code and playing with it, you can find today’s version here. The current repo is still small, but the important change is visible in the structure: configs, pipelines, manifests, live runners, ops tools, and tests now sit around ML4T rather than replacing it. The screenshot below is SAM after the rebuild.

This version is still early, but it gives SAM the right shape for further development. Check out the repo and let me know what you think.

Here’s a video describing how I came to know about tabular foundation models, why I talk exclusively about them and not about other developments in the broad field of AI, and what I will be doing in my videos.

Subscribe to the channel to receive updates and share with other data, AI, and finance professionals who might be interested.

In my posts, I have tried my best to make the examples as close to reality as possible because I am interested in applying these lessons, conceptually and programmatically, to real-world problems.

I am happy with the depth my posts cover, and I have several more planned. However, after P22, I took a strategic break to reflect on the quality of the posts and code so I could improve future posts. One of the things I wanted to do was extract reusable concepts and save them in a self-contained page for future reference.

But concepts by themselves are of limited use. That led me to think it would be better to have a place where I could practice them regularly and solve a real problem at the same time.

This week, I took a step toward that and started building a quantitative research and engineering platform to analyze financial markets continuously and in an increasingly sophisticated manner over time. I am calling it SAM, and it is available under the Apache 2.0 license.

README:

SAM is an open-core personal quant research and engineering platform and is a companion to the DSAIEngineering Newsletter. The workflows and primitives described in the newsletter are implemented in SAM. Currently, it focuses on production-style US-listed ETF allocation, volatility/risk scoring, and future US cross-sectional equity ranking workflows. More functionality will be integrated from the newsletter into SAM to make it more capable over time.

The idea for creating this platform has been on my mind since last year. I always envisioned using it in combination with Perplexity Finance: SAM could do comprehensive data analysis, modeling, simulations, and related quantitative workflows; Perplexity Finance could do broader market and geopolitical analysis; and together, they could be used to develop investment or trading strategies that a person could test with paper money to learn the process and test their understanding of the field.

Currently, SAM is capable of generating a daily report with the following contents:

# Daily SAM Brief - 2026-05-21

Research only. Not investment advice.

## Data Freshness and Validation

- Price last date: 2026-05-20
- Data freshness days: 1
- Validation failures: 0
- Model status: reused

## SPY Risk Regime

- Risk level: normal
- Realized volatility 20d: 0.1058
- Threshold: 0.2074
- Volatility / threshold: 0.5103
- Drawdown 20d: -0.0106
- MA distance 20d: 0.0165

## ETF Ranking

| score_rank | symbol | prediction | selected |
| --- | --- | --- | --- |
| 1 | SLV | 0.1835 | True |
| 2 | GLD | 0.0254 | True |
| 3 | SPY | 0.0237 | True |
| 4 | TLT | 0.0200 | True |
| 5 | EFA | 0.0187 | True |
| 6 | XLU | 0.0164 | False |
| 7 | DIA | 0.0146 | False |
| 8 | XLF | 0.0126 | False |
| 9 | LQD | 0.0123 | False |
| 10 | HYG | 0.0098 | False |

## Target Research Weights

| symbol | weight | score | score_rank |
| --- | --- | --- | --- |
| SLV | 0.2000 | 0.1835 | 1 |
| GLD | 0.2000 | 0.0254 | 2 |
| SPY | 0.2000 | 0.0237 | 3 |
| TLT | 0.2000 | 0.0200 | 4 |
| EFA | 0.2000 | 0.0187 | 5 |

## Turnover and Cost Diagnostics

- Previous weight date: 2026-05-20 00:00:00
- Turnover: 0.4000
- Estimated cost: 0.0002
- Gross exposure: 1.0000
- Net exposure: 1.0000
- Turnover breach: False

## Limitations

- Public market data can be revised and may differ from institutional data.
- Costs are simple basis-point estimates, not a market-impact model.
- The allocation view is a daily research snapshot for a monthly-horizon ETF workflow.
- TabPFN and TabICL are not required for this CPU-first daily brief.

Over time, I want SAM to perform more sophisticated analyses using traditional quantitative finance methods and ML methods.

If you are interested in playing with it, I invite you to check out the repository: github.com/msaharan/sam, and let me know what you think.

In P21, I made that first workflow more demanding. The universe expanded from 9 ETFs to 25 ETFs, the target changed from top 3 of 9 to top 5 of 25, the base positive rate fell from about 0.333 to 0.200, the return convention moved from close-to-close to next-open-to-next-open, and the main feature set removed static ticker identity features.

That follow-up made the workflow more credible, but it also made the remaining gaps easier to see. The liquidity diagnostic path was thin. Feature sensitivity was mostly a lightweight linear-model check. Alternative objectives were still closer to audits of the one-month score than to retrained objective-specific experiments. The portfolio section had improved, but it still needed a cleaner distinction between score quality and allocation quality.

So this post is the final follow-up for this tactical-allocation mini-series. I keep the same 25-ETF, top-5, next-open-to-next-open problem from P21, but I strengthen the experimental design around the remaining issues that are addressable in a public notebook. I leave three limitations outside today’s scope: TabICL remains disabled, the dataset is still public rather than institutional point-in-time data, and tabular foundation model (TFM) embeddings are not tested. Those are real limitations, but this post is about making the rest of the testbench more complete.

I am treating this as a learning-in-public research notebook and worked example, not as a claim to have the final word on tactical allocation or tabular foundation models. I am learning as I go, and I want that to make the write-up more careful, not more tentative. The value of the post is in making the workflow inspectable enough that TFM labs, ML practitioners, data and AI teams, and quant readers can see the setup, the potential benefits, the tradeoffs, and the limitations clearly.

The empirical result is mixed in an informative way. TabPFN has the best completed row-level ranking result on the main target in this run. The TabPFN calibration-base scorer has holdout Average Precision of 0.302, and the final direct TabPFN scorer has AP 0.298, against a 0.200 base rate. XGBoost is above base rate but lower on this row-level view, with AP 0.252. However, the highest headline portfolio Sharpe is not TabPFN. It is the 12-month momentum top-k rule, with Sharpe 1.11. TabPFN portfolios are competitive and often close to the top, but they do not settle the allocation question against compact finance rules in this workflow.

For me, the conclusion is methodological rather than promotional. Direct TabPFN scoring still looks useful after the P21 workflow is strengthened, while the best deterministic allocation rule remains highly competitive at the portfolio layer. That is why this kind of notebook is worth building: it separates the modeling question, “can a pretrained tabular model rank asset-month rows?”, from the allocation question, “does that score become the best portfolio rule after turnover, risk, and portfolio construction are included?”

The equity-curve figure previews the central tension in the post. The learned scores are useful, but the portfolio paths are close enough that the allocation layer has to be evaluated on its own terms. I discuss the Sharpe, drawdown, turnover, and uncertainty details after setting up the notation and experimental changes.

You can find the notebook in my GitHub repository here or you can clone it directly on Kaggle.

Minimal notation for this follow-up

P20 gives the full formulation, and P21 restates the notation for the broader universe and next-open target. I will only repeat the pieces needed to read this post, so the later tables and diagnostics have a common language.

Each row is an asset-month pair \((i,t)\). The index \(i\) denotes an ETF, and \(t\) denotes a monthly signal date. The information set available at the signal date is \(\mathcal{F}_t\). The feature vector is:

where \(\phi_i(\cdot)\) is the feature-generation process for asset \(i\). In this notebook, the main features include trailing returns, volatility, downside volatility, drawdown, moving-average distance, beta, volume and dollar-volume proxies, high-low range proxies, VIX features, Treasury features, and other public market-state features. The main feature policy is identity-ablated, meaning static ticker identity and asset-group metadata are excluded from the headline predictive matrix.

The target uses the next-open-to-next-open convention introduced in P21. Let \(O^+_{i,t+1}\) denote the adjusted first tradable open after signal month \(t\), and let \(O^+_{i,t+2}\) denote the adjusted first tradable open after the next month. The one-month forward return used for the main target is:

The label is:

Here, \(\mathbf{1}\{\cdot\}\) is the indicator function. It equals 1 when the condition inside the braces is true and 0 otherwise. Let \(N_t\) be the number of assets with a usable forward return in month \(t\), and let \(K\) be the number selected as positives. In the full holdout months, \(N_t=25\) and \(K=5\), so the monthly positive-class rate is:

A scorer then produces:

where \(s_{i,t}\) is the model score and \(\mathcal{D}_{\text{train}}\) is the labelled training or context data available to the scorer. This is a top-group membership score, not a price forecast. A high score does not mean the asset must go up next month. It means the model ranks the asset as more likely to belong to the next-month top group within the available ETF cross-section.

The supervised table is familiar, but TabPFN and XGBoost use it differently. XGBoost learns task-specific parameters from the labelled allocation table. Conceptually, it selects a scoring function from a model family \(\mathcal{G}\):

Here, \(\mathcal{G}\) is the candidate family of scoring functions, \(f\) is one candidate scorer, \(\ell\) is the loss function, and \(\hat{f}\) is the fitted scorer selected from that family.

Direct TabPFN uses a pretrained tabular model and labelled rows as task context:

where \(\theta\) denotes the pretrained TabPFN parameters, \(x_\ast\) is a query row, \(y_\ast\) is its unknown label, and \((X_{\text{context}}, y_{\text{context}})\) is the labelled context. This is the TFM use case I am testing: whether a pretrained in-context tabular learner can produce useful allocation scores under the same chronology, leakage discipline, and portfolio diagnostics used for the classical supervised ML baseline.

With that notation in place, the rest of the post is about what changed in the testbench and how those changes affect the interpretation.

What is new in this experiment

Liquidity and spread are now part of the feature and diagnostic story

P21 added portfolio stress diagnostics, but the liquidity path still needed work. A strategy that looks good on returns alone can become less interesting once turnover, capacity, trading frictions, and liquidity are considered. Even in a public ETF notebook, I want the workflow to start carrying these concerns through the experiment.

Today’s notebook adds dollar-volume and high-low range proxy features to the model frame. This is both a feature-set change and a diagnostic improvement. The final main feature matrix grows to 1,060 features after retained numeric columns and missingness indicators. That matters because P22 is not only rerunning P21 with extra reports; it is also giving the models additional liquidity-related inputs.

The portfolio diagnostics now include average weighted ADV, where ADV means average daily dollar volume, a high-low range proxy reported in basis points, and turnover-weighted participation at a USD 1 million notional portfolio size. These are proxies, not institutional execution data. The high-low proxy is based on intraday range and should not be read as a quoted bid-ask spread. The participation calculation is not a market-impact model. But the diagnostic is now an interpretable liquidity screen rather than a placeholder.

Feature-policy sensitivity is now a model-family check

P21 used Logistic Regression as a lightweight sensitivity diagnostic. That was useful, but it did not answer whether the feature policy mattered similarly for TabPFN and for a tree-based classical tabular model.

Today’s notebook reruns direct TabPFN and a fixed GPU XGBoost sensitivity model across five feature policies:

• full;

• ticker-ablated;

• identity-ablated;

• metadata-only;

• strict-time-series.

In this naming scheme, “full” keeps the complete candidate feature set, “ticker-ablated” removes ticker dummy features, “identity-ablated” removes ticker identity and asset-group metadata, “metadata-only” keeps only those static identity and metadata fields, and “strict-time-series” keeps the non-identity time-series and market-state features. In this run, the strict-time-series and identity-ablated matrices end up with the same retained feature count after filtering.

This is a fixed XGBoost sensitivity model rather than a second full XGBoost search. It uses a fixed GPU XGBoost configuration for the feature-policy comparison, while the main XGBoost row still uses the searched model. That is a reasonable compromise for this notebook because the purpose is to test whether the main identity-ablated conclusion is fragile, not to run a full nested model-selection study for every feature variant.

Alternative objectives are retrained, not only audited

P21 made a useful start on alternative objectives, but the more direct version is to retrain and evaluate models on those objectives. Today’s notebook does that for three targets:

• 1-month benchmark-relative outperformance versus SPY;

• 3-month top-k membership;

• 6-month top-k membership.

For the benchmark-relative target:

Here, \(R^{\text{open}}_{\text{SPY},t+1}\) is SPY’s one-month forward open-to-open return under the same convention. This benchmark-relative target uses SPY as the label benchmark. Later, the portfolio benchmark-relative table uses a 60/40 SPY/TLT benchmark for active-risk diagnostics; those are related but distinct benchmark choices.

For an \(h\)-month top-k target, where \(h\) is the horizon length in months:

Here, \(m\) indexes the monthly return offsets inside the horizon. The \(h\)-month label then applies the same top-\(K\) cross-sectional rule to \(R^{(h)}_{i,t}\), with \(K=5\) in this experiment. This matters because a model trained to predict one-month top-k membership is not necessarily the right model for a six-month target or a benchmark-relative target.

The notebook also changes how multi-horizon results are interpreted. Overlapping 3-month and 6-month forward returns are not compounded as if they were independent monthly portfolio returns. They are reported as horizon-aware score-to-return diagnostics. This is less flashy, but it is conceptually cleaner.

The portfolio layer is treated as a separate object

The score \(s_{i,t}\) is not the portfolio. A portfolio requires a selection rule, a weighting rule, turnover accounting, transaction costs, and some view of risk. The same row-level score can look better or worse after this translation.

Let \(w_{i,t}\) be the portfolio weight assigned to asset \(i\) after the signal at month \(t\). In the main top-k equal-weight portfolio, selected assets receive weight \(1/K\), unselected assets receive weight 0, and the portfolio return is:

with turnover:

and net diagnostic return:

where the summation is over the available assets, \(R^p_{t+1}\) is the pre-cost portfolio return, \(\tau_t\) is the traded-weight turnover implied by the change in portfolio weights, \(\widetilde{R}^p_{t+1}\) is the cost-adjusted diagnostic return, and \(c=0.0005\), corresponding to 5 basis points per unit of traded weight. Under this convention, opening a fully invested long-only portfolio from cash has \(\tau_t=1\), and replacing the entire portfolio with non-overlapping long-only holdings has \(\tau_t=2\).

Today’s notebook also adds benchmark-relative diagnostics, tax-drag sensitivity, turnover constraints, liquidity and participation proxies, drawdown summaries, and a trailing-covariance ex ante risk proxy. These do not make the notebook a production allocator. They keep the interpretation from resting only on a row-level model metric.

The next section walks through the resulting data frame, score quality, robustness checks, and portfolio translation in that order.

Code discussion and experimental results

Dataset and chronological splits

I start with the data shape because the rest of the metrics depend on this cross-section and chronology. The final model frame contains 6,059 asset-month rows across 243 months and 25 ETFs. The holdout period contains 1,875 rows across 75 months, from January 2020 through March 2026. The holdout positive rate is 0.200 because the target selects five assets out of twenty-five in each full holdout month.

The ETF universe is:

SPY, QQQ, DIA, IWM, EFA, EEM, TLT, IEF, SHY, LQD, HYG, GLD, SLV, DBC, VNQ, IYR, XLB, XLE, XLF, XLI, XLK, XLP, XLU, XLV, XLY.

This is still a public ETF universe, not an institutional production universe. It is broad enough to include US equity styles, international equity, rates, credit, commodities, real estate, and US sectors, while remaining small enough that the full notebook is inspectable. The rows are then split chronologically as follows:

Chronological split matters because random splits in market data can leak future regimes into model selection and make a score look more stable than it would be in a real research process. Feature filtering and imputation are fitted on the model-selection history before being applied to calibration and holdout rows.

Main row-level score quality

The first empirical question is row-level: can a scorer separate future top-five asset-month rows from the other rows in the holdout set? Because the holdout label selects 5 assets out of 25 each month, a non-informative scorer has an Average Precision reference level of 0.200.

I keep the same row-level metrics used in P20 and P21. Average Precision summarizes precision-recall ranking quality, ROC AUC summarizes pairwise positive-versus-negative ranking, Brier score measures squared probability error, log loss penalizes confident wrong probability estimates, and ECE means expected calibration error, computed here with 10 quantile bins.

For comparability, the TabPFN runs use model version 2.6, the same model line used for P20 and P21. A newer TabPFN version was released while this post was being prepared, so pinning the version keeps the experiment from mixing model-version change with testbench change.

There are also three learned-score variants to keep separate. “Direct” means the final scorer fitted or contextualized on all pre-holdout rows and then evaluated on holdout. “Calibration base” means the scorer excludes the calibration-window labels; this gives a cleaner object for calibration-window evaluation and for checking what changes when the calibration window is not used as labelled context. “Calibrated” means a sigmoid calibration layer was fitted on the calibration window. The calibration-base score series is a distinct diagnostic series, not a separate model family. The deterministic rules are raw ranking scores, not calibrated probability models, so Brier score, log loss, and ECE are not reported for those rows.

The main row-level result is that TabPFN has clear lift over the 0.200 base rate. The calibration-base TabPFN row has AP 0.302, and direct TabPFN has AP 0.298. I do not treat that small difference as a meaningful contest between two TabPFN variants; the important point is that both are materially above the non-informative reference.

XGBoost is also above base rate, but lower than TabPFN on AP and ROC AUC in this run. The momentum rules are close to XGBoost on AP, which reinforces the value of deterministic finance baselines even before the portfolio layer is introduced.

The next two figures show the row-level ranking and calibration diagnostics visually, but I keep them to a small set of headline curves for readability. The tables carry the fuller variant comparison, including TabPFN calibration base.

The precision-recall figure should be read against the 0.200 base-rate reference. TabPFN does not need to produce a perfect ranking to be useful. It needs to keep the ranked list above the non-informative reference in the part of the list used for top-five selection.

Calibration is a separate question, so I read it from the calibration curve and the probability-error columns rather than from AP alone.

The calibration result tells a different story from the ranking result. XGBoost calibrated has a better Brier score and expected calibration error than XGBoost final, but it gives up AP. TabPFN direct has stronger ranking quality and reasonable calibration. This is a common supervised ML tradeoff: a score can become more probability-like without becoming a better top-k ranker.

The XGBoost calibration curve can look visually odd because the uncalibrated XGBoost probabilities are poorly scaled for this holdout. That is not the same as an invalid ranker. The score sanity checks show valid probabilities with no NaNs and no values outside \([0,1]\), and the XGBoost final row still has AP 0.252 versus the 0.200 base rate. The issue is probability reliability: the uncalibrated XGBoost score ranks some rows usefully, but its probability scale is not trustworthy. The sigmoid-calibrated XGBoost row fixes much of that probability-scale problem, as shown by the lower Brier score and ECE, but its AP falls to 0.240.

Feature-policy sensitivity

Before translating scores into portfolios, I first check whether the main row-level result depends on a fragile feature policy. The main run is identity-ablated, so the model does not receive ticker identity or asset-group metadata in the headline feature matrix. The sensitivity study asks whether this choice changes the result sharply.

The identity-ablated TabPFN result does not appear to depend heavily on that feature policy. Full features improve TabPFN AP only slightly, from 0.298 to 0.302. That suggests the main conclusion is not just a static-identity artifact.

The metadata-only result is still important. Metadata-only TabPFN reaches AP 0.290, and metadata-only fixed XGBoost reaches AP 0.282. This does not mean metadata-only is a sufficient allocation strategy. It means persistent asset-class structure is meaningful in this fixed ETF universe. That is worth noticing rather than treating feature design as an implementation detail. Feature design is part of the scientific result.

Objective-specific retraining

The next sensitivity is the target itself. The objective-specific reruns ask whether the models still find useful structure when the label changes. This is more direct than the P21 audit because the models are retrained for the alternative labels.

The benchmark-relative target has a higher base rate, so AP 0.466 is a modest lift above 0.422. The 3-month and 6-month targets are more encouraging because their base rates remain 0.200 and TabPFN direct reaches AP 0.325 and 0.337. The table above shows the best holdout scorer for each objective, which is why calibration variants are not listed there unless they are the best row-level scorer. The table below is different: it shows selected score-to-return diagnostics, so it includes calibration-base variants when they produce a relevant portfolio diagnostic.

The return columns in the next table depend on the objective. For the benchmark-outperform rows, selected and universe horizon returns are SPY-relative one-month returns, and active horizon return is selected SPY-relative return minus universe SPY-relative return. For the 3-month and 6-month top-k rows, selected and universe horizon returns are raw cumulative open-to-open returns over the stated horizon.

The horizon-aware score-to-return diagnostics are:

These are not independent tradable monthly portfolio returns. They are horizon-aware diagnostics of selected forward returns. That is the right interpretation because 3-month and 6-month forward windows overlap. The positive active horizon returns therefore say that the selected baskets had better subsequent horizon returns than the universe average in this historical holdout; they do not define an immediately compounding monthly strategy.

Monthly cross-sectional ranking

After the row-level robustness checks, I return to the main one-month target and ask how the scores behave inside each monthly cross-section. Row-level AP pools all holdout rows together, but tactical allocation is applied month by month. The monthly rank diagnostic asks whether high scores line up with high next-month returns inside each monthly cross-section.

For month \(t\), the Spearman information coefficient is:

The table below reports the mean and median of this monthly rank correlation, the top-k hit rate, and selected-minus-universe returns. The top-k hit rate is the fraction of selected assets that actually land in the realized top-five group, so a non-informative top-five selection would be expected to sit near 0.20 in a 25-asset month.

This view supports the TabPFN result, but it also keeps the interpretation grounded. TabPFN calibration-base and direct TabPFN have positive monthly rank diagnostics and positive active selected returns. The 12-month momentum rule is very close, with the highest mean IC in this table. XGBoost final is positive, but its allocation translation is less favorable than its row-level AP alone might suggest.

The low-volatility rule has a low result under this target in this holdout. I read that as period-specific. The 2020-2026 holdout includes the COVID crash, a rapid recovery, an inflation and rate shock, and a strong growth-equity period. A defensive rule can look less favorable when the target rewards relative top-five returns in a risk-on or momentum-dominated path.

Portfolio translation

The monthly ranking view is still not the same as a portfolio. The main portfolio diagnostic converts each scorer into a monthly top-five equal-weight portfolio and subtracts 5 basis points per unit of traded-weight turnover. This is not a production backtest. It is a controlled score-to-allocation diagnostic. CAGR means compound annual growth rate. Sharpe is computed from annualized mean monthly net return divided by annualized monthly volatility, using a zero risk-free-rate simplification, so it does not have to equal CAGR divided by annual volatility exactly.

Because the calibration-base score series is distinct, I also include its portfolio translation when it is part of the headline strategy set. I do not draw every such variant in the PR and calibration figures because that would make the figures harder to read; the tables are the authoritative place for the variant-level numbers.

This is the key table for interpreting the experiment because it prevents the row-level metric from becoming the whole headline. TabPFN has the best row-level ranking result, but the 12-month momentum rule has the highest portfolio Sharpe point estimate. TabPFN calibration-base has the highest final equity in this subset, but it also has higher volatility and a larger drawdown than the 12-month momentum rule.

This is a plausible finance result, not a warning sign by itself. In ETF allocation, deterministic rules such as 6-month or 12-month momentum are meaningful baselines. They are compact expressions of persistent return continuation effects that many tactical-allocation systems compare against after costs, turnover, and risk are included. A supervised model or TFM score can improve row-level ranking and still not translate into the best portfolio result if the score creates more turnover, selects a more volatile basket, or concentrates in assets with less favorable drawdown timing.

This is not a contradiction. The row-level classification problem and the portfolio problem are connected but not identical. A model can identify more top-group rows overall and still produce a less attractive portfolio after monthly selection, turnover, risk concentration, and drawdown enter the picture.

Portfolio realism and uncertainty

The score and allocation results need to be checked against broader portfolio diagnostics. The benchmark-relative view compares strategies against a 60/40 SPY/TLT benchmark. This is a portfolio-reporting benchmark, not the SPY benchmark used in the benchmark-outperformance label above. Annual active return is the annualized strategy-minus-benchmark return. Tracking error is the annualized volatility of that active return, and information ratio is annual active return divided by tracking error. This is not benchmark-relative optimization, but it is closer to how allocation results are often discussed in practice.

The liquidity proxy also now carries useful information. I include both TabPFN direct and TabPFN calibration-base here because they are both headline portfolio strategies in the main portfolio table, and liquidity is a property of the resulting traded weights rather than only a property of the model family. The spread-proxy column is the weighted high-low range proxy reported in basis points, not an estimate of realized transaction cost:

At the USD 1 million diagnostic notional, the participation values are small. That does not prove the strategies are scalable. It says the notebook now has a working path for asking the capacity question instead of ignoring it. A real capacity study would need actual bid-ask quotes, order-book depth, creation-redemption mechanics, fund-specific spreads, and market-impact modeling.

The trailing-covariance ex ante risk proxy is also populated:

This is a proxy, not a production risk model. It still checks whether the portfolio construction is creating risk differences that a row-level metric would miss. In this table, the TabPFN direct portfolio’s higher realized volatility is visible before looking at return alone, while the 12-month momentum portfolio realized less volatility than this trailing-covariance proxy expected.

The month-block bootstrap intervals remain wide:

Selected portfolio intervals are similarly wide:

These intervals are the reason I keep the conclusion modest. TabPFN is competitive in this run and has the best main row-level ranking view, but the portfolio-level uncertainty overlaps with strong baselines. This is evidence from one public historical path, not a claim of stable model-family superiority.

Runtime and research cost

After accuracy, portfolio behavior, and uncertainty, there is one more practical dimension: research cost. The final direct TabPFN scorer takes about 78 seconds. XGBoost final takes about 463 seconds after the early-stopping changes. This is not a pure speed comparison because TabPFN and XGBoost are doing different things, but it is relevant for notebook-first research workflows.

The practical lesson is not that one should replace XGBoost with TabPFN everywhere. XGBoost remains useful when a team wants task-specific fitting, tree-based diagnostics, feature-importance workflows, and controlled hyperparameter search. TabPFN is useful when direct pretrained tabular scoring is informative enough to justify its inclusion as a baseline or component in the research workflow.

Known limitations

Those results still need boundaries. TabICL is disabled in today’s empirical comparison. That is an engineering limitation of this public notebook run, not a statement about TabICL as a model family. I still want TabICL in the broader DSAIEngineering testbench because open-source usability matters for practical commercial work.

The data is still public. yfinance, public Cboe VIX history, and public FRED series make the notebook reproducible, but they are not a licensed point-in-time institutional data stack. A production version would need stronger data lineage, corporate-action controls, vendor quality checks, point-in-time macro release handling, and operational monitoring.

TFM embeddings are not tested. This post evaluates direct TabPFN scoring. A separate experiment could ask whether TabPFN or TabICL embeddings improve downstream XGBoost, a ranking model, a portfolio optimizer, or a strategy-selection meta-model.

The liquidity and high-low range diagnostics are proxies. The high-low range proxy is not a real quoted bid-ask spread. The participation diagnostic is useful for scale awareness, but it is not a market-impact model.

The objective-specific reruns are more direct than P21, but they are still focused diagnostics. A production benchmark-relative allocator would optimize active risk directly. A production multi-horizon allocator would define how overlapping signals map into actual position changes and rebalance timing.

The portfolio diagnostics are not production backtests. They do not include real execution, tax lots, mandate constraints, borrow constraints, capacity limits, market-impact estimation, a production ex ante risk model, model registry, or live monitoring. They are useful for testing whether model scores survive controlled allocation translation, not for approving live capital.

Finally, the holdout is one historical path. The month-block bootstrap intervals are wide. I read the result as evidence that direct TabPFN scoring is useful in this workflow, not as evidence that this TFM-based allocation workflow has a stable edge.

Conclusion

This final follow-up puts the tactical asset-allocation notebook in a more complete and better-qualified state. P20 introduced the supervised ranking framing. P21 made the problem more demanding with a larger universe, a lower base-rate target, a next-open convention, and identity-ablated main features. Today’s version strengthens the remaining experimental layers: liquidity, feature-policy sensitivity, objective-specific retraining, portfolio realism, and uncertainty.

The result is intentionally not a single-winner model-family story. TabPFN has the best completed row-level ranking result on the main target in this run. It also remains useful in monthly rank and portfolio diagnostics. But the best headline portfolio Sharpe belongs to the 12-month momentum rule, not to TabPFN, in this run. XGBoost remains an important classical supervised baseline, even though its main holdout AP is lower than TabPFN’s in this run.

I read the deterministic-rule result as a normal bar in this domain. If an ML or TFM allocator does not improve on strong deterministic finance rules after chronology, transaction costs, turnover, drawdown, and uncertainty are carried through the workflow, then the appropriate conclusion is that the model is useful in some parts of the workflow but not the leading allocation rule in that experiment.

That mixed result is the point. Tabular foundation models can be evaluated as useful components inside quantitative workflows, while still being compared with classical ML baselines, deterministic financial rules, calibration checks, leakage controls, uncertainty intervals, and portfolio translation.

This closes the tactical-allocation mini-series for now. The next step is to carry the same evaluation discipline into a new quantitative workflow.

That first version was intentionally controlled. It used a nine-ETF universe, a top-three target, ticker identity features, a close-to-close return convention, and a simple transaction-cost-aware portfolio diagnostic. The result was useful, but the limitations were also clear. A nine-asset universe makes the top-group label coarse. Static ticker identity can make the model learn persistent ETF identity rather than reusable market-state relationships. Close-to-close evaluation is easy to audit, but it is not the same as asking what would happen if a signal were acted on after month-end information is available. The portfolio diagnostic also needed more stress checks.

This post is the follow-up. I keep the same supervised tabular framing, but I make the allocation testbench more demanding in the directions that mattered most after P20:

• The universe is expanded from 9 ETFs to 25 ETFs.

• The target changes from top 3 of 9 to top 5 of 25.

• The base positive rate falls from about 0.333 to about 0.200.

• The target return convention changes from close-to-close to next-open-to-next-open.

• The main feature set removes ticker identity, asset-group metadata, and risk-bucket metadata.

• Feature-set sensitivity is checked with lightweight Logistic Regression diagnostics.

• The notebook adds multi-horizon and benchmark-relative audits.

• The portfolio section adds turnover, tax-drag, drawdown, benchmark-relative, and constrained-weight diagnostics.

The empirical result changes in an interesting way. In P20, direct TabPFN and TabICL had the strongest row-level Average Precision point estimates, around 0.409 against a 0.333 base rate, while XGBoost had stronger portfolio diagnostic point estimates. In this follow-up, TabICL is not part of the final empirical comparison because the full TabICL run hit RAM limits on Kaggle. Direct TabPFN remains the strongest row-level ranker among the models I completed, with holdout Average Precision of 0.313 against a 0.200 base rate. It also leads the main portfolio diagnostic in this run. XGBoost and the 12-month momentum rule remain serious baselines.

The most important interpretation is not that one model family wins. The useful result is that a more demanding allocation workflow changes the comparison while preserving the central lesson from P20: tabular foundation model scoring can be evaluated inside a realistic supervised ML workflow, but the conclusion depends on ranking quality, monthly cross-sectional behavior, allocation translation, uncertainty, and known limitations.

The equity-curve figure gives the high-level result before the tables. TabPFN direct finishes with the highest final equity among the plotted score-driven portfolios, but the path should not be read as a trading recommendation. It is a visual summary of how the monthly score-to-portfolio diagnostic behaved after the same top-five allocation rule and transaction-cost assumption were applied across scorers.

You can find the notebook in my GitHub repository here or you can clone it directly on Kaggle. The results discussed in this post come from the latest notebook version and its generated artifacts.

Minimal notation for this follow-up

P20 gives the full formulation, so I will only restate the notation needed to read this follow-up. Each row is still an asset-month pair \((i,t)\). The feature vector \(x_{i,t}\) contains information available at or before the monthly signal date, and the score \(s_{i,t}\) is used to rank assets within that month. The target \(y_{i,t}\) is 1 when the asset belongs to the next-period top group and 0 otherwise.

The core object is unchanged:

where \(\mathcal{F}_t\) is the information set available at the signal date and \(\phi_i(\cdot)\) is the feature-generation process for asset \(i\). A scorer then produces:

where \(\mathcal{D}_{\text{train}}\) is either the training data for a fitted supervised model or the labelled context for a direct tabular foundation model scorer. This is the only TFM recap needed here. XGBoost learns task-specific parameters from the allocation table. TabPFN uses a pretrained tabular model and the labelled context rows to score query rows:

where \(\theta\) denotes the pretrained TFM parameters, \(x_{\ast}\) is a query row, and \((X_{\text{context}}, y_{\text{context}})\) is the labelled context. The new capability I am exercising is this pretrained in-context scoring mechanism. The old discipline still applies: the target, chronology, leakage controls, baselines, calibration, and portfolio translation remain part of the experiment.

What is new in this experiment

A broader universe and a less coarse target

The P20 universe had 9 ETFs:

SPY, QQQ, IWM, TLT, IEF, GLD, HYG, EEM, VNQ.

This follow-up uses 25 ETFs:

SPY, QQQ, DIA, IWM, EFA, EEM, TLT, IEF, SHY, LQD, HYG, GLD, SLV, DBC, VNQ, IYR, XLB, XLE, XLF, XLI, XLK, XLP, XLU, XLV, XLY.

This adds more US equity style exposure, developed and emerging international equity, rates, credit, commodities, real estate, and US sectors. It is still not a production institutional universe, but it is a more demanding cross-sectional ranking problem than the first notebook.

Let \(N_t\) be the number of available assets in month \(t\). In the holdout period, \(N_t=25\) for the usable months. The target selects \(K=5\) assets. The monthly base rate is therefore approximately:

This matters for Average Precision. In P20, a non-informative scorer had a reference level near 0.333 because three out of nine assets were positive each month. In this post, the reference level is near 0.200. A raw AP number from P20 and a raw AP number from this post should not be compared without remembering that the target is different.

A next-open target convention

P20 used a close-to-close convention. That was easy to audit, but it left an execution caveat: if features are computed at month-end, what price could actually be traded after the signal is formed?

P20 used \(P_{i,t}\) for the adjusted month-end close price. I use a new symbol here because the execution convention changes from close-to-close to open-to-open. Let \(O^+_{i,t+1}\) denote the adjusted first tradable open after signal month \(t\), and let \(O^+_{i,t+2}\) denote the adjusted first tradable open after the next month. The selected one-month target return is:

The label is then:

Here \(\mathbf{1}\{\cdot\}\) is the indicator function: it equals 1 when the condition inside the braces is true and 0 otherwise. The phrase “top \(K\)” means that assets are ranked within the same signal month by the selected forward return convention.

The notebook still saves both close-to-close and next-open returns for audit. This is important because a change in execution convention can change rankings. The goal is not to claim that this is now a complete execution model. It is a more realistic diagnostic convention than the P20 close-to-close target, but it still does not model intraday slippage, bid-ask spreads, order timing, market impact, or data release timing in production detail.

Identity-ablated main features

The P20 notebook included ticker identity indicators and asset-group metadata. That choice was defensible for a fixed-universe allocation exercise, but it raised a good interpretation problem. If the model performs well, is it learning reusable relationships between market-state features and future relative returns, or is it learning persistent differences among named ETFs?

This post makes the main run identity-ablated. If \(z_i\) denotes static identity and metadata features for asset \(i\), the main feature policy is closer to:

The notebook excludes 34 static identity or metadata columns from the main model matrix. The final main feature matrix has 956 model features after retained base features and missingness indicators.

This does not mean identity information is unimportant. In fact, one diagnostic suggests it may be very important. The metadata-only Logistic Regression sensitivity model has a holdout AP of 0.282, which is above the 0.200 base rate and also slightly above the XGBoost final AP of 0.277 in this run. That does not mean metadata-only is a better allocation system. It means persistent asset identity and group structure contain meaningful information in this historical sample. For a practitioner, that is a useful warning: removing identity features makes the main experiment more conservative, but studying identity sensitivity remains part of the interpretation.

Feature sensitivity as a diagnostic, not a full rerun

The notebook compares these feature policies with lightweight CPU Logistic Regression:

• full

• ticker-ablated

• identity-ablated

• metadata-only

• strict time-series

The important phrase is “lightweight diagnostic.” I did not rerun the full TabPFN and XGBoost workflows for every feature variant because the practical constraint in this iteration was compute time. I had to rerun the notebook several times while working around TabICL memory failures and XGBoost tuning costs; each full attempt took close to two hours, with roughly 1.5 hours spent in the XGBoost tuning part. A full model-by-feature-variant study would be stronger, but that is a separate experiment. The current sensitivity check is useful for inspecting whether feature-family choices matter, but it should not be presented as a complete model-family sensitivity study.

The holdout feature sensitivity table is:

The metadata-only result is the surprising line. It suggests that persistent group and identity information can rank the holdout rows better than a weak linear model using the larger noisy feature set. This is not a reason to abandon time-series features. It is a reason to be careful with interpretation and to treat feature design as part of the scientific result.

Alternative-objective audits

P20 noted that the top-group target was only one possible allocation objective. A production allocator might care about benchmark-relative active return, multi-month horizons, drawdown risk, tax-aware turnover, or risk-adjusted return.

This notebook does not retrain the main models for all of those objectives. Instead, it asks a narrower question: how do the existing one-month scores relate to alternative labels?

For a 3-month or 6-month horizon \(h\), the notebook compounds the selected one-month return convention. In this run, that selected convention is next-open-to-next-open:

where \(m\) indexes the one-month return offset inside the \(h\)-month compounding window. The notebook then assigns a top-\(K\) label using the same cross-sectional ranking idea. It also creates a benchmark-relative label against SPY:

These are audits of the same scores, not new objective-specific training runs. This distinction matters. If a reader wants to know whether TabPFN is best for a six-month allocation target, the correct experiment would retrain and validate on that target. The current notebook asks whether the one-month next-open score contains some information about those alternative outcomes.

Experimental results

Dataset and splits

The final dataset contains 6,059 rows across 243 months and 25 assets. The holdout period contains 1,875 rows across 75 months, from January 2020 through March 2026. The holdout positive rate is exactly 0.200 because the target selects five assets out of twenty-five in each full holdout month.

The chronological split follows the same design logic as P20:

This is a time-ordered split, not a random split.

Row-level classification

P20 used Average Precision, ROC AUC, Brier score, calibration diagnostics, monthly rank diagnostics, and portfolio diagnostics because the target is a top-group ranking label rather than an ordinary balanced classification target. Average Precision summarizes precision-recall ranking quality, ROC AUC summarizes pairwise positive-versus-negative ranking, and Brier score measures squared probability error. I keep those metrics here so the comparison is continuous. The new pieces in this post are not new row-level metrics; they are the lower base-rate target, the next-open target convention, identity-ablated features, feature-policy sensitivity, alternative-objective audits, and the extra portfolio stress diagnostics.

The direct TabPFN scorer has the highest holdout AP and ROC AUC among the completed main models. The comparison with P20 needs care. P20's TabPFN and TabICL AP values were around 0.409, but the base rate was 0.333. Here the TabPFN AP is 0.313, but the base rate is 0.200. On the new, less coarse target, TabPFN is further above the non-informative reference than the raw AP alone would suggest.

The result is useful for the TFM question because the main TabPFN run is identity-ablated. The model is not being handed ticker identity or asset-group labels in the main feature matrix. It is using the labelled context rows and the time-varying feature table to score holdout asset-month rows.

In the precision-recall plot, the useful comparison is against the lower horizontal base-rate reference. The curves do not need to reach extreme precision to be useful in this setting; they need to stay meaningfully above the 0.200 reference over the part of the ranked list that matters for top-group selection. TabPFN direct does that most clearly among the completed main models.

That does not prove that TabPFN has discovered a stable allocation law. It shows that, in this public-data experiment, direct TabPFN scoring remains competitive after making the target harder, removing static identity features, and changing the execution convention.

Monthly cross-sectional ranking

P20 showed why monthly rank diagnostics matter: TabPFN and TabICL had the strongest row-level AP point estimates, while XGBoost looked stronger in the monthly rank and portfolio views. That same separation of questions matters here. Row-level AP pools all holdout rows together, while allocation is applied month by month. A scorer can have useful pooled AP and still produce a weaker within-month ordering.

For each month \(t\), Spearman information coefficient is:

The mean of \(\rho_t\) across holdout months summarizes whether higher scores tended to align with higher next-open returns within the same monthly cross-section.

TabPFN direct leads on selected return and active return. XGBoost is close. The 12-month momentum rule has the highest mean IC among these rows, but a lower top-k hit rate and slightly lower selected return. This continues the P20 lesson: mean rank correlation, top-k hit rate, selected return, and portfolio performance are related but not identical.

The low-volatility rule performs poorly in this holdout under this target. That should be read as a period-specific result, not as a general claim against low-volatility investing. The 2020 to 2026 holdout contains a sharp crash, a rapid recovery, an inflation and rates shock, and a strong mega-cap growth period. A defensive rule can look weak under a relative top-five target if the period rewards risk-on or momentum-like exposures.

Portfolio translation

The portfolio diagnostic converts scores into monthly top-five equal-weight portfolios and subtracts a 5 basis point cost per unit of one-way turnover. If \(w_{i,t}\) is the portfolio weight assigned to asset \(i\) after signal month \(t\), the pre-cost diagnostic portfolio return is:

Turnover is:

In these two equations, the summation is over the available assets in the monthly universe. The turnover \(\tau_t\) is one-way turnover: a move from 20 percent in one asset to 0 percent contributes 20 percentage points, and a move from 0 percent to 20 percent in another asset contributes another 20 percentage points.

The net return used in the main portfolio table is:

where \(c=0.0005\) for the 5 basis point transaction-cost assumption.

This is the biggest empirical change from P20. In P20, XGBoost had the strongest portfolio diagnostic point estimates. In this follow-up, TabPFN direct leads the main portfolio table by Sharpe and final equity. It also remains ahead after the basic 5 basis point turnover cost.

I do not treat this as a stable trading result. The holdout is one historical path. The confidence intervals are wide. The portfolio is still a simple monthly top-five diagnostic, not a production optimizer. But the result is useful because it shows that the TabPFN row-level advantage is not only a row-level AP artifact in this run. It also survives a simple allocation translation.

Turnover, tax drag, and weight constraints

The notebook adds several stress views that were missing or incomplete in P20.

The turnover-cap diagnostic blends previous weights toward new target weights. With a 50 percent turnover cap, the resulting portfolio can hold more than five assets because old positions decay rather than disappearing immediately. This is not a pure top-five portfolio. It is better described as post-score turnover-smoothed weights.

Selected turnover-cap results:

The stress result is favorable for TabPFN and XGBoost, but it changes portfolio meaning. A turnover-smoothed portfolio is not the same object as a strict top-five monthly portfolio. That is why the notebook saves the turnover-cap outputs separately.

The tax-drag sensitivity applies additional drag per unit of turnover. It is not a tax engine. It is a way to ask whether high-turnover strategies are fragile to additional frictions.

For TabPFN direct, CAGR falls from 18.3 percent with no extra tax drag to 14.8 percent with 50 basis points of tax drag per turnover. For XGBoost final, CAGR falls from 16.0 percent to 11.0 percent under the same stress. The larger drop for XGBoost is consistent with its higher annual turnover.

The weight-constraint artifact confirms that the main top-five equal-weight portfolios have 20 percent maximum asset weights and gross leverage of 1.0. It also shows that turnover is not trivial. XGBoost final has 52 months above 50 percent turnover and 17 months above 100 percent turnover. TabPFN direct has 24 months above 50 percent turnover and 5 months above 100 percent turnover. This is one reason portfolio diagnostics are necessary even when row-level AP looks good.

Benchmark-relative diagnostics

The notebook also compares strategy returns against benchmark portfolios such as 60/40 SPY/TLT. Against 60/40, TabPFN direct has annualized active return of about 10.5 percent, tracking error of about 11.8 percent, and information ratio of about 0.89. XGBoost final has annualized active return of about 8.9 percent, tracking error of about 12.5 percent, and information ratio of about 0.71.

These numbers are diagnostics, not mandate-ready active-risk controls. A real benchmark-relative allocation process would usually include explicit tracking-error budgets, sector or asset-class constraints, capacity assumptions, and ex ante risk models. Still, adding the benchmark-relative view is useful because it moves the evaluation closer to how allocation results are often discussed in practice.

Calibration

Calibration remains separate from ranking. If a score is only used to sort assets within a month, a monotonic calibration transform may leave the selected top-five set unchanged. If a score is described as a probability of top-group membership, probability quality matters. ECE means expected calibration error; in the notebook artifact used here, it is computed with 10 quantile bins.

The main holdout calibration view is:

TabPFN direct has the best AP among the listed scorers, while XGBoost final has the lowest expected calibration error among the listed learned scorers. That is not a contradiction. Ranking quality and probability reliability are different properties.

The metadata-only logistic score is a useful caution. Its AP is good, but its Brier score, log loss, and ECE are poor. That means it can rank some positives above negatives while still producing poor probability estimates. A practitioner familiar with supervised ML will recognize this pattern: a model can be directionally useful as a score and still poorly calibrated as a probability model.

Runtime and research cost

Runtime is part of the research result. The direct TabPFN holdout scorer takes about 55 seconds in this run. XGBoost takes about 4,352 seconds because the workflow performs a GPU-accelerated XGBoost model-selection process. This is not a perfectly fair runtime contest because the workflows are doing different things. It is still practically relevant.

For a practitioner, the TFM capability is attractive when quick direct scoring is valuable. XGBoost remains attractive when task-specific fitting, feature importance workflows, and controlled hyperparameter searches are important. I do not see these as mutually exclusive tools. In a serious research workflow, I would want both kinds of baselines.

Uncertainty

The notebook uses month-block bootstrap intervals. It resamples months rather than individual rows because rows from the same month share the same market environment and are tied together by the cross-sectional top-\(K\) target.

Selected row-level bootstrap intervals:

Selected portfolio bootstrap intervals:

The intervals explain the cautious language. TabPFN has the best point estimates in the main diagnostics, but the uncertainty intervals overlap with strong baselines. This is evidence of competitiveness in one public-data holdout path, not proof of stable superiority.

Known limitations

This is still a public-data research workflow. It uses yfinance, public Cboe VIX history, and public FRED series. That is useful for reproducibility, but it is not equivalent to a licensed point-in-time institutional market data system. Vendor corrections, corporate-action controls, survivorship handling, data release timing, and operational monitoring are outside the scope of this post.

The liquidity, spread, and market-impact proxy artifact should not be interpreted in this run. The notebook created portfolio_liquidity_impact_proxy.csv, but the artifact values are all zero because the underlying dollar-volume and high-low-spread proxy fields were missing. I am leaving the issue documented rather than fixing it today because the main post can stand without that artifact. Any claim about liquidity, spread, or participation should wait for a corrected rerun.

The feature sensitivity section is not a full model-family sensitivity study. It uses Logistic Regression diagnostics to inspect feature-policy effects. A stronger version would rerun TabPFN, XGBoost, and any enabled TFM scorers across all feature variants under the same chronological validation design.

The multi-horizon and benchmark-relative sections are audits of existing one-month scores. They do not retrain the main models for 3-month, 6-month, or benchmark-relative objectives. A proper alternative-objective study would define those labels as the main targets and repeat model selection and validation.

The portfolio diagnostics are not production backtests. They do not include a real execution model, tax lots, mandate constraints, borrow constraints, capacity, market-impact estimation, ex ante risk models, or benchmark-relative optimization. They are useful for testing whether model scores survive simple allocation translation, not for approving live capital.

TabICL is not part of the final empirical comparison in this post. That is an engineering limitation of my current public notebook run, not a statement about TabICL as a model family. The guarded TabICL code path remains in the notebook, but the completed results here should be read as TabPFN, XGBoost, Logistic Regression diagnostics, and deterministic-rule results.

TFM embeddings are still out of scope. This post evaluates direct TabPFN scoring. It does not test whether TabPFN or TabICL embeddings improve a downstream XGBoost, neural ranking, or portfolio optimization workflow.

Finally, my own goal in this series is learning by building. I am using the notebook to make assumptions visible and testable. The claims are intentionally limited to what this public experiment can support.

Conclusion

This follow-up makes the P20 tactical allocation experiment more demanding. The universe grows from 9 to 25 ETFs, the target becomes top 5 of 25, the execution convention changes to next-open-to-next-open, static identity features are removed from the main model matrix, and the portfolio diagnostics become more realistic.

Within this configuration, direct TabPFN produces the strongest completed row-level ranking result: AP 0.313 against a 0.200 base rate. It also leads the main portfolio diagnostic, with 18.3 percent CAGR, 1.12 Sharpe, and final equity of 2.86 over the 2020 to 2026 holdout. XGBoost and 12-month momentum remain competitive. The bootstrap intervals overlap enough that I would not claim stable superiority for any model family.

The most useful conclusion is methodological. A tabular foundation model can be inserted into a supervised financial tabular workflow as a direct scorer, but it still has to be evaluated like any serious model: with chronological splits, leakage checks, baseline rules, calibration diagnostics, uncertainty intervals, feature sensitivity, execution assumptions, and portfolio translation. The newer capability is the pretrained in-context scoring mechanism. The older discipline of supervised ML and financial validation is still necessary.

Given information available at a monthly signal date, can TabPFN, TabICL, XGBoost, and simple allocation rules rank assets in a public ETF universe by next-month relative attractiveness?

More precisely, I build a supervised tabular workflow that converts monthly tactical asset allocation into a cross-sectional top-group classification problem. Each row is an asset-month observation. The label is whether that asset is among the next-month top three assets in a nine-ETF universe. The models are evaluated as allocation scorers, not as price forecasters. A useful score should help rank assets within each month, remain credible under chronological validation, and survive comparison with simple momentum and benchmark allocation rules.

This is a useful place to test direct TFM scoring because the input-output table is familiar to a supervised ML practitioner, but the workflow mechanism is different. XGBoost learns task-specific parameters from the labelled allocation table. TabPFN and TabICL use labelled rows as context for a pretrained tabular learner and then score later query rows. The notebook therefore asks whether that context-based prediction capability remains useful once the workflow includes chronological splits, leakage checks, calibration diagnostics, deterministic finance rules, and portfolio diagnostics.

The main result is deliberately modest. Direct TabICL and TabPFN are competitive row-level rankers, with holdout Average Precision around 0.409 against a 0.333 base rate. XGBoost has lower row-level Average Precision in this run but stronger monthly portfolio diagnostic point estimates. The 12-month momentum rule remains a serious baseline. The uncertainty intervals overlap enough that I do not treat the exercise as a model-ranking contest. The useful conclusion is that direct tabular foundation model scoring can be evaluated seriously inside a tactical allocation workflow, but classical GPU-accelerated tabular ML and simple finance rules still need to be part of the comparison.

I treat this as a learning-by-building research workflow. The goal is to make the modeling choices explicit enough that a data or AI practitioner can inspect them, and modest enough that a quant finance professional would recognize what is still outside scope. I am not trying to present a finished allocation system. I am trying to build a careful public example of how TFMs can be placed inside a realistic validation and diagnostic structure.

You can find the notebook in my GitHub repository here, or you can clone it directly on Kaggle. The notebook is meant to be run with GPU enabled, and the results discussed in this post come from the full research configuration.

Background and scope

Series concepts reused here

The most useful prerequisites from earlier posts are P4, P12, P14, P15 to P17, and P18 to P19. I use three ideas from those posts here. First, TabPFN and TabICL can be viewed operationally through a posterior-predictive or in-context prediction lens: labelled rows provide task context for later query rows. Second, time-indexed market problems should be converted into supervised tables without breaking chronological validation. Third, model quality in an applied workflow is not one number; ranking, calibration, runtime, leakage, drift, and downstream decision diagnostics can disagree.

The allocation task

This post keeps the same supervised tabular framing but changes the finance task. Volatility-regime forecasting asked whether a single market object, SPY, was likely to enter a high-risk state. Tactical asset allocation asks a relative allocation question across several assets:

Given information available at a monthly signal date, can a model rank assets in a public ETF universe by their next-month attractiveness?

In the notebook, “attractiveness” is defined narrowly and mechanically. An asset is attractive for a given month if its next-month total return is in the top group within the configured ETF universe. This is not the only way to define tactical allocation. A production team might care about excess return over cash, drawdown risk, risk-adjusted return, benchmark-relative active return, mandate constraints, or tax-aware turnover. I use the top-group definition because it turns the allocation problem into a clear supervised tabular classification task while preserving the cross-asset ranking structure that matters for allocation.

The shift from volatility-regime forecasting to tactical allocation changes the interpretation of the supervised task. In volatility-regime forecasting, each row was a market date and the positive class was “future volatility is high.” The score was naturally an alert-prioritization or risk-attention score. In tactical allocation, each row is an asset-month pair. The positive class is not “this asset has a good absolute return.” It is “this asset belongs to the top group among the assets available in that month.” This makes the target relative. The model is not asked to forecast the whole market direction. It is asked to help rank assets within the same market environment.

This is closer to many practical allocation and cross-sectional equity workflows than a single-asset price forecast. A portfolio process often needs to decide how to distribute capital across alternatives, not only whether one asset’s return will be positive. Ranking, relative strength, turnover, and allocation stability therefore become central parts of the evaluation.

Mathematical setup

Let \(i\) index assets in a universe of size \(N\), let \(N_t\) denote the number of assets available at month \(t\), and let \(t\) index monthly signal dates. Let \(P_{i,t}\) be the adjusted close price of asset \(i\) at the end of month \(t\). The next-month simple return is:

At signal date \(t\), \(R_{i,t+1}\) is unknown. It is the future outcome used to define the training label and later evaluate the score. The feature vector for asset \(i\) at date \(t\) is:

where \(\mathcal{F}_t\) denotes the information available at or before the signal date, and \(\phi_i(\cdot)\) is the feature-generation process for that asset and market state. In the notebook, examples include recent asset returns, realized volatility, downside volatility, drawdown, moving-average distance, volume z-scores, beta to SPY, VIX features, Treasury yields, yield-curve features, availability-checked public macro features, and static asset metadata.

For each month, the assets are ranked by their next-month returns. If \(K\) is the number of assets selected as the positive group, the label is:

Here, \(y_{i,t}=1\) means that asset \(i\) was in the next-month top group, and \(y_{i,t}=0\) means it was not. The indicator \(\mathbf{1}\{\cdot\}\) returns 1 when the condition is true and 0 otherwise.

The supervised learning table is then:

The label construction implies:

Here, \(\bar{y}_t\) is the positive-class rate in month \(t\). In the full holdout run, \(N_t=9\) and \(K=3\), so the monthly base rate is \(3/9=0.333\). This is why the notebook treats 0.333 as the non-informative Average Precision reference.

This notation is close to earlier posts, but there is one important difference. In the volatility-regime posts, the row index was essentially one date \(t\), because the target was a single market-level event for SPY. In this notebook, the row index is a pair \((i,t)\), because each month contributes one row per asset. This turns the problem into a panel-style tabular dataset: many assets observed through time, with a target that is defined relative to other assets in the same month.

What the score means

A price forecast would ask for a prediction such as:

That is not what this notebook does. The model produces a score:

The score can be read as a probability estimate if calibration is adequate, or more conservatively as a ranking score. In the allocation diagnostic, the practical use is ranking: for each month, sort assets by \(s_{i,t}\), select the top \(K\), and allocate equal weight to those selected assets.

This distinction matters. A high score does not say that an asset must go up next month. It says that, according to the fitted workflow, the asset has features associated with membership in the next-month top group within the configured universe. An asset can have a high score in a month where every asset loses money. Conversely, an asset can have a positive next-month return and still not be in the top group if several other assets did better.

That is why this notebook evaluates the workflow through both prediction diagnostics and allocation diagnostics. Average Precision and ROC AUC summarize row-level classification quality. Monthly rank correlation and top-\(K\) hit rate ask whether the scores are useful within each monthly cross-section. Portfolio diagnostics ask what happens when those scores are translated into a simple allocation rule with transaction costs.

Chronological validation and leakage

Financial data is ordered in time, so random train-test splits are inappropriate. They can leak future regimes into model selection and make a model look more stable than it would be in a real research process.

The core rule is the same as in P18 and P19:

At signal date \(t\), features may use information available at or before \(t\), but the label uses returns after \(t\).

The notebook uses chronological windows. Earlier data is used for model selection, a later pre-holdout window is used for calibration diagnostics, and the final period is reserved for holdout evaluation. Feature filtering and imputation are fitted before the calibration and holdout windows. That matters because even preprocessing can leak information if it is fitted on the full dataset.

The target also has a cross-sectional leakage risk. The label for asset \(i\) in month \(t\) is defined by comparing \(R_{i,t+1}\) with the other assets’ \(R_{j,t+1}\) in the same future month. That future cross-sectional ranking is allowed for label construction, but none of those future returns can enter the feature vector \(x_{i,t}\). The notebook therefore excludes forward returns, forward excess returns, future ranks, and target columns from model features.

This design is still a public-data research workflow, not a production market-data system. Public ETF data from yfinance, public VIX history, and public FRED series are useful for reproducibility, but they do not provide the same guarantees as a licensed point-in-time institutional data store. That limitation is part of the reason the notebook includes an explicit leakage and reuse checklist.

XGBoost versus direct TFMs

The input-output table is the same for XGBoost and the direct TFM scorers: rows contain features \(x_{i,t}\), and labels contain top-\(K\) membership \(y_{i,t}\). The difference is how each model uses that table. For a classical supervised model such as XGBoost, fitting means learning task-specific parameters from the current labelled dataset. Conceptually, the model is selected from a function class \(\mathcal{G}\) by minimizing a training objective:

Here, \(\mathcal{G}\) is the candidate class of scoring functions, \(\ell\) is the loss function, and \(\hat{f}\) is the fitted allocation scorer. Hyperparameter tuning then selects a configuration using chronological validation folds. In the notebook, XGBoost is the main classical supervised benchmark because it can run on GPU and is a strong practical baseline for tabular data.

For TabPFN and TabICL, the workflow is different. The large model is already pretrained. The labelled rows supplied through .fit() are better understood as task context rather than as the same kind of task-specific weight training used by XGBoost. In notation similar to P4 and P18, the prediction is closer to:

where \(x_{\text{new}}\) is a query row, \(y_{\text{new}}\) is the unknown label, \(X_{\text{context}}\) is the context feature matrix, and \(y_{\text{context}}\) is the corresponding vector of context labels.

This difference is the main reason to test tabular foundation models in this kind of notebook. A standard model such as XGBoost asks whether a task-specific learner can find a useful mapping from engineered market features to the top-\(K\) label. A tabular foundation model asks a related but different question: whether a pretrained in-context tabular learner can use the labelled allocation history as context and produce useful scores for later asset-month rows without the same task-specific training process.

The new capability being exercised here is therefore not that TFMs remove the need for careful data construction. They do not. The capability being tested is whether a pretrained tabular learner can use a relatively small labelled context and produce useful task-specific scores without the same hyperparameter-search and task-specific fitting procedure used by XGBoost. That is why the notebook still includes deterministic rules, chronological validation, calibration checks, and portfolio diagnostics. The TFM component is only one part of the workflow.

This is not a claim that tabular foundation models are automatically better for allocation. The value has to be demonstrated empirically against strong baselines, simple finance rules, runtime costs, and validation constraints. In this notebook, TabPFN and TabICL are evaluated as direct scorers. They are not used as embedding generators in the default path. That keeps today’s comparison focused on a simpler operational question: can direct TFM scoring be useful in a monthly allocation workflow?

Evaluation scope

The row-level classification target is imbalanced because only the top \(K\) assets in each month receive label 1. If the universe has \(N\) assets and all assets are available, the base rate is approximately \(K/N\). Accuracy is therefore not a very informative primary metric. A model can look accurate by mostly predicting the majority class and still be unhelpful for allocation.

Average Precision is useful because it evaluates the ranked list of asset-month rows by summarizing precision as recall increases. Precision is the fraction of selected or flagged rows that are true positives, and recall is the fraction of all true positives that have been selected or flagged. For a non-informative ranking, the expected precision is close to the positive-class base rate, which is why the 0.333 top-k label rate is the main reference point here. ROC AUC is also reported because it asks how often a randomly chosen positive row is ranked above a randomly chosen negative row. For allocation, however, row-level metrics are not sufficient. The model is used month by month, so the cross-sectional ordering inside each month matters.

A monthly rank diagnostic asks whether high scores correspond to higher next-month returns within the same month. One summary is Spearman rank correlation between scores and realized next-month returns:

Here, \(N_t\) is the number of available assets in month \(t\). A positive value means that, in that month, higher model scores tended to align with higher next-month returns. This is only one diagnostic; it does not include transaction costs or portfolio constraints.

The allocation diagnostic converts scores into portfolio weights. In the simple top-\(K\) equal-weight rule used here:

The next-month portfolio return before costs is:

The notebook also subtracts a simple transaction-cost term based on absolute weight turnover. If \(c\) is the transaction-cost rate and:

then the net diagnostic return is:

The first month is treated as an initial allocation in the notebook. This does not make the diagnostic a production backtest. It is a controlled way to ask whether the score remains interesting after a basic cost penalty and monthly reallocation mechanics are included.

Calibration is a separate question. If the score is used only to rank assets within a month, a monotonic transformation of the score may not change the allocation. If the score is interpreted as:

then probability quality matters. The notebook therefore reports Brier score, log loss, expected calibration error, and reliability-bin artifacts. Brier score is the mean squared error of predicted probabilities, log loss penalizes confident wrong probabilities, and expected calibration error compares binned predicted probabilities with observed frequencies. As in previous posts, I treat calibration diagnostics as complementary to ranking diagnostics rather than as replacements for them.

With these diagnostics defined, the scope of the post is intentionally narrow. This is a first tactical asset-allocation workflow test. It is in scope to ask whether deterministic allocation rules, XGBoost, direct TabPFN, and direct TabICL produce useful next-month top-group scores under chronological validation.

It is also in scope to inspect row-level classification quality, monthly rank quality, calibration, runtime, month-block bootstrap uncertainty, feature drift, leakage checks, and a simple transaction-cost-aware allocation diagnostic.

It is not in scope to claim a trading strategy, forecast exact asset prices, optimize a production portfolio, account for taxes, model liquidity in detail, or produce a definitive benchmark for financial ML.

The useful contribution is narrower: a reproducible workflow for testing tabular foundation models as components inside a realistic, but still public-data-based, tactical allocation research pipeline.

Code discussion and results

Dataset and chronological splits

The notebook builds a monthly panel from nine liquid ETFs: SPY, QQQ, IWM, TLT, IEF, GLD, HYG, EEM, and VNQ. Each row is an asset-month observation. The target is whether that asset is among the next-month top three assets in the universe. With nine assets and three positives per full month, the natural class base rate is close to 33.3 percent. This base rate is important when reading the results: an Average Precision near 0.33 is approximately what a non-informative scorer should achieve, while values above that level indicate some ranking information.

The final modeling frame contains 2,172 asset-month rows across 243 monthly signal dates. The holdout window contains 675 rows across 75 months from January 2020 through March 2026. This period is a useful stress period for evaluation because it includes the COVID shock, the 2022 inflation and rate-hiking cycle, the subsequent recovery, and several large cross-asset rotations. It is also a difficult period for stable model selection, so the results should be interpreted as evidence from one demanding holdout window rather than as a universal ranking of methods.

The split design is deliberately chronological. Model selection uses rolling validation folds inside the pre-holdout history, calibration is evaluated after the selection window, and the final holdout remains untouched until the final scoring step. The feature policy is also fitted before calibration and holdout evaluation. That detail matters because feature filtering, missingness decisions, and imputation can otherwise leak holdout distribution information into the model.

Feature construction and data availability

The full run starts from market, volatility, rate, volume, relative-strength, drawdown, beta, and asset-identity features. The feature policy keeps candidate features only if they have enough non-missing observations in the model-selection source window and do not exceed the configured missingness threshold. After the FRED availability gate, 209 numeric candidate features are reviewed, all 209 base features are retained, 187 missingness indicators are added, and the final model matrix contains 396 columns.

This is a practical compromise. Public market and macro series are useful for reproducibility, but they are not equivalent to a licensed point-in-time data warehouse. A concrete example is credit-spread data. In this run, five FRED series have enough selection-window coverage and are included in features. The two public OAS series attempted through FRED, BAMLH0A0HYM2 and BAMLC0A0CM, start only on 2023-05-09 in the downloaded file and have zero non-missing observations in the model-selection window. The notebook records their availability and excludes them from the model matrix rather than silently imputing unavailable credit-spread history.

The static asset-identity signal is encoded through ticker indicators rather than an ordinal numeric asset_id. This avoids giving the model an artificial numeric ordering among ETFs. A fixed identity signal is still a modeling choice: it lets the model learn persistent differences between asset classes, but it also means the result is tied to the configured universe rather than to a fully universe-agnostic allocation rule.

Scorers compared

The notebook compares five scorer families. The deterministic rules provide finance-domain references, including momentum and low-volatility rules. The dummy prior provides a non-informative classification reference at the 0.333 base rate, but it is excluded from portfolio diagnostics because constant scores create arbitrary top-k tie-breaking. XGBoost is the GPU-accelerated classical supervised learner with rolling chronological model selection and calibration variants. TabPFN and TabICL are evaluated as direct allocation scorers: they receive labelled asset-month rows as context and score later holdout rows without the same task-specific fitting process used by XGBoost.

Row-level ranking results

The first result view is row-level classification performance on the holdout period. Average Precision is the main metric because the target is a top-group label with a 33.3 percent base rate. ROC AUC, Brier score, expected calibration error, and runtime are shown here because they answer different questions. The notebook also saves log loss in the full artifact table.

The direct TabICL and TabPFN scores have the highest holdout Average Precision point estimates, both around 0.409. This is above the 0.333 base rate and above the deterministic momentum rules. The magnitude should be read carefully, though. The lift is meaningful for a difficult monthly cross-sectional task, but it is not large enough to support a strong claim of stable model-family superiority.

XGBoost illustrates why several metrics are needed. Its calibrated variant has a lower Average Precision point estimate than TabPFN and TabICL, but its ROC AUC is slightly higher. Average Precision focuses more directly on the positive class and on early ranking quality, while ROC AUC averages pairwise ranking behavior over the whole score distribution. In an allocation workflow that selects only the top few assets each month, Average Precision and monthly top-k diagnostics are usually more directly relevant than ROC AUC alone.

The deterministic rules are important baselines. The 12-month momentum rule is competitive with the learned models, which is not surprising in an asset-allocation setting. A learned model that cannot improve on simple relative-strength rules would not be very persuasive. The low-volatility rule performs poorly in this particular holdout window, which is also informative: it shows that not every simple financial heuristic is rewarded in the 2020-2026 regime.

The dummy prior is retained for classification calibration context, but it is not used as a portfolio scorer. A constant-prior score creates ties across all assets; converting those ties into a top-k portfolio would depend on arbitrary tie-breaking and can produce misleading deterministic selections.

The precision-recall figure shows the same result visually. The horizontal reference line is the holdout base rate. Curves that stay above this line are ranking positives better than a non-informative scorer. The TabPFN and TabICL curves sit modestly above the baseline over useful parts of the recall range, but they are not separated enough from the other competitive methods to justify strong superiority language.

Monthly cross-sectional ranking results

Row-level classification metrics pool all asset-month rows together. Tactical allocation is implemented month by month, so the notebook also evaluates whether each scorer ranks assets well inside each monthly cross-section. The monthly diagnostics include mean Spearman information coefficient, top-k hit rate, average selected next-month return, average universe next-month return, and the difference between selected and universe returns.

This view changes the emphasis. XGBoost final has the strongest monthly rank diagnostic point estimates in the full run, even though TabPFN and TabICL have stronger row-level Average Precision point estimates. That is not a contradiction. Average Precision pools rows across the full holdout. Monthly rank diagnostics ask a narrower allocation question: in each month, did the scorer put the better next-month assets closer to the top of that month’s list?

The 12-month momentum rule remains a serious baseline here as well. Its mean monthly IC and top-k hit rate are close to the direct TabPFN result. For a professional reader, this is an important result rather than an inconvenience. In tactical allocation, a model has to be compared against simple, interpretable allocation rules that already encode domain structure.

The negative low-volatility result is also useful. It indicates that this holdout period rewarded relative strength and risk-on selection more than defensive low-volatility selection. That interpretation is period-specific; it should not be generalized into a permanent statement about low-volatility investing.

Portfolio translation

The portfolio diagnostic converts each score into a simple monthly top-three equal-weight portfolio and subtracts a transaction-cost assumption of 5 basis points per unit of absolute weight turnover. This is not a production backtest. It does not model taxes, market impact, execution timing beyond the close-to-close convention, fund-specific trading costs, mandate constraints, or liquidity rules. It is a controlled diagnostic for asking whether a score remains useful after it is translated into a repeatable allocation rule.

The portfolio table gives a different perspective from the row-level model table. XGBoost final has the strongest portfolio diagnostic point estimates in this run, with a 18.1 percent CAGR, 1.03 Sharpe ratio, and final equity of 2.83 over the holdout period. SPY-only has a high final equity of 2.44, but it is a concentrated single-asset benchmark and has a lower Sharpe ratio than the XGBoost final diagnostic portfolio. The 12-month momentum rule remains strong and is a serious reference point for the learned models.

The direct TabPFN and TabICL portfolios finish above initial capital and above equal weight in final equity, but they do not dominate the stronger classical and deterministic allocation baselines. Their higher row-level Average Precision does not automatically translate into the strongest allocation outcome. This is why the notebook reports several diagnostic layers. A scorer can rank positives well in aggregate, yet still produce a higher-turnover or less stable top-k portfolio.

The equity-curve figure helps separate level, volatility, and drawdown behavior. The relevant reading is not only which line ends highest. A professional allocation review would also ask whether the path is stable, whether performance is concentrated in a few months, how much turnover is required, and whether a simpler benchmark offers a similar result with less operational complexity. The figure plots the final XGBoost scorer as well as its calibrated counterpart because the portfolio table shows that the distinction matters. The notebook also excludes the constant-prior dummy and duplicated deterministic rule portfolios from this score-driven portfolio view. In the post table, I present one calibrated XGBoost portfolio view when calibration-base and calibrated selections produce the same monthly allocation path.

Calibration and probability quality

Calibration is evaluated separately from ranking. If scores are used only to sort assets within a month, a monotonic calibration transform may leave the top-k selection unchanged. If scores are presented as probabilities of next-month top-group membership, then calibration becomes part of the model quality assessment.

The XGBoost calibrated score improves the Brier score from 0.21875 to 0.21797 and expected calibration error from 0.0520 to 0.0366, while leaving Average Precision unchanged at 0.3877 for the calibrated-base score. This is the expected behavior for a monotonic probability calibration method: it can improve probability reliability without changing the ranking order materially.

TabPFN direct and TabICL direct have similar Brier scores around 0.219. Their expected calibration errors are about 0.045 and 0.061, respectively. These are not poor numbers for this task, but they also do not support an interpretation that the TFM scores are already production-grade probabilities. In this workflow, the most defensible interpretation of the scores is as ranking scores, with calibration diagnostics reported for transparency.

The calibration figure compares predicted probability bins with observed positive rates. A perfectly calibrated model would lie close to the diagonal. The zoomed view is useful because most probabilities are concentrated in a relatively narrow range rather than across the full 0 to 1 interval. That concentration is reasonable in this task: with three positives out of nine assets each month, even useful scores should not usually produce extreme probabilities.

Runtime and research cost

Runtime is part of the research result, not just an engineering detail. XGBoost performs a broad randomized hyperparameter search under rolling chronological validation, so its workflow time is much longer than the direct TFM scorers in this run. TabPFN direct takes about 25 seconds and TabICL direct about 13 seconds, while the full XGBoost search takes about 39 minutes.

The runtime versus Average Precision figure shows the main engineering tradeoff. Direct TabPFN and TabICL provide competitive Average Precision at much lower workflow time. XGBoost is slower because it is being tuned more extensively, but in this run its final scorer produces the strongest portfolio diagnostic point estimates. The practical implication is not that one method is always better. It is that the right model choice depends on the research objective: quick direct scoring, probability quality, portfolio behavior, interpretability, or a more exhaustive supervised tuning process.

Uncertainty, drift, and leakage

The notebook uses month-block bootstrap intervals so that uncertainty is resampled at the month level rather than by treating individual asset rows as independent observations. This matters because rows from the same month share the same market regime and are linked by the top-k target definition.

The intervals are the main reason to avoid strong superiority language. TabICL and TabPFN have the highest AP point estimates, but their AP intervals overlap with the intervals for momentum and XGBoost. XGBoost final has the highest monthly IC and top-k hit-rate point estimates, and its IC interval is positive in this run, but it still should be read as holdout evidence rather than proof of a stable trading edge.

The low-volatility rule has the least favorable diagnostic profile in this comparison. Its Average Precision interval is below most other methods, and its mean monthly IC is negative. Even there, the correct interpretation is period-specific: the holdout regime did not favor this defensive rule under the notebook’s top-k target and universe.

The feature-drift report shows substantial distribution changes between model-selection history and holdout. The largest population stability index values are concentrated in rate, yield-curve, beta, and volatility-related features. Examples include EEM beta to SPY, the 10-year minus 2-year yield curve, fed funds rate changes, Treasury yields, and ETF beta features.

This is not a defect in the notebook. It is the market environment the model has to face. A holdout from 2020 to 2026 is materially different from much of the 2006 to 2017 selection period. The implication is that model performance should be interpreted together with drift: a scorer that looks useful here has been evaluated through a demanding regime shift, but the same drift also warns against assuming stationarity.

The leakage checklist passes the main mechanical checks: target columns are excluded from the feature matrix, chronological splitting is preserved, feature policy and imputation are fitted before calibration and holdout evaluation, and next-month returns are used only as labels. The notebook also records the close-to-close convention. Features are computed from month-end values and next-month returns are measured from that same adjusted close to the next month-end adjusted close. This is a diagnostic convention rather than a fully executable live trading assumption. A production implementation would need an explicit trade-date and execution-price model.

Result interpretation

The empirical story is deliberately mixed. Direct TabPFN and TabICL have the strongest row-level Average Precision point estimates, which is useful evidence that direct TFM scoring can be competitive in this allocation framing. XGBoost final has the strongest monthly rank and portfolio diagnostic point estimates, which shows why a classical supervised baseline remains essential. The 12-month momentum rule remains close enough to the learned models that it should be treated as a serious domain benchmark rather than a weak baseline. The bootstrap intervals overlap materially, so the right reading is not that one model family wins. The right reading is that direct TFM scoring, GPU XGBoost, and simple allocation rules each answer part of the workflow question, and the final judgment depends on ranking quality, allocation behavior, calibration, runtime, uncertainty, and data limitations together.

Known limitations

The most important limitation is that this is a public-data research workflow, not an institutional production allocation system. The notebook uses yfinance, public Cboe VIX history, and public FRED files. That makes the workflow reproducible, but it does not provide the data lineage, point-in-time guarantees, vendor corrections, corporate-action controls, or operational monitoring that a production investment process would require.

The execution convention is also simplified. Features are computed from month-end close information, and the allocation diagnostic applies scores to the next close-to-close monthly return. The notebook documents this assumption, but it does not model when the month-end data becomes available, what price could actually be traded, whether a next-open execution convention would change the result, or how intraday slippage and market impact would affect rebalancing.

The universe is intentionally small. A nine-ETF universe is useful for a controlled notebook because the target is easy to inspect and the model behavior can be reviewed asset by asset. It is not large enough to represent the full complexity of a professional tactical allocation universe. The top-three label is also coarse: with nine assets, one third of the universe is labelled positive each month. A broader universe would make the ranking problem more realistic and would likely change the relative value of different models.

The portfolio diagnostic should not be read as a deployable backtest. It uses equal-weight top-k selection, a simple turnover-cost assumption, and a small number of benchmark portfolios. It does not include taxes, borrow constraints, fund-specific bid-ask spreads, market impact, capacity, mandate constraints, benchmark-relative risk budgets, turnover penalties during model selection, or drawdown-aware optimization. The diagnostic is useful for testing whether scores have allocation relevance; it is not sufficient for live portfolio approval.

The feature design includes ticker identity indicators and asset-group metadata. That is defensible for a fixed-universe allocator because the model may need to distinguish persistent asset-class behavior. It also limits the interpretation. Some of the signal may come from learning persistent ETF identities rather than from fully reusable market-state relationships. A stronger follow-up should include an identity-ablated run and compare how much performance remains when ticker identity is removed.

The uncertainty analysis is helpful but not definitive. Month-block bootstrap intervals are more appropriate than row-level independent resampling for this panel, but they still summarize only the observed holdout period. The 2020-2026 holdout contains important market regimes, yet it is still one historical path. The overlapping confidence intervals across TabPFN, TabICL, XGBoost, and momentum rules mean that the results should be interpreted as evidence of competitiveness, not as proof of stable model superiority.

Finally, the notebook evaluates direct TabPFN and TabICL scoring, not every possible way to use tabular foundation models. It does not test TFM embeddings, hybrid TFM-plus-XGBoost pipelines, larger universes, multi-horizon targets, benchmark-relative objectives, or constrained portfolio optimization. Those are natural extensions, but they are outside the scope of this post.

Summary and conclusion

This post treated tactical asset allocation as a supervised tabular ranking problem. The objective was not to forecast exact prices or claim a deployable trading strategy. The objective was narrower: build a reproducible public-data workflow that asks whether different scorers can identify the next-month top group within a small ETF universe under chronological validation.

The empirical picture is mixed in a useful way. Direct TabICL and TabPFN produce the strongest holdout Average Precision point estimates, around 0.409 versus a 0.333 base rate. That is consistent with the pretrained tabular foundation models using the labelled asset-month context in a way that contains some cross-sectional ranking information. At the same time, the portfolio diagnostic favors the final XGBoost scorer, and the 12-month momentum rule remains highly competitive. This is the kind of result I would expect from a serious workflow test: the newer model class can be useful in this setup, but it does not make strong classical baselines or simple domain rules irrelevant.

The uncertainty analysis also matters. The confidence intervals overlap materially across the main competitive methods. XGBoost has the strongest monthly IC and top-k hit-rate point estimates in the holdout, while TabICL and TabPFN have the strongest row-level Average Precision point estimates. These are not contradictory findings; they show that different evaluation views answer different questions. For allocation work, row-level ranking, monthly cross-sectional ranking, portfolio behavior, turnover, calibration, and drift should be read together.

My conclusion is that direct TabPFN and TabICL scoring deserves attention in applied financial tabular workflows, but the evidence should be presented carefully. In this notebook, TFMs are credible components in a tactical allocation research pipeline. They are not shown to be universally better than XGBoost or momentum rules, and the notebook does not establish live trading suitability. The stronger contribution is the workflow itself: chronological splits, leakage checks, GPU-first model comparison, saved artifacts, calibration diagnostics, transaction-cost-aware portfolio diagnostics, month-block uncertainty, and explicit documentation of public-data limitations.

Outlook

The next step is to make the allocation testbench more demanding. The current universe has nine ETFs and a top-three monthly target. That is enough for a controlled blog example, but it is still a small universe. A broader ETF universe, sector funds, international exposures, commodities, currencies, and duration buckets would create a more realistic cross-sectional ranking problem and make top-k selection less coarse.

The second direction is model-design sensitivity. The current notebook uses ticker identity indicators and asset-group metadata. That is defensible for a fixed-universe allocator, but it should be tested. A useful follow-up would compare the current feature set with an identity-ablated version, a metadata-only version, and a strictly time-series feature version. If performance depends heavily on persistent ticker identity, the interpretation is different from a model that generalizes mainly from market-state and asset-behavior features.

The third direction is portfolio realism. The current diagnostic includes monthly turnover costs, but it still uses a simple close-to-close convention. A stronger research version would include next-open or volume-weighted execution assumptions, wider transaction-cost sensitivity, liquidity filters, concentration limits, turnover constraints, drawdown-aware objectives, and benchmark-relative risk controls. Those additions would move the notebook closer to an institutional research prototype, although still not to a production trading system.

Finally, there is room to compare direct TFM scoring with the embedding workflows from earlier posts. In this notebook, TabPFN and TabICL are used as direct allocation scorers. A later version could test whether TFM embeddings improve a supervised XGBoost or neural ranking pipeline, and whether that hybrid approach is more stable than direct TFM scoring under regime drift. That would connect this allocation workflow back to the embedding-based fraud workflows from P16 and P17, while keeping the evaluation discipline from the volatility-regime and allocation posts.

Given the information available at the close of date t, can a model score whether SPY’s realized volatility over the next 20 trading days will fall into a high-volatility regime?

That kind of score can be useful in a business workflow in several ways. A portfolio team might use it to decide when to review gross exposure. A risk team might use it to prioritize stress checks. A model-risk team might use it to monitor whether market conditions are moving outside the regime where a strategy was validated. A product team building investing tools might use it as a risk-state feature rather than as a trading signal. The value is in improving the timing and quality of risk attention.

To explore this question, I built a supervised tabular workflow that used information available at the close date t to score whether SPY’s realized volatility over the next 20 trading days would be high. In that workflow, I compared the performance and workflow-related pros and cons of direct TabPFN and TabICL scoring, XGBoost on raw features, XGBoost on raw features plus TabPFN or TabICL embeddings, and simple volatility-domain rules.

The main result was that direct TabPFN and TabICL looked more useful as scoring models than the embedding-enhanced XGBoost workflows, while raw XGBoost and volatility-domain rules formed competitive baselines.

This post follows up on P18 while focusing on the business aspects. At the end of that post, I highlighted several limitations of the workflow. In this version, I addressed the most important ones: the validation window is less thin, the volatility-domain baselines are stronger, and the diagnostics now look at threshold sensitivity and market regimes more directly. Some limitations remain, especially around public data, overlapping targets, and regime-dependent validation.

Because the volatility-domain rules were so competitive in P18, I added four more rules to learn how much simple finance structure can explain before a learned model is worth the added complexity. This is important for business evaluation. If a simple volatility-domain rule gives almost the same risk-ranking value as a learned model for almost no expense, the learned model has to justify itself through a clearer alert queue, better robustness, better probability quality, better complementarity, or better downstream decision impact.

I also removed the embedding-enhanced XGBoost workflows from this version. Across P16, P17, and P18, those embedding-enhanced workflows did not consistently improve performance. My guess is that I was doing something wrong there, so I took them out until I understand the reason.

You can find the notebook in my GitHub repository here or you can also clone it directly on Kaggle. The notebook is meant to be run with GPU enabled.

Discussion and results

Technical changes

The target and final holdout are kept stable. The target is still future 20-trading-day SPY realized volatility crossing an 80th percentile threshold estimated only from data through 2017-12-29. The threshold is 0.200603 annualized realized volatility, and the final holdout is still 2020-01-02 to 2026-04-02.

The important changes are:

• The post-context tuning window now starts in 2010 instead of 2012. That gives optional benchmark tuning more high-volatility examples.

• The volatility-domain baseline set is stronger. P18 used a narrower baseline set; this version reports VIX close, VIX z-score, close-to-close realized volatility, Parkinson realized volatility, Garman-Klass realized volatility, and a VIX-minus-realized-volatility gap rule. These rules have the following definitions:

  • VIX close: the market’s implied volatility gauge for the S&P 500. It is a natural forward-looking risk baseline.

  • VIX z-score 252d: VIX relative to its own one-year history. This asks whether implied volatility is unusually high compared with its recent context.

  • SPY realized volatility 20d: recent close-to-close volatility of SPY. This asks whether volatility has already been elevated.

  • SPY Parkinson realized volatility 20d: a high-low range based volatility estimate. It uses intraday range information rather than only close-to-close moves.

  • SPY Garman-Klass realized volatility 20d: another range-based volatility estimate that uses open, high, low, and close information.

  • VIX/realized volatility gap 20d: the spread between implied and recently realized volatility. In this run, that particular construction is not well aligned with the future high-volatility target.

• The comparison is now direct: TabPFN, TabICL, raw all-history XGBoost, and domain rules.

• The notebook reports threshold sensitivity, named market-regime behavior, score-decile checks, runtime, calibration, uncertainty, drift/leakage checks, and a risk-control diagnostic.

• The feature policy also stays close to P18 but expands slightly with the additional volatility features.

The split is:

This split matters because a business user does not care whether a model only works in a quiet 2012-2017 slice. The score has to survive a holdout that includes COVID, the inflation/rate shock, quiet years, and the more recent 2024-2026 period.

Main holdout result

The holdout base rate is 23.3%. Average Precision is the main ranking metric because this is an alert-prioritization problem: sort dates from highest expected risk to lowest expected risk.

The business interpretation is:

1. Direct TabPFN has the best broad ranking point estimate.

2. VIX close is almost as useful and costs essentially nothing.

3. TabICL is competitive and faster than TabPFN in this run.

4. XGBoost is respectable but expensive because the workflow includes randomized hyperparameter search.

5. The VIX-realized-volatility gap is a useful negative control: plausible finance features still need validation.

Although the foundation models and the classical ML benchmark perform well at this task, performance alone would not justify their use in production if simple domain rules provide nearly the same ranking value at almost no additional cost. The learned models need to improve the alert queue, add complementary coverage, improve probability quality, or support a better downstream decision.

Precision-recall curve

The precision-recall curve shows the performance visually. Direct TabPFN, VIX close, Parkinson volatility, Garman-Klass volatility, TabICL direct, XGBoost, close-to-close realized volatility, and VIX z-score all sit well above the random baseline for much of the useful recall range. The curve shows that there are several viable risk-scoring candidates; the operating-point table shows what the queue would actually look like.

Alert queue impact

The strongest business view is the alert queue.

At 50% recall, direct TabPFN captures half of future high-volatility rows with the shortest queue:

That is a practical point in TabPFN’s favor. A smaller queue means fewer dates requiring review for the same captured share of future high-volatility events.

At 80% recall, the story changes:

At high recall, VIX is still slightly more efficient than TabPFN. The difference between VIX close and TabPFN is only seven alerts, so I would not overstate it. The practical message is that the simple implied-volatility signal remains extremely hard to beat when the business goal is broad risk coverage.

At 90% recall, VIX z-score has the shortest queue: 751 alerts with precision 0.439. That is a reminder that “best model” depends on the operating requirement. If the business goal is to capture nearly all high-volatility periods, a normalized VIX signal can be more useful than the model with the best overall AP.

Threshold sensitivity

A production definition of “high volatility” is not sacred. Risk teams may care about moderately elevated volatility, severe tail volatility, or multiple tiers.

The notebook tests this by holding scores fixed and changing the high-volatility threshold from the 70th to the 90th percentile. This is not a refit. It asks whether the same scores still rank risk well under alternate definitions.

This is valuable from a business perspective. TabPFN is robust across the 70th to 85th percentile definitions. TabICL becomes strongest at the stricter 90th percentile definition. That suggests TabICL may be especially worth inspecting for more severe tail-risk definitions, even though TabPFN is stronger at the primary threshold.

Market regime behavior

The 2020-forward holdout is not one market. It contains different regimes.

The quiet 2023 regime is intentionally absent from this table because it has no positive high-volatility rows under the primary target definition, so AP and ROC AUC are undefined. That absence is itself informative: some market blocks do not contain enough target events to support a meaningful model comparison.

This is probably the most important risk-management lesson in the run. Model quality is regime-dependent.

In 2020, a VIX z-score is highly informative because market-implied volatility was extreme relative to its own history. In 2022, range-based realized-volatility estimators work very well because the high-volatility label dominates the year. In the later 2024-2026 period, where the positive rate is much lower, direct TabPFN has the best AP.

For a business user, this means the model should not be monitored only with one full-holdout AP number. It should be monitored by market state. A risk score that works in crisis conditions may not be the best score in quiet conditions, and the reverse can also be true.

Score-decile sanity check

A risk score should concentrate future high-volatility events in its highest score buckets. The decile check makes that visible.

This is the kind of diagnostic one might want in a risk dashboard. The top bucket should be meaningfully different from the bottom bucket. Most of the strong models pass that test. The VIX-realized-volatility gap fails it badly in this construction, which explains its poor AP and ROC AUC.

Runtime and complexity

Runtime should be interpreted carefully. This is notebook workflow time, not a controlled benchmark. Still, the cost signal is useful.

The domain rules are effectively free. VIX close gives AP 0.633 with no model fitting. Parkinson volatility gives AP 0.628 with no model fitting. That is a serious business baseline.

Direct TabICL takes 32.0 seconds in this run and gives AP 0.627. Direct TabPFN takes 69.4 seconds and gives AP 0.664. Those are reasonable workflow costs for an offline research notebook.

The raw all-history XGBoost incumbent takes 1592.9 seconds because it includes randomized hyperparameter search. It gives AP 0.611. That does not mean XGBoost is bad. It means that in this notebook configuration, the extra tuning cost does not buy a stronger holdout ranking than the simpler domain rules or direct TabPFN/TabICL.

Calibration and probability quality

Ranking and probability quality are different. A model can sort high-risk dates well while still producing probabilities that are too high or too low.

For the selected model configurations that output probabilities:

Lower is better for these diagnostics. The probability diagnostics are reasonably close across the learned models. I would still treat these scores primarily as ranking and risk-prioritization tools unless a downstream workflow explicitly needs calibrated probabilities.

Risk-control diagnostic

This section is not a trading strategy. It is a decision diagnostic.

The notebook asks: if a score marks a date as high risk, what happens if exposure is reduced according to a pre-specified rule? The selected policy uses a 20% target action rate and a 0.5 high-risk weight. The threshold is chosen from the calibration window, so the realized action rate in the holdout can differ from 20%. Transaction costs are included at 1 basis point.

The result shows the tradeoff clearly:

Buy-and-hold has the highest final equity in this period, but it also has the largest drawdown. Several risk-score policies reduce drawdown materially while giving up some final equity. That is the actual business tradeoff: risk control versus participation.

The best Sharpe in the selected policy comes from VIX z-score, not from TabPFN. TabPFN gives a more compact classification queue at 50% recall and the highest AP point estimate, but risk-policy utility is not identical to classification ranking.

Uncertainty

The notebook uses 300 calendar-month block-bootstrap samples because the 20-day target windows overlap and market regimes cluster in time.

The AP confidence intervals overlap heavily:

This limits the claim. I would say TabPFN has the best AP point estimate in this run. I would not say it is conclusively better than VIX, TabICL, XGBoost, or the strongest realized-volatility rules.

Summary and conclusion

This post is less about proving that one model wins and more about understanding how a volatility-regime score could create business value.

Direct TabPFN looks promising as a compact risk-ranking model. It has the strongest holdout AP point estimate, the most compact 50% recall queue, and good score-decile separation. TabICL is also worth attention: it is competitive at the primary threshold, faster than TabPFN in this run, and strongest at the 90th percentile sensitivity threshold.

XGBoost is a useful classical incumbent but not the best business tradeoff in this notebook configuration. Its AP is respectable, but the randomized-search workflow cost is high relative to the value added over domain rules and direct TabPFN/TabICL scoring.

VIX and realized-volatility rules remain essential baselines. VIX is transparent, economically meaningful, free to compute, and nearly tied with TabPFN at the 80% recall operating point. VIX z-score is especially useful at very high recall and in the risk-control diagnostic.

The risk-control diagnostic shows that useful classification scores do not automatically translate into the best portfolio-style tradeoff, and the regime diagnostics show that the full holdout is not one stable market state. Moreover, no single metric is enough. AP, alert queues, threshold sensitivity, regime behavior, calibration, runtime, and risk-control diagnostics each answer a different business question.

The conclusion is that tabular foundation models can add value in a finance risk-scoring workflow, but only when they are evaluated next to simple domain rules, operating constraints, regime shifts, and decision diagnostics.

In this post, I am turning the direction of this series toward quantitative finance. I am starting with volatility-regime forecasting because it is a natural bridge between supervised tabular ML and a practical finance workflow. The question is:

Can we identify market states where future risk is unusually high?

More precisely, I build a supervised tabular workflow that uses information available at the close of date t to score whether SPY’s realized volatility over the next 20 trading days will be high. This is not a price forecast, a trading signal, or an investment recommendation. It is a workflow test: can TabPFN, TabICL, XGBoost, TFM embeddings, and simple volatility-domain rules produce useful high-volatility rankings under a chronological market-regime split?

The main result is that direct TabPFN and TabICL look more useful as scoring models than the appended-embedding workflows look as feature generators for XGBoost. In this run, direct TabPFN gives the strongest holdout Average Precision point estimate, VIX remains a very strong domain baseline, and the downstream XGBoost embedding variants do not improve ranking under substantial market-regime shift.

The second point is about evaluation. Tabular foundation models should be evaluated as workflow components, not only as model names on a leaderboard. Direct prediction, offline embeddings, calibration, runtime, and risk-control thresholds can tell different stories. This notebook is part of my learning-by-building process, and the goal is to make those workflow tradeoffs visible in a quantitative-finance setting.

Because this is the first version of the workflow, I frame the results as exploratory evidence under regime shift, not as a definitive benchmark or trading claim.

You can find the notebook in my GitHub repository here or you can also clone it directly on Kaggle. The notebook is meant to be run with GPU enabled.

Background and scope

To follow this notebook, the most useful references from earlier posts are P4, P10, P12, P14, P15, P16, and P17.

P4 introduced the posterior predictive distribution viewpoint. That remains useful here because TabPFN and TabICL are not trained from scratch in the same way as XGBoost. They use labelled rows as context and produce predictions for query rows. P10 introduced row embeddings, which are reused here as offline features. P12 showed how a time-indexed problem can be converted into a tabular prediction problem. P14 compared TabPFN, TabICL, and standard supervised ML models. P15 to P17 moved the discussion toward practical workflow tests: chronological splits, rare-event ranking, alert queues, calibration, runtime, and embedding-enhanced XGBoost workflows.

Today’s notebook keeps that workflow-testing style but changes the domain. Instead of fraud detection, the task is volatility-regime forecasting. The models are still used in supervised tabular form, but the data now comes from public market, volatility, macro, and credit series. The evaluation is also more finance-specific: VIX, the Cboe Volatility Index that summarizes market-implied 30-day volatility expectations for the S&P 500, and recent realized volatility become substantive domain baselines. The split must respect time, and the final diagnostic asks whether scores can support a simple risk-control rule. That rule is only a diagnostic. It is not a trading recommendation.

Volatility-regime forecasting as tabular classification

The raw financial object is a price series. Let \(P_t\) be the adjusted close price of SPY on trading day \(t\). A common daily log return is:

Volatility is the variability of returns. In this notebook, the target is based on future realized volatility over the next \(H=20\) trading days. A simple way to write the future realized volatility from signal date \(t\) is:

Here, \(H=20\) is the forecast horizon, and \(\bar{r}_{t,H}\) is the average of the future returns \(r_{t+1}, \ldots, r_{t+H}\). The multiplier \(\sqrt{252}\) annualizes daily volatility under the usual convention of about 252 trading days per year. This matches the notebook’s use of a rolling standard deviation of future log returns. Some finance texts define realized volatility from sums of squared returns without demeaning; this post uses the rolling-standard-deviation convention because that is what the notebook computes. Because this formula uses future returns, \(RV_{t,H}\) is not a feature. It is the target we are trying to forecast.

The notebook turns this into a binary classification problem. Let \(q\) be a high-volatility threshold estimated only from pre-calibration history. Then the label is:

Here, \(y_t=1\) means that the next 20 trading days fall into a high-volatility regime, and \(y_t=0\) means they do not. The indicator \(\mathbf{1}\{\cdot\}\) returns 1 when the condition is true and 0 otherwise. The notebook uses \(\geq q\), so rows exactly at the threshold are included in the high-volatility class.

The supervised learning table then has the familiar form:

Here, \(\mathcal{D}\) is the labelled tabular dataset and \(n\) is the number of rows. The row \(x_t\) contains features available at or before the close of date \(t\). Examples include recent returns, recent realized volatility, drawdowns, moving-average distances, VIX, interest rates, credit spreads, and cross-asset relationships. The label \(y_t\) uses future SPY realized volatility and is only used for training and evaluation.

This is similar to the time-series framing in P12: the sequence problem is converted into a supervised tabular problem. The difference is that P12 predicted future values directly, while today’s notebook predicts whether a future risk measure crosses a regime threshold.

Why this is not a price forecast

A price forecast asks for a predicted future price or return, for example:

That is not what this notebook does. It asks for a score:

Here, \(s_t\) is the model score for date \(t\), \(\mathbb{P}(\cdot)\) denotes probability, and \(\mathcal{D}_{\text{train}}\) is the training subset of the labelled table. The score is an estimate of the probability, or at least the rank-order risk, that the next 20 trading days will be high-volatility. This distinction matters. A high score does not say that SPY will go up or down. It says that the row has features associated with higher future realized volatility in the context of the fitted workflow.

This is why the evaluation focuses on ranking, calibration, and operating points rather than directional price accuracy. A useful volatility-regime score should put many future high-volatility dates near the top of the ranked list, and if it is interpreted as a probability, its probability scale should also be checked.

Chronological validation and leakage

In ordinary tabular demos, it is common to use random train-test splits. That would be inappropriate here. Financial data is time ordered, market regimes change, and future information must not leak into past decisions.

The core rule is:

At signal date \(t\), features may use information available at or before \(t\), but the label uses returns after \(t\).

So the notebook uses chronological windows rather than random splits. The earlier windows provide context, tuning data, and calibration data. The final 2020-forward period is held out as the future-facing evaluation window.

This design handles several separate questions:

• Can the model learn from earlier market conditions and generalize to later conditions?

• Does the high-volatility threshold come from pre-holdout history rather than from the final holdout?

• Are TabPFN and TabICL context labels kept separate from downstream XGBoost labels when embeddings are used?

• Does calibration use a separate window rather than the same rows used for final holdout evaluation?

This does not remove every limitation. The future 20-day realized-volatility targets overlap across nearby dates. For example, the target for date \(t\) and the target for date \(t+1\) share many future return observations. That is why independent-and-identically-distributed, or IID, assumptions are weak here and why the notebook later uses calendar-month block bootstrap for uncertainty.

Ranking metrics and operating points

The positive class is not fraud-rare in the same way as the credit-card fraud dataset in P15 to P17, but this is still an alert-style problem. The model produces a score for each date, and we sort dates from highest predicted risk to lowest predicted risk.

Precision asks:

Among dates flagged as high risk, what fraction were actually followed by high realized volatility?

Recall asks:

Among all future high-volatility dates, what fraction did the score identify within the flagged set?

Average Precision, or AP, summarizes the precision-recall curve. It is useful because it evaluates the ranked list across many possible thresholds. ROC AUC is also reported. ROC AUC asks how often a randomly chosen high-volatility row is ranked above a randomly chosen low-volatility row. I still treat AP and operating-point metrics as more directly relevant here because the output is naturally used as a ranked risk queue.

Operating points ask more concrete questions:

• If I inspect the top 10% or top 20% highest-risk dates, how much future high volatility do I capture?

• If I want 80% recall, how many dates must be placed into the alert queue?

• If I want 90% recall, does the queue become too broad to be useful?

These questions are important because two models can have similar AP but behave differently at the threshold a practitioner actually uses.

Calibration is different from ranking

Ranking quality and probability quality are different. A model can sort high-risk dates above low-risk dates while still producing probability values that are too high or too low.

If the score is only used for ordering, monotonic transformations do not matter much. If the score is interpreted as:

then calibration matters. A calibrated model should have the property that, among rows receiving scores near 0.30, roughly 30% should actually be positive. The notebook therefore reports probability diagnostics such as Brier score, log loss, and expected calibration error.

Brier score is the mean squared error of predicted probabilities:

Log loss penalizes confident wrong probabilities more sharply:

Expected calibration error, or ECE, bins predictions by score and compares average predicted probability with observed event frequency inside each bin. The notebook reports ECE with 10 score bins. These are probability-quality diagnostics, not replacements for AP.

For Brier score, log loss, and ECE, lower values are better. For AP and ROC AUC, higher values are better.

Classical supervised ML versus tabular foundation models

For a classical supervised model such as XGBoost, fitting means learning a task-specific model from the current training table. Conceptually, this is:

Here, \(\mathcal{F}\) is the model class, \(\ell\) is the training loss, and \(\hat{f}\) is the fitted model. Hyperparameter tuning then selects a configuration using validation data.

For TabPFN and TabICL, the mental model is different. The large model is pretrained. The labelled rows in .fit() are better understood as task context rather than as a full weight-training dataset. Using notation closer to P4, the prediction is closer to:

where \(x_{\text{new}}\) is a new query row, \(y_{\text{new}}\) is its unknown label, \(X_{\text{context}}\) is the context feature matrix, and \(y_{\text{context}}\) is the corresponding context-label vector. This is the same posterior-predictive-distribution idea discussed in P4, adapted to the volatility-regime label used here.

This gives two possible workflow roles for tabular foundation models:

• Direct scorer: TabPFN or TabICL receives labelled context rows and directly predicts high-volatility probabilities or scores for later rows.

• Representation generator: TabPFN or TabICL produces row embeddings, and those embeddings are appended to raw features for a downstream model such as XGBoost.

The second role is the same integration pattern explored in P16 and P17. It is not guaranteed to help. Embeddings may add useful information, but they may also be redundant with the raw features, interact poorly with a small or quiet tuning window, or make the downstream feature space harder to tune. That is why this notebook evaluates direct TFM scoring and embedding-enhanced XGBoost as separate workflow components.

What is in scope today

This post is a first volatility-regime workflow test. It is in scope to ask whether direct TabPFN, direct TabICL, raw XGBoost, embedding-enhanced XGBoost, VIX, and recent realized volatility produce useful high-volatility rankings under a chronological holdout.

It is also in scope to inspect probability calibration, runtime, uncertainty, feature drift, leakage checks, and a simple risk-control diagnostic.

It is not in scope to claim a trading strategy, forecast exact future prices, optimize a portfolio, or produce a definitive benchmark for all financial datasets. The useful contribution is narrower: a reproducible workflow for testing tabular foundation models as components in a realistic volatility-regime forecasting pipeline.

Code and results

With the scope in place, this section walks through the notebook outputs. The practical question is:

Given the information available at the close of date t, can a model score whether SPY’s realized volatility over the next 20 trading days will fall into a high-volatility regime?

I discuss the outputs as a workflow test under regime shift. The notebook compares direct TabPFN and TabICL scoring, XGBoost on raw features, XGBoost on raw features plus TabPFN or TabICL embeddings, simple volatility-domain rules, calibration diagnostics, operating-point behavior, uncertainty, runtime, risk-control diagnostics, drift, and leakage checks.

1. Data and target

The target asset is SPY. The feature set uses public daily data from yfinance, VIX history from Cboe, and macro/credit series from FRED. The market features include SPY, QQQ, IWM, TLT, IEF, GLD, HYG, EEM, and VNQ. The macro and credit inputs include 10-year Treasury yield, 2-year Treasury yield, the 10y-2y yield curve, high-yield OAS, investment-grade OAS, and the effective federal funds rate.

This public-data choice matters. It makes the notebook reproducible, but it is not the same thing as an institutional point-in-time market data system. So I read the results as a research workflow demonstration rather than as a deployable market-data pipeline.

The target is a binary high-volatility label. The notebook computes SPY’s future 20-trading-day realized volatility, annualizes it, and labels a row as high-volatility if that future realized volatility is at least a pre-holdout threshold. The threshold is estimated only from history ending on 2017-12-29. In this run, the threshold is the 80th percentile of pre-calibration history, which gives:

• Target asset: SPY.

• Forecast horizon: 20 trading days.

• High-volatility threshold: 0.200602 annualized realized volatility.

• Threshold source window: data available through 2017-12-29.

• Rows after target filtering: 5,094.

This thresholding policy is important because it prevents the 2020-forward holdout from defining what “high volatility” means. The holdout can then be used as a future-facing test period rather than as part of the label-design procedure. This is a practical approximation for a public-data notebook, not a claim that this threshold would be the right operational definition in every volatility workflow.

2. Feature table

The feature table is daily and indexed by signal date. The notebook builds features that would be available at or before the signal date: returns, realized volatility, downside volatility, drawdowns, moving-average distances, volume changes, VIX features, rates, credit-spread features, and cross-asset correlations.

The feature policy is intentionally broad but audited. The notebook starts with 297 candidate features. Seven sparse credit-spread-derived features are dropped because they do not have enough usable pre-calibration history. The retained feature table has:

• 290 base features.

• 284 missingness indicators.

• 574 final model features.

The missingness indicators are part of the feature policy rather than a cosmetic detail. Public macro and credit series do not all start at the same time or update with the same density. If missingness itself is informative, the model can use those indicators. If missingness is mainly a data artifact, later leakage and drift checks help keep that visible.

3. Chronological split design

The split design is central to the notebook because volatility forecasting is not an iid random-split problem.

The notebook uses these chronological windows:

• TabPFN/TabICL representation context: 2006-01-03 to 2011-12-30, with 1,511 rows and 36.0% high-volatility rows.

• Fair downstream tuning window: 2012-01-03 to 2017-12-29, with 1,509 rows and 4.0% high-volatility rows.

• Calibration window: 2018-01-02 to 2019-12-31, with 503 rows and 22.3% high-volatility rows.

• Holdout: 2020-01-02 to 2026-04-02, with 1,571 rows and 23.3% high-volatility rows.

• Raw all-history final training window: 2006-01-03 to 2019-12-31, with 3,523 rows and 20.3% high-volatility rows.

The split rates already tell us why this is a hard experiment. The fair downstream tuning window is unusually quiet: only 60 of 1,509 rows are high-volatility rows. The holdout then includes COVID, the 2022 inflation/rate shock, the post-2022 lower-volatility period, and the 2025/early-2026 episodes. A model selected in a quiet 2012-2017 validation setting may not transfer cleanly into a noisier 2020-forward regime.

The representation-context split also keeps the embedding comparison honest. TabPFN and TabICL use the early context labels to condition their representations. Therefore, the fair downstream XGBoost workflows using those embeddings do not reuse that same context window as downstream XGBoost training labels. The raw all-history XGBoost model is reported separately because it answers a different practical question: what if a classical model simply uses all pre-holdout labels instead of reserving some of them for a TFM context?

4. Model configurations

The notebook compares three kinds of scorers.

First, it includes simple domain rules:

• Rule[VIX close], which uses VIX as a volatility-domain score.

• Rule[SPY realized volatility 20d], which uses recent SPY realized volatility as a score.

Second, it includes XGBoost workflows:

• XGBoost[Raw], trained on the fair raw feature set.

• XGBoost[Raw all-history incumbent], a stronger raw-feature reference that uses all pre-holdout labels.

• XGBoost[Raw + TabPFN embeddings], with 574 raw features plus 192 TabPFN embedding columns, for 766 total features.

• XGBoost[Raw + TabICL embeddings], with 574 raw features plus 512 TabICL embedding columns, for 1,086 total features.

Third, it includes direct TFM scoring:

• TabPFN[Direct all-history].

• TabICL[Direct all-history].

This distinction is important. Direct TFM scoring and TFM embeddings answer different workflow questions. Direct scoring asks whether TabPFN or TabICL can act as the classifier. The embedding path asks whether a TFM representation improves a downstream XGBoost model. This run gives different answers to those two questions.

5. Tuning and validation caveat

The XGBoost workflows use randomized hyperparameter search with chronological validation folds. This is better aligned with the time-ordered nature of the problem than random cross-validation. However, the fair raw-vs-embedding path has only one usable chronological fold after the positive-count checks. That fold has 920 training rows with 26 high-volatility rows and 196 validation rows with 33 high-volatility rows.

That is the main experimental caveat in this section. The run is internally coherent, but the fair XGBoost model-selection signal is fragile. I would not read the XGBoost rankings as a definitive statement about all possible XGBoost configurations. I read them as the result of this specific, audited workflow under a difficult regime shift.

6. Main holdout results

The final holdout has 1,571 rows and 366 high-volatility rows, so the high-volatility base rate is 23.3%. Average Precision is the main ranking metric because this is naturally an alert-style problem: the score is useful if it pushes future high-volatility periods toward the top of the ranked list.

The main holdout results are:

The result is not “TFM embeddings improve XGBoost.” In this run, they do not. The more useful result is that direct TabPFN, direct TabICL, and simple volatility-domain scores are competitive with the tuned XGBoost workflows, and in some views they are ahead of them.

The clearest positive result is direct TabPFN’s ranking quality. It has the highest AP and the highest Top 20% Recall. The most important baseline result is VIX. VIX has the highest ROC AUC and the most efficient 80% recall queue overall. This is a useful reminder that, in a finance workflow, a simple domain signal can be a substantive baseline.

The direct TabICL result is also useful, but less dominant. Its AP is essentially tied with the raw all-history XGBoost incumbent, while its runtime is much lower. The embedding-enhanced XGBoost result is the clearest non-improvement result: adding TabPFN or TabICL row embeddings to raw features lowers downstream XGBoost AP in this holdout.

7. Precision-recall curves

This figure shows precision on the vertical axis and recall on the horizontal axis. The dashed horizontal line is the holdout base rate, about 0.233. A model above that line is ranking high-volatility rows better than random ordering.

The first thing to notice is that all models are above the base-rate line for much of the curve. This suggests that the scores contain useful ranking signal in this holdout, rather than behaving like random ordering.

The second thing to notice is the grouping. Direct TabPFN, VIX, raw all-history XGBoost, TabICL direct, and recent SPY realized volatility form the stronger group. They are not separated by a huge margin, but they sit above the fair raw XGBoost and the embedding-enhanced XGBoost workflows through much of the useful recall range.

The third thing to notice is the embedding result. If TabPFN or TabICL embeddings were improving XGBoost as general-purpose features in this workflow, I would expect their curves to move above raw XGBoost or at least approach the stronger group. Instead, the embedding curves sit lower. The TabPFN embedding workflow has some early precision strength, but it does not hold up through the middle of the curve. The TabICL embedding workflow has the lowest AP in this run.

This figure is why I would frame the experiment as a workflow-component test. Direct TFM scores can be useful even when offline TFM embeddings do not help a downstream tree model.

8. Operating points

AP summarizes the whole ranked list, but an actual alerting or risk-control workflow usually operates at a finite queue size. That is why the notebook also reports target-recall and fixed-top-rate views.

At 50% recall, the most efficient queues are:

• TabPFN direct: 271 alerts, precision 0.675.

• XGBoost raw all-history: 273 alerts, precision 0.670.

• VIX rule: 282 alerts, precision 0.649.

At this moderate-recall operating point, TabPFN direct, raw all-history XGBoost, and VIX are very close. A practitioner choosing among them would probably care about stability, calibration, cost, and interpretability, not only the two-alert difference between the first two rows.

At 80% recall, the ordering changes:

• VIX rule: 611 alerts, precision 0.480.

• TabPFN direct: 639 alerts, precision 0.459.

• SPY realized-volatility rule: 659 alerts, precision 0.445.

• XGBoost raw all-history: 725 alerts, precision 0.404.

Here VIX is the most efficient score. This is an important practical result. Direct TabPFN has the best AP, but VIX gives the shortest 80% recall queue.

At 90% recall, VIX has the shortest queue among the reported scores:

• VIX rule: 865 alerts, precision 0.382.

• XGBoost raw all-history: 1037 alerts, precision 0.318.

• TabPFN direct: 1061 alerts, precision 0.311.

• TabICL direct: 1065 alerts, precision 0.310.

The fixed top-rate view tells a similar story but emphasizes TabPFN’s broad ranking strength:

• At the top 5% of holdout rows, TabPFN direct, TabICL direct, and XGBoost raw plus TabPFN embeddings each reach 18.0% recall.

• At the top 10%, TabPFN direct leads with 33.1% recall.

• At the top 20%, TabPFN direct leads with 56.0% recall, narrowly ahead of VIX and raw all-history XGBoost.

• At the top 30%, TabPFN direct leads with 71.9% recall, narrowly ahead of the recent-SPY-realized-volatility rule.

So the operating-point conclusion is more nuanced than the AP ranking. Direct TabPFN has the best broad-ranking point estimate in this run. VIX is the strongest high-recall queue baseline. The best model depends on the operating point.

9. Uncertainty

The notebook uses 300 calendar-month block-bootstrap iterations. This is more appropriate than IID row bootstrap here because the 20-day target windows overlap and market regimes are temporally clustered.

For AP, the point estimates and 95% intervals are:

The intervals overlap heavily among the stronger models. That does not erase the point-estimate story, but it limits the claim. I would not say that TabPFN “wins” as a definitive benchmark result. The more defensible statement is that TabPFN direct has the strongest point AP in this run, while VIX, TabICL direct, and raw all-history XGBoost are close enough that the result should be treated as exploratory.

The embedding result is still meaningful in a workflow sense. The embedding-enhanced XGBoost variants do not show a clear downstream ranking benefit here, and they cost more than raw XGBoost.

10. Year-by-year behavior

The yearly breakdown explains why a single holdout AP number is not enough.

High-volatility labels are concentrated unevenly:

• 2020: 131 high-volatility rows out of 253.

• 2021: 3 high-volatility rows out of 252.

• 2022: 178 high-volatility rows out of 251.

• 2023: 0 high-volatility rows out of 250.

• 2024: 14 high-volatility rows out of 252.

• 2025: 38 high-volatility rows out of 250.

• 2026 partial: 2 high-volatility rows out of 63.

This is not a stable IID classification problem. Entire years can be mostly high-volatility, mostly quiet, or impossible to evaluate with AP and ROC AUC because there are no positive labels.

The yearly AP leaders also change:

• In 2020, XGBoost raw plus TabICL embeddings has the highest AP at 0.778.

• In 2021, there are only 3 positives; raw XGBoost has the highest AP at 0.071, but the sample is too small for a strong conclusion.

• In 2022, raw XGBoost leads AP at 0.842, with VIX close behind at 0.826.

• In 2023, there are no positives, so AP and ROC AUC are undefined.

• In 2024, there are only 14 positives; VIX has the highest AP at 0.062.

• In 2025, raw all-history XGBoost has the highest AP at 0.563.

• In the partial 2026 window, raw all-history XGBoost has the highest AP at 0.171, but there are only 2 positives.

This is one of the most useful scientific lessons from the notebook. The full-holdout AP aggregates several different market regimes. A model can look useful overall while behaving differently across the COVID shock, the 2022 inflation/rate shock, and quieter years. That is why I read this as a regime-sensitivity workflow, not a final model ranking.

11. Runtime versus ranking quality

The runtime plot is a workflow-cost diagnostic, not a controlled timing benchmark. It puts holdout AP on the vertical axis and total workflow seconds on the horizontal axis, so the most attractive region is the upper-left: higher ranking quality with lower measured workflow cost.

The caveat is that the timings are not all produced under the same validation workload. In this run, the raw all-history XGBoost incumbent used three chronological validation folds, while the fair raw-vs-embedding XGBoost workflows had only one usable fold after the positive-count checks. Direct TabPFN and direct TabICL were not tuned through rolling CV in this notebook; each was fitted once on the all-history pre-holdout context and then scored on holdout. A fair timing benchmark would rerun comparable workflows under a matched evaluation policy. For this post, I read the figure as a record of the cost of this specific notebook execution, alongside the ranking results.

Under that interpretation, the domain rules remain important. They sit at essentially zero workflow time and are strong, especially VIX with AP 0.633 and ROC AUC 0.832. This reinforces the earlier point that a simple market-implied volatility signal is a substantive baseline.

The direct TFM points are also useful in this view, with that one-fit interpretation in mind. TabICL direct finishes in 25.6 seconds with AP 0.612. TabPFN direct takes 61.0 seconds and has the highest AP point estimate at 0.655. These numbers are not meant to be a universal speed comparison against every possible XGBoost setup, but they show that direct TFM scoring is not expensive in this particular workflow.

The embedding-enhanced XGBoost points are less attractive in this run because they add feature width without improving holdout ranking. TabPFN embeddings add 192 columns, and TabICL embeddings add 512 columns. The final feature sets grow to 766 and 1,086 columns respectively, but the larger feature matrices do not improve AP in the 2020-forward holdout.

So the practical takeaway is measured rather than absolute: in this volatility-regime workflow, direct TFM scoring looks more promising than using TFM row embeddings as appended features for downstream XGBoost, while the exact runtime comparison should be revisited with a matched fold policy.

12. Calibration and probability quality

The calibration figure plots mean predicted probability against observed high-volatility rate by score bin. The diagonal line is perfect calibration: if the model says 30%, about 30% of those rows should be positive.

The direct TFM probability diagnostics are relatively clean among the reported configurations:

• TabPFN direct: Brier 0.125, log loss 0.406, ECE 0.039.

• TabICL direct: Brier 0.134, log loss 0.427, ECE 0.041.

• XGBoost raw all-history: Brier 0.133, log loss 0.424, ECE 0.040.

• XGBoost raw: Brier 0.188, log loss 0.601, ECE 0.154.

• XGBoost raw plus TabPFN embeddings: Brier 0.167, log loss 0.701, ECE 0.105.

• XGBoost raw plus TabICL embeddings: Brier 0.194, log loss 0.638, ECE 0.163.

The main calibration issue is the fair XGBoost probability scale. The calibration-quality view shows that sigmoid calibration improves those probability diagnostics. For raw XGBoost, ECE improves from 0.203 to 0.093 and log loss improves from 0.735 to 0.432 on the holdout rows in the calibration-quality comparison. For raw plus TabPFN embeddings, ECE improves from 0.162 to 0.094. For raw plus TabICL embeddings, ECE improves from 0.189 to 0.095.

However, sigmoid calibration is monotonic, so it does not change the ranking. It can make the scores more interpretable as probabilities, but it does not change a lower AP curve into a higher AP curve. This is another example of why the notebook evaluates the workflow through multiple lenses: ranking quality, probability quality, runtime, and operating thresholds can tell different stories.

13. Risk-control diagnostic

This figure needs careful framing. It is not evidence of a trading strategy. It is a diagnostic that asks whether a high-volatility score can support a simple pre-specified risk-control rule.

The policy uses calibration-window score quantiles to define high-risk periods. On holdout, the policy reduces SPY exposure during periods scored as high risk. The selected view discussed below uses the notebook’s 20% target action-rate setting and a 0.5 high-risk weight, meaning the policy holds half SPY exposure during periods classified as high risk. The figure compares growth of 1.0 before taxes, with transaction costs included in the notebook’s policy accounting.

For learned models in this diagnostic, the selected policy rows use calibration-base score variants where needed because the calibration window is reserved for threshold selection. I shorten the labels below for readability.

Buy-and-hold has the highest final equity at 2.219, but it also has the largest max drawdown at -0.337 and Sharpe 0.726. Several score-driven risk-control policies have lower final equity but smoother paths:

• XGBoost raw plus TabICL embeddings: Sharpe 0.888, final equity 2.155, max drawdown -0.277.

• XGBoost raw plus TabPFN embeddings: Sharpe 0.838, final equity 2.098, max drawdown -0.263.

• XGBoost raw: Sharpe 0.824, final equity 2.030, max drawdown -0.255.

• TabPFN direct: Sharpe 0.817, final equity 1.843, max drawdown -0.199.

• TabICL direct: Sharpe 0.807, final equity 1.835, max drawdown -0.199.

• VIX rule: Sharpe 0.742, final equity 1.690, max drawdown -0.199.

This is interesting because the best high-volatility classifiers are not automatically the best risk-control policies. The policy result depends on threshold timing, exposure reduction, transaction costs, and the path of SPY returns, not only on classification AP. The right conclusion is not “use this as a strategy.” The right conclusion is that score utility can differ from classification ranking, so a production research workflow should test both.

14. Drift and leakage checks

The drift table confirms that the holdout is a real distribution-shift test. The notebook reports population stability index, or PSI, as a drift diagnostic. Larger PSI values indicate larger differences between the pre-holdout and holdout feature distributions. The largest drift features include:

• Computed 10y-2y yield curve: preholdout mean 1.333, holdout mean 0.240, PSI 5.42.

• FRED 10y-2y yield curve: preholdout mean 1.333, holdout mean 0.240, PSI 5.41.

• SPY-TLT 126-day correlation: preholdout mean -0.393, holdout mean -0.043, PSI 3.02.

• Federal funds rate: preholdout mean 1.300, holdout mean 2.783, PSI 2.43.

• QQQ-TLT 126-day correlation: preholdout mean -0.340, holdout mean -0.003, PSI 2.36.

This drift is not incidental. The 2020-forward holdout contains a different macro environment from much of the pre-holdout period: COVID, the zero-rate aftermath, inflation, rate hikes, and changed stock-bond correlation behavior. That helps explain why validation and holdout behavior can diverge, and why simple market-implied signals like VIX remain strong.

The leakage checks are also mostly clean:

• Target columns are excluded from the feature set.

• The high-volatility threshold is estimated before calibration and holdout.

• The chronological split order passes.

• The TabPFN/TabICL representation context is not reused as downstream XGBoost training labels in the fair embedding path.

• The raw all-history XGBoost incumbent is reported separately from the fair raw-vs-embedding comparison.

• Overlapping 20-day labels are recorded as a known limitation, which is why the uncertainty view uses calendar-month block bootstrap.

• The risk-policy section is explicitly marked as educational and diagnostic, not an investment recommendation.

This is the final reason I would keep the claims cautious. The notebook passes the checks needed for a clean public-data experiment, but the holdout is a hard regime-shift period and the target windows overlap. That makes the result scientifically useful, but not definitive.

Known limitations

The issues are experimental rather than mechanical:

1. The fair raw-vs-embedding XGBoost path has only one usable chronological CV fold. This makes the downstream XGBoost tuning less stable than ideal.

2. The fair tuning window has a much lower high-volatility rate than the later calibration and holdout windows. This can create model-selection mismatch.

3. The raw all-history incumbent is not strictly fair to embedding workflows because it uses more labels, but it is an important practical baseline. I keep that distinction explicit rather than treating it as the same comparison as the fair raw-vs-embedding path.

4. Direct all-history TFM models and embedding-enhanced XGBoost models answer different questions. Direct TFM scores can be strong even if embeddings do not help XGBoost.

5. The risk-control section is path-dependent and should stay framed as a diagnostic. It should not be described as investment evidence.

6. Year-level metrics are unstable in years with very few positives. 2023 has zero positives, so AP and ROC AUC are undefined. 2021 and 2026 have too few positives for strong model comparisons.

   Thanks for reading DSAIEngineering Newsletter! Subscribe for free to receive new posts and support my work.

Summary and conclusion

This first volatility-regime notebook gives a useful but cautious result. Direct tabular foundation model scoring looks promising for this workflow, especially direct TabPFN. On the 2020-forward holdout, TabPFN direct has the strongest Average Precision point estimate and the strongest Top 20% Recall. Direct TabICL is also competitive with the raw all-history XGBoost incumbent while being much faster in this run.

The result is not a simple “foundation models beat classical ML” story. VIX remains a very strong domain baseline and is the most efficient score at some high-recall operating points. That is an important practical anchor. If a market-implied volatility signal is already strong, then any ML workflow has to justify its added complexity against that baseline.

The clearest non-improvement result is the embedding path. In this run, appending TabPFN or TabICL row embeddings to raw features does not improve downstream XGBoost ranking. That does not mean TFM embeddings are useless in general. It means that, in this volatility-regime setup, useful direct TFM scores do not automatically translate into useful offline features for a downstream tree model.

The broader lesson is that tabular foundation models should be evaluated as components inside a workflow. Direct scoring, embedding generation, calibration, runtime, alert thresholds, risk-policy diagnostics, and drift checks answer different questions. A model can look strong in one view and weaker in another. For this reason, I read today’s result as an exploratory workflow test under market-regime shift, not as a definitive benchmark or trading claim.

Outlook

This is the first attempt at this volatility-regime workflow, and it needs more work. The next step is to make the experiment more robust before drawing stronger conclusions.

The first improvement is validation depth. The fair raw-vs-embedding XGBoost path had only one usable chronological CV fold after the positive-count checks. I want to redesign the split or validation policy so that the downstream XGBoost comparison has a stronger model-selection signal.

The second improvement is regime analysis. The yearly results show that 2020, 2022, quiet years, and partial 2026 behave very differently. I want to inspect performance by market phase more carefully rather than relying mainly on full-holdout aggregates.

The third improvement is workflow realism. I want to add stronger feature-lag checks, richer drift diagnostics, sensitivity tests for the high-volatility threshold, and comparisons against more finance-standard volatility baselines. I also want to keep the risk-control section clearly diagnostic while making it more rigorous.

My plan is to continue iterating on this example until it becomes a reusable volatility-regime testbench for TabPFN, TabICL, and classical supervised tabular models.

This post builds on that last idea. The question is not whether TabPFN or TabICL should replace XGBoost as the fraud model. The question is more practical:

Can offline row embeddings from TabPFN or TabICL improve a production-style XGBoost fraud-detection workflow enough to justify the extra representation step?

I am learning these models by building examples and testing workflow patterns, so I treat this as a workflow demonstration rather than a benchmark claim. The goal is to make the integration pattern, the possible benefits, and the caveats visible. That can be useful both for labs building tabular foundation models and for data and AI practitioners who want to understand how these models might fit into workflows they already know.

You can find the notebook in my GitHub repository, and you can also clone it directly on Kaggle. The notebook is meant to be run on Kaggle with GPU enabled. It installs cuDF for pandas acceleration, uses CUDA for TabPFN and TabICL, and uses a CuPy-backed XGBoost path so that the downstream scorer also runs on GPU.

Background and scope

To follow the work presented here, a few concepts from earlier posts are useful. I have discussed them in more detail before, so I will use this section to connect the current notebook to that background.

If you need a refresher on row embeddings from TabPFN, you can refer to P10. If you want the broader comparison between TabPFN, TabICL, and standard supervised ML models, P14 is the relevant reference. For the fraud-detection setup, rare-event metrics, and the first direct use of TabPFN and TabICL as fraud scorers, see P15. For the earlier version of the embedding-plus-XGBoost workflow, including the motivation for using embeddings as offline representation features, see P16.

What changed in this notebook

The main integration pattern is the same one introduced in P16: TabPFN and TabICL are still used as offline representation generators, and XGBoost remains the downstream fraud scorer. The models themselves have not changed: the notebook still compares raw XGBoost, a raw all-history XGBoost incumbent, raw + TabPFN embeddings, and raw + TabICL embeddings.

The main changes are in the comparison design and the notebook structure.

First, the combined Raw + TabPFN + TabICL feature set is no longer part of the main comparison. This version evaluates one embedding source at a time. That makes the adoption question cleaner because using both embedding systems together would add a different level of operational cost and complexity.

Second, the tuning setup has been strengthened. Yesterday’s notebook intended to use chronological cross-validation, but the fair raw-vs-embedding path effectively had only one usable fold after the fraud-count checks. That made the tuning signal weaker than the design intended. In this version, the fair raw-vs-embedding comparison uses the downstream tuning history after excluding the representation-context rows, and it has five valid chronological folds. The raw all-history incumbent also uses five chronological folds, so the main model configurations are selected under a more consistent tuning protocol before the holdout results are inspected.

Third, the calibration section is organized more carefully. In this workflow, a calibrated model cannot use the calibration window for base model fitting, because that window is held out for fitting the post-hoc sigmoid calibration map. If I compare a calibrated configuration only against a fully trained uncalibrated configuration, two things change at once: the score transformation and the amount of data used to fit the base XGBoost model. This version therefore keeps calibration-base configurations separate from sigmoid-calibrated configurations. The calibration-base configuration shows the uncalibrated model trained on the same pre-calibration data as the calibrated model. The calibrated configuration then shows what changes after applying the sigmoid calibration step. That makes calibration easier to inspect as its own diagnostic.

Fourth, the notebook now leaves behind cleaner review artifacts: curated result tables, bootstrap uncertainty tables, provenance information, embedding matrix summaries, CUDA memory summaries, and final figures. These files are saved in the Kaggle output directory and are available to download as a zip file after the run completes. That makes the run easier to audit after execution instead of relying only on displayed notebook output.

Evaluation setup reminders

The evaluation logic follows the same principles discussed in P15 and P16. The public fraud dataset has a Time column, so the notebook uses chronological windows rather than random splitting. The percentage partitioning is the same as yesterday’s embedding workflow: earliest 20% as the source window for the TabPFN/TabICL representation context, next 40% for downstream XGBoost training, next 10% for validation, next 10% for calibration, and final 20% for holdout evaluation. The actual TabPFN/TabICL context passed to the embedding models is sampled from the earliest window, while the fair downstream comparison still excludes that full earliest window from XGBoost tuning and fitting. The important point is that embeddings for later rows are generated only from earlier labelled context rows.

The metric logic is also the same as before. Accuracy is not useful for this rare-event fraud dataset, so the notebook reports Average Precision and alert-queue metrics: top-alert recall, top-percent recall, and the number of alerts needed to reach target recall levels.

Calibration remains a diagnostic rather than the main comparison target. The difference in this notebook is not the definition of calibration, but the cleaner organization of calibration-base and sigmoid-calibrated configurations described above. The results section reports Brier score, log loss, ECE 10, and reliability artifacts where they help interpret probability quality. Here, ECE 10 means expected calibration error computed with 10 bins.

With that scope in place, the rest of the post focuses on the experimental results rather than re-explaining the earlier fraud-detection setup.

Results

Comparison design

The notebook compares four model configurations. Raw XGBoost is the standard supervised-learning baseline using the 29 raw model features. Raw all-history XGBoost is a stronger incumbent-style baseline: it still uses only raw features, but it can use all pre-holdout labelled history because it does not need to reserve an earlier window for TabPFN or TabICL context. Raw + TabPFN adds 192 TabPFN embedding features to the raw features, and Raw + TabICL adds 512 TabICL embedding features to the raw features.

This setup separates two practical questions. The raw baselines ask whether standard XGBoost is already strong enough, especially when it can use all available pre-holdout raw history. The embedding configurations ask whether a context-conditioned representation from TabPFN or TabICL adds useful information beyond those raw features.

The tuning design is meant to keep that comparison clear. The embedding configurations cannot use the representation-context labels again as downstream training labels, because those labels were already used to condition the embedding model. The fair raw-vs-embedding comparison therefore excludes the representation-context window from downstream tuning. The all-history raw incumbent is reported separately because it answers a different question: how competitive is a strong raw-feature XGBoost workflow if it simply uses more labelled history instead of adding a foundation-model representation step?

Compared with yesterday’s notebook, the important tuning change is qualitative rather than just numeric: model selection is no longer based on a single usable chronological fold. The fair raw-vs-embedding path and the raw all-history incumbent both use five valid chronological folds before the final holdout is inspected.

The next sections read the results through several views: full-holdout ranking, alert-queue behavior, uncertainty, runtime and memory, and calibration. I use those views together because no single metric captures the whole workflow tradeoff.

Full-holdout results

The full holdout is the deployment-facing view because it keeps the final-window fraud base rate. It contains 56,962 transactions and only 75 fraud cases, so small movements in the top of the ranking can change the interpretation.

The first question is whether the embedding features improve the overall fraud ranking. Average Precision is the main summary for that question. The compact result list is:

• Raw all-history XGBoost: AP 0.8097, workflow time 173.3 seconds, Top 0.5% Recall 0.8533, Top 1% Recall 0.8533, Brier 0.000424, ECE 10 0.000104.

• Raw XGBoost: AP 0.8034, workflow time 147.0 seconds, Top 0.5% Recall 0.8400, Top 1% Recall 0.8667, Brier 0.000403, ECE 10 0.000231.

• Raw + TabICL: AP 0.7970, workflow time 1147.8 seconds, Top 0.5% Recall 0.8267, Top 1% Recall 0.9067, Brier 0.000451, ECE 10 0.000485.

• Raw + TabPFN: AP 0.7909, workflow time 840.2 seconds, Top 0.5% Recall 0.8000, Top 1% Recall 0.8400, Brier 0.000381, ECE 10 0.000184.

I read these numbers in three layers.

First, the point AP ranking favors the raw-feature baselines. Raw all-history XGBoost is highest, and standard raw XGBoost is close behind. If the TabPFN or TabICL embeddings were adding a broadly useful ranking signal, I would expect one of the embedding configurations to move above the raw baselines on AP. That does not happen in this run.

Second, runtime changes the practical bar. Raw XGBoost finishes in about 147 seconds, while the embedding workflows take much longer because they add representation generation and wider downstream feature matrices. For an embedding workflow to be attractive in this kind of setting, I would want to see either a clear ranking improvement, a meaningful operating-point improvement, or some other production value that justifies the additional representation step.

Third, the metrics are seeing different things. TabICL has the best Top 1% recall even though it does not have the best AP. TabPFN has the best Brier score among these four configurations, but weaker ranking and alert behavior. That means the result is not a single-metric result where one configuration is best on every view. It separates ranking quality, alert-queue behavior, probability diagnostics, and engineering cost.

The precision-recall figure shows the same story visually. The dashed horizontal line is the fraud base rate. Because fraud is rare, that line sits very low; a useful model should lift precision far above that line for the transactions it ranks near the top.

The four curves are all well above the base-rate line in the high-score region, which means all four XGBoost-based configurations are learning useful fraud rankings. But the curves are close to each other. There is no clean visual separation where an embedding curve stays clearly above both raw baselines across most of the recall range. If that had happened, it would be stronger evidence that the TabPFN or TabICL representation was improving the ranking broadly.

Instead, the figure suggests a narrower interpretation. The raw all-history and raw XGBoost curves are already strong. The embedding curves are competitive, but they do not visibly dominate. When curves are this close, the figure should not be read alone; the alert-budget results below are needed because fraud teams operate at specific review capacities, not across the whole precision-recall curve at once.

Alert-budget interpretation

The alert-budget view asks a more operational question than AP. Instead of summarizing the whole precision-recall curve, it asks how many transactions a team would need to review to recover a target share of the known fraud cases.

On this full holdout, 80% recall means finding 60 of the 75 fraud cases. At that target:

• Raw all-history XGBoost needs 94 alerts to find 60 frauds, with precision 0.6383.

• Raw XGBoost needs 116 alerts to find 60 frauds, with precision 0.5172.

• Raw + TabICL needs 127 alerts to find 60 frauds, with precision 0.4724.

• Raw + TabPFN needs 199 alerts to find 60 frauds, with precision 0.3015.

At this operating point, the raw all-history incumbent is the most efficient queue. This is what I would expect if the standard raw-feature workflow already captures most of the easy-to-rank fraud cases. If TabPFN or TabICL embeddings had helped strongly at this level, they would have reduced the alert count needed to find those same 60 frauds. They do not do that here.

The picture changes at 90% recall. Here the target is to find 68 of the 75 fraud cases, so the model has to rank deeper into the difficult part of the fraud set:

• Raw + TabICL needs 490 alerts to find 68 frauds, with precision 0.1388.

• Raw all-history XGBoost needs 1,064 alerts to find 68 frauds, with precision 0.0639.

• Raw XGBoost needs 1,177 alerts to find 68 frauds, with precision 0.0578.

• Raw + TabPFN needs 1,598 alerts to find 68 frauds, with precision 0.0426.

This is the main operational nuance. AP favors the raw all-history incumbent, but the 90% recall operating point favors TabICL. One way to read this is that TabICL does not improve the whole ranking enough to lead on AP, but it may move some additional fraud cases into a useful part of the high-recall review queue.

If the AP leader and the 90% recall leader had been the same model, the conclusion would be simpler. Here they differ. That is why I would not summarize this notebook with only AP. A fraud team targeting a compact 80% recall queue might prefer the raw all-history incumbent. A team targeting very high recall could reasonably investigate the TabICL queue behavior further.

For fixed alert budgets on the full holdout:

• At 100 alerts, Raw all-history XGBoost finds 60 frauds and reaches 80.0% recall.

• At 500 alerts, Raw + TabICL finds 68 frauds and reaches 90.7% recall.

• At 1000 alerts, Raw + TabICL still finds 68 frauds and remains at 90.7% recall.

• At the top 0.5% of transactions, Raw all-history XGBoost finds 64 frauds and reaches 85.3% recall.

• At the top 1% of transactions, Raw + TabICL finds 68 frauds and reaches 90.7% recall.

These fixed-budget numbers give the same intuition from the opposite direction. If the team can only review about 100 transactions, raw all-history XGBoost gives the best result. If the team can review around 500 transactions or the top 1% of this holdout, TabICL finds more of the hard-to-catch fraud cases. AP, fixed alert budgets, and target-recall alert counts each answer a different question.

Uncertainty

The notebook uses bootstrap resampling on the full holdout to estimate uncertainty. This matters because the full holdout has only 75 fraud cases. With so few positives, a small number of transactions moving up or down the ranked list can change AP or alert-count estimates.

For full-holdout AP:

• Raw XGBoost has point AP 0.8034, bootstrap median 0.8032, and 95% interval from 0.6972 to 0.8709.

• Raw all-history XGBoost has point AP 0.8097, bootstrap median 0.8140, and 95% interval from 0.7189 to 0.8947.

• Raw + TabPFN has point AP 0.7909, bootstrap median 0.7905, and 95% interval from 0.7061 to 0.8687.

• Raw + TabICL has point AP 0.7970, bootstrap median 0.7973, and 95% interval from 0.7063 to 0.8723.

These intervals overlap heavily. That does not make the point estimates useless, but it does change the strength of the claim. My reading is that raw all-history XGBoost has the best AP point estimate, not that it is clearly separated from every other configuration. If the intervals had been well separated, I would be more comfortable making a stronger ranking claim.

For alerts needed at 90% recall:

• Raw XGBoost has point estimate 1,177, bootstrap median 1,230, and 95% interval from 212 to 9,822 alerts.

• Raw all-history XGBoost has point estimate 1,064, bootstrap median 1,046, and 95% interval from 129 to 7,311 alerts.

• Raw + TabPFN has point estimate 1,598, bootstrap median 1,598, and 95% interval from 451 to 7,972 alerts.

• Raw + TabICL has point estimate 490, bootstrap median 495, and 95% interval from 254 to 3,449 alerts.

The TabICL point estimate remains interesting because it is much lower than the raw baselines, but the intervals are still wide. I would not read this as a deployment recommendation. I read it as a signal that this operating point is worth a more careful follow-up test on richer and larger fraud data.

Runtime and memory

The runtime plot shows the engineering tradeoff:

In this figure, the vertical axis is AP and the horizontal axis is workflow time. The most attractive region is the upper-left: high AP with low runtime. A point far to the right needs a meaningful quality improvement to justify its extra cost.

The raw-feature models sit much closer to that attractive region. They are fast and have the best AP point estimates. The embedding workflows move far to the right because they add offline representation generation and wider downstream feature matrices. In this run, they do not move upward enough on AP to compensate.

The workflow times are:

• Raw XGBoost: 147.0 seconds.

• Raw all-history XGBoost: 173.3 seconds.

• Raw + TabPFN: 840.2 seconds, including 371.4 seconds of shared TabPFN embedding preparation.

• Raw + TabICL: 1147.8 seconds, including 120.5 seconds of shared TabICL embedding preparation.

One subtle point is that TabICL embedding preparation is faster than TabPFN embedding preparation in this run, but the full Raw + TabICL workflow is slower. The likely reason is the downstream XGBoost search over a wider feature matrix: TabICL contributes 512 embedding columns, while TabPFN contributes 192. This is an important workflow lesson. Embedding extraction time alone is not the whole cost. The downstream model also has to tune and fit on the expanded feature set.

The embedding matrix sizes also matter:

• TabPFN final training embeddings: 170,884 rows, 192 columns, about 125.2 MB.

• TabPFN full-holdout embeddings: 56,962 rows, 192 columns, about 41.7 MB.

• TabICL final training embeddings: 170,884 rows, 512 columns, about 333.8 MB.

• TabICL full-holdout embeddings: 56,962 rows, 512 columns, about 111.3 MB.

Both embedding paths fit on the Kaggle two-T4 GPU runtime used for the notebook. TabPFN used both CUDA devices and reached about 936.8 MB maximum allocated memory per device during extraction. TabICL used device 0 more heavily, reaching about 3657.3 MB maximum allocated memory and 4548.0 MB reserved memory after extraction.

For practitioners, the lesson is that representation quality should be judged together with representation cost. If the embedding point had moved clearly upward in the runtime plot, the extra cost might be easy to defend. Here the AP view does not justify the added cost by itself, so the main reason to investigate TabICL further is the high-recall operating-point behavior seen above.

Calibration diagnostics

Calibration remains diagnostic rather than a clear improvement. This section asks a different question from ranking: if the model gives a transaction a low or high fraud probability, should that score be interpreted as a calibrated probability?

The clean comparison is between each calibration-base model and its sigmoid-calibrated version. AP stays the same for those pairs because sigmoid calibration is monotonic: it changes the probability scale but does not change the ranking order.

The probability-quality picture is mixed:

• For raw XGBoost, sigmoid calibration keeps AP at 0.8047 but worsens Brier score, log loss, and ECE 10 relative to the raw calibration-base configuration.

• For raw all-history XGBoost, sigmoid calibration keeps AP at 0.7997, improves Brier score, worsens log loss, and worsens ECE 10 relative to the raw all-history calibration-base configuration.

• For Raw + TabPFN, sigmoid calibration keeps AP at 0.7918 but worsens Brier score, log loss, and ECE 10 relative to the TabPFN calibration-base configuration.

• For Raw + TabICL, sigmoid calibration keeps AP at 0.7978 and improves Brier score and ECE 10 relative to the TabICL calibration-base configuration, but worsens log loss.

• Among the uncalibrated configurations, Raw all-history XGBoost has the best ECE 10, while Raw + TabPFN has the best Brier score and log loss.

That is why I do not treat calibration as a clear improvement in this notebook. If calibration had consistently improved Brier score, log loss, and ECE without harming the workflow, I would read it as useful probability cleanup. Here it improves some diagnostics for TabICL but not enough to make a broad claim.

The calibration figure is also harder to read than the precision-recall figure. Most scores and observed fraud rates are very close to zero because the event is so rare. Visually, that compresses the reliability curves near the origin. The diagonal reference line shows ideal calibration: predicted probability and observed fraud rate would match along that line.

The curves do not provide a clean visual story where one calibrated model clearly tracks the diagonal and the others clearly do not. For this reason, I treat the calibration plot as a warning to inspect probability quality, not as decisive evidence. The reliability-bin CSVs saved by the notebook are more useful than the figure when reviewing calibration in detail.

Public-data constraints

The leakage checks are included to keep the result in perspective. The notebook verifies the checks that are possible in this public dataset: the target is excluded from features, Time is used for chronological splitting, and Time is not used as a model feature in the default run.

The dataset is anonymized, so important production checks remain unavailable. I cannot test customer-level, card-level, merchant-level, or account-level leakage. I also cannot verify raw feature lineage or label availability timing. That means the results should be read as a workflow demonstration, not as a production fraud benchmark.

For a real fraud dataset, I would repeat this same workflow with entity-aware splits, delayed-label handling, feature timestamp checks, and drift monitoring before trusting the result.

Known limitations

The public-data constraints above are not the only caveats. There are also limits in this particular experiment design.

This is one public dataset, so I would not generalize the result to all fraud datasets, transaction workflows, or tabular foundation models.

The full holdout has only 75 fraud cases. The bootstrap intervals help, but they also show why point estimates should be interpreted cautiously.

The TabPFN and TabICL representation context is sampled from the earliest 20% source window rather than using every row in that window. The sampling keeps the rare fraud rows but caps the number of normal rows, so the context seen by the embedding models is smaller and more fraud-enriched than the full chronological source window. That could affect the learned row representations and may be one reason the embedding configurations do not improve AP here. This notebook does not isolate that factor, so I treat it as a hypothesis for follow-up rather than as an explanation proven by the run.

The TabICL embedding path uses model internals rather than a stable public embedding method comparable to TabPFN’s get_embeddings. That does not make the experiment invalid, but I would version-pin and review that code path.

I have not yet added interpretability methods such as SHAP, missing-data stress tests, categorical stress tests, drift-by-period analysis, or group-aware splitting. Those are important next steps for a broader testbench.

Summary and conclusion

This notebook tests a practical integration pattern:

1. Keep XGBoost as the downstream fraud scorer.

2. Use TabPFN or TabICL as an offline row-embedding generator.

3. Append those embeddings to raw transaction features.

4. Evaluate the result with chronological splits, AP, alert counts, runtime, memory, calibration diagnostics, and leakage checks.

The result does not show tabular foundation model embeddings outperforming the raw baselines across the main summary view. Raw all-history XGBoost has the best full-holdout point AP, and raw XGBoost is close while being much faster. Single-source TabPFN and TabICL embeddings do not beat the raw baselines by AP in this run. The bootstrap intervals overlap heavily, so I read the AP ranking cautiously.

The useful nuance is the high-recall operating point. At 90% recall, Raw + TabICL needs 490 alerts to recover 68 of 75 fraud cases, compared with 1,064 alerts for raw all-history XGBoost and 1,177 alerts for raw XGBoost. That makes TabICL worth investigating for high-recall review-queue settings, even though it is not the best AP/runtime configuration overall.

My interpretation is that tabular foundation model embeddings are best evaluated as workflow components, not only as standalone model scores. The important question is not only whether an embedding configuration has the highest AP. It is whether the embedding improves a business-relevant operating point enough to justify the added representation path.

For practitioners, the reusable lesson is the evaluation design. A tabular foundation model embedding experiment needs comparison against a strong classical baseline, with time-aware splitting, alert-budget metrics, calibration diagnostics, runtime accounting, memory accounting, and leakage checks.

For labs and researchers, this kind of notebook can be useful as a field-facing test. It does not replace formal benchmarks, but it can show how model capabilities appear when inserted into workflows that data teams already understand.

Outlook

With this dataset, I am reaching the point where the next useful experiments may need richer transaction context than the public file provides. I plan to keep extending the workflow in directions that matter for real data science teams, including representation-context ablations, interpretability, missing-data behavior, categorical features, time-derived feature policy, drift by time period, and group-aware splitting when entity IDs are available. For the embedding workflow specifically, I would like to test larger context samples, different normal-row sampling policies, repeated context draws, and context choices that preserve the source-window class balance more closely when model limits allow it.

However, these experiments take time, and I may change direction if I find a better experiment or a more useful problem to work on. If this line of testing is useful to you, comments are a good place to tell me whether you want to see these experiments carried through and what parts of the workflow you think would be most worth testing next.

My current goal is not to prove that one model family is always better. It is to build reusable examples that make benefits, costs, and caveats visible enough for both model builders and practitioners to reason about them.

You can find the notebook in my GitHub repository, and you can also clone it directly on Kaggle. The notebook is meant to be run on Kaggle with GPU enabled. It installs cuDF for pandas acceleration, uses CUDA for TabPFN and TabICL, and uses a CuPy-backed XGBoost path so that the downstream scorer also runs on GPU.

Conceptual background

The practical question

In many tabular ML systems, especially in fraud, credit risk, churn, pricing, and transaction monitoring, the production model is often a classical supervised model. XGBoost, LightGBM, CatBoost, Random Forest, and Logistic Regression remain common because they are fast, stable, easy to monitor, and familiar to data teams.

Tabular foundation models are interesting because they can learn useful representations from tabular data. But for an industry team, replacing an existing production scorer is a high bar. A replacement model has to be better, fast enough, stable under drift, explainable enough for the decision process, compatible with existing feature stores and monitoring, and acceptable under licensing and infrastructure constraints.

The notebook uses a lower-friction integration pattern:

1. Keep the production-style downstream model as XGBoost.

2. Use TabPFN and TabICL offline to generate row embeddings.

3. Append those embeddings to the raw transaction features.

4. Compare XGBoost trained on raw features against XGBoost trained on raw + embedding features.

This is closer to how a team might pilot tabular foundation models without rebuilding its whole fraud system.

Before looking at the dataset and results, it is useful to make the integration point precise. The downstream model is still ordinary supervised ML. The tabular foundation model contribution is an extra representation step before XGBoost.

Where tabular foundation model embeddings enter the supervised workflow

For a supervised model such as XGBoost, the workflow is familiar. We start with raw features \(x_i\) for transaction \(i\), a binary label \(y_i \in \{0,1\}\), and train a task-specific model

where \(\hat{p}_i\) is the predicted fraud score or fraud probability, and \(\theta\) represents the parameters learned from this dataset.

That part is still present in this notebook. XGBoost remains the downstream supervised model. The new part is the feature-generation step before XGBoost.

A tabular foundation model (TFM) is a pretrained model intended to work across many tabular tasks. In this notebook, TabPFN and TabICL are given an earlier labelled context

where \(j\) indexes the context rows and \(m\) is the number of rows in the context set. An embedding is a dense numerical vector that represents how a model internally encodes a row. The TFM maps a later row \(x_i\) into an embedding vector

where \(f_{\text{TFM}}\) is the embedding function after conditioning on \(C\), and \(z_i\) is the resulting representation produced for row \(i\). The downstream XGBoost model is then trained on an augmented feature vector:

where the brackets mean feature concatenation. So the similarity to ordinary supervised ML is that we still train and evaluate a supervised fraud model with labels, validation data, calibration data, and a future holdout. The difference is that the model receives additional representation features from a pretrained context-conditioned tabular model. That embedding-generation capability is the part that comes from TabPFN and TabICL.

Dataset

The notebook uses a public copy of the credit-card fraud dataset loaded from TensorFlow’s storage bucket. It contains:

• 284,807 transactions;

• 492 fraud transactions;

• 284,315 non-fraud transactions;

• a fraud rate of about 0.1727%;

• anonymized PCA-style features V1 to V28;

• Amount;

• Time;

• the binary target Class.

This is not a perfect production dataset. The features are anonymized, and there are no customer, card, merchant, device, or account identifiers. That limits what we can test. For example, a production fraud model should usually check entity-level leakage, delayed labels, future aggregate features, and drift by customer or merchant segment. This public dataset does not expose enough raw business context for that.

Even with those limitations, it is useful for this notebook because it has the basic shape of a fraud problem: severe class imbalance, a time column, and a binary rare-event target.

Fraud detection is a ranking problem before it is a classification problem

Because the target is so rare, metric choice matters before model choice. A model can look good under a broad metric and still be useless for an alert queue.

Accuracy is not a useful headline metric here. A model that predicts “not fraud” for every transaction would be more than 99% accurate, but it would catch no fraud.

The more useful deployment question is whether the model ranks fraud cases near the top of the risk queue. This is why the notebook focuses on precision, recall, Average Precision, and alert counts.

For binary labels:

Here, \(TP\) means true positives, \(FP\) means false positives, and \(FN\) means false negatives.

Precision asks: among the transactions we flag, what fraction are truly fraud?

Recall asks: among all fraud transactions, what fraction did we catch?

Average Precision summarizes the precision-recall curve. One common way to write it is:

where the steps are ordered by increasing recall, \(K\) is the number of evaluated threshold steps, and \(P_k\) and \(R_k\) are precision and recall at step \(k\). In rare-event problems, Average Precision is usually more informative than ROC AUC because it focuses directly on the positive class and the quality of the alert queue.

Still, even Average Precision is not the end of the story. Fraud teams work with review capacity. A team may ask:

• How many alerts do we need to review to catch 80% of fraud cases?

• How many alerts do we need to review to catch 90%?

• At a fixed alert budget, how many fraud cases are found?

• Does the model score behave like a calibrated probability, or only like a ranking score?

The notebook therefore includes both model-quality metrics and operating-point tables.

Chronological evaluation

Good metrics are still not enough if the split does not match deployment. For fraud, the model is trained on past transactions and used on future transactions.

Random train-test splits can be misleading because the real use case is future prediction. So the notebook sorts the data by Time and uses chronological windows:

• Earliest 20%: TabPFN/TabICL representation context.

• Next 40%: downstream XGBoost training window.

• Next 10%: validation window for model selection.

• Next 10%: calibration window.

• Final 20%: full holdout.

The representation-context window is important. TabPFN and TabICL need labelled rows to condition their embeddings. To avoid giving a row access to its own label, the TFM context is earlier than the downstream training, validation, calibration, and holdout rows.

The downstream model sees these feature sets:

• Raw: 29 features.

• Raw + TabPFN embeddings: 221 features.

• Raw + TabICL embeddings: 541 features.

• Raw + TabPFN + TabICL embeddings: 733 features.

The raw features exclude the target. In this default run, Time is used for splitting but not used as a model feature. That is a conservative default because Time can encode period-specific artifacts in public fraud datasets. A production team may use timestamp-derived features, but those features should be reviewed carefully.

Why the training windows are sampled

The full dataset has a very low fraud rate. If we train or condition a model on a small chronological slice without sampling, the positive class may be too sparse for useful learning or tuning. The notebook therefore uses fraud-enriched sampled windows for two parts of the workflow:

• the TFM representation context;

• the fair raw-vs-embedding downstream training window.

The sampling keeps all fraud rows in those windows and caps the number of normal rows. This is a case-control style design. It is common in rare-event modeling, but it must be interpreted correctly: the sampled training prevalence is not the deployment prevalence.

That is why validation, calibration, and the full holdout are kept at full prevalence. The final comparison is made on the future holdout with the real final-window base rate. The sampled training window also should not be confused with the final XGBoost fit size: after tuning, the uncalibrated fair raw-vs-embedding models are refit on the full downstream pre-holdout history for which embeddings were generated, excluding the earlier TFM context rows.

Model choices

With the representation, split, and sampling policy defined, the remaining question is what should score the transactions.

The notebook uses XGBoost as the production-style scorer. That choice is deliberate. XGBoost is a strong and widely used supervised model for tabular data, and it is a credible incumbent for fraud workflows.

The raw all-history incumbent is a separate XGBoost baseline that uses raw features only and can use every pre-holdout row, including the earlier representation-context rows. This is a stronger classical baseline than the fair raw-vs-embedding row because it asks whether using more ordinary historical data is already competitive with adding TFM embeddings.

TabPFN and TabICL are used as embedding generators rather than as final scorers. In this setup:

• TabPFN exposes a public embedding API through get_embeddings.

• TabICL does not expose the same sklearn-level public embedding API, so the notebook extracts row representations through the fitted model’s internal representation-cache path and records the TabICL version.

That TabICL detail is important. It means the TabICL embedding path is useful for exploration, but if someone uses it as benchmark infrastructure, the version should be pinned and reviewed.

The notebook also keeps Logistic Regression as an optional CPU benchmark, but it is not enabled in the default run. The main workflow uses GPU XGBoost because the industrial question is about a strong deployable scorer, and because sklearn Logistic Regression would make the notebook slower without answering the main question.

In implementation terms, the notebook does the following:

1. Load the fraud CSV and validate that Time and Class are present.

2. Sort rows by Time.

3. Build the chronological context, training, validation, calibration, and holdout windows.

4. Extract TabPFN and TabICL embeddings from the earlier context into later windows.

5. Build raw and embedding-enhanced feature bundles.

6. Tune GPU XGBoost with chronological validation folds.

7. Refit the selected models, evaluate the full holdout, and inspect operating points, calibration, runtime, and leakage checks.

Hyperparameter tuning and calibration

The notebook uses randomized hyperparameter search with chronological folds. Randomized search is used because it can explore a wider parameter space than a small manual grid without testing every possible combination. The folds preserve time order: each candidate is trained on earlier rows and validated on a later contiguous slice.

Because fraud is rare, the code also checks that each tuning fold has enough fraud rows in both the training and validation sides. If a fold has too few fraud cases, it is not a useful fold for model selection. In this run, the fair raw-vs-embedding tuning had only one valid chronological fold after those checks. That is a limitation, and I discuss it again in the outlook section.

The notebook also evaluates sigmoid-calibrated variants of XGBoost. Calibration is different from ranking. A model can rank fraud cases well but produce scores that should not be interpreted as probabilities.

Two probability-quality metrics are reported:

and

Here, \(N\) is the number of evaluated rows, \(i\) indexes those rows, \(y_i\) is the true label, and \(\hat{p}_i\) is the model’s predicted fraud probability. Lower is better for both metrics. In a fraud workflow, calibration matters if the score is used as a probability for thresholds, policies, or downstream decisions. If the score is used only for ranking a queue, calibration is still useful to inspect but not necessarily the primary objective.

Hands-on demo

The rest of the post reports the completed notebook run and focuses on the full holdout because that is the closest view to deployment.

Run integrity

The completed notebook ran without execution errors. The environment used Python 3.12.12, pandas 2.3.3, cuDF 26.2.1, CuPy 14.0.1, scikit-learn 1.6.1, XGBoost 3.2.0, torch 2.10.0+cu128, TabPFN 7.1.1 (pip version), and TabICL 2.1.1. The run used two CUDA devices.

The notebook also checks basic data quality and leakage conditions. The target is excluded from the features, the chronological split is ordered by Time, and the default feature set does not include Time. The duplicate diagnostics are more nuanced: exact duplicate rows exist, but exact duplicate groups do not cross time windows. Model-feature duplicate groups do cross time windows, but the detected cross-window groups did not contain fraud rows. I treat this as a review item rather than a fatal leakage finding.

Splits

The actual row counts after chronological splitting and sampling were:

• TFM embedding context, sampled: 4,157 rows, 157 fraud rows, 3.7768% fraud rate.

• Classical training window, sampled: 4,203 rows, 203 fraud rows, 4.8299% fraud rate.

• Validation window, full prevalence: 28,480 rows, 24 fraud rows, 0.0843% fraud rate.

• Calibration window, full prevalence: 28,481 rows, 33 fraud rows, 0.1159% fraud rate.

• Test holdout, sampled: 20,075 rows, 75 fraud rows, 0.3736% fraud rate.

• Test holdout, full prevalence: 56,962 rows, 75 fraud rows, 0.1317% fraud rate.

The full holdout is the deployment-facing view because it keeps the natural final-window fraud rate. The sampled holdout is useful for quick inspection, but the main conclusions should come from the full holdout.

For scale, the fair raw-vs-embedding tuning matrix has 32,683 rows, and the fair uncalibrated final-training matrix has 170,884 rows. The raw all-history incumbent’s uncalibrated final-training matrix has 227,845 rows because it can use the representation-context window as ordinary raw-feature history.

Full-holdout model quality

On the full holdout, the main results were:

Here, Workflow seconds means the one-path time for the feature set: shared offline embedding preparation, when applicable, plus downstream XGBoost tuning, fitting, optional calibration, and prediction time.

• Raw + TabICL embeddings, no calibration: Average Precision 0.8128, workflow 236.6 seconds.

• Raw all-history incumbent, no calibration: Average Precision 0.8097, workflow 160.4 seconds.

• Raw + TabICL embeddings, sigmoid calibration: Average Precision 0.8046, workflow 249.0 seconds.

• Raw + TabPFN + TabICL embeddings, no calibration: Average Precision 0.8029, workflow 640.4 seconds.

• Raw, sigmoid calibration: Average Precision 0.7995, workflow 21.6 seconds.

• Raw, no calibration: Average Precision 0.7850, workflow 21.0 seconds.

• Raw + TabPFN embeddings, no calibration: Average Precision 0.7811, workflow 428.3 seconds.

The main read is not that every embedding helps. The main read is more specific:

• TabICL embeddings improved XGBoost’s full-holdout AP over the fair raw baseline: 0.8128 versus 0.7850.

• The raw all-history incumbent was also strong: 0.8097 AP.

• TabPFN embeddings alone did not help in this run.

• Combining TabPFN and TabICL was slower than using TabICL alone and did not improve the full-holdout AP.

That makes the conclusion more practical than dramatic. The useful pattern in this run is not “add all foundation-model embeddings.” It is “TabICL embeddings were useful as additional features for XGBoost, but the raw all-history XGBoost incumbent remained very competitive.”

Precision-recall curves

The precision-recall plot shows the top XGBoost variants on the full holdout. The dashed baseline is near zero because the fraud base rate is about 0.13% in the final window.

The curves are close, which matters. This is not a case where the embedding-enhanced model completely changes the problem. The best rows are all strong XGBoost variants. Still, the TabICL-enhanced curve is slightly better in AP than the raw all-history incumbent and the raw fair baseline.

The honest interpretation is this: TabICL embeddings improved the ranking signal in this run, but the improvement is incremental and should be judged against runtime, feature-generation complexity, and the strength of the existing raw-feature workflow.

Runtime versus AP

The runtime plot is one of the most useful figures in the notebook. It shows that the best AP is not the only thing that matters.

Raw XGBoost is fastest. Raw + TabICL takes longer because the TabICL embeddings have to be generated, but it improves AP. Raw + TabPFN + TabICL is much slower and does not improve AP over TabICL alone in this run. Raw + TabPFN alone is also not attractive here because it is slower than raw XGBoost and lower AP than the raw baseline in the full-holdout results.

This is the tradeoff the notebook is meant to surface. If embeddings are produced offline in a batch-scored workflow, several minutes of feature generation may be acceptable. If the use case needs low-latency online scoring, the engineering burden is different.

Alert-count view

The operating-point results make the comparison more concrete. On the full holdout:

At 80% recall:

• Raw + TabICL embeddings: 91 alerts, 60 frauds found, 65.93% precision.

• Raw all-history incumbent: 94 alerts, 60 frauds found, 63.83% precision.

• Raw: 109 alerts, 60 frauds found, 55.05% precision.

• Raw + TabPFN + TabICL embeddings: 116 alerts, 60 frauds found, 51.72% precision.

At 90% recall:

• Raw all-history incumbent: 1,064 alerts, 68 frauds found, 6.39% precision.

• Raw + TabICL embeddings: 1,087 alerts, 68 frauds found, 6.26% precision.

• Raw + TabPFN + TabICL embeddings: 2,147 alerts, 68 frauds found, 3.17% precision.

• Raw: 2,284 alerts, 68 frauds found, 2.98% precision.

At 80% recall, Raw + TabICL embeddings required the fewest alerts among these rows. At 90% recall, the raw all-history incumbent was slightly better than Raw + TabICL embeddings, with 1,064 alerts versus 1,087. Both were much better than the fair raw baseline.

This is a useful result because it prevents a simplistic conclusion. Depending on the operating point, the best practical choice may be Raw + TabICL or the raw all-history incumbent. The embedding path helps, but the strong incumbent baseline deserves respect.

Calibration

The calibration results are less decisive than the ranking results.

On the full holdout:

• Raw + TabICL embeddings: AP 0.8128, Brier score 0.0004, log loss 0.0023.

• Raw all-history incumbent: AP 0.8097, Brier score 0.0004, log loss 0.0028.

• Raw + TabICL embeddings, calibrated: AP 0.8046, Brier score 0.0004, log loss 0.0031.

• Raw: AP 0.7850, Brier score 0.0004, log loss 0.0030.

• Raw, calibrated: AP 0.7995, Brier score 0.0004, log loss 0.0032.

The calibration curves are compressed near the origin because the positive class is extremely rare. They do not, by themselves, prove that calibration improved probability quality. In this run, calibration often lowered AP, and log loss did not consistently improve.

This does not mean calibration is useless. It means this notebook should treat calibration as a diagnostic, not as a guaranteed improvement. In a production fraud workflow, I would want more reliability diagnostics: score-bin tables, expected calibration error, calibration by segment, and careful separation between the model used before calibration and the same model after calibration.

What I take from the demo

The notebook supports a practical hypothesis: tabular foundation model embeddings can be useful as additional features for an existing classical fraud workflow. It also shows why the comparison has to include strong classical incumbents, operating-point metrics, runtime, calibration diagnostics, and leakage checks. This is the kind of evidence I want to build: not a leaderboard claim, but a reusable workflow that shows what improves, what does not, and what still needs review.

Known shortcomings in this version

The most important shortcomings I see are:

1. The exact ranking should be treated cautiously. The current completed run makes Raw + TabICL embeddings the best full-holdout AP row, with the raw all-history incumbent close behind. A previous completed execution of the same workflow put Raw + TabPFN + TabICL embeddings ahead by about 0.001 AP. The stable lesson is that TabICL embeddings look useful and TabPFN adds non-trivial cost here; the unstable lesson would be claiming that one embedding combination is always best.

2. The fair raw-vs-embedding tuning is not yet as strong as I want. The notebook requests multiple chronological folds, but after enforcing minimum fraud counts, the fair raw-vs-embedding comparison has only one valid fold. The raw all-history incumbent has more valid chronological folds. This means the hyperparameter-selection protocol is not equally strong for all rows yet.

3. The all-history raw incumbent is a serious baseline. This is good, but it also raises the bar. A production team may prefer a strong raw-feature XGBoost model trained on all available pre-holdout history over an embedding-enhanced path if the embedding lift is small or unstable. Future versions should keep this incumbent and make the comparison even cleaner.

4. Calibration is not isolated cleanly enough yet. The calibrated rows use a calibration window, while the uncalibrated final models can train on more pre-holdout labels. That means the comparison mixes two effects: calibration and different training-window sizes. I want to add an uncalibrated calibration-base row trained on the same rows as the calibrated base model, so the calibration effect can be evaluated more directly.

5. The calibration plots are visually weak. Because fraud is extremely rare, the reliability curves are compressed near the origin. They are useful as a warning, but not strong enough as production-grade calibration evidence. Better diagnostics would include Brier score, log loss, expected calibration error, and reliability tables by score quantile.

6. The sampling design needs to remain explicit. The TFM context and downstream training windows are intentionally fraud-enriched, while validation, calibration, and the full holdout remain full-prevalence windows. That is a reasonable rare-event design, but it should always be stated clearly so the reader does not confuse training prevalence with deployment prevalence.

7. The public dataset limits the leakage analysis. The notebook checks chronological order, target exclusion, duplicate rows, and duplicate model-feature rows. But the dataset does not expose customer, account, merchant, device, or label-timing information. In production, those would be required for stronger entity-level leakage and delayed-label checks.

8. The TabICL embedding path needs version discipline. TabPFN exposes a public embedding path. TabICL row representations are extracted through model internals in this notebook. That is acceptable for exploration, but a benchmark or production testbench should pin the version and review that extraction path whenever TabICL changes.

I include this section because I do not want the post to read like a finished benchmark. It is a working notebook that already gives useful signal, but the output review also shows exactly where the next engineering iterations should go.

Summary and Conclusion

This post asks a different question from direct fraud scoring: can TabPFN or TabICL improve an existing XGBoost fraud workflow through embeddings?

The answer from this run is cautious but useful:

1. TabICL embeddings improved XGBoost ranking quality on the full holdout.

2. The raw all-history XGBoost incumbent remained very competitive.

3. TabPFN embeddings did not help in this particular run.

4. Combining TabPFN and TabICL embeddings was not worth the extra runtime here.

5. Operating-point metrics are essential because AP alone does not tell a fraud team how large the review queue will be.

6. Calibration should be evaluated separately from ranking and should not be assumed to improve the workflow automatically.

The most practical conclusion is the following. If a team already has a strong XGBoost fraud workflow, TabICL embeddings are worth testing as offline representation features. But they should be tested against a strong raw-feature incumbent, under chronological validation, with alert-count metrics and runtime included.

That is a more modest conclusion than saying tabular foundation models replace classical ML. It is also, in my view, more useful.

Outlook

This work is still in progress. The notebook already gives a more practical view of TFM adoption in a fraud workflow, but the known-shortcomings section above is the clearest map for the next iteration.

The next step is not simply to add more models. The next step is to make the evaluation protocol stronger: better chronological tuning, cleaner calibration comparisons, clearer figures, and more datasets. After that, it will be easier to say when TFM embeddings are genuinely useful for an existing tabular ML workflow and when a strong classical incumbent is still the better engineering choice.

I will continue improving this version in that direction. Direct prediction is one path for tabular foundation models. Offline embeddings are another. Feature auditing, data-quality checks, uncertainty estimation, and cold-start modelling may be others. I am learning by building these notebooks step by step, and the goal is to turn that learning into workflows that are useful to both model developers and data teams trying to understand where these models fit.

Yesterday, I created a notebook that brought TabPFN and TabICL into the same supervised-learning examples. That was a useful first comparison, but the examples were still close to toy examples. I later felt that I wanted to move on to examples that are closer to realistic data science workflow as soon as possible.

Today I moved one step in that direction. I built a fraud-detection workflow using TabICL, TabPFN, Logistic Regression, and XGBoost. I am still learning these models by building and running examples, so I treat this notebook as a practical exploration rather than a definitive evaluation. Fraud detection is a useful first workflow because it is widely used. In addition, it is an interesting example from a technical perspective because the positive class is rare, plain accuracy is misleading, and the practical question is not only whether a model has a good score, but whether it produces a usable review queue under runtime, calibration, and deployment constraints.

You can find the notebook here on my GitHub repository. As explained in the notebook, it could also be run using time as a model feature (USE_TIME_AS_FEATURE = True) as the only change, and that notebook is available here. The Time=True notebook is a sensitivity check because timestamp-derived behavior can be useful in fraud workflows, but it can also encode period-specific artifacts that need careful production review.

Usually, I write these posts with the conceptual background preceding the work of that day. Today, I am writing this post differently because I have included a lot of conceptual background and interpretation of the results in the notebook. Therefore, the conceptual background given in the following section is meant to supplement that given in the notebook, should the reader need it.

Today’s work is meant to be read directly from the notebook. In future posts, I plan to develop this notebook further to ensure a realistic data science workflow is used by testing it in various ways.

Conceptual Background

Tabular Foundation Models in This Notebook

A tabular foundation model, or TFM, is a pretrained model intended to work across many tabular prediction tasks. In this post, the two TFMs are TabICLv2 and TabPFN.

For a practitioner familiar with supervised ML, the main difference is this: when I call .fit() on Logistic Regression or XGBoost, I am training a task-specific model from the current dataset. When I call .fit() on TabICL or TabPFN in this notebook, I am not training their large pretrained model weights from scratch; I am giving the model labelled rows as task context, and then the model uses that context to predict new rows.

This is why the context set matters for TFMs. The labelled rows given during .fit() are part of the task description the model uses at prediction time. Later in the notebook discussion, I distinguish between the context rows given to TabICL/TabPFN and the holdout rows used for evaluation.

Fraud Detection Is a Rare-Event Ranking Problem

The public credit-card fraud dataset used in the notebook has 284,807 transactions and 492 fraud cases. That is a fraud rate of about 0.17%.

In a dataset like this, accuracy is not a good primary metric by itself, where accuracy is the fraction of all predictions that are correct. A model that predicts “not fraud” for every transaction would be more than 99% accurate, but it would not help a fraud team find fraud cases.

A more relevant question is whether the model can put fraud cases near the top of a ranked list of risky transactions. For that, the useful concepts are precision, recall, and Average Precision.

Precision answers: among the transactions the model flags, what fraction are actually fraud?

Recall answers: among all fraud transactions, what fraction did the model catch?

Average Precision, or AP, summarizes the precision-recall curve across many possible thresholds. Higher AP is better. AP is more informative than accuracy here because it focuses on the rare positive class.

I also report ROC AUC. ROC AUC measures how often the model ranks a random fraud transaction above a random non-fraud transaction. Higher ROC AUC is better. However, I do not use it as the main result because ROC AUC can look good even when the actual fraud review queue still needs separate inspection.

Fraud Teams Deploy Alert Queues, Not Average Precision

Average Precision is useful for comparing rankers, but fraud teams usually do not deploy “Average Precision.” They deploy review queues.

For example, a team might ask:

• If we review the top 100 transactions, how many frauds do we catch?

• If we can review the top 0.5% of transactions, what is precision and recall?

• If we want to catch 80% of fraud cases, how many alerts must we investigate?

• If we want to catch 90% of fraud cases, does the review queue become too large?

Here, the phrase “alerts needed to reach 80% recall” means this: sort all transactions by model score from highest risk to lowest risk, then count how many transactions must be reviewed before 80% of the known fraud cases have appeared in that review list. Fewer alerts are better at a fixed recall target. Higher precision is also better because it means fewer false alerts in the review queue.

This is why the notebook includes alert-budget outputs. The goal is not only to ask which model has the highest AP, but also to inspect whether similar AP values lead to different operational outcomes.

Time-Aware Validation Matters

In ordinary tabular ML examples, we often use random train-test splits or random cross-validation. That is often fine for introductory demos, but it is less realistic for fraud detection. Fraud systems care about future transactions. Therefore, model selection should be based on earlier data and evaluation should happen on later data.

The notebook sorts the data by Time and uses four windows:

• earliest 60% of transactions: training period;

• next 10% of transactions: validation period for classical model selection;

• next 10% of transactions: calibration period for probability calibration;

• final 20% of transactions: future holdout.

This is still a simplified public-data workflow, but it is closer to a real fraud setting than random cross-validation. This is why the notebook has train, validation, calibration, and final holdout windows rather than one random train-test split.

Calibration Is Different From Ranking

A model can rank fraud cases well while producing poorly calibrated probabilities. This distinction matters.

If a model score is used only to sort transactions into a review queue, ranking quality is the main issue. If a score is interpreted as “this transaction has a 12% fraud probability,” then probability calibration matters.

A calibrated model is a model whose predicted probabilities match observed frequencies reasonably well. For example, among transactions that receive a predicted fraud probability near 10%, roughly 10% should actually be fraud if the model is well calibrated.

In the notebook, the calibrated Logistic Regression and calibrated XGBoost rows are not entirely new model families. They are the selected classical models plus a post-hoc sigmoid calibration step. The sigmoid calibration learns a mapping from the model’s scores to probabilities using the separate calibration window.

The notebook uses two calibration-related metrics:

• Brier score: the mean squared error between predicted probabilities and the true 0/1 labels. Lower is better.

• Log loss: a probability-sensitive loss that strongly penalizes confident wrong probabilities. Lower is better.

Because the TFM context is sampled differently from the full holdout, I treat probability calibration as something to inspect rather than assume.

Why Compare Against Tuned Classical Baselines?

If I compare TabICL and TabPFN against untuned classical models, the comparison would be weak. In normal data science workflows, a team would tune Logistic Regression or XGBoost, select hyperparameters on validation data, refit the selected model, and check calibration if probabilities are used for decisions.

So the notebook uses:

• Logistic Regression as a simple classical baseline.

• XGBoost as a widely used boosted-tree baseline.

• Calibrated variants of both classical models using a separate calibration window.

• TabICLv2 and TabPFN as foundation-model comparators.

I intentionally leave out Random Forest and CatBoost today to avoid leaning toward a broad benchmark. For this workflow test, I use fewer models and spend more space on validation design, full-holdout evaluation, alert-budget analysis, calibration diagnostics, runtime notes, and leakage checks.

Closing thoughts

This is all for today. I will pick it up next week from here. In the meantime, give the notebook a spin in Kaggle and let me know what you think.

For a new reader, the minimum background is this: TabPFN is a pretrained tabular foundation model. Unlike XGBoost or Random Forest, its ordinary .fit() call does not update model weights to learn a fresh model from scratch. Instead, .fit() prepares the labelled rows as context for the current task, and TabPFN uses that context to predict new rows. That is why I have described TabPFN as a context-conditioned predictor throughout this series.

In previous posts, I completed the examples given in TabPFN’s hands-on demo notebook, except for unsupervised learning, which I left out intentionally. Today, I moved on to TabICLv2, which is a different tabular foundational model. I chose TabICLv2 because it’s almost as good as TabPFN and is free and open source, whereas TabPFN requires commercial license.

TabICLv2’s GitHub repository contains examples that demonstrate its usability, but there’s no comprehensive Jupyter notebook like that of TabPFN, and I felt that the examples are also basic. To test TabICLv2 and compare it with TabPFN, I felt that I first needed to develop a test bench. In this post, I am sharing with you the Jupyter notebook I created for this purpose. It contains the TabPFN-related code from my previous posts and the TabICL-related code from their GH repo. You can find the notebook here in my GitHub repo.

Using the Notebook

To use the notebook, you can download it from my GH repo and run it in Kaggle. Once you open Kaggle, - you can create a notebook as follows,

and import my notebook.

Make sure to use the GPU because both TabPFN and TabICL are extremely slow on the CPU.

Once you have the setup ready, you would need Prior Labs' API token to download the weights of TabPFN and Hugging Face access token (optional) to download TabICL. You can specify them in Kaggle secrets, and they will be imported automatically when the notebook runs.

The secrets are always available on Kaggle (privately, only to you). So once you set them, you can always import any notebook I share on GitHub and use it immediately with your secrets.

Contents

The first four sections (0-3) of the notebook contain the code from my previous posts. I felt that the examples given in the TabICL repository were not as good as the ones I looked at in previous posts. Since I had already worked on making the classification, regression, and model interpretability (SHAP and embeddings) examples work in previous posts, I thought I would use them here and include TabICL instead of reinventing new examples. So that’s what I did.

The examples in section 4 and 5 are taken from the TabICL repository. For now, I only added them to the notebook and made them work. I wi…ll look into the details in future posts.

Classification and Regression

The following figures show the comparison of all the models for classification and regression tasks. I verified that the performance of TabPFN and other classical ML models is consistent with previous blog posts. TabICL shows improvement compared to TabPFN, but the difference is statistically insignificant. However, please keep in mind that this (and other observations mentioned below) is a preliminary observation, and I could have a closer look in future posts.

Model Interpretability: SHAP and Embeddings

For now, I think there’s no need to comment on the SHAP plot other than the fact that this plot is in a different format than previous blog posts to enable the side-by-side comparison. I could look into it in future posts, but I don’t think SHAP is a priority, for now, because I am currently more interested in developing a holistic picture of tabular foundation models rather than digging deep into one topic.

The embeddings section highlighted an important point. While TabPFN exposes embeddings through TabPFNEmbedding, TabICLv2 does not currently expose the same public embedding API. However, it can cache row representations when kv_cache="repr" is used, and that was used to produce the comparison shown below.

Probability Surfaces

The following figure shows how each classifier distributes class probability across a noisy two-dimensional input space. The point is not to ask which model has the best score, but how each model behaves between observed training points. For now, I won’t go into the details and leave them for future posts.

Quantile Regression

The following figure shows the comparison of predictive intervals on a regression problem. Again, I will discuss the details in future post.

Closing Thoughts

With this post, this series has moved on from using only TabPFN to comparing TabPFN and TabICL. Today, I only shared the code and output. I will go deeper in comparing TabPFN and TabICL in future posts. In the meantime, I encourage you to play with the notebook. If you do, let me know in the comments what you think of these models and the modifications you made.

This post continues my series on tabular foundation models. So far, I have covered the basic vocabulary of tabular foundation models in P3, the posterior predictive distribution in P4, the architecture in P5, pre-training in P6, the TabPFN repository in P7, the hands-on demo’s classification and regression examples in P8, TabPFN Client in P9, TabPFN embeddings in P10, TabPFN’s predictive behavior in P11, and time series forecasting with TabPFN in P12.

For a new reader, the minimum background is this: TabPFN is a pretrained tabular foundation model. Unlike XGBoost or Random Forest, its ordinary .fit() call does not update model weights to learn a fresh model from scratch. Instead, .fit() prepares the labelled rows as context for the current task, and TabPFN uses that context to predict new rows. That is why I have described TabPFN as a context-conditioned predictor throughout this series.

Today I cover the causal inference section of the official TabPFN hands-on demo notebook. You can find my local version of the notebook here.

The interesting question is different from the previous posts. In classification and regression, the goal was to predict an observed target. In causal inference, the goal is to estimate what would change if we intervened. For example: if two customers have similar covariates, but one receives a treatment and the other does not, how much of the outcome difference should be attributed to the treatment rather than to pre-existing differences between the customers?

A practical caveat is important here. TabPFN does not automatically make a causal claim true. It is still a predictive model. In the notebook, TabPFN is used as a base learner inside causal estimators from econml. The causal identification comes from the assumptions and estimator design. TabPFN contributes flexible outcome and propensity modeling.

The notebook places this example in a broader research direction: choosing base models for CATE estimators with AutoML (Vandershueren et al., 2025), using TabPFN as an out-of-the-box base learner in CATE pipelines (Zhang et al., 2025), and pretraining PFN-style models directly for causal effect estimation (Robertson et al.). I will not review those papers here. I use them as signposts for why this notebook section belongs in the TabPFN ecosystem.

The post is organized as follows:

1. Introduction: why causal inference is different from ordinary prediction.

2. Conceptual background: potential outcomes, CATE, confounding, propensity models, outcome models, and PEHE.

3. Hands-on demo: generating a synthetic CATE dataset, fitting CATE estimators with TabPFN, and reading the evaluation.

4. Summary and conclusion: what this example shows, what it does not show, and what to keep in mind.

2. Conceptual Background

Before going to the hands-on demo, I want to set up the concepts that make the example meaningful. This section does five things:

1. It explains why causal inference is not the same as prediction.

2. It defines potential outcomes and the conditional average treatment effect.

3. It explains the observed-confounder setting used in the notebook.

4. It shows where outcome models and propensity models enter the workflow.

5. It defines the PEHE metric used to evaluate CATE estimates in the synthetic demo.

2.1 Working Vocabulary

The key terms for this post are:

• Treatment: the action or exposure whose effect we want to estimate. In the notebook, this is the binary variable \(T \in \{0,1\}\).

• Outcome: the response variable affected by treatment. In the notebook, this is \(Y\).

• Covariates: observed features measured before treatment. In the notebook, these are the columns in \(X\).

• Potential outcomes: \(Y(1)\) and \(Y(0)\), the outcomes that would be observed under treatment and no treatment.

• Individual treatment effect: \(\tau_i = Y_i(1) - Y_i(0)\), the effect for one individual unit.

• CATE: the conditional average treatment effect, \(\tau(x) = \mathbb{E}[Y(1) - Y(0) | X=x]\). Here, \(\mathbb{E}\) means expectation, or the average value under the relevant probability distribution.

• Propensity score: \(e(x) = \mathbb{P}(T=1 | X=x)\), the probability of receiving treatment given covariates. Here, \(\mathbb{P}\) means probability.

• Outcome model: a model for \(\mathbb{E}[Y | T=t, X=x]\), usually estimated separately or jointly for \(t=0\) and \(t=1\).

• Confounder: a variable that affects both treatment assignment and outcome.

• Unconfoundedness: the assumption that, after conditioning on observed covariates \(X\), treatment assignment is independent of the potential outcomes.

I use \(i\) to index one row or unit, \(x\) for a specific covariate value, \(X\) for the covariate random variable or matrix depending on context, \(T\) for treatment, and \(Y\) for the observed outcome. This is consistent with the earlier posts, where lowercase symbols denote concrete values and uppercase symbols denote random variables or matrices.

In ordinary supervised learning, we usually learn a prediction rule:

This is useful, but it is associational. It answers: given features \(x\), what outcome do we expect?

Causal inference asks a different question:

The notation \(do(T=1)\) means that treatment is set by intervention, not merely observed. This distinction matters because people who receive a treatment may differ systematically from people who do not receive it.

2.2 Potential Outcomes and CATE

For each unit \(i\), imagine two possible outcomes:

The individual treatment effect is:

The fundamental problem of causal inference is that, for a real unit, we usually observe only one of these two outcomes. If \(T_i=1\), we observe \(Y_i(1)\) and not \(Y_i(0)\). If \(T_i=0\), we observe \(Y_i(0)\) and not \(Y_i(1)\). The observed outcome is:

Since individual effects are not directly observed in ordinary data, many causal workflows estimate average effects. The object in the notebook is the conditional average treatment effect:

This tells us how the treatment effect changes across the covariate space. In other words, CATE estimation is about treatment-effect heterogeneity.

This is different from the average treatment effect, which averages over the population:

If the treatment effect is the same for everyone, CATE and ATE carry similar information. If the effect varies by \(X\), CATE is more informative because it tells us where the treatment works more or less strongly.

The same object can be written using the intervention notation:

This is the equation shown in the notebook’s causal inference section. The point is not only to predict \(Y\). The point is to compare two counterfactual worlds for the same covariate profile \(x\).

2.3 Confounding and Identification

The notebook uses a simple observed-confounder graph:

The graph has three arrows:

• \(X \rightarrow T\): covariates affect treatment assignment.

• \(X \rightarrow Y\): covariates affect the outcome.

• \(T \rightarrow Y\): treatment affects the outcome.

This is a confounded observational setting because \(X\) affects both \(T\) and \(Y\). If we ignore \(X\), the observed association between \(T\) and \(Y\) mixes at least two effects:

1. The causal effect of treatment on outcome.

2. The pre-existing difference in covariates between treated and untreated units.

The notebook is constructed so that all confounding is observed in \(X\). This is the setting where the unconfoundedness assumption can hold:

Here, \(\perp\!\!\!\perp\) means statistical independence, and the vertical bar \(|\) means “conditional on.” So this expression says that, after conditioning on \(X\), treatment assignment behaves as if it were independent of the two potential outcomes.

In practice, this is a strong assumption. It cannot be guaranteed by using TabPFN, XGBoost, Random Forest, or any other predictive model. It is a causal assumption about the data-generating process. In randomized experiments, randomization can help justify it. In observational data, it depends on whether the relevant confounders were measured and adjusted for.

The consistency assumption is also needed. It says that the observed outcome equals the potential outcome under the treatment actually received:

So, if \(T=1\), the observed outcome is \(Y(1)\), and if \(T=0\), the observed outcome is \(Y(0)\).

There is another important assumption called overlap or positivity:

This means that for the covariate profiles we care about, there must be some treated and some untreated examples. If everyone with a certain covariate profile always receives treatment, then we do not have empirical support for the untreated counterfactual in that region.

For a binary treatment, under assumptions such as consistency, unconfoundedness, and overlap, the CATE can be identified from observed data:

This equation is where predictive modeling enters the workflow. We need to estimate conditional outcome functions from data.

2.4 Outcome Models, Propensity Models, and TabPFN

Many CATE estimators use one or both of the following nuisance models:

Here, \(m_t(x)\) is an outcome model and \(e(x)\) is a propensity model. They are called nuisance models because they are not always the final object of interest, but the causal estimator uses them to estimate the treatment effect.

This is exactly where TabPFN can be useful. TabPFN is a pretrained tabular model that can serve as:

• a regression model for the outcome \(Y\);

• a classification model for the treatment \(T\);

• a flexible base learner inside a causal estimator.

The notebook uses TabPFN in two ways:

1. As the overall model in an S-learner.

2. As the outcome and treatment model inside CausalForestDML.

The S-learner fits one model for the outcome:

Then it estimates CATE by changing only the treatment input:

This is simple and intuitive. The same fitted outcome model is asked two counterfactual prediction questions.

The CausalForestDML approach uses a different strategy. At a high level, double machine learning first estimates nuisance components such as the baseline outcome model and treatment model, then uses residualized variation to estimate treatment effects. A simplified residual idea is:

Here, \(g(x) = \mathbb{E}[Y|X=x]\) is the baseline outcome function, \(\hat{g}(X)\) is its estimate, and \(\hat{e}(X)\) is the estimated treatment propensity. The final stage then uses the part of \(Y\) and \(T\) not already explained by \(X\) to estimate heterogeneous treatment effects. The causal forest part allows the treatment effect to vary over the feature space.

The important point for this post is: TabPFN is not replacing causal identification. It is replacing or supporting the predictive sub-models inside the causal pipeline.

This split is useful for practitioners who already know supervised ML. The causal workflow is mostly familiar model-fitting machinery, but used for a different target:

• Standard supervised ML part: fit flexible models for outcomes or treatment assignment from tabular data.

• Causal inference part: decide whether the graph and assumptions justify interpreting adjusted differences as causal effects.

• TabPFN-specific part: use a pretrained, context-conditioned tabular model as a strong default base learner without training a fresh task-specific model from scratch.

So the new capability is not “causality from TabPFN.” The practical capability is that a tabular foundation model can be plugged into existing causal estimators as a low-tuning base learner for the predictive components.

2.5 PEHE

Since the notebook uses synthetic data, the true CATE \(\tau(x)\) is known. That means we can evaluate the estimated CATE directly. The metric used is PEHE, which stands for precision in estimation of heterogeneous effects.

For test points \(x_1,\ldots,x_n\), PEHE is:

This is simply the root mean squared error between the estimated treatment effect and the true treatment effect. Lower PEHE is better.

In real observational data, we usually do not know \(\tau(x)\), so PEHE cannot be computed directly. That is why synthetic examples are useful for learning: they let us test whether the estimator can recover a known causal effect under a controlled data-generating process.

3. Hands-on Demo

The conceptual background prepared us for the notebook example. The causal section of the notebook does four things:

1. It creates a synthetic observed-confounder setting.

2. It generates potential outcomes and a true CATE value for each row.

3. It fits three CATE estimators.

4. It compares the estimated CATE values with the known true values using PEHE.

The full notebook contains the install commands and imports. Below, I show the parts that matter for understanding the workflow.

3.1 Generating the Causal Graph

The notebook first draws the graph:

import networkx as nx

edges = [["X", "Y"], ["T", "Y"], ["X", "T"]]
graph = nx.DiGraph(edges)
nx.draw(graph, with_labels=True, node_size=2000)

This graph makes the identification setup explicit. \(X\) affects treatment assignment, \(X\) also affects the outcome, and treatment affects the outcome. So a naive comparison of treated and untreated outcomes would be confounded.

The reason the example is still manageable is that \(X\) is observed. The causal estimator can adjust for \(X\).

3.2 Creating the Synthetic CATE Dataset

The notebook creates a dataset with 500 rows and 5 covariates:

import numpy as np

num_samples, num_features, train_test_split = 500, 5, 75

noise_scale = 0.01
exogenous_scale = 0.1
heterogeneity_scale = 2

X = np.random.normal(loc=0, scale=exogenous_scale, size=(num_samples, num_features)).astype(np.float32)
prop_eps = np.random.normal(loc=0, scale=noise_scale, size=(num_samples)).astype(np.float32)
Y_eps = np.random.normal(loc=0, scale=noise_scale, size=(num_samples)).astype(np.float32)

w_X_Y = np.random.uniform(size=num_features)
w_X_T = np.random.uniform(size=num_features)
w_X_T_effect = np.random.uniform(size=num_features)

T_0 = np.zeros(shape=num_samples).astype(np.float32)
T_1 = np.ones(shape=num_samples).astype(np.float32)

base_X_Y = np.sum(w_X_Y * X, axis=1)
base_X_T = np.sum(w_X_T * X, axis=1)
heterogeneity_term = np.sum(w_X_T_effect * X, axis=1)

propensity = np.sin(base_X_T) + prop_eps
T = (propensity > propensity.mean()).astype(np.float32)

Y_0 = np.cos(base_X_Y) + np.sin(0.3 * T_0) + Y_eps
Y_1 = np.cos(base_X_Y) + np.sin(0.3 * T_1 + heterogeneity_scale * heterogeneity_term) + Y_eps

Y = Y_0 * (1 - T) + Y_1 * T
tau = Y_1 - Y_0

Here, train_test_split is the notebook's variable name for the number of training rows. It is not the train_test_split function from sklearn.

The treatment assignment depends on \(X\) through base_X_T, which is why \(X \rightarrow T\) appears in the graph. The outcome depends on \(X\) through base_X_Y, which is why \(X \rightarrow Y\) appears. The treated and untreated potential outcomes differ through the treatment term, which is why \(T \rightarrow Y\) appears.

This code is useful because it makes the hidden causal quantities visible. In real data, we would observe \(X\), \(T\), and \(Y\), but not both \(Y_0\) and \(Y_1\). In this synthetic dataset, we know both potential outcomes, so we can compute:

The notebook stores this row-level ground truth treatment effect as tau. In this synthetic construction, tau is the target used to evaluate how well the estimators recover the heterogeneous treatment effect.

The following plot shows the true treatment-effect heterogeneity against two features, with points colored by observed treatment assignment:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = pd.DataFrame({**{f"X{i}": X[:, i] for i in range(num_features)}, "T": T, "Y": Y, "Y_0": Y_0, "Y_1": Y_1, "tau": tau})
fig, axes = plt.subplots(ncols=2, figsize=(6, 3))
sns.scatterplot(data=df, x="X0", y="tau", hue="T", ax=axes[0], edgecolor="black", palette="Set1")
sns.scatterplot(data=df, x="X1", y="tau", hue="T", ax=axes[1], edgecolor="black", palette="Set1")
fig.tight_layout()

The treatment effect is not constant. It changes with the covariates. This is what makes the task a CATE estimation task rather than only an average treatment effect estimation task.

The notebook then creates a train/test split:

perm = np.random.permutation(num_samples)
train_idx, test_idx = perm[:train_test_split], perm[train_test_split:]
X_train, X_test = X[train_idx], X[test_idx]
Y_train, Y_test = Y[train_idx], Y[test_idx]
T_train, T_test = T[train_idx], T[test_idx]
tau_test = tau[test_idx]

In this run, the training set has only 75 rows and the test set has 425 rows. That makes the example a useful small-data setting for TabPFN.

3.3 Fitting CATE Estimators with TabPFN

The notebook compares three estimators:

• s_tabpfn: an S-learner using TabPFNRegressor as the overall outcome model.

• cfdml_tabpfn: CausalForestDML using TabPFNRegressor for the outcome model and TabPFNClassifier for the treatment model.

• cfdml_default: CausalForestDML with its default nuisance models.

The imports are:

from econml.metalearners import SLearner
from econml.dml import CausalForestDML
from tabpfn import TabPFNClassifier, TabPFNRegressor

The S-learner is fitted as:

s_tabpfn = SLearner(overall_model=TabPFNRegressor())

s_tabpfn.fit(Y=Y_train, X=X_train, T=T_train)
s_tabpfn_cate = s_tabpfn.effect(X=X_test)

Conceptually, this estimates a single outcome function \(\hat{m}(x,t)\), then evaluates the difference:

The TabPFN-backed causal forest is fitted as:

cfdml_tabpfn = CausalForestDML(model_y=TabPFNRegressor(), model_t=TabPFNClassifier(), discrete_treatment=True)

cfdml_tabpfn.fit(Y=Y_train, X=X_train, T=T_train)
cfdml_tabpfn_cate = cfdml_tabpfn.effect(X=X_test)

Here, TabPFN is used for the nuisance models. The final heterogeneous treatment-effect estimation is handled by CausalForestDML.

The default comparison is:

cfdml_default = CausalForestDML(discrete_treatment=True)

cfdml_default.fit(Y=Y_train, X=X_train, T=T_train)
cfdml_default_cate = cfdml_default.effect(X=X_test)

This comparison is useful because it separates two questions:

1. Does TabPFN work well as a direct base learner in a simple meta-learner?

2. Does TabPFN improve the nuisance-modeling stage inside a more structured causal estimator?

3.4 Evaluating CATE Recovery

Since the synthetic data contains the true test-set treatment effect tau_test, the notebook computes PEHE:

from sklearn.metrics import root_mean_squared_error

s_tabpfn_pehe = root_mean_squared_error(tau_test, s_tabpfn_cate)
cfdml_tabpfn_pehe = root_mean_squared_error(tau_test, cfdml_tabpfn_cate)
cfdml_default_pehe = root_mean_squared_error(tau_test, cfdml_default_cate)

print(f"s_tabpfn PEHE: {s_tabpfn_pehe}")
print(f"cfdml_tabpfn PEHE: {cfdml_tabpfn_pehe}")
print(f"cfdml_default PEHE: {cfdml_default_pehe}")

In my notebook run, the output was:

• s_tabpfn PEHE: 0.0715.

• cfdml_tabpfn PEHE: 0.1956.

• cfdml_default PEHE: 0.2803.

Lower PEHE means better recovery of the true heterogeneous treatment effect. In this run, the S-learner with TabPFN performs best.

The final plot compares the true CATE on the x-axis with each model’s predicted CATE on the y-axis:

df_test = pd.DataFrame({"T": T_test, "Y": Y_test, "tau": tau_test, **{f"X{i}": X_test[:, i] for i in range(num_features)}})
df_test["s_tabpfn_pred"], df_test["cfdml_tabpfn_pred"], df_test["cfdml_default_pred"] = s_tabpfn_cate, cfdml_tabpfn_cate, cfdml_default_cate

fig, axes = plt.subplots(ncols=3, figsize=(9, 3))
sns.scatterplot(data=df_test, x="tau", y="s_tabpfn_pred", hue="T", ax=axes[0], edgecolor="black", palette="Set1")
sns.scatterplot(data=df_test, x="tau", y="cfdml_tabpfn_pred", hue="T", ax=axes[1], edgecolor="black", palette="Set1")
sns.scatterplot(data=df_test, x="tau", y="cfdml_default_pred", hue="T", ax=axes[2], edgecolor="black", palette="Set1")
fig.tight_layout()

The left panel shows that s_tabpfn_pred closely tracks the true \tau. The middle and right panels show more compressed and noisier estimates for the two causal forest variants in this particular run.

This result should be read as a controlled notebook example, not as a general benchmark. The notebook uses a small synthetic dataset, no fixed random seed in the shown code, and one data-generating process. What it shows is narrower but still useful: in this setting, TabPFN can be a strong base learner for recovering a smooth heterogeneous treatment effect from a small context dataset.

For a stronger evaluation template, I would add two diagnostics:

1. Add a diagonal y=x reference line to the predicted-vs-true CATE plots.

2. Repeat the experiment over many random seeds and report mean PEHE with uncertainty intervals.

That would make the comparison more stable and easier to read.

4. Summary and Conclusion

In this post, I used the causal inference section of the official TabPFN hands-on demo to understand how TabPFN can be used inside CATE estimation workflows.

The conceptual section separated prediction from causal inference. Ordinary supervised learning estimates associations such as \(\mathbb{E}[Y|X=x]\). Causal inference asks intervention questions such as \(\mathbb{E}[Y|do(T=1),X=x] - \mathbb{E}[Y|do(T=0),X=x]\). This distinction is the main reason we needed potential outcomes, confounding, unconfoundedness, overlap, outcome models, and propensity models before reading the code.

The hands-on demo then created a synthetic observed-confounder setting with \(X \rightarrow T\), \(X \rightarrow Y\), and \(T \rightarrow Y\). Because the data were synthetic, the notebook had access to both potential outcomes \(Y_0\) and \(Y_1\), so it could compute the true CATE \(\tau = Y_1 - Y_0\). That made PEHE available as a direct evaluation metric.

The notebook compared three estimators: an S-learner using TabPFN, a CausalForestDML estimator using TabPFN for nuisance models, and a default CausalForestDML estimator. In my run, the S-learner with TabPFN had the lowest PEHE and the clearest predicted-vs-true CATE pattern.

The main takeaway is not that TabPFN solves causal inference by itself. It does not. The causal assumptions still matter. The estimator still matters. The graph still matters. What TabPFN may offer is a strong tabular base learner for the predictive subproblems inside causal estimators, especially in small-to-medium tabular settings where ordinary model selection and tuning can become tedious.

This also connects back to the pre-training discussion from P6. TabPFN’s synthetic pretraining already uses structural causal models as part of the prior over tabular tasks. The current demo uses TabPFN as a base model inside standard CATE estimators. The next step in the field is even more direct: pretraining PFN-style models specifically for causal effect estimation, such as the Do-PFN direction mentioned in the notebook.

With this post, I have covered another remaining section of the TabPFN hands-on demo. In the upcoming posts, I will continue exploring the parts of the TabPFN ecosystem that are most relevant for practical tabular data workflows: when TabPFN should be used directly, when it should be used as a component inside a larger method, and when classical tabular ML remains the simpler choice.

This post continues my series on tabular foundation models. So far, I have covered the basic vocabulary of tabular foundation models in P3, the posterior predictive distribution in P4, the architecture in P5, pre-training in P6, the TabPFN repository in P7, the hands-on demo’s classification and regression examples in P8, TabPFN Client in P9, TabPFN embeddings in P10, and TabPFN’s predictive behavior in P11.

For a new reader, the minimum background is this: TabPFN is a pretrained tabular foundation model. Unlike XGBoost or Random Forest, its ordinary .fit() call does not update model weights to learn a new model from scratch for the current dataset. Instead, .fit() prepares the labelled rows as context, and TabPFN uses that context to predict new rows. This is why I have repeatedly described TabPFN as a context-conditioned predictor rather than just another sklearn-like estimator.

Today I cover the time series forecasting section of the official TabPFN hands-on demo notebook. You can find my local version of the notebook here.

In P3, I left time series forecasting as “Later.” This post fills that gap. The interesting question is: if TabPFN is a tabular foundation model, how can it forecast a sequence? At first, this is not obvious because ordinary tabular models usually treat rows as examples in a table, while time series forecasting depends on the order of observations, seasonality, and future horizons. The answer is not that TabPFN suddenly becomes an ARIMA model, a recurrent neural network, or a native time-series transformer. The main idea in TabPFN-TS is to convert forecasting into a tabular regression problem.

This is where TabPFN-TS comes in. The notebook cites the work of Hoo et al., whose current arXiv paper is listed as From Tables to Time: Extending TabPFN-v2 to Time Series Forecasting. That work is the reference behind the TabPFN-TS workflow used in the notebook. The TabPFN-TS repository summarizes the workflow as:

1. Transform a time series into a table.

2. Extract temporal features and add them to the table.

3. Perform regression on the table using TabPFNv2.

4. Use the regression output as the time series forecast.

The post is organized as follows:

1. Introduction: why time series forecasting belongs in this TabPFN series.

2. Conceptual background: the forecasting vocabulary and the tabular-regression formulation.

3. Hands-on demo: loading the Chronos data, adding features, predicting, and reading the forecast plot.

4. Summary and conclusion: what this example shows and how to evaluate forecasts in practice.

   Thanks for reading DSAIEngineering! Subscribe for free to receive new posts and support my work.

2. Conceptual Background

Before going to the hands-on demo, I want to set up the concepts that make the example meaningful. This section does five things:

1. It defines the basic vocabulary of time series forecasting.

2. It explains how a sequence can be represented as a supervised tabular regression problem.

3. It separates what is standard supervised ML from what is specific to TabPFN-TS.

4. It explains why temporal features matter.

5. It connects point forecasts and quantile forecasts back to the predictive-distribution language from earlier posts.

2.1 Working Vocabulary

The key terms for this post are:

• Time series: observations indexed by time, such as monthly tourism demand, hourly electricity load, daily sales, or sensor readings.

• Forecast horizon: the future window we want to predict. In the notebook, prediction_length = 24, so the model predicts 24 future monthly values.

• History/context window: the observed part of the time series that is available before the forecast starts.

• Point forecast: a single predicted value for each future timestamp.

• Probabilistic forecast: a forecast that describes uncertainty, often through quantiles.

• Quantile forecast: a prediction for a chosen quantile level, such as the 0.1 or 0.9 quantile.

• Covariates/features: extra columns known at prediction time, such as calendar features, holidays, weather, promotions, or a running time index.

• Zero-shot forecasting: applying a pretrained model to a new forecasting problem without training a task-specific forecasting model from scratch.

In ordinary supervised regression, we usually have a table:

Each row \(x_i\) contains features, and \(y_i\) is the target value. A regressor learns or uses a mapping from rows to targets.

Here, \(X\) is the feature matrix, \(y\) is the target vector, \(n\) is the number of rows, and \(x_i^\top\) means that the feature vector for row \(i\) is written as a row vector. The superscript \(\top\) denotes transpose.

In time series forecasting, the data initially looks different. For one item, we observe:

and want to predict:

Here, \(T\) is the last observed time index, and \(H\) is the forecast horizon.

The key move in TabPFN-TS is to make the second problem look like the first problem.

2.2 Forecasting as Tabular Regression

Suppose we have multiple time series indexed by item \(i\). For item \(i\), let \(y_{i,t}\) be the observed value at time \(t\), and let \(T_i\) be the last observed time index available before forecasting starts. The forecasting task is to estimate future values:

In the single-series notation above, the last observed index was \(T\). With multiple series, I write this as \(T_i\) because each item \(i\) may have its own last observed timestamp. Here, \(H\) is the forecast horizon, and \(h\) is the number of steps ahead from the end of the observed history for item \(i\). To use a tabular model, we build a feature vector for each item-time pair:

Here, \(g(\cdot)\) is the feature-construction function. It turns time information and any known covariates into ordinary tabular columns. The training table contains rows where the target is known:

Here, \(\mathcal{I}\) is the set of item IDs included in the forecasting task. The future table contains rows where the target is unknown:

Now the forecasting problem has become a tabular regression problem:

Here, \(f\) is a prediction function, and \(\hat{y}_{i,T_i+h}\) is the predicted value for item \(i\) at forecast step \(h\).

For a classical supervised model, \(f\) would usually be a model fitted specifically to the current training table. For example, a supervised regressor might choose:

Here, \(\mathcal{F}\) is the model class, \(\ell\) is the regression loss, and \(\hat{f}\) is the fitted task-specific model. Random Forest, XGBoost, LightGBM, and CatBoost all differ in how they define and optimize \(\mathcal{F}\), but in this workflow they are still learning a fresh model from the current transformed table.

For TabPFN, the meaning is different. TabPFN is already pretrained, and the current training rows become context. Conceptually, the prediction is closer to:

Here, \(p(\cdot)\) denotes a predictive distribution over the future target value.

This is the same posterior-predictive-distribution viewpoint I discussed in P4 and reused in P11. The difference is that the row \(x_{i,T_i+h}\) now represents a future timestamp, not a generic tabular row.

This framing also explains why the time-series package can support point forecasts and probabilistic forecasts. If TabPFN produces a predictive distribution for the target at a future row, then the output can be summarized as a mean, median, or quantiles. This is a useful conceptual lens, not a guarantee that the output is perfectly calibrated for every dataset.

2.3 Standard Supervised ML vs What Is New Here

The conversion from a time series to a tabular regression problem is not unique to TabPFN. A practitioner could build the same kind of table and fit XGBoost, LightGBM, Random Forest, CatBoost, or a linear model on the generated rows. In that sense, the feature-engineering idea is a standard supervised-ML move.

What is new in the TabPFN-TS workflow is the model used after the transformation. Instead of training and tuning a new forecasting model from scratch, TabPFN-TS uses a pretrained tabular foundation model as the regression engine. The training rows act as context, the future rows act as queries, and the model returns point and quantile predictions through the time-series wrapper.

So the split is:

• Standard supervised ML part: turn timestamps into rows, create temporal features, define known-target training rows and unknown-target future rows.

• TabPFN-specific part: use a pretrained, context-conditioned tabular model instead of fitting a task-specific model from scratch.

• TabPFN-TS convenience: return both point forecasts and quantile forecasts through one forecasting interface.

There is an important caveat. TabPFN-TS relies on temporal featurization; TabPFN is not modeling sequence order natively in the same way as a dedicated sequence model. The sequence structure becomes available to the model through columns such as running index, calendar features, seasonal features, and known covariates.

2.4 Why Temporal Features Matter

If we only create rows without useful time-derived features, a tabular model has no direct way to know that January 1980 and January 1981 are related, or that December and January are adjacent months. This is why temporal feature engineering is central to the TabPFN-TS workflow.

The notebook uses three feature groups:

selected_features = [
    RunningIndexFeature(),
    CalendarFeature(),
    AutoSeasonalFeature(),
]

The running index gives each timestamp an ordered numeric position within each item. If item \(i\) has \(n_i\) observed rows, the running index over the observed history is:

This \(n_i\) counts rows in the observed history; it is separate from \(T_i\), which denotes the last observed time index used in the forecasting equations. The running index helps the model see trend-like behavior. Calendar features encode timestamp information such as year, month, day of week, and similar components. Seasonal features encode repeated patterns. A standard way to encode cyclic seasonality is:

where \(P\) is the period. For monthly data with annual seasonality, \(P=12\). Using both sine and cosine is useful because it represents the cycle on a circle. This avoids treating the end of a period and the beginning of the next period as far apart.

In the notebook output, the transformed table contains columns such as:

target, running_index, year, second_of_minute_sin, second_of_minute_cos, ...,
sin_#0, cos_#0, sin_#1, cos_#1, sin_#2, cos_#2

The target column is known in the training rows and missing in the future rows. The time-derived features are known for both training and future rows. That is exactly what forecasting needs: at prediction time, we do not know the future target, but we do know the future timestamps.

2.5 Point Forecasts, Quantiles, and Coverage

For item \(i\) and forecast step \(h\), the future row is \(x_{i,T_i+h}\), and the random future target is \(Y_{i,T_i+h}\). A point forecast gives one value:

A probabilistic forecast gives more information. The conditional cumulative distribution function is:

Here, \(\mathbb{P}\) denotes probability, and \(F_{i,h}(y)\) is the probability that the future target is less than or equal to the candidate value \(y\), given the future row and the training context.

The \(\alpha\)-quantile is:

Here, \(\alpha\) is a quantile level between 0 and 1, and \(\inf\) means the infimum: the smallest value, or limiting lower bound, where the cumulative probability reaches at least \(\alpha\).

For example, the interval:

is an 80% central prediction interval. In the demo, TabPFN-TS returns the point forecast and quantile columns from 0.1 to 0.9.

As in yesterday’s post, quantile intervals should not be treated as automatic guarantees. They need to be checked on held-out data. Let the held-out future points be indexed by \((i_j,h_j)\) for \(j=1,\ldots,m\), where \(m\) is the number of held-out item-horizon pairs being evaluated. The empirical 80% coverage is:

Here, \(\mathbf{1}\{\cdot\}\) is the indicator function: it equals 1 when the condition is true and 0 otherwise.

For context, when quantile models are trained directly, a common loss is the pinball loss. For quantile level \(\alpha\), true value \(y\), and quantile prediction \(q\), it is:

This loss penalizes under-prediction and over-prediction asymmetrically, which is exactly what is needed for quantile estimation.

The coverage calculation answers a different question from the pinball loss. If the empirical coverage value is close to 0.8, the interval is roughly calibrated on that held-out sample. If it is much lower, the forecast intervals are overconfident. If it is much higher, the intervals may be too wide to be useful.

3. Hands-on Demo

The conceptual background gave us the main objects: a time-indexed sequence, a transformed tabular representation, a future table with unknown targets, and point/quantile forecasts. Now I use the notebook to walk through the time-series example.

The mental model for the demo is:

• Training rows: past timestamps with known target values.

• Future rows: future timestamps with target = NaN.

• Features: running index, calendar features, seasonal features, and any known covariate columns.

• Output: point forecast plus quantile columns for each future row.

The full notebook contains the setup code and imports. Below, I show the parts that matter for understanding the workflow.

3.1 Loading the Time Series Data

The demo uses a dataset from the Chronos datasets collection on Hugging Face. To keep the example small, it uses only two time series from monash_tourism_monthly.

dataset_metadata = {
    "monash_tourism_monthly": {"prediction_length": 24},
    "m4_hourly": {"prediction_length": 48},
}

dataset_choice = "monash_tourism_monthly"
num_time_series_subset = 2

The notebook then loads the dataset, converts it into a TimeSeriesDataFrame, keeps only two item IDs, and creates a train/test split. The last 24 months are held out as the future window.

from datasets import load_dataset
from tabpfn_time_series import TimeSeriesDataFrame
from tabpfn_time_series.data_preparation import generate_test_X, to_gluonts_univariate

prediction_length = dataset_metadata[dataset_choice]["prediction_length"]
dataset = load_dataset("autogluon/chronos_datasets", dataset_choice)

tsdf = TimeSeriesDataFrame(to_gluonts_univariate(dataset["train"]))
tsdf = tsdf[
    tsdf.index.get_level_values("item_id").isin(tsdf.item_ids[:num_time_series_subset])
]

train_tsdf, test_tsdf_ground_truth = tsdf.train_test_split(
    prediction_length=prediction_length
)
test_tsdf = generate_test_X(train_tsdf, prediction_length)

The first important object is train_tsdf: the observed history. The second is test_tsdf_ground_truth: the future values that we hide from the model but keep for evaluation. The third is test_tsdf: the future table that contains the timestamps where predictions are needed. The function generate_test_X creates those future timestamp rows for the forecast horizon, with unknown targets.

The following plot shows the two tourism series and the train/test split.

Both series show strong yearly seasonality. The vertical dashed red line marks the point where the training history ends and the held-out future window starts. Since the forecast horizon is 24 months, the model is asked to forecast two full seasonal cycles.

3.2 Adding Time Features

The next step is the most important conceptual step in the demo. The raw time series is transformed into a tabular regression problem by adding features.

from tabpfn_time_series import FeatureTransformer
from tabpfn_time_series.features import (
    AutoSeasonalFeature,
    CalendarFeature,
    RunningIndexFeature,
)

selected_features = [
    RunningIndexFeature(),
    CalendarFeature(),
    AutoSeasonalFeature(),
]

feature_transformer = FeatureTransformer(selected_features)

train_tsdf, test_tsdf = feature_transformer.transform(train_tsdf, test_tsdf)

After this transformation, the training table has a known target column and many feature columns. The future table has the same feature columns, but the target column is missing:

item_id  timestamp    target     running_index    year    ...    sin_#0    cos_#0
0        1979-01-31   1149.8700  0                1979    ...    0.0000    1.0000
0        1979-02-28   1053.8002  1                1979    ...    0.5000    0.8660
...
0        1992-08-31   NaN        163              1992    ...   -0.5000   -0.8660

This is the point where forecasting becomes tabular. The rows with known targets form the context. The rows with unknown targets form the query set.

3.3 Predicting with TabPFN-TS

In my run, I used local mode, which runs TabPFN on my local GPU, instead of client mode, which uses GPUs hosted in Prior Labs’ cloud:

from tabpfn_time_series import TabPFNMode, TabPFNTimeSeriesPredictor

predictor = TabPFNTimeSeriesPredictor(
    tabpfn_mode=TabPFNMode.LOCAL,
)

pred = predictor.predict(train_tsdf, test_tsdf)

The output pred is again indexed by item_id and timestamp. It contains a point forecast in the target column and quantile forecasts in columns such as 0.1, 0.2, ..., 0.9.

                         target          0.1          0.2  ...          0.8          0.9
item_id timestamp
0       1992-08-31  6632.519531  6147.241211  6321.268066  ...  6938.606445  7118.754395
        1992-09-30  4159.460938  3881.989502  3977.088379  ...  4355.097656  4479.076172
        1992-10-31  3012.987549  2780.682861  2859.992432  ...  3172.838623  3264.242920

This output format is useful because it gives both a central forecast and uncertainty bands without first setting up a separate conformal wrapper or separately trained quantile model.

3.4 Visualizing the Forecast

The notebook visualizes the history, the held-out future values, the TabPFN-TS point forecast, and the 0.1 to 0.9 quantile band.

from tabpfn_time_series.plot import plot_pred_and_actual_ts

plot_pred_and_actual_ts(
    train=train_tsdf,
    test=test_tsdf_ground_truth,
    pred=pred,
)

The blue curve is the observed history. The purple curve is the held-out future, which is available only because this is a demo. The red curve is the TabPFN-TS forecast. The shaded red region is the 0.1 to 0.9 quantile interval.

The forecast captures the most obvious structure in both series: strong annual seasonality, sharp yearly peaks, and a recurring drop after the peak. This is exactly where the feature transformation matters. The model is not seeing a raw sequence alone; it is seeing a tabular representation that exposes time position and seasonal phase.

The forecast is not perfect. For example, the sharpness and height of some future peaks are difficult to match exactly. That is expected because monthly tourism demand is not deterministic. The useful question is not whether every point lands exactly on the future curve. The useful question is whether the model captures the seasonal structure, gives sensible point forecasts, and expresses uncertainty that is reasonable for the held-out window.

3.5 Evaluating Forecasts in Practice

The demo is useful as a first look, but a real forecasting workflow would need numerical evaluation. At minimum, a practitioner should compute point forecast errors and quantile coverage on the held-out window.

import numpy as np

eval_df = test_tsdf_ground_truth[["target"]].rename(
    columns={"target": "actual"}
).join(
    pred.rename(columns={"target": "forecast"})
)

q10_col = 0.1 if 0.1 in eval_df.columns else "0.1"
q90_col = 0.9 if 0.9 in eval_df.columns else "0.9"

mae = (eval_df["actual"] - eval_df["forecast"]).abs().mean()
rmse = np.sqrt(((eval_df["actual"] - eval_df["forecast"]) ** 2).mean())
coverage_80 = (
    (eval_df["actual"] >= eval_df[q10_col])
    & (eval_df["actual"] <= eval_df[q90_col])
).mean()

print(f"MAE: {mae:.3f}")
print(f"RMSE: {rmse:.3f}")
print(f"80% interval coverage: {coverage_80:.3f}")

For a more complete evaluation, it is also useful to break the errors down by item ID and forecast horizon. In forecasting, average error can hide important behavior. A model may be good for short horizons but weak for longer horizons, or good for one item but poor for another.

eval_df = eval_df.reset_index()
eval_df["horizon"] = eval_df.groupby("item_id").cumcount() + 1
eval_df["absolute_error"] = (eval_df["actual"] - eval_df["forecast"]).abs()

display(
    eval_df.groupby("horizon")["absolute_error"]
    .mean()
    .rename("MAE by horizon")
)

4. Summary and Conclusion

In this post, I used the time series forecasting section of the official TabPFN hands-on demo to understand how TabPFN can be applied outside ordinary static tabular prediction.

The conceptual section made the key step explicit: TabPFN-TS frames univariate time series forecasting as tabular regression. This transformation is not unique to TabPFN; supervised ML models such as XGBoost, LightGBM, Random Forest, and CatBoost can also use time-derived tabular features. What changes with TabPFN-TS is the regression engine: a pretrained, context-conditioned tabular foundation model is used instead of fitting a new task-specific model from scratch.

Operationally, the observed history becomes rows with known targets. Future timestamps become rows with missing targets. Running-index, calendar, and seasonal features give the tabular model information about trend, time position, and cyclic structure.

The hands-on demo then showed this idea in code. We loaded two monthly tourism time series from the Chronos datasets collection, held out the last 24 months, added temporal features, predicted with TabPFNTimeSeriesPredictor, and visualized both point forecasts and 0.1 to 0.9 quantile intervals.

The main takeaway is that TabPFN is not being used as a native sequence model here. The bridge is feature engineering. Once time is represented as tabular features, TabPFNv2 can be used as a zero-shot tabular regressor for future timestamps.

This makes the workflow conceptually simple and practically interesting. It also creates clear evaluation questions: how accurate are the point forecasts, how well calibrated are the quantile intervals, and how does the model behave across different horizons, seasonalities, and item IDs?

With this post, I have covered another remaining section of the official TabPFN hands-on demo. In the upcoming posts, I will continue exploring the parts of the TabPFN ecosystem that can translate into useful workflows for real tabular and time-dependent data problems. As I continue this series, I welcome feedback and requests from readers: what did you find most useful in this post, and which aspects of tabular foundation models should I explore next?