Records of the !mmortal Data Scientist

Deleting every feed-forward branch removes 8.5 billion parameters and makes local Gemma 4 inference roughly three times faster.

Cut the image into patches, run one shared kernel bank over every patch, average the results, and classify.

A network whose hidden units are kernel prototypes is supposed to be legible.

An empirical feature covariance gives a trained kernel network ranked orthogonal axes.

Every hidden representation induces an empirical kernel.

A runnable companion to the lazy-training post: the Yat layer with its frozen softplus scalars, the one-line NNX filter that trains a readout while the bank stays frozen, the per-arm learning-rate bracketing, and the movement telemetry that caught the ant…

Freeze a bank of randomly initialized Yat units and train only the linear readout.

A runnable companion to the geometry-of-attention post: the telemetry that exports a trained head's actual query and key vectors, the offline replay that recomputes every row's winner under rescaled queries, the linear program that asks whether any key si…

An attention head is only ever evaluated at the tokens of the sequence, but its formula accepts any query vector at all.

A runnable companion to the compatibility-kernel post: the attention module where one branch computes softmax and the other computes the Yat kernel with no exponential anywhere, the parameter-matched training harness, the telemetry that measured our bound…

The QK post in this series ended on a construction it refused to build: keep the query and key roles, but replace the bilinear-then-exponential score with a genuine Mercer kernel between them.

A runnable companion to the depth-on-demand post: the leapfrog classifier trained with lax.scan at fixed depth, the step-doubling controller that re-renders it to tolerance at inference, the honest work accounting (probes included), and the measured tol^(…

The last post made depth a resolution: layers are time steps of a learned flow, and running more of them just renders the same trajectory finer.

A runnable companion to the memory post: the momentum block and its exact inverse, the custom_vjp whose backward pass reconstructs the trajectory instead of storing it, XLA's memory_analysis as the measuring instrument, and the (1/mu)^L float cliff reprod…

Training memory is a tax nobody chose: backprop must hold every activation of the forward pass hostage until the backward pass consumes it, so depth costs memory even when it costs little compute.

A runnable companion to the energy-conservation post: the HNN pendulum field as the symplectic gradient of one learned scalar, the plain field model it beats on drift, and the leapfrog classifier whose residual block is a kick-drift-kick step of a learned…

A pendulum organizes its motion around energy, while an ordinary residual network has no comparable scalar.

A runnable companion to the solve-vs-descend post: the Yat kernel and its Gram matrix, the exact kernel ridge solve via Cholesky, the same kernel as a Flax NNX module trained by AdamW with LR sweeps and best-epoch selection, the measured timing wall, mini…

A runnable companion: the pre-norm Transformer block as a forward-Euler step, then the residual-stream velocity ledger as one line of Flax NNX state (mu = 0 recovers plain), the ngpt-lite retraction variant, best-val early-stopped training, and the depth…

Build the Yat DeepSurv trunk and Cox loss in Flax NNX, recover exact prototype contributions and explicit row edits, then run the LR-fair five-dataset benchmark with concordance, calibration, Brier, AUC, and classical baselines.

A dense kernel-ridge solve and a learned-center Yat expansion use the same kernel family but optimize different hypothesis classes.

A pre-norm Transformer's residual stream is forward Euler: x += Attn(norm x); x += MLP(norm x).

Build a survival network from learned prototype patients, derive its exact risk decomposition, and benchmark it on five datasets against Cox, penalized Cox, Random Survival Forest, and ReLU DeepSurv.

A runnable companion: the residual block as a forward-Euler step, then the momentum residual network as a Flax NNX module with one extra state, a velocity the blocks write into.

A runnable companion: build the weight-tied Yat equilibrium operator in Flax NNX, then teach a class by appending rows to the readout (F untouched, exact) or into the shared dynamics (one paste, present at every depth).

A runnable companion: the five-run distillation experiment in JAX/Flax NNX.

A runnable companion: build a whole second feature layer by hand in JAX, on top of the hand-built first.

A runnable companion: build the matched Yat and ReLU MLPs in Flax NNX with the same softmax head, then measure their honesty.

A residual block x + F(x) is one forward-Euler step: depth is time, the block is a velocity, position moves directly.

One post taught and forgot classes by editing rows of a Yat network, with proofs that nothing else moved.

What crosses the wire in knowledge distillation besides the winning class?

One hand-built feature layer matched a trained backbone at 83.3% on Fashion-MNIST, and real networks are deep.

A network hands you a probability with every answer, and the number is the part you act on.

A runnable companion: build the Yat deep-equilibrium network in JAX/Flax NNX.

A deep network makes you choose its depth before you have seen the problem, and gives every layer its own weights.

A runnable companion: build a transformer whose feed-forward block is a finite learned-center kernel expansion.

Replace an MLP activation with a kernel and its feed-forward block becomes an explicit learned-center expansion.

A kernel is a spectral price list: it decides which functions are affordable, and regularization sets the budget.

A runnable companion: build the representer-theorem weight in JAX.

A standard neuron's weight and its input never actually meet: one is a point you can see, the other an arrow off in its own space, joined only by a shadow.

Train a small backbone and place its prototype head, then remove training entirely: implement fixed Sobel orientation channels, pool them into 343 named features, and classify with a constructed Yat head.

Construct the prototype head on random and learned features, then replace the backbone with named edge and corner measurements.

A runnable companion: build the prototype Yat-MLP in Flax NNX, then add a class by concatenating a few prototype rows and forget a class by masking them out, with no gradient steps.

Construct valid periodic kernel spectra, audit truncated RKHS norms for convergence, then solve kernel ridge regression and read regularization as spectral shrinkage, effective dimension, and a measured generalization curve.

If a neuron is a labelled picture, a classifier is a list of them, and a list is something you edit.

Build a finite bank of Yat-kernel sections in JAX/Flax NNX, verify the kernel, train it on two moons and Fashion-MNIST, inspect exact prototype contributions, and test the initialization control that separates visible centers from noisy ones.

Replace the activation with a finite bank of learned kernel sections.

A runnable companion to The Three States of Information: train tiny models in JAX and measure the three states directly: the feature-covariance spectrum collapsing from high-rank (random) to a C−1-mode frame (structured), the distributional simplicity bias

In these training runs, representation geometry moves through three recognizable regimes: random, organized into local clusters, and globally structured around separated class means.

A runnable companion to Latent on the Spectrum: build a codebook as the spectral embedding of a label kernel in JAX (classical MDS with square-root eigenvalue scaling), watch a flat spectrum become the simplex and a graded one become the horseshoe, measure