These are my background notes on a corner of continual learning I find unusually clean. I wrote them the way I would brief a new labmate: start from the problem, keep the math honest but minimal, and stop at the point where my own research begins. If you read to the end you will understand exactly what question my paper picks up.
The problem: forgetting
Class-incremental learning trains a model on a sequence of tasks, each bringing new classes, and then tests it on everything seen so far. The hard constraint is that once a task has passed, its data is gone. Train a neural network the ordinary way on this stream and it suffers catastrophic forgetting: the gradient updates for new classes overwrite what made the old ones work.
The two familiar defenses both have a cost. Replay keeps a buffer of old examples and mixes them back in — but storing data is exactly what many real settings forbid, for privacy or for space. Regularization tries to protect important weights from changing, which slows learning of the new. Both fight forgetting while still doing the thing that causes it: updating the whole network by gradient descent.
The move: freeze, then solve
Analytic continual learning makes a sharp bet: don't update the network at all. Take a backbone f — a pretrained feature extractor — freeze it, and learn only a linear classifier on top of its features. Forgetting came from the gradient updates, so a method that never takes them cannot forget in that way.
That sounds like it gives up too much, until you notice what a frozen backbone buys you. A linear classifier trained with squared error is ridge regression, and ridge regression has a closed-form solution — one equation, no iteration. If X stacks the features of your data and Y their one-hot labels, the best weights are:
The important part is what this equation depends on. It does not need the individual examples — only two summaries: XTX, how the features correlate (a Gram matrix), and XTY, how features relate to labels. Both are just running sums. When a new task arrives, you add its contribution to each sum and re-solve. No old images, no gradients.
ACIL: learning by recursion
ACIL [1] turned this idea into a full method, and added two things worth knowing.
First, a feature expansion: before the readout, it pushes the backbone's features through one extra layer with random, frozen weights and a ReLU. This lifts the features into a higher dimension where the classes are easier to separate with a straight line. Nothing about it is learned — it is a fixed random map — but it makes the cheap linear classifier much stronger.
Second, and elegantly, a recursive update. ACIL stores the inverse of the regularized Gram matrix — it calls this the regularized feature autocorrelation matrix, R — and when a new task arrives it updates R and the weights with a rank-update identity, without ever re-inverting from scratch:
The payoff is a property ACIL calls absolute memorization: learning the tasks one at a time, this way, gives exactly the same classifier as training on all of them together. Incremental learning with no approximation and no forgetting — because the recursion is algebraically identical to the joint solution. And since only R and the weights are kept, never the past images, the method is private by construction.
This is the part I find beautiful: forgetting is not fought here, it is defined away. The update is the joint answer.
RanPAC: why width helps
RanPAC [2] brought the same recipe to modern pretrained models and sharpened the role of the expansion. On top of a frozen ViT, it inserts a random projection into a much wider space, followed by a nonlinearity, and then builds class prototypes with second-order statistics — the same Gram-matrix machinery.
Why does projecting into a wider random space help, when it adds no new information? Because width makes classes more linearly separable: in a high enough dimension, a simple linear readout can carve boundaries that were tangled in the original features. RanPAC's gains are large — double-digit error reductions across benchmarks, with no replay — and they come mostly from this expansion plus a decorrelation step that removes the bias pretrained features carry. The lesson that matters here: more width, better accuracy.
What you actually carry
Now the tension that pulls me in. The expansion wants to be wide. But the thing the learner must carry between tasks — the Gram matrix of those expanded features — is a square matrix. Double the width and you quadruple its size. At ten thousand coordinates it is hundreds of megabytes, all of it just the record of how coordinates interact.
So "more width, better accuracy" collides with a budget. If the state you carry has to stay small — imagine many separate task streams sharing one frozen backbone, each keeping its own readout — you cannot simply make the expansion as wide as you like. Something has to give: fewer coordinates, or a cheaper way to store how they interact.
Where this goes
That collision is exactly the question my current work takes up: given a fixed memory budget, should you spend it on more width, or on storing the interactions among coordinates more faithfully? The method and the answer are in a paper under review, so I am not posting them here yet — but the next note frames the question itself.
If the setup in these notes now feels natural, that is the point.
Read: Width or Interaction Fidelity? →
References
- H. Zhuang et al. ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection. NeurIPS 2022.
- M. D. McDonnell et al. RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023.
These are personal study notes; any errors are mine. I have simplified the equations for readability — see the papers for the exact statements.