Continual learning in context
A trained language model is frozen: whatever it reads, its weights stay where training left them. Nested Learning (Behrouz et al., 2025) asks what happens if some of them keep learning while the model reads. Its continuum memory system gives a model levels of parameters that update at different frequencies: a fast level adapts every few dozen tokens and forgets as quickly, a slow one every few hundred and keeps what it learned for longer. Each level takes a step of gradient descent on the model’s own next-token loss over the tokens it just read, and the model reads on with the stepped weights.
In this notebook we take a small pretrained model, SmolLM2-135M, and turn the feed-forward blocks of its last four layers into such levels, each at its own frequency. We measure what learning in context buys it on held-out text, train the levels’ starting points by continued pre-training through their own steps, and then put the model through class-incremental learning: ten tasks of intent classification, learned one after another, with the forgetting of earlier tasks measured after each.
Install dependencies
Section titled “Install dependencies”Dew from GitHub with its cuda13 and streaming extras. Dew streams the text and the intent dataset from the Hub.
%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@7715ecf18adf6827aeae6025f95203f78df0b707" --extra-index-url https://pypi.dewml.dev/simpleSettings
Section titled “Settings”MODEL = "HuggingFaceTB/SmolLM2-135M"SEQUENCE_LENGTH = 512BATCH_SIZE = 16LEVEL_LAYERS = (26, 27, 28, 29)LEVEL_WINDOWS = (32, 64, 128, 256)LEVEL_RATES = (1e-3, 5e-4, 2.5e-4, 1.25e-4)PRETRAIN_STEPS = 600TASKS = 10INTENTS_PER_TASK = 15TASK_STEPS = 60TEST_PER_INTENT = 4RUN_DIR = "runs/13-continual"SEED = 0import jaximport jax.numpy as jnpimport numpy as np
from dew import figuresfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())A pretrained model and its levels
Section titled “A pretrained model and its levels”Pretrained.load reads SmolLM2-135M from the Hub as a native CausalTransformer. Nested Learning’s section 7.3 starts a continuum memory system from a pretrained model’s own blocks: each level is one of its feed-forward blocks, its trained weights the level’s start. So the model itself does not change at all. What changes is the objective, which now steps those blocks in context.
ChunkAdaptation(layers=...) names the layers whose feed-forward blocks become levels, with a window and a learning rate each. The block of layer 26 steps every 32 tokens, layer 29’s every 256. Within a row, the model reads the text a chunk of 32 tokens at a time, through its own decode cache, and each chunk reads every level at the weights its window started with. At the end of a level’s window, its weights take one step down the gradient of the loss over the window’s tokens.
from dew.interop import Pretrainedfrom dew.objectives.lm import ChunkAdaptation, LMObjective
loaded = Pretrained.load(MODEL, dtype=jnp.bfloat16, attention_impl="auto", max_seq_len=SEQUENCE_LENGTH)model, pretrained = loaded.model, loaded.variablesadaptation = ChunkAdaptation(window=LEVEL_WINDOWS, rate=LEVEL_RATES, layers=LEVEL_LAYERS)adapted = LMObjective(model, SEQUENCE_LENGTH, adaptation=adaptation, variables=pretrained)plain = LMObjective(model, SEQUENCE_LENGTH)print(f"{sum(leaf.size for leaf in jax.tree.leaves(pretrained['params'])) / 1e6:.0f}M parameters, " f"{sum(leaf.size for layer in LEVEL_LAYERS for leaf in jax.tree.leaves(pretrained['params'][f'layers_{layer}']['mlp'])) / 1e6:.1f}M of them in levels")Learning while reading
Section titled “Learning while reading”We read FineWeb-Edu text the model never saw, windows of 512 tokens, and score every token twice: with the model as it is, and with its levels stepping in context. The difference is what test-time training on the model’s own loss gives a frozen model.
from dew.data import Hub, presetsfrom dew.data.compile import compile as compile_datafrom dew.data.dataset import DataPartition
fineweb = Hub("HuggingFaceFW/fineweb-edu", name="sample-10BT")crawl = Hub("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2025-26")windows = {"seq_len": SEQUENCE_LENGTH, "batch": BATCH_SIZE, "pack": None}data = compile_data(fineweb.shuffle(seed=0, window=64).repeat().pipe(presets.lm, MODEL, **windows), validation=crawl.pipe(presets.lm, MODEL, **windows, records=4), read="streamed")held_out = [batch["text"] for batch in data.val(DataPartition())]
def perplexity(objective, variables): losses = [objective.token_scores(variables, jnp.asarray(rows)).losses for rows in held_out] return float(jnp.exp(jnp.mean(jnp.stack(losses))))
before = {"frozen": perplexity(plain, pretrained), "in context": perplexity(adapted, pretrained)}print(before)figures.bars([list(before.values())], labels=list(before), y_max=max(before.values()), y_label="validation perplexity")Training the levels’ starts
Section titled “Training the levels’ starts”The levels start from the pretrained blocks, which were trained to be read, not to be stepped. Continued pre-training trains them, and every other weight, through the levels’ own steps: the outer loss differentiates through each in-context step back to the starting weights, a second-order gradient. This is the meta-learning step of Nested Learning’s outermost level.
from dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfigfrom dew.optim import Cosine
schedule = Cosine(peak=3e-5, warmup_steps=PRETRAIN_STEPS // 10, decay_steps=PRETRAIN_STEPS, end=3e-6)trainer = Trainer(adapted, OptimConfig(schedule=schedule), key=jax.random.key(SEED), checkpoints=Checkpoints(f"{RUN_DIR}/pretrain"), tracker=LocalTracker(f"{RUN_DIR}/tracking"))state = trainer.fit(data, steps=PRETRAIN_STEPS, log_every=PRETRAIN_STEPS // 10, checkpoint_every=PRETRAIN_STEPS)trained = state.variablesafter = {"frozen": perplexity(plain, trained), "in context": perplexity(adapted, trained)}print(after)figures.bars([list(before.values()), list(after.values())], labels=list(before), captions=["pretrained", "after continued pre-training"], y_max=max(*before.values(), *after.values()), y_label="validation perplexity")Class-incremental learning
Section titled “Class-incremental learning”CLINC150 (Larson et al., 2019) has 150 intents of short user queries. We split its intents into ten tasks of fifteen, and fine-tune on the tasks one after another, each on its own training queries alone. After every task we test on every task’s held-out queries, choosing among every intent seen so far: the model reads Query: ... and we score the log-likelihood it gives each intent’s name after Intent:, picking the highest (dew.eval.label_scores). That is class-incremental learning, the setting Nested Learning’s section 9.1 measures.
continual_run collects the accuracies into a matrix: row i is the accuracies after task i, column j task j’s. Its last row’s average is the final accuracy, and a task’s forgetting is the most it lost from its best.
from dew.data import HFTokenizer, Memory
tokenizer = HFTokenizer(MODEL)clinc = Hub("clinc_oos", name="plus")names = clinc.builder.info.features["intent"].namestrain_rows = [record for record in clinc.records("indexed")]test_rows = [record for record in clinc.with_split("test").records("indexed")]rng = np.random.default_rng(SEED)intents = [index for index, name in enumerate(names) if name != "oos"]rng.shuffle(intents)tasks = [intents[at * INTENTS_PER_TASK:(at + 1) * INTENTS_PER_TASK] for at in range(TASKS)]
def query(text): return tokenizer.encode(f"Query: {text}\nIntent:", add_special_tokens=False)
def label(intent): return tokenizer.encode(" " + names[intent].replace("_", " "), add_special_tokens=False)
def task_data(task): kept = [{"text": f"Query: {row['text']}\nIntent: {names[row['intent']].replace('_', ' ')}"} for row in train_rows if row["intent"] in task] return compile_data(Memory(kept).shuffle(seed=0).repeat().pipe(presets.lm, MODEL, seq_len=64, batch=BATCH_SIZE, pack="first-fit"))
def tests(task): picked = {} for row in test_rows: if row["intent"] in task and len(picked.setdefault(row["intent"], [])) < TEST_PER_INTENT: picked[row["intent"]].append(row["text"]) return [(query(text), intent) for intent, texts in picked.items() for text in texts]from dew.eval import continual_run, label_accuracy, label_scores
def run(steps_in_context, name): objective = LMObjective(model, 64, adaptation=adaptation if steps_in_context else None, variables=trained) learner = Trainer(objective, OptimConfig(schedule=Cosine(peak=1e-4, warmup_steps=5, decay_steps=TASKS * TASK_STEPS, end=1e-5)), key=jax.random.key(SEED), checkpoints=Checkpoints(f"{RUN_DIR}/{name}")) held = {}
def fit(stage, task): held["state"] = learner.fit(task_data(task), steps=TASK_STEPS * (stage + 1), log_every=TASK_STEPS, checkpoint_every=TASK_STEPS)
def accuracy(stage, scored): if scored > stage: # A task not yet trained has no intent among those seen: class-incremental scores it 0. return 0.0 seen = [intent for task in tasks[:stage + 1] for intent in task] examples = tests(tasks[scored]) variables = held["state"].variables scores = label_scores(lambda rows: objective.token_scores(variables, rows).losses, [prompt for prompt, _ in examples], [label(intent) for intent in seen], pad_id=tokenizer.eos_id or 0) return label_accuracy(scores, [seen.index(intent) for _, intent in examples])
return continual_run(tasks, fit, accuracy)
matrices = {"frozen": run(False, "frozen"), "in context": run(True, "in-context")}for name, matrix in matrices.items(): print(name, matrix.record())figures.heatmap([matrix.values for matrix in matrices.values()], rows=[str(at) for at in range(TASKS)], columns=[str(at) for at in range(TASKS)], captions=list(matrices), x_label="task scored", y_label="after task")figures.bars([[matrix.average(), matrix.average_forgetting()] for matrix in matrices.values()], labels=["final accuracy", "forgetting"], captions=list(matrices))Where to go next
Section titled “Where to go next”The levels here are a pretrained model’s own blocks. CausalTransformer(continuum=...) gives a model separate levels on its final states instead, a chain of them at their own frequencies, which ChunkAdaptation(window=..., rate=...) steps the same way; ChunkAdaptation(carry="run") keeps what they learned across documents and training steps, the memory a continual learner carries. dew.nn.memory has the in-context memories of Titans, TTT and Miras, which learn from their own reconstruction losses rather than the model’s.