Skip to content

Load a pretrained decoder and keep training it

In notebook 05 we trained a language model from random weights. But most of the time, you would start from a model someone else already trained, on far more data than we have. So here we load SmolLM2-360M into Dew’s own CausalTransformer. It is a small Llama-style model from Hugging Face, described in the paper “SmolLM2: When Smol Goes Big” by Allal et al. (2025). We continue its training on Python code from GitHub and export the result back to the Hugging Face format. At the end, we load the export with the transformers library and check that it predicts the same next token as Dew.

The round trip. A Hugging Face checkpoint loads into Dew, Dew's next-token probabilities match transformers', Dew trains it on Python code, and the export loads back into transformers with the same prediction.Hugging Faceconfig.jsonmodel.safetensorsloadDewCausalTransformerparams treesame P(next token)as transformerstrain on PythonDew, trainedlower perplexityon held-out codeexportHugging Faceconfig.jsonmodel.safetensorstransformersreads it back:same next token

interop adds safetensors support, and torch is for transformers, which checks the loaded model and reads the exported checkpoint.

%pip install -q "dewml[cuda13,interop,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" torch --extra-index-url https://pypi.dewml.dev/simple
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheel for dewml (pyproject.toml) ... done
CHECKPOINT = "HuggingFaceTB/SmolLM2-360M"
SEQUENCE_LENGTH = 1024
BATCH_SIZE = 16
STEPS = 2000
LEARNING_RATE = 3e-5
MAX_NEW_TOKENS = 80
PROMPT = "def is_palindrome(text):"
RUN_DIR = "runs/08-continued"
EXPORT_DIR = "runs/08-exported"
SEED = 0
import os
import jax
import jax.numpy as jnp
import numpy as np
from dew import figures
from dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX call
print(jax.devices())
[CudaDevice(id=0)]

PretrainedDecoder.load takes a Hub repo id or a local folder in the Hugging Face layout. It translates the checkpoint’s config.json into CausalTransformer settings and its weights into Dew’s parameter tree. The weights stay in float32, and the model computes in the dtype we ask for. If the config has a setting that changes the computation and Dew has no equivalent for it, load raises an error naming it, instead of silently dropping it.

It returns a PretrainedDecoder, the kind of Pretrained a decoder checkpoint loads as, with the model on .model, its weights on .variables, and the Dew settings it derived on .model_config.

from dew.interop import PretrainedDecoder
pretrained = PretrainedDecoder.load(CHECKPOINT, dtype=jnp.bfloat16, attention_impl="auto", max_seq_len=SEQUENCE_LENGTH)
model, variables = pretrained.model, pretrained.variables
n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables))
print(f"{n_params / 1e6:.0f}M parameters")
print({key: pretrained.model_config[key] for key in
("vocab_size", "emb_features", "num_layers", "num_heads", "num_kv_heads", "head_dim")})
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
362M parameters
{'vocab_size': 49152, 'emb_features': 960, 'num_layers': 32, 'num_heads': 15, 'num_kv_heads': 5, 'head_dim': 64}

The model was trained on the ids of its own tokenizer, so we must use that tokenizer too; any other tokenizer would feed it ids that mean something else entirely in its vocabulary. HFTokenizer wraps any Hugging Face tokenizer behind the same encode and decode as Dew’s byte tokenizer.

from dew.data import HFTokenizer
tokenizer = HFTokenizer(CHECKPOINT)
print("vocabulary:", tokenizer.vocab_size)
print(tokenizer.encode(PROMPT))
vocabulary: 49152
[1604, 314, 79, 22162, 526, 4522, 24, 2692, 727]

Before we train anything, let’s see what the model writes as it is. SmolLM2 learned from web text, code and textbooks, so it already knows some Python. We generate greedily, taking the most likely token every time, so the output is the same on every run.

from dew.sampling import Sampling, generate
prompt = jnp.asarray([tokenizer.encode(PROMPT)], jnp.int32)
before = generate(model, variables, prompt, max_new_tokens=MAX_NEW_TOKENS,
key=jax.random.key(0), sampling=Sampling(temperature=0.0))
print(tokenizer.decode(before.tokens[0]))
def is_palindrome(text):
    return text == text[::-1]

def is_palindrome_2(text):
    return text == text[::-1]

def is_palindrome_3(text):
    return text == text[::-1]

def is_palindrome_4(text):
    return text == text[::-1]

def is_palindrome_

Its first function is a correct palindrome check, but then it can’t stop: it writes the same function again as is_palindrome_2, is_palindrome_3 and so on. Greedy decoding from a small model often falls into a loop like this.

A loaded checkpoint is only right if it computes what the original does. So before we change anything, let’s put both side by side: the same prompt through Dew in float32 and through transformers in float32, the reference implementation, and the probabilities each gives the next token.

import torch
from transformers import AutoModelForCausalLM
reference = AutoModelForCausalLM.from_pretrained(CHECKPOINT, dtype=torch.float32)
with torch.no_grad():
theirs = torch.softmax(reference(input_ids=torch.tensor(np.asarray(prompt))).logits[0, -1], -1)
full = PretrainedDecoder.load(CHECKPOINT, dtype=jnp.float32, max_seq_len=SEQUENCE_LENGTH)
ours = jax.nn.softmax(full.model.apply(full.variables, prompt)[0, -1])
theirs, ours = theirs.numpy(), np.asarray(ours)
top = np.argsort(theirs)[::-1][:6]
print("largest difference in any probability:", float(np.abs(theirs - ours).max()))
figures.bars([theirs[top], ours[top]],
labels=[repr(tokenizer.decode([token]))[1:-1] for token in top],
captions=["transformers", "Dew"], y_max=float(theirs[top[0]]),
y_label="P(next token)")
largest difference in any probability: 0.00023849308490753174
P(next token): \n , \r\n , \n, \n , \n , \n\n P(next token)\n 0.750.75\r\n 0.080.08\n0.080.08\n 0.020.02\n 0.010.01\n\n 0.010.01transformersDew

They agree. The largest difference in any of the 49,152 probabilities is 0.00024, and both put 0.75 on the same next token, a newline followed by indentation.

The corpus is codeparrot-clean-valid, about 61,000 Python files from GitHub, with exact duplicates and auto-generated files removed. Its code is in a column called content, which text.tokenize turns into ids with the loaded decoder’s 49,152-entry vocabulary. We hold out the first 1% of that token stream, and the language-model preset cuts each part into windows (pack=None); the ids are tokens already, so it takes no tokenizer. The training windows are cut first and then shuffled, so every epoch reads each window once.

from dew.data import Hub, Loading, presets, text
from dew.data.compile import compile as compile_data
code = Hub("codeparrot/codeparrot-clean-valid").map(text.tokenize(CHECKPOINT, field="content", eos=True))
train, held = code.holdout(tokens=0.01, fields=("text",))
windows = {"seq_len": SEQUENCE_LENGTH, "pack": None}
train_windows = train.pipe(presets.lm, None, **windows, batch=None).shuffle(seed=0).repeat()
data = compile_data(train_windows.batch(BATCH_SIZE),
validation=held.pipe(presets.lm, None, **windows, batch=BATCH_SIZE),
read="memory", provision=True, loading=Loading(workers=0))
stderr
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
22:23:18 WARNING dew.data: preparing hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train (size not stated) into /root/.cache/huggingface/datasets/codeparrot___codeparrot-clean-valid/default/0.0.0/4db92d2ec0c1b4c41eeb439cfae16854511d9dcd
stderr
22:23:26 WARNING dew.data: preparing hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train (135.6 MiB) into /root/.cache/huggingface/datasets/codeparrot___codeparrot-clean-valid/default/0.0.0/4db92d2ec0c1b4c41eeb439cfae16854511d9dcd
stderr
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (26303 > 8192). Running this sequence through the model will result in indexing errors
print("training windows:", data.records)
training windows: 165581

LMObjective takes the loaded weights through variables=, so training starts from them instead of from a fresh random draw. It keeps no moving average of the weights, so validation scores the weights that trained. We keep the learning rate small, so that the model adapts to this code while staying close to what it already knew. The validation perplexity, measured every few hundred steps, shows the adaptation.

from dew import Checkpoints, LocalTracker, Trainer
from dew.config import OptimConfig
from dew.objectives.lm import LMObjective, Perplexity
objective = LMObjective(model, SEQUENCE_LENGTH, variables=variables)
trainer = Trainer(objective, OptimConfig(learning_rate=LEARNING_RATE),
key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR),
tracker=LocalTracker(f"{RUN_DIR}/tracking"))
state = trainer.fit(data, steps=STEPS, log_every=STEPS // 10, eval_every=STEPS // 4,
checkpoint_every=STEPS, metrics=(Perplexity(),))
Training CausalTransformer from step 0 to 2000: 361.8M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 16, float32 parameters, bfloat16 compute
22:39:42 WARNING dew.nn.kernels.tuning: 6 kernel launches have no config measured on NVIDIA A100-SXM4-40GB, so they run the nearest measured shape's or their heuristic: measure them on a GPU to yourself with Trainer(tune_kernels=True) or `python tools/kernel_picks.py`
step 200/2000  loss 1.251  ce 1.251  perplexity 3.495  token_accuracy 72.5%  data_wait_ms 0.03575  grad_norm 0.5024  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.34  step_time_ms 243.6  samples_per_sec 65.68  tokens 16,384  tokens_per_s 67,254  mfu 50.9%  hlo_flops_util_lower_bound 50.9%  hfu_lower_bound 50.9%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:07:13 left
step 400/2000  loss 1.260  ce 1.260  perplexity 3.526  token_accuracy 72.8%  data_wait_ms 0.03565  grad_norm 0.4938  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.34  step_time_ms 241.8  samples_per_sec 66.18  tokens 16,384  tokens_per_s 67,766  mfu 51.2%  hlo_flops_util_lower_bound 51.2%  hfu_lower_bound 51.2%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:06:27 left
eval val at step 500: perplexity 3.465   loss 1.243 (1672 records in 18.90 s)
step 600/2000  loss 1.221  ce 1.221  perplexity 3.389  token_accuracy 73.7%  data_wait_ms 0.03680  grad_norm 0.4966  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.35  step_time_ms 358.8  samples_per_sec 44.59  tokens 16,384  tokens_per_s 45,662  mfu 34.5%  hlo_flops_util_lower_bound 34.5%  hfu_lower_bound 34.5%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:05:37 left
step 800/2000  loss 1.199  ce 1.199  perplexity 3.316  token_accuracy 73.4%  data_wait_ms 0.03581  grad_norm 0.4948  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.36  step_time_ms 241.7  samples_per_sec 66.19  tokens 16,384  tokens_per_s 67,780  mfu 51.3%  hlo_flops_util_lower_bound 51.3%  hfu_lower_bound 51.3%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:04:50 left
step 1000/2000  loss 1.289  ce 1.289  perplexity 3.628  token_accuracy 72.7%  data_wait_ms 0.03567  grad_norm 0.4932  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.36  step_time_ms 242.3  samples_per_sec 66.03  tokens 16,384  tokens_per_s 67,614  mfu 51.1%  hlo_flops_util_lower_bound 51.1%  hfu_lower_bound 51.1%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:04:03 left
eval val at step 1000: perplexity 3.401 ↓ 1.8%   loss 1.224 ↓ 1.5% (1672 records in 9.20 s)
step 1200/2000  loss 1.107  ce 1.107  perplexity 3.024  token_accuracy 75.4%  data_wait_ms 0.03610  grad_norm 0.4917  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.37  step_time_ms 294.2  samples_per_sec 54.38  tokens 16,384  tokens_per_s 55,681  mfu 42.1%  hlo_flops_util_lower_bound 42.1%  hfu_lower_bound 42.1%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:03:14 left
step 1400/2000  loss 1.225  ce 1.225  perplexity 3.404  token_accuracy 73.2%  data_wait_ms 0.03591  grad_norm 0.5096  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.37  step_time_ms 242.3  samples_per_sec 66.04  tokens 16,384  tokens_per_s 67,623  mfu 51.1%  hlo_flops_util_lower_bound 51.1%  hfu_lower_bound 51.1%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:02:25 left
eval val at step 1500: perplexity 3.366 ↓ 1.0%   loss 1.214 ↓ 0.9% (1672 records in 9.65 s)
step 1600/2000  loss 1.134  ce 1.134  perplexity 3.108  token_accuracy 74.4%  data_wait_ms 0.04808  grad_norm 0.4880  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.38  step_time_ms 294.9  samples_per_sec 54.26  tokens 16,384  tokens_per_s 55,564  mfu 42.0%  hlo_flops_util_lower_bound 42.0%  hfu_lower_bound 42.0%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:01:38 left
step 1800/2000  loss 1.250  ce 1.250  perplexity 3.489  token_accuracy 73.3%  data_wait_ms 0.03581  grad_norm 0.4855  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.39  step_time_ms 242.2  samples_per_sec 66.06  tokens 16,384  tokens_per_s 67,643  mfu 51.2%  hlo_flops_util_lower_bound 51.2%  hfu_lower_bound 51.2%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  0:00:48 left
step 2000/2000  loss 1.122  ce 1.122  perplexity 3.070  token_accuracy 75.2%  data_wait_ms 0.03559  grad_norm 0.4882  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 23.39  step_time_ms 242.3  samples_per_sec 66.04  tokens 16,384  tokens_per_s 67,627  mfu 51.1%  hlo_flops_util_lower_bound 51.1%  hfu_lower_bound 51.1%  unsized_kernels 194.0  accepted 100.0%  learning_rate 3.000e-05  accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581  attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373  chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0  truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0
eval val at step 2000: perplexity 3.340 ↓ 0.8%   loss 1.206 ↓ 0.6% (1672 records in 9.21 s)
Trained 2000 steps in 0:09:43: first step after 31.48 s, then 4.0 step/s
85.4% of the wall time in steps, final loss 1.122
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]
train = [(row["step"], row["scalars"]["train/loss"]) for row in rows
if "train/loss" in row["scalars"]]
val = [(row["step"], row["scalars"]["val/perplexity"]) for row in rows
if "val/perplexity" in row["scalars"]]
print("validation perplexity:", ", ".join(f"{ppl:.2f} at {step}" for step, ppl in val))
figures.curve({"training": [loss for _, loss in train]}, x=[step for step, _ in train],
x_label="step", y_label="loss (nats per token)")
validation perplexity: 3.46 at 500, 3.40 at 1000, 3.37 at 1500, 3.34 at 2000
loss (nats per token) against step1.151.21.251.3500100015002000steploss (nats per token)1.12
after = generate(model, state.variables, prompt, max_new_tokens=MAX_NEW_TOKENS,
key=jax.random.key(0), sampling=Sampling(temperature=0.0))
print(tokenizer.decode(after.tokens[0]))
def is_palindrome(text):
    """
    Check if a string is a palindrome.

    :param text: The string to check.
    :type text: str
    :return: True if the string is a palindrome, False otherwise.
    :rtype: bool
    """
    if not text:
        return False
    if len(text) <= 1:
        return True
    if text[0]

Validation perplexity falls from 3.46 at the first check, step 500, to 3.34 at the end. The model was already good at this kind of code: a perplexity near 3.4 is roughly the uncertainty of choosing among three or four equally likely tokens on average, and the training sharpens it a little. The generation changes more. Instead of looping, the model now writes one function with a docstring in the style of the files it trained on, a description, :param and :rtype lines, and then the checks themselves.

Where did the model change its mind? Here is a held-out file’s first lines, each token shaded by the probability the model gave it, before training and after: green means more than 0.5, blue between 0.1 and 0.5, and orange 0.1 or less.

from dew.data.dataset import DataPartition
held_out = next(iter(data.val(DataPartition())))["text"][0][:33]
def shaded(variables):
logits = model.apply(variables, jnp.asarray(held_out[None, :-1]))[0]
probs = np.asarray(jax.nn.softmax(logits.astype(jnp.float32)))
probs = probs[np.arange(len(held_out) - 1), held_out[1:]]
return ["accent_fill" if p > 0.5 else "tint1" if p > 0.1 else "warm_fill" for p in probs]
pieces = [repr(tokenizer.decode([token]))[1:-1] or "·" for token in held_out[1:]]
rows = [pieces[i:i + 8] for i in range(0, len(pieces), 8)]
fills = [shaded(variables), shaded(state.variables)]
for label, shades in zip(("before", "after"), fills):
print(label, {shade: shades.count(shade) for shade in ("accent_fill", "tint1", "warm_fill")})
figures.tokens(rows * 2, fills=[f[i:i + 8] for f in fills for i in range(0, len(pieces), 8)],
notes=["before"] + [""] * (len(rows) - 1) + ["after"] + [""] * (len(rows) - 1))
before {'accent_fill': 19, 'tint1': 8, 'warm_fill': 5}
after {'accent_fill': 23, 'tint1': 5, 'warm_fill': 4}
token rows django.conf import settings\nfrom djangobefore.core.paginator import Paginator, InvalidPage, EmptyPage\nfrom django.http import HttpResponse\n\nfrom django.conf import settings\nfrom djangoafter.core.paginator import Paginator, InvalidPage, EmptyPage\nfrom django.http import HttpResponse\n\nfrom

Before training, 19 of these 32 tokens get more than half the model’s probability; after, 23 do, and the tokens it gave less than 0.1 drop from 5 to 4. These lines are Django imports, exactly the kind of boilerplate the training data is full of.

PretrainedDecoder.from_model wraps the trained weights for export, and its save writes them in the Hugging Face layout: config.json, model.safetensors, and the tokenizer files. It runs the same name and shape translation as PretrainedDecoder.load, just backwards. Loading the export again gives back exactly the weights we trained.

PretrainedDecoder.from_model(model, state.variables, tokenizer=CHECKPOINT).save(EXPORT_DIR)
print(sorted(os.listdir(EXPORT_DIR)))
reloaded = PretrainedDecoder.load(EXPORT_DIR, dtype=jnp.bfloat16, attention_impl="auto", max_seq_len=512)
same = all(np.array_equal(np.asarray(a), np.asarray(b))
for a, b in zip(jax.tree_util.tree_leaves(reloaded.variables["params"]),
jax.tree_util.tree_leaves(state.variables["params"])))
print("reloaded weights equal the trained ones:", same)
['config.json', 'generation_config.json', 'model.safetensors', 'tokenizer.json', 'tokenizer_config.json']
reloaded weights equal the trained ones: True

An export is only useful if another library can actually read it. So we load the folder with transformers, run the prompt through it in float32, and compare its most likely next token with Dew’s.

import torch
from transformers import AutoModelForCausalLM
hf_model = AutoModelForCausalLM.from_pretrained(EXPORT_DIR, dtype=torch.float32)
with torch.no_grad():
hf_next = int(hf_model(input_ids=torch.tensor(np.asarray(prompt))).logits[0, -1].argmax())
dew_next = generate(model, state.variables, prompt, max_new_tokens=1,
key=jax.random.key(0), sampling=Sampling(temperature=0.0))
dew_next = int(dew_next.tokens[0, -1])
print("transformers:", hf_next, repr(tokenizer.decode([hf_next])))
print("dew: ", dew_next, repr(tokenizer.decode([dew_next])))
transformers: 472 '\n   '
dew:          472 '\n   '

The README model list names the decoder families Dew loads, trains, generates with and exports. The recipe flag --pretrained runs this flow at scale, for example python recipes/lm/train.py --set data.source.url=<dir> --pretrained HuggingFaceTB/SmolLM2-360M --tokenizer HuggingFaceTB/SmolLM2-360M.