Load a pretrained decoder and keep training it
In notebook 05 we trained a language model from random weights. But most of the time, you would start from a model someone else already trained, on far more data than we have. So here we load SmolLM2-360M into Dew’s own CausalTransformer. It is a small Llama-style model from Hugging Face, described in the paper “SmolLM2: When Smol Goes Big” by Allal et al. (2025). We continue its training on Python code from GitHub and export the result back to the Hugging Face format. At the end, we load the export with the transformers library and check that it predicts the same next token as Dew.
Install Dependencies
Section titled “Install Dependencies”interop adds safetensors support, and torch is for transformers, which checks the loaded model and reads the exported checkpoint.
%pip install -q "dewml[cuda13,interop,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" torch --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheel for dewml (pyproject.toml) ... done
Settings
Section titled “Settings”CHECKPOINT = "HuggingFaceTB/SmolLM2-360M"SEQUENCE_LENGTH = 1024BATCH_SIZE = 16STEPS = 2000LEARNING_RATE = 3e-5MAX_NEW_TOKENS = 80PROMPT = "def is_palindrome(text):"RUN_DIR = "runs/08-continued"EXPORT_DIR = "runs/08-exported"SEED = 0import os
import jaximport jax.numpy as jnpimport numpy as np
from dew import figuresfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())[CudaDevice(id=0)]
Loading the checkpoint
Section titled “Loading the checkpoint”PretrainedDecoder.load takes a Hub repo id or a local folder in the Hugging Face layout. It translates the checkpoint’s config.json into CausalTransformer settings and its weights into Dew’s parameter tree. The weights stay in float32, and the model computes in the dtype we ask for. If the config has a setting that changes the computation and Dew has no equivalent for it, load raises an error naming it, instead of silently dropping it.
It returns a PretrainedDecoder, the kind of Pretrained a decoder checkpoint loads as, with the model on .model, its weights on .variables, and the Dew settings it derived on .model_config.
from dew.interop import PretrainedDecoder
pretrained = PretrainedDecoder.load(CHECKPOINT, dtype=jnp.bfloat16, attention_impl="auto", max_seq_len=SEQUENCE_LENGTH)model, variables = pretrained.model, pretrained.variablesn_params = sum(x.size for x in jax.tree_util.tree_leaves(variables))print(f"{n_params / 1e6:.0f}M parameters")print({key: pretrained.model_config[key] for key in ("vocab_size", "emb_features", "num_layers", "num_heads", "num_kv_heads", "head_dim")})stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
362M parameters
{'vocab_size': 49152, 'emb_features': 960, 'num_layers': 32, 'num_heads': 15, 'num_kv_heads': 5, 'head_dim': 64}The model was trained on the ids of its own tokenizer, so we must use that tokenizer too; any other tokenizer would feed it ids that mean something else entirely in its vocabulary. HFTokenizer wraps any Hugging Face tokenizer behind the same encode and decode as Dew’s byte tokenizer.
from dew.data import HFTokenizer
tokenizer = HFTokenizer(CHECKPOINT)print("vocabulary:", tokenizer.vocab_size)print(tokenizer.encode(PROMPT))vocabulary: 49152 [1604, 314, 79, 22162, 526, 4522, 24, 2692, 727]
Generating before training
Section titled “Generating before training”Before we train anything, let’s see what the model writes as it is. SmolLM2 learned from web text, code and textbooks, so it already knows some Python. We generate greedily, taking the most likely token every time, so the output is the same on every run.
from dew.sampling import Sampling, generate
prompt = jnp.asarray([tokenizer.encode(PROMPT)], jnp.int32)before = generate(model, variables, prompt, max_new_tokens=MAX_NEW_TOKENS, key=jax.random.key(0), sampling=Sampling(temperature=0.0))print(tokenizer.decode(before.tokens[0]))def is_palindrome(text):
return text == text[::-1]
def is_palindrome_2(text):
return text == text[::-1]
def is_palindrome_3(text):
return text == text[::-1]
def is_palindrome_4(text):
return text == text[::-1]
def is_palindrome_Its first function is a correct palindrome check, but then it can’t stop: it writes the same function again as is_palindrome_2, is_palindrome_3 and so on. Greedy decoding from a small model often falls into a loop like this.
Dew against transformers
Section titled “Dew against transformers”A loaded checkpoint is only right if it computes what the original does. So before we change anything, let’s put both side by side: the same prompt through Dew in float32 and through transformers in float32, the reference implementation, and the probabilities each gives the next token.
import torchfrom transformers import AutoModelForCausalLM
reference = AutoModelForCausalLM.from_pretrained(CHECKPOINT, dtype=torch.float32)with torch.no_grad(): theirs = torch.softmax(reference(input_ids=torch.tensor(np.asarray(prompt))).logits[0, -1], -1)full = PretrainedDecoder.load(CHECKPOINT, dtype=jnp.float32, max_seq_len=SEQUENCE_LENGTH)ours = jax.nn.softmax(full.model.apply(full.variables, prompt)[0, -1])theirs, ours = theirs.numpy(), np.asarray(ours)top = np.argsort(theirs)[::-1][:6]print("largest difference in any probability:", float(np.abs(theirs - ours).max()))figures.bars([theirs[top], ours[top]], labels=[repr(tokenizer.decode([token]))[1:-1] for token in top], captions=["transformers", "Dew"], y_max=float(theirs[top[0]]), y_label="P(next token)")largest difference in any probability: 0.00023849308490753174
They agree. The largest difference in any of the 49,152 probabilities is 0.00024, and both put 0.75 on the same next token, a newline followed by indentation.
Data pipeline
Section titled “Data pipeline”The corpus is codeparrot-clean-valid, about 61,000 Python files from GitHub, with exact duplicates and auto-generated files removed. Its code is in a column called content, which text.tokenize turns into ids with the loaded decoder’s 49,152-entry vocabulary. We hold out the first 1% of that token stream, and the language-model preset cuts each part into windows (pack=None); the ids are tokens already, so it takes no tokenizer. The training windows are cut first and then shuffled, so every epoch reads each window once.
from dew.data import Hub, Loading, presets, textfrom dew.data.compile import compile as compile_data
code = Hub("codeparrot/codeparrot-clean-valid").map(text.tokenize(CHECKPOINT, field="content", eos=True))train, held = code.holdout(tokens=0.01, fields=("text",))windows = {"seq_len": SEQUENCE_LENGTH, "pack": None}train_windows = train.pipe(presets.lm, None, **windows, batch=None).shuffle(seed=0).repeat()data = compile_data(train_windows.batch(BATCH_SIZE), validation=held.pipe(presets.lm, None, **windows, batch=BATCH_SIZE), read="memory", provision=True, loading=Loading(workers=0))stderr
Repo card metadata block was not found. Setting CardData to empty. WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty. Repo card metadata block was not found. Setting CardData to empty. WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty. 22:23:18 WARNING dew.data: preparing hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train (size not stated) into /root/.cache/huggingface/datasets/codeparrot___codeparrot-clean-valid/default/0.0.0/4db92d2ec0c1b4c41eeb439cfae16854511d9dcd
stderr
22:23:26 WARNING dew.data: preparing hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train (135.6 MiB) into /root/.cache/huggingface/datasets/codeparrot___codeparrot-clean-valid/default/0.0.0/4db92d2ec0c1b4c41eeb439cfae16854511d9dcd
stderr
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (26303 > 8192). Running this sequence through the model will result in indexing errors
print("training windows:", data.records)training windows: 165581
Continued training
Section titled “Continued training”LMObjective takes the loaded weights through variables=, so training starts from them instead of from a fresh random draw. It keeps no moving average of the weights, so validation scores the weights that trained. We keep the learning rate small, so that the model adapts to this code while staying close to what it already knew. The validation perplexity, measured every few hundred steps, shows the adaptation.
from dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfigfrom dew.objectives.lm import LMObjective, Perplexity
objective = LMObjective(model, SEQUENCE_LENGTH, variables=variables)trainer = Trainer(objective, OptimConfig(learning_rate=LEARNING_RATE), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR), tracker=LocalTracker(f"{RUN_DIR}/tracking"))state = trainer.fit(data, steps=STEPS, log_every=STEPS // 10, eval_every=STEPS // 4, checkpoint_every=STEPS, metrics=(Perplexity(),))Training CausalTransformer from step 0 to 2000: 361.8M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 16, float32 parameters, bfloat16 compute 22:39:42 WARNING dew.nn.kernels.tuning: 6 kernel launches have no config measured on NVIDIA A100-SXM4-40GB, so they run the nearest measured shape's or their heuristic: measure them on a GPU to yourself with Trainer(tune_kernels=True) or `python tools/kernel_picks.py` step 200/2000 loss 1.251 ce 1.251 perplexity 3.495 token_accuracy 72.5% data_wait_ms 0.03575 grad_norm 0.5024 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.34 step_time_ms 243.6 samples_per_sec 65.68 tokens 16,384 tokens_per_s 67,254 mfu 50.9% hlo_flops_util_lower_bound 50.9% hfu_lower_bound 50.9% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:07:13 left step 400/2000 loss 1.260 ce 1.260 perplexity 3.526 token_accuracy 72.8% data_wait_ms 0.03565 grad_norm 0.4938 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.34 step_time_ms 241.8 samples_per_sec 66.18 tokens 16,384 tokens_per_s 67,766 mfu 51.2% hlo_flops_util_lower_bound 51.2% hfu_lower_bound 51.2% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:06:27 left eval val at step 500: perplexity 3.465 loss 1.243 (1672 records in 18.90 s) step 600/2000 loss 1.221 ce 1.221 perplexity 3.389 token_accuracy 73.7% data_wait_ms 0.03680 grad_norm 0.4966 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.35 step_time_ms 358.8 samples_per_sec 44.59 tokens 16,384 tokens_per_s 45,662 mfu 34.5% hlo_flops_util_lower_bound 34.5% hfu_lower_bound 34.5% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:05:37 left step 800/2000 loss 1.199 ce 1.199 perplexity 3.316 token_accuracy 73.4% data_wait_ms 0.03581 grad_norm 0.4948 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.36 step_time_ms 241.7 samples_per_sec 66.19 tokens 16,384 tokens_per_s 67,780 mfu 51.3% hlo_flops_util_lower_bound 51.3% hfu_lower_bound 51.3% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:04:50 left step 1000/2000 loss 1.289 ce 1.289 perplexity 3.628 token_accuracy 72.7% data_wait_ms 0.03567 grad_norm 0.4932 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.36 step_time_ms 242.3 samples_per_sec 66.03 tokens 16,384 tokens_per_s 67,614 mfu 51.1% hlo_flops_util_lower_bound 51.1% hfu_lower_bound 51.1% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:04:03 left eval val at step 1000: perplexity 3.401 ↓ 1.8% loss 1.224 ↓ 1.5% (1672 records in 9.20 s) step 1200/2000 loss 1.107 ce 1.107 perplexity 3.024 token_accuracy 75.4% data_wait_ms 0.03610 grad_norm 0.4917 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.37 step_time_ms 294.2 samples_per_sec 54.38 tokens 16,384 tokens_per_s 55,681 mfu 42.1% hlo_flops_util_lower_bound 42.1% hfu_lower_bound 42.1% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:03:14 left step 1400/2000 loss 1.225 ce 1.225 perplexity 3.404 token_accuracy 73.2% data_wait_ms 0.03591 grad_norm 0.5096 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.37 step_time_ms 242.3 samples_per_sec 66.04 tokens 16,384 tokens_per_s 67,623 mfu 51.1% hlo_flops_util_lower_bound 51.1% hfu_lower_bound 51.1% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:02:25 left eval val at step 1500: perplexity 3.366 ↓ 1.0% loss 1.214 ↓ 0.9% (1672 records in 9.65 s) step 1600/2000 loss 1.134 ce 1.134 perplexity 3.108 token_accuracy 74.4% data_wait_ms 0.04808 grad_norm 0.4880 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.38 step_time_ms 294.9 samples_per_sec 54.26 tokens 16,384 tokens_per_s 55,564 mfu 42.0% hlo_flops_util_lower_bound 42.0% hfu_lower_bound 42.0% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:01:38 left step 1800/2000 loss 1.250 ce 1.250 perplexity 3.489 token_accuracy 73.3% data_wait_ms 0.03581 grad_norm 0.4855 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.39 step_time_ms 242.2 samples_per_sec 66.06 tokens 16,384 tokens_per_s 67,643 mfu 51.2% hlo_flops_util_lower_bound 51.2% hfu_lower_bound 51.2% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 0:00:48 left step 2000/2000 loss 1.122 ce 1.122 perplexity 3.070 token_accuracy 75.2% data_wait_ms 0.03559 grad_norm 0.4882 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 23.39 step_time_ms 242.3 samples_per_sec 66.04 tokens 16,384 tokens_per_s 67,627 mfu 51.1% hlo_flops_util_lower_bound 51.1% hfu_lower_bound 51.1% unsized_kernels 194.0 accepted 100.0% learning_rate 3.000e-05 accepted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 165,581 attempted/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 61,373 chunks_per_record/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 truncated/hf:codeparrot/codeparrot-clean-valid@4db92d2ec0c1b4c41eeb439cfae16854511d9dcd:None:train 0 eval val at step 2000: perplexity 3.340 ↓ 0.8% loss 1.206 ↓ 0.6% (1672 records in 9.21 s) Trained 2000 steps in 0:09:43: first step after 31.48 s, then 4.0 step/s 85.4% of the wall time in steps, final loss 1.122
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]train = [(row["step"], row["scalars"]["train/loss"]) for row in rows if "train/loss" in row["scalars"]]val = [(row["step"], row["scalars"]["val/perplexity"]) for row in rows if "val/perplexity" in row["scalars"]]print("validation perplexity:", ", ".join(f"{ppl:.2f} at {step}" for step, ppl in val))figures.curve({"training": [loss for _, loss in train]}, x=[step for step, _ in train], x_label="step", y_label="loss (nats per token)")validation perplexity: 3.46 at 500, 3.40 at 1000, 3.37 at 1500, 3.34 at 2000
after = generate(model, state.variables, prompt, max_new_tokens=MAX_NEW_TOKENS, key=jax.random.key(0), sampling=Sampling(temperature=0.0))print(tokenizer.decode(after.tokens[0]))def is_palindrome(text):
"""
Check if a string is a palindrome.
:param text: The string to check.
:type text: str
:return: True if the string is a palindrome, False otherwise.
:rtype: bool
"""
if not text:
return False
if len(text) <= 1:
return True
if text[0]Validation perplexity falls from 3.46 at the first check, step 500, to 3.34 at the end. The model was already good at this kind of code: a perplexity near 3.4 is roughly the uncertainty of choosing among three or four equally likely tokens on average, and the training sharpens it a little. The generation changes more. Instead of looping, the model now writes one function with a docstring in the style of the files it trained on, a description, :param and :rtype lines, and then the checks themselves.
Where did the model change its mind? Here is a held-out file’s first lines, each token shaded by the probability the model gave it, before training and after: green means more than 0.5, blue between 0.1 and 0.5, and orange 0.1 or less.
from dew.data.dataset import DataPartition
held_out = next(iter(data.val(DataPartition())))["text"][0][:33]
def shaded(variables): logits = model.apply(variables, jnp.asarray(held_out[None, :-1]))[0] probs = np.asarray(jax.nn.softmax(logits.astype(jnp.float32))) probs = probs[np.arange(len(held_out) - 1), held_out[1:]] return ["accent_fill" if p > 0.5 else "tint1" if p > 0.1 else "warm_fill" for p in probs]
pieces = [repr(tokenizer.decode([token]))[1:-1] or "·" for token in held_out[1:]]rows = [pieces[i:i + 8] for i in range(0, len(pieces), 8)]fills = [shaded(variables), shaded(state.variables)]for label, shades in zip(("before", "after"), fills): print(label, {shade: shades.count(shade) for shade in ("accent_fill", "tint1", "warm_fill")})figures.tokens(rows * 2, fills=[f[i:i + 8] for f in fills for i in range(0, len(pieces), 8)], notes=["before"] + [""] * (len(rows) - 1) + ["after"] + [""] * (len(rows) - 1))before {'accent_fill': 19, 'tint1': 8, 'warm_fill': 5}
after {'accent_fill': 23, 'tint1': 5, 'warm_fill': 4}Before training, 19 of these 32 tokens get more than half the model’s probability; after, 23 do, and the tokens it gave less than 0.1 drop from 5 to 4. These lines are Django imports, exactly the kind of boilerplate the training data is full of.
Exporting back to Hugging Face
Section titled “Exporting back to Hugging Face”PretrainedDecoder.from_model wraps the trained weights for export, and its save writes them in the Hugging Face layout: config.json, model.safetensors, and the tokenizer files. It runs the same name and shape translation as PretrainedDecoder.load, just backwards. Loading the export again gives back exactly the weights we trained.
PretrainedDecoder.from_model(model, state.variables, tokenizer=CHECKPOINT).save(EXPORT_DIR)print(sorted(os.listdir(EXPORT_DIR)))
reloaded = PretrainedDecoder.load(EXPORT_DIR, dtype=jnp.bfloat16, attention_impl="auto", max_seq_len=512)same = all(np.array_equal(np.asarray(a), np.asarray(b)) for a, b in zip(jax.tree_util.tree_leaves(reloaded.variables["params"]), jax.tree_util.tree_leaves(state.variables["params"])))print("reloaded weights equal the trained ones:", same)['config.json', 'generation_config.json', 'model.safetensors', 'tokenizer.json', 'tokenizer_config.json'] reloaded weights equal the trained ones: True
Reading the export with transformers
Section titled “Reading the export with transformers”An export is only useful if another library can actually read it. So we load the folder with transformers, run the prompt through it in float32, and compare its most likely next token with Dew’s.
import torchfrom transformers import AutoModelForCausalLM
hf_model = AutoModelForCausalLM.from_pretrained(EXPORT_DIR, dtype=torch.float32)with torch.no_grad(): hf_next = int(hf_model(input_ids=torch.tensor(np.asarray(prompt))).logits[0, -1].argmax())
dew_next = generate(model, state.variables, prompt, max_new_tokens=1, key=jax.random.key(0), sampling=Sampling(temperature=0.0))dew_next = int(dew_next.tokens[0, -1])print("transformers:", hf_next, repr(tokenizer.decode([hf_next])))print("dew: ", dew_next, repr(tokenizer.decode([dew_next])))transformers: 472 '\n ' dew: 472 '\n '
Where to go next
Section titled “Where to go next”The README model list names the decoder families Dew loads, trains, generates with and exports. The recipe flag --pretrained runs this flow at scale, for example python recipes/lm/train.py --set data.source.url=<dir> --pretrained HuggingFaceTB/SmolLM2-360M --tokenizer HuggingFaceTB/SmolLM2-360M.