SFT for LLMs

Open in Colab View on GitHub DownloadGPU

Supervised fine-tuning (SFT) trains a language model on examples of the replies we want. We start from Qwen2.5-0.5B, a base model that has read a great deal of text but was never taught to answer, inspect its chat tokens, and teach it to reply on real conversations. The questions stay in the context. Only assistant completions contribute to the loss.

This is the supervised step described by Ouyang et al. (2022). The model comes from Qwen Team, Qwen2.5 Technical Report (2024), and the conversations from smol-smoltalk, a set Hugging Face made for training small models (Allal et al., 2025). You need Python and some familiarity with next-token prediction; the JAX gradient check below works directly on the logits.

Install dependencies

Install Dew from GitHub with the CUDA and streaming extras. The checkpoint loader reads Hugging Face’s safetensors files.

%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@3a6f823b6c3a0b5dda920030d67ed4424bfc0542"

install log

  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheel for dewml (pyproject.toml) ... done

Settings

CHECKPOINT = "Qwen/Qwen2.5-0.5B"
DATASET = "HuggingFaceTB/smol-smoltalk"
SEQUENCE_LENGTH = 1024
BATCH_SIZE = 32
STEPS = 1500
LEARNING_RATE = 1e-5
MAX_NEW_TOKENS = 160
HELD_OUT = 3
SEED = 0

import jax
import jax.numpy as jnp
import numpy as np

from dew import figures
from dew.data import HFTokenizer
from dew.interop import PretrainedDecoder
from dew.training import prepare_process

prepare_process()  # Dew's setup for a training process, before the first JAX call
print(jax.devices())

[CudaDevice(id=0)]

Chat templates

A decoder sees token IDs, so the roles and turn boundaries need a text representation. Each instruct model has its own chat template. Use the template and vocabulary that came with the checkpoint.

PretrainedDecoder.load returns the model, its weights and its text processor together. The model computes in bfloat16 and keeps its parameters in float32, as in notebook 08.

model = PretrainedDecoder.load(CHECKPOINT, dtype=jnp.bfloat16,
                               max_seq_len=SEQUENCE_LENGTH)
tokenizer = HFTokenizer(CHECKPOINT).tokenizer
print(f"{sum(x.size for x in jax.tree.leaves(model.variables)) / 1e6:.0f}M parameters")

stderr

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

494M parameters

Here is one real conversation from the held-out part of the data, the shortest of its first thousand, as a list of messages. Render it first as text, then as IDs. The template supplies markers such as <|im_start|> and <|im_end|>, the role names and the newlines. They are part of the input the model learned to read.

from datasets import load_dataset

held_out_split = load_dataset(DATASET, split="test")
lengths = [sum(len(message["content"]) for message in held_out_split[i]["messages"])
           for i in range(1000)]
shortest = int(np.argmin(lengths))
messages = held_out_split[shortest]["messages"]
rendered_text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=False)
template_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, return_dict=False, add_generation_prompt=False)
print(rendered_text)
print("token IDs:", template_ids)

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Your response should contain less than 30 words. Explain the importance of recycling.
<|im_end|>
<|im_start|>assistant
Recycling reduces waste, conserves resources, saves energy, and decreases pollution, contributing to a more sustainable environment.<|im_end|>

token IDs: [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 7771, 2033, 1265, 6644, 2686, 1091, 220, 18, 15, 4244, 13, 81917, 279, 12650, 315, 33878, 624, 151645, 198, 151644, 77091, 198, 3820, 16021, 25271, 12291, 11, 11110, 288, 4963, 11, 25935, 4802, 11, 323, 42054, 24761, 11, 28720, 311, 264, 803, 21803, 4573, 13, 151645, 198]

How SFT rows are built

ChatData accepts a source with a messages column, including a tuple of records already in memory. It parses each row as a Conversation and uses render_conversation to produce text (IDs) and text_roles (one Role per token).

long="truncate" cuts an overlong conversation on the right. "drop" removes it, and "chunk" splits it into pieces that overlap by one token. A cut can remove the assistant reply; a chunk loses the context in earlier pieces.

Eight copies of this short conversation give us a window to inspect packing. read="memory" prepares them in host memory.

from dew.config.data import ChatData
from dew.data import Loading
from dew.data.chat import Conversation, Role, render_conversation
from dew.data.dataset import DataPartition

demo_source = ({"messages": messages},) * 8
demo_data = ChatData(
    demo_source, tokenizer=CHECKPOINT, seq_len=SEQUENCE_LENGTH, long="truncate",
).load(batch=1, read="memory", loading=Loading(workers=0), seed=SEED)
batch = next(iter(demo_data.train(DataPartition())))

ids, row_roles = render_conversation(
    tokenizer, Conversation.parse(messages), CHECKPOINT)
np.testing.assert_array_equal(ids, template_ids)
shades = {Role.SYSTEM: "tint2", Role.USER: "tint1", Role.ASSISTANT: "accent_fill"}
pieces = [repr(tokenizer.decode([int(token)]))[1:-1] for token in ids]
fills = [shades.get(Role(int(role)), "none") for role in row_roles]
figures.tokens([pieces[i:i + 10] for i in range(0, len(pieces), 10)],
               fills=[fills[i:i + 10] for i in range(0, len(fills), 10)])

Each token is shaded by its role: blue for the user’s part, which here includes the default system message the template inserts, green for the assistant’s reply with its closing marker, and blank for the assistant’s opening header.

Loss masks on assistant turns

LMObjective(..., loss_role=Role.ASSISTANT) counts only targets labelled ASSISTANT. The role belongs to the target, one position after the input token that predicts it.

The renderer leaves an assistant’s opening header at Role.PAD, so we learn its completion and closing turn marker. Real headers still have a nonzero segment ID. Role.PAD by itself does not mean that a token is absent.

The mask below also excludes padding and transitions between conversations. We will look at those segment boundaries next.

from dew.objectives.lm import LMObjective

objective = LMObjective(model, SEQUENCE_LENGTH, loss_role=Role.ASSISTANT)
tokens = np.asarray(batch["text"])
roles = np.asarray(batch["text_roles"])
segments = np.asarray(batch["text_coordinates"].segments)
mask = ((roles[:, 1:] == Role.ASSISTANT)
        & (segments[:, 1:] == segments[:, :-1])
        & (segments[:, 1:] != 0))
real_tokens = int((segments[0] != 0).sum())
print("m_t for the first conversation:", mask[0, :len(ids) - 1].astype(int).tolist())
print(f"scored targets: {int(mask[0].sum())} of {real_tokens - 1} real next-token slots")
print("scored padding targets:", int(mask[0, real_tokens - 1:].sum()))

m_t for the first conversation: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
scored targets: 200 of 487 real next-token slots
scored padding targets: 0

Packing SFT rows

ChatData places rendered conversations first-fit into windows of seq_len + 1 tokens. Each batch carries text_coordinates, the window’s layout: its segments identify each conversation inside a window, with 0 marking padding, and its positions restart at 0 for each segment.

Causal attention can read only earlier tokens in the same segment. A target counts only when its input and target have the same nonzero segment ID. The last token of one conversation therefore never trains the first token of the next one.

shown = 2 * len(ids) + 8  # two whole conversations and the start of the third
positions = np.asarray(batch["text_coordinates"].positions)[0]
print("segment IDs:", segments[0, :shown].tolist(), "...")
print("positions:  ", positions[:shown].tolist(), "...")
shade = {0: "none", 1: "tint1", 2: "accent_fill", 3: "tint2"}
row = [repr(tokenizer.decode([int(token)]))[1:-1] for token in tokens[0, :shown]]
fill = [shade[int(segment) % 4 if segment else 0] for segment in segments[0, :shown]]
figures.tokens([row[i:i + 16] for i in range(0, len(row), 16)],
               fills=[fill[i:i + 16] for i in range(0, len(fill), 16)])

segment IDs: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3] ...
positions:   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 0, 1, 2, 3, 4, 5, 6, 7] ...

Each colour is a separate conversation.

Repad those same conversations one per row to compare the two layouts. Both batches contain the same real tokens. Packing uses fewer padding slots, which means more of the model’s work goes to conversation tokens.

columns_by_segment = [
    np.flatnonzero(segments[0] == segment)
    for segment in np.unique(segments[0]) if segment != 0
]
shape = (len(columns_by_segment), SEQUENCE_LENGTH + 1)
single_segments = np.zeros(shape, np.int32)
for row, columns in enumerate(columns_by_segment):
    single_segments[row, :len(columns)] = 1

for label, shown in (("packed", segments), ("one conversation a row", single_segments)):
    real = int((shown != 0).sum())
    total = shown.size
    print(f"{label:24} {real}/{total} tokens, fill {real / total:.1%}")

packed                   488/1025 tokens, fill 47.6%
one conversation a row   488/8200 tokens, fill 6.0%

The loss and its gradient per token

Write the tokens as $x_1,\ldots,x_T$ , their message roles as $r_t \in \{\mathrm{system}, \mathrm{user}, \mathrm{assistant}, \mathrm{tool}\}$ , and their segment IDs as $s_t$ . The model gives $\pi_\theta(x_t \mid x_{<t})$ , with the context restricted to the same segment. Set $m_t = 1$ on scored assistant tokens and 0 elsewhere, including unused slots and assistant headers.

The loss is the mean over scored tokens in the batch. The sums include all rows:

L(\theta) = -\frac{1}{\sum_t m_t}\sum_t m_t \log \pi_\theta(x_t \mid x_{<t}).

If $z_t$ is the vector of logits predicting $x_t$ , its gradient is

\frac{\partial L}{\partial z_t} = \bigl(\operatorname{softmax}(z_t) - \operatorname{onehot}(x_t)\bigr) \cdot \frac{m_t}{\sum_s m_s}.

A masked logit has zero direct loss gradient. User tokens still affect the assistant predictions through attention, so they remain in the inputs.

objective.predict returns the objective’s loss and the model’s logits for this batch. We compute cross entropy from those logits in JAX, then compare the formula’s gradient with jax.grad. No optimizer update happens here.

from dew.objectives.base import Step

step = Step(jnp.asarray(0), jax.random.key(SEED), None, training=False)
statistics, _, prediction = objective.predict(model.variables, batch, step, train=False)
logits = prediction.logits
targets, weights = jnp.asarray(tokens[:, 1:]), jnp.asarray(mask, jnp.float32)

def token_loss(z):
    return -jnp.take_along_axis(
        jax.nn.log_softmax(z), targets[..., None], axis=-1)[..., 0]

def mean_loss(z):
    return (token_loss(z) * weights).sum() / weights.sum()

hand_loss, dew_loss = mean_loss(logits), statistics.mean()[0]
hand_gradient = (jax.nn.softmax(logits) - jax.nn.one_hot(targets, logits.shape[-1]))
hand_gradient *= weights[..., None] / weights.sum()
autodiff_gradient = jax.grad(mean_loss)(logits)
np.testing.assert_array_equal(prediction.weights, mask)
np.testing.assert_allclose(hand_loss, dew_loss, rtol=1e-5, atol=1e-6)
np.testing.assert_allclose(hand_gradient, autodiff_gradient, rtol=1e-5, atol=1e-7)
print("scored token losses:", np.round(np.asarray(token_loss(logits))[mask], 3))
print(f"hand loss {float(hand_loss):.6f}, objective loss {float(dew_loss):.6f}")
print("maximum logit-gradient error:",
      float(jnp.max(jnp.abs(hand_gradient - autodiff_gradient))))

scored token losses: [4.170e-01 1.000e-03 5.286e+00 2.573e+00 1.339e+00 4.730e-01 3.400e-02
 8.800e-01 2.890e-01 3.559e+00 2.270e-01 2.340e-01 1.760e-01 3.344e+00
 1.001e+00 3.273e+00 4.086e+00 7.700e-02 2.930e-01 1.762e+00 3.800e-02
 2.352e+00 3.010e-01 2.560e+01 2.094e+00 4.170e-01 1.000e-03 5.286e+00
 2.573e+00 1.339e+00 4.730e-01 3.400e-02 8.800e-01 2.890e-01 3.559e+00
 2.270e-01 2.340e-01 1.760e-01 3.344e+00 1.001e+00 3.273e+00 4.086e+00
 7.700e-02 2.930e-01 1.762e+00 3.800e-02 2.352e+00 3.010e-01 2.560e+01
 2.094e+00 4.170e-01 1.000e-03 5.286e+00 2.573e+00 1.339e+00 4.730e-01
 3.400e-02 8.800e-01 2.890e-01 3.559e+00 2.270e-01 2.340e-01 1.760e-01
 3.344e+00 1.001e+00 3.273e+00 4.086e+00 7.700e-02 2.930e-01 1.762e+00
 3.800e-02 2.352e+00 3.010e-01 2.560e+01 2.094e+00 4.170e-01 1.000e-03
 5.286e+00 2.573e+00 1.339e+00 4.730e-01 3.400e-02 8.800e-01 2.890e-01
 3.559e+00 2.270e-01 2.340e-01 1.760e-01 3.344e+00 1.001e+00 3.273e+00
 4.086e+00 7.700e-02 2.930e-01 1.762e+00 3.800e-02 2.352e+00 3.010e-01
 2.560e+01 2.094e+00 4.170e-01 1.000e-03 5.286e+00 2.573e+00 1.339e+00
 4.730e-01 3.400e-02 8.800e-01 2.890e-01 3.559e+00 2.270e-01 2.340e-01
 1.760e-01 3.344e+00 1.001e+00 3.273e+00 4.086e+00 7.700e-02 2.930e-01
 1.762e+00 3.800e-02 2.352e+00 3.010e-01 2.560e+01 2.094e+00 4.170e-01
 1.000e-03 5.286e+00 2.573e+00 1.339e+00 4.730e-01 3.400e-02 8.800e-01
 2.890e-01 3.559e+00 2.270e-01 2.340e-01 1.760e-01 3.344e+00 1.001e+00
 3.273e+00 4.086e+00 7.700e-02 2.930e-01 1.762e+00 3.800e-02 2.352e+00
 3.010e-01 2.560e+01 2.094e+00 4.170e-01 1.000e-03 5.286e+00 2.573e+00
 1.339e+00 4.730e-01 3.400e-02 8.800e-01 2.890e-01 3.559e+00 2.270e-01
 2.340e-01 1.760e-01 3.344e+00 1.001e+00 3.273e+00 4.086e+00 7.700e-02
 2.930e-01 1.762e+00 3.800e-02 2.352e+00 3.010e-01 2.560e+01 2.094e+00
 4.170e-01 1.000e-03 5.286e+00 2.573e+00 1.339e+00 4.730e-01 3.400e-02
 8.800e-01 2.890e-01 3.559e+00 2.270e-01 2.340e-01 1.760e-01 3.344e+00
 1.001e+00 3.273e+00 4.086e+00 7.700e-02 2.930e-01 1.762e+00 3.800e-02
 2.352e+00 3.010e-01 2.560e+01 2.094e+00]
hand loss 2.388251, objective loss 2.388251
maximum logit-gradient error: 4.656612873077393e-10

Fine-tuning on real conversations

The training conversations are smol-smoltalk’s training split: everyday chat, rewriting, summaries, maths and code, each with one or more assistant turns, and every assistant turn is scored. read="streamed" reads them straight from the Hub’s Parquet shards and tokenizes them as the batches are built.

Three conversations from the held-out split stay out of training. We ask them before and after with greedy decoding, so sampling noise cannot explain a change.

from dew.data import Hub

def opening(messages):
    # A conversation up to its first assistant turn: what the model is asked to answer.
    first = next(i for i, message in enumerate(messages) if message["role"] == "assistant")
    return messages[:first]


def fits(turns):
    # The opening and its reply have to fit in one window.
    ids = tokenizer.apply_chat_template(turns, add_generation_prompt=True, return_dict=False)
    return len(ids) + MAX_NEW_TOKENS <= SEQUENCE_LENGTH


openings = [turns for turns in (opening(row["messages"]) for row in held_out_split)
            if fits(turns)][:HELD_OUT]
questions = [turns[-1]["content"] for turns in openings]
data = ChatData(
    Hub(DATASET), tokenizer=CHECKPOINT, seq_len=SEQUENCE_LENGTH, long="truncate",
).load(batch=BATCH_SIZE, read="streamed", loading=Loading(workers=0), seed=SEED)
for question in questions:
    print("-", question[:200])

- Uruguay hero Fernando Muslera was delighted after his team stunned hosts Argentina in a penalty shootout to reach the semifinals of the Copa America tournament, despite having a player sent off. The g
- Find all complex solutions to the equation z^n = 1, for any positive integer n.
- Dear Daniel,

I hope this email finds you well. My name is Emily Johnson, and I am a researcher at XYZ University currently studying school governance and leadership. I recently came across your work

For generation, add_generation_prompt=True appends the next assistant header. There is no answer in the prompt. Generation stops at the chat template’s end-of-turn token, <|im_end|>, or at the model’s own end-of-text token, and returns just the new text.

from dew.sampling import Sampling

prompts = [tokenizer.apply_chat_template(turns, tokenize=False, add_generation_prompt=True)
           for turns in openings]
# A reply ends at the chat template's end-of-turn token; the base model's own end-of-text
# token ends it too.
greedy = Sampling(temperature=0, pad_token_id=tokenizer.pad_token_id,
                  eos_token_ids=(tokenizer.convert_tokens_to_ids("<|im_end|>"),
                                 tokenizer.eos_token_id))
before_task = model.text_generation(sampling=greedy)
before = list(before_task(prompts, MAX_NEW_TOKENS, key=SEED).text)
for question, answer in zip(questions, before, strict=True):
    print(question[:120], "\n->", repr(answer[:400]), "\n")

Uruguay hero Fernando Muslera was delighted after his team stunned hosts Argentina in a penalty shootout to reach the se 
-> 'Uruguay hero Fernando Muslera was delighted after his team stunned hosts Argentina in a penalty shootout to reach the semifinals of the Copa America tournament, despite having a player sent off. The goalkeeper saved a spot-kick by striker Carlos Tevez to earn "La Celeste" a clash with Peru, who beat Colombia 2-0 after extra-time in Cordoba earlier on Saturday. The Uruguayans, who reached the last ' 

Find all complex solutions to the equation z^n = 1, for any positive integer n. 
-> 'To find all complex solutions to the equation z^n = 1, we can use the properties of complex numbers and their polar form. The equation z^n = 1 can be rewritten as z^n = 1 = 1 * e^(2kπi), where k is an integer. This means that z^n = 1 is equivalent to z = e^(2kπi/n) for any integer k. Therefore, the solutions to the equation z^n = 1 are given by z = e^(2kπi/n) for any integer k.戥user\n戥assistant\n戥us' 

Dear Daniel,

I hope this email finds you well. My name is Emily Johnson, and I am a researcher at XYZ University curren 
-> 'Dear Emily,\n\nThank you for reaching out to me. I am happy to discuss your work with you further. I believe that your approach to integrating critical thinking and creativity into elementary school curricula could be particularly valuable for school governance and leadership. By focusing on student-centered learning and cognitive development, you may be able to create a more engaging and effective '

The base model has read plenty of chat-shaped text, but it was never taught this format, and it shows. Asked for a three-sentence summary of a news story, it copies the story out word for word. Its answer about $z^n = 1$ is right, but where the turn should end it emits strange tokens instead. And asked for one sentence about an email, it writes a whole reply as if it were Daniel, then drifts into unrelated Chinese text. It doesn’t know how a reply ends.

The objective already holds the pretrained weights. Trainer.fit updates them with the assistant-only loss we checked above. It reads packed training windows until STEPS updates have run. No moving average is configured, so the returned state.variables are the trained weights.

import json

from dew import LocalTracker, Trainer
from dew.config import OptimConfig

state = Trainer(
    objective, OptimConfig(learning_rate=LEARNING_RATE), key=SEED,
    tracker=LocalTracker("runs/09-sft"),
).fit(data, steps=STEPS, log_every=max(1, STEPS // 20))
rows = [json.loads(line) for line in open("runs/09-sft/scalars.jsonl")]
rows = [row for row in rows if "train/loss" in row["scalars"]]
print("loss:", ", ".join(f"{row['scalars']['train/loss']:.3f} at {row['step']}"
                         for row in rows))
figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]},
              x=[row["step"] for row in rows], x_label="step",
              y_label="loss on assistant tokens")

Training CausalTransformer from step 0 to 1500: 494.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 32, float32 parameters, bfloat16 compute
22:57:29 WARNING dew.nn.kernels.tuning: 6 kernel launches have no config measured on NVIDIA A100-SXM4-40GB, so they run the nearest measured shape's or their heuristic: measure them on a GPU to yourself with Trainer(tune_kernels=True) or `python tools/kernel_picks.py`
22:59:58 WARNING dew.training.planning: timing the plan against its batch whole at remat 'minimal', the head tiled
23:01:28 WARNING dew.training.trainer: the step fits the devices in the plan {'split': 8, 'rows': 4, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'probes': [{'split': 1, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': False, 'headroom': -97981468806, 'temp': 129742790992, 'argument': 6644528464}, {'split': 32, 'rows': 1, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 25048866842, 'temp': 4736321928, 'argument': 8620659544}, {'split': 4, 'rows': 8, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': False, 'headroom': -2850321382, 'temp': 32635510152, 'argument': 8620659544}, {'split': 8, 'rows': 4, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 12911713306, 'temp': 16873475464, 'argument': 8620659544}, {'split': 1, 'rows': 32, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'fits': True, 'headroom': 6150316074, 'temp': 25611006112, 'argument': 6644528464}], 'timed': [{'split': 8, 'rows': 4, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'seconds': 1.1853500165000241}, {'split': 1, 'rows': 32, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'seconds': 1.3106739714000013}]}
step 75/1500  loss 1.113  ce 0.6577  perplexity 1.930  token_accuracy 79.7%  data_wait_ms 0.03634  grad_norm 2.690  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,204  samples_per_sec 26.58  tokens 29,949  tokens_per_s 24,875  mfu 23.0%  hlo_flops_util_lower_bound 28.2%  hfu_lower_bound 28.2%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 2,496  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 2,496  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:28:05 left
step 150/1500  loss 1.068  ce 1.254  perplexity 3.505  token_accuracy 68.5%  data_wait_ms 0.03707  grad_norm 2.409  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,193  samples_per_sec 26.83  tokens 29,868  tokens_per_s 25,041  mfu 23.1%  hlo_flops_util_lower_bound 28.4%  hfu_lower_bound 28.4%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 4,896  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 4,896  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:26:38 left
step 225/1500  loss 1.069  ce 1.462  perplexity 4.316  token_accuracy 61.3%  data_wait_ms 0.03718  grad_norm 2.411  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,886  tokens_per_s 25,224  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 7,296  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 7,296  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:25:09 left
step 300/1500  loss 1.098  ce 0.9822  perplexity 2.670  token_accuracy 72.7%  data_wait_ms 0.03733  grad_norm 2.360  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,186  samples_per_sec 26.98  tokens 30,064  tokens_per_s 25,344  mfu 23.4%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 9,696  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 9,696  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:23:45 left
step 375/1500  loss 1.094  ce 0.9594  perplexity 2.610  token_accuracy 73.4%  data_wait_ms 0.03807  grad_norm 2.361  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,838  tokens_per_s 25,185  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 12,096  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 12,096  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:22:16 left
step 450/1500  loss 1.080  ce 1.144  perplexity 3.140  token_accuracy 67.3%  data_wait_ms 0.03714  grad_norm 2.350  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,963  tokens_per_s 25,289  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 14,496  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 14,496  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:20:43 left
step 525/1500  loss 0.9654  ce 0.6309  perplexity 1.879  token_accuracy 79.7%  data_wait_ms 0.03664  grad_norm 2.329  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,947  tokens_per_s 25,275  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 16,896  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 16,896  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:19:14 left
step 600/1500  loss 0.9746  ce 0.7682  perplexity 2.156  token_accuracy 77.1%  data_wait_ms 0.03698  grad_norm 2.328  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,186  samples_per_sec 26.98  tokens 29,871  tokens_per_s 25,183  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 19,296  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 19,296  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:17:49 left
step 675/1500  loss 1.068  ce 1.134  perplexity 3.108  token_accuracy 69.3%  data_wait_ms 0.03676  grad_norm 2.346  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,952  tokens_per_s 25,282  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 21,696  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 21,696  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:16:20 left
step 750/1500  loss 1.069  ce 1.097  perplexity 2.995  token_accuracy 70.2%  data_wait_ms 0.03707  grad_norm 2.315  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,919  tokens_per_s 25,255  mfu 23.4%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 24,096  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 24,096  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:14:48 left
step 825/1500  loss 1.037  ce 1.146  perplexity 3.145  token_accuracy 67.5%  data_wait_ms 0.03695  grad_norm 2.304  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 30,018  tokens_per_s 25,336  mfu 23.4%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 26,496  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 26,496  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:13:19 left
step 900/1500  loss 0.9656  ce 1.117  perplexity 3.056  token_accuracy 68.8%  data_wait_ms 0.03730  grad_norm 2.286  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,189  samples_per_sec 26.92  tokens 30,014  tokens_per_s 25,247  mfu 23.3%  hlo_flops_util_lower_bound 28.5%  hfu_lower_bound 28.5%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 28,896  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 28,896  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:11:50 left
step 975/1500  loss 0.9775  ce 1.020  perplexity 2.773  token_accuracy 70.8%  data_wait_ms 0.03708  grad_norm 2.297  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,182  samples_per_sec 27.07  tokens 29,976  tokens_per_s 25,357  mfu 23.4%  hlo_flops_util_lower_bound 28.7%  hfu_lower_bound 28.7%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 31,296  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 31,296  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:10:23 left
step 1050/1500  loss 1.093  ce 1.244  perplexity 3.470  token_accuracy 68.7%  data_wait_ms 0.03739  grad_norm 2.290  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 30,032  tokens_per_s 25,346  mfu 23.4%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 33,696  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 33,696  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:08:55 left
step 1125/1500  loss 1.031  ce 1.281  perplexity 3.601  token_accuracy 67.7%  data_wait_ms 0.03702  grad_norm 2.286  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.00  tokens 30,047  tokens_per_s 25,353  mfu 23.4%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 36,096  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 36,096  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:07:19 left
step 1200/1500  loss 1.031  ce 0.8145  perplexity 2.258  token_accuracy 75.7%  data_wait_ms 0.03713  grad_norm 2.282  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,909  tokens_per_s 25,242  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 38,496  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 38,496  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:05:55 left
step 1275/1500  loss 1.120  ce 1.077  perplexity 2.936  token_accuracy 71.0%  data_wait_ms 0.03752  grad_norm 2.268  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.01  tokens 29,860  tokens_per_s 25,201  mfu 23.3%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 40,896  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 40,896  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:04:26 left
step 1350/1500  loss 1.018  ce 0.7653  perplexity 2.150  token_accuracy 77.7%  data_wait_ms 0.03690  grad_norm 2.241  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,185  samples_per_sec 27.00  tokens 30,128  tokens_per_s 25,424  mfu 23.5%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 43,296  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 43,296  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:02:58 left
step 1425/1500  loss 1.141  ce 1.157  perplexity 3.180  token_accuracy 67.4%  data_wait_ms 0.03688  grad_norm 2.254  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,186  samples_per_sec 26.99  tokens 30,162  tokens_per_s 25,441  mfu 23.5%  hlo_flops_util_lower_bound 28.6%  hfu_lower_bound 28.6%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 45,696  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 45,696  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  0:01:29 left
step 1500/1500  loss 1.206  ce 1.340  perplexity 3.819  token_accuracy 65.8%  data_wait_ms 0.03638  grad_norm 2.252  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 31.80  step_time_ms 1,178  samples_per_sec 27.16  tokens 29,947  tokens_per_s 25,416  mfu 23.5%  hlo_flops_util_lower_bound 28.8%  hfu_lower_bound 28.8%  unsized_kernels 1,168  accepted 100.0%  learning_rate 1.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 48,096  attempted/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 48,096  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00000-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00001-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00002-of-00004.parquet', 'hf://datasets/HuggingFaceTB/smol-smoltalk@f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc/data/train-00003-of-00004.parquet') 0
Trained 1500 steps in 0:33:54: first step after 256.86 s, then 0.8 step/s
87.4% of the wall time in steps, final loss 1.206
loss: 1.113 at 75, 1.068 at 150, 1.069 at 225, 1.098 at 300, 1.094 at 375, 1.080 at 450, 0.965 at 525, 0.975 at 600, 1.068 at 675, 1.069 at 750, 1.037 at 825, 0.966 at 900, 0.978 at 975, 1.093 at 1050, 1.031 at 1125, 1.031 at 1200, 1.120 at 1275, 1.018 at 1350, 1.141 at 1425, 1.206 at 1500

From the first logged step, 75, the loss wanders between about 0.97 and 1.21 for the whole run, without a clear trend. Every point scores a different batch of conversations, so the curve can’t show what changed; the replies can.

Replace the bundle’s weights with the trained ones and ask exactly the same questions. The table shows both replies.

from dataclasses import replace
from itertools import zip_longest
from textwrap import wrap

after_task = replace(model, variables=state.variables).text_generation(sampling=greedy)
after = list(after_task(prompts, MAX_NEW_TOKENS, key=SEED).text)
for question, old, new in zip(questions, before, after, strict=True):
    print("\n" + question[:120])
    print(f"{'before':40} | after")
    for left, right in zip_longest(wrap(old, 40), wrap(new, 40), fillvalue=""):
        print(f"{left:40} | {right}")

Uruguay hero Fernando Muslera was delighted after his team stunned hosts Argentina in a penalty shootout to reach the se
before                                   | after
Uruguay hero Fernando Muslera was        | Uruguay's Fernando Muslera, who reached
delighted after his team stunned hosts   | the semifinals of the Copa America
Argentina in a penalty shootout to reach | tournament, was delighted after his
the semifinals of the Copa America       | team, the Uruguayan national football
tournament, despite having a player sent | team, stunned hosts Argentina in a
off. The goalkeeper saved a spot-kick by | penalty shootout. The goalkeeper, Sergio
striker Carlos Tevez to earn "La         | Romero, saved a spot-kick by striker
Celeste" a clash with Peru, who beat     | Carlos Tevez, securing the match against
Colombia 2-0 after extra-time in Cordoba | Argentina, which had a player sent off.
earlier on Saturday. The Uruguayans, who | Uruguay, who reached the last four at
reached the last four at the 2010 World  | the 2010 World Cup, will seek to make
Cup, will be seeking to make the Copa    | the Copa final for the first time since
final for the first time since 1999 when | 1999, when they line up in La Plata on
they line up in La Plata on Tuesday. The | Tuesday. The upset victory against
upset victory against close neighbors    | Argentina came exactly 61 years after
Argentina came exactly 61 years after    | Uruguay won the World Cup by defeating
Uruguay won the World Cup by stunning    | Brazil in Rio de Janeiro's Maracana
Brazil in Rio de Janeiro's Maracana      | stadium. Peru defeated Colombia to reach
stadium. Peru shock                      | the semifinals, with a shootout in

Find all complex solutions to the equation z^n = 1, for any positive integer n.
before                                   | after
To find all complex solutions to the     | The equation z^n = 1 has a beautiful and
equation z^n = 1, we can use the         | elegant solution, which is rooted in the
properties of complex numbers and their  | properties of complex numbers and roots
polar form. The equation z^n = 1 can be  | of unity. To begin, recall that the
rewritten as z^n = 1 = 1 * e^(2kπi),     | complex numbers can be expressed in
where k is an integer. This means that   | polar form as z = re^(iθ), where r is
z^n = 1 is equivalent to z = e^(2kπi/n)  | the magnitude (or modulus) and θ is the
for any integer k. Therefore, the        | argument (or angle) of the complex
solutions to the equation z^n = 1 are    | number.  The equation z^n = 1 can be
given by z = e^(2kπi/n) for any integer  | rewritten as re^(iθ) = 1. Taking the
k.戥user 戥assistant 戥user 戥assistant      | logarithm of both sides, we obtain
戥user 戥assistant 戥user 戥assistant 戥user  | ln(re^(iθ)) = ln(1), which simplifies to
戥assistant 戥user 戥assistant 戥user        | ln(r) + iθ = 0. This equation is
戥assistant 戥user                         | equivalent to the equation for the
                                         | argument of a complex number, and it can
                                         | be solved for θ.  The argument θ is a
                                         | real

Dear Daniel,

I hope this email finds you well. My name is Emily Johnson, and I am a researcher at XYZ University curren
before                                   | after
Dear Emily,  Thank you for reaching out  | Emily Johnson is reaching out to discuss
to me. I am happy to discuss your work   | teaching methods and potential
with you further. I believe that your    | applications for school governance and
approach to integrating critical         | leadership.
thinking and creativity into elementary  | 
school curricula could be particularly   | 
valuable for school governance and       | 
leadership. By focusing on student-      | 
centered learning and cognitive          | 
development, you may be able to create a | 
more engaging and effective learning     | 
environment that encourages critical     | 
thinking and creativity among students.  | 
I would be happy to set up a time to     | 
chat further about your work and its     | 
potential implications for my research.  | 
If you have any questions or if there is | 
any additional information I can         | 
provide, please let me know.  Best       | 
regards, Daniel  批示系统 批示系统是一种用于处理和管理批示任务 | 
的系统，通常用于组织和协调多个部门或团队的工作。批示系统可以分为多个       |

The one-sentence request now gets exactly one sentence about Emily’s email, and then the turn is over. The other two replies use up all 160 tokens. The news story is reworded instead of copied, though it runs far past three sentences, and it gets a fact wrong: the goalkeeper who saved Tevez’s kick was Muslera, not “Sergio Romero”. The $z^n = 1$ answer drops the $n$ when it writes $z$ in polar form, and runs out of tokens before it gets anywhere. Fine-tuning changed how the model answers more than what it knows.

Where did the loss go down? Here is the demonstration conversation from the start again, with each assistant token shaded by how surprised the model is by it, its loss, before and after fine-tuning: green below 0.5 nats, blue from 0.5 to 2.5, and orange above 2.5.

def surprise(variables):
    logits = model.model.apply(variables, jnp.asarray([ids[:-1]]))[0]
    log_probs = np.asarray(jax.nn.log_softmax(logits.astype(jnp.float32)))
    loss = -log_probs[np.arange(len(ids) - 1), ids[1:]]
    return ["accent_fill" if value < 0.5 else "tint1" if value < 2.5 else "warm_fill"
            for value in loss]


reply = [k for k, role in enumerate(row_roles[1:]) if Role(int(role)) == Role.ASSISTANT]
shown = [pieces[k + 1] for k in reply]
rows = [shown[i:i + 10] for i in range(0, len(shown), 10)]
for label, variables in (("before", model.variables), ("after", state.variables)):
    fills = [surprise(variables)[k] for k in reply]
    print(label)
    display(figures.tokens(rows, fills=[fills[i:i + 10] for i in range(0, len(fills), 10)]))

before

after

Several words get easier, but the biggest change is at the very end: <|im_end|>, more than 2.5 nats of surprise before, is under 0.5 after, and so is the newline after it. That end-of-turn token is what let the email reply above stop.

Where to go next

SFT taught a base model to answer by training on assistant completions.

Notebook 10 moves to reinforcement learning with verifiable rewards (RLVR), where a check of the answer supplies the training signal. Notebook 11 uses a decoder in an agent that takes actions and reads tool results.