Skip to content

Text to image with classifier-free guidance

This is part 3 of the series, and it builds directly on notebook 02, so I would recommend going through that one first.

The model we trained in notebook 02 learned what flowers look like, but there was no way to tell it which flower to draw. You simply got whatever flower the starting noise happened to lead to. But what we usually want is to describe the image in words and get that image. So in this notebook, we give the model a caption with every image, so that at the end we can type “a red rose” and get something red and rose-like.

Classifier-free guidance. From the noisy image, the model's prediction without a caption and with the caption 'a red rose'; their difference points where the caption pulls, and the guided prediction follows it further, here at scale 2.2.xₜno caption“a red rose”the caption's pullguided, s = 2.2ε_guided = ε_uncond + s (ε_cond − ε_uncond)

Same as notebook 02: Dew from GitHub with its cuda13 and streaming extras.

%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simple
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
IMAGE_SIZE = 128
BATCH_SIZE = 64
STEPS = 30000
LEARNING_RATE = 2e-4
UNCONDITIONAL_PROB = 0.12
SAMPLE_STEPS = 40
PROMPTS = ["a red rose", "a yellow sunflower", "a white daisy", "a purple iris"]
GUIDANCE_SCALES = [1.0, 3.0, 6.0]
RUN_DIR = "runs/03-text-to-image"
SEED = 0
import jax
import numpy as np
from dew import figures
from dew.artifacts import uint8_pixels
from dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX call
print(jax.devices())
[CudaDevice(id=0)]
def show_images(images, titles, columns=4):
# Each row's captions, then the grid.
for row in range(0, len(titles), columns):
print(" | ".join(titles[row:row + columns]))
return figures.images([np.asarray(images)], columns=columns, size=440 / columns - 4)

A neural network can’t read words, so a text encoder first turns each caption into a sequence of vectors. We use the text half of CLIP (ViT-L/14), from the paper “Learning Transferable Visual Models From Natural Language Supervision” by Radford et al. (2021). CLIP was trained to match captions with images, so its vectors already say something about what a caption looks like, which is exactly what we want here. We keep CLIP frozen; only the diffusion model trains.

The InputSpec now has a condition as well as the image. Condition(encoder, field="text", unconditional="") basically says: read tokens from the batch field text, encode them with CLIP, and use the empty caption as “no caption”. The key textcontext is the keyword the model receives the encoded caption under.

from dew import Condition, Field, InputSpec
from dew.inputs import CLIPText
encoder = CLIPText.from_pretrained("openai/clip-vit-large-patch14")
inputs = InputSpec(
sample=Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)),
conditions={"textcontext": Condition(encoder, field="text", unconditional="")},
)
print("context length:", encoder.context)
print(inputs.tokenize(["a red rose"])["text"].keys())
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
context length: 77
dict_keys(['input_ids', 'attention_mask'])

The data is the same Hugging Face copy of Oxford Flowers as in notebook 02, and every photo carries a caption written by an image captioning model (BLIP). We build the same chain through the images preset, and passing host=(EncodeCaptions(inputs.tokenize),) to compile turns each caption into CLIP tokens while the batch is being built, so the batches carry token arrays under text instead of strings.

from dew.data import Hub, Loading, presets
from dew.data.compile import compile as compile_data
from dew.data.dataset import DataPartition
from dew.data.images import EncodeCaptions
flowers = Hub("pranked03/flowers-blip-captions").shuffle(seed=0).repeat()
data = compile_data(flowers.pipe(presets.images, IMAGE_SIZE, batch=BATCH_SIZE, augment=True, rejects=False),
read="memory", provision=True,
loading=Loading(workers=0, threads=16, read_buffer=64),
host=(EncodeCaptions(inputs.tokenize),))
stream = data.train(DataPartition())
try:
batch = next(stream)
finally:
stream.close()
captions = encoder.captions(batch["text"])
show_images(batch["image"][:8], captions[:8])
a pelargonium in the garden | pink azaleas in the garden | a red and yellow columbine in a garden | a wild pansy in the garden
a yellow sunflower in the sky | a passion flower in a pot on a planter | a white windflower in the grass | a red red ginger in the garden
8 images

The model is the same DiT from notebook 02, except that now it takes the caption vectors through the textcontext keyword and mixes them into every layer.

DiffusionObjective gets two new arguments. unconditional_prob makes it swap the caption for the empty one on 12% of training rows, and guidance=CFG(3.0) is how its previews sample. We keep the CLIP weights in the training state as frozen parameters, so they are saved in the checkpoint but never updated.

import jax.numpy as jnp
from dew import Checkpoints, LocalTracker, Trainer
from dew.config import OptimConfig
from dew.diffusion.presets import EDM
from dew.nn.backbones import SimpleDiT
from dew.objectives.diffusion import DiffusionObjective
from dew.sampling import CFG, EulerAncestral
preset = EDM(regime="pixel")
model = SimpleDiT(
patch_size=4,
emb_features=384,
num_layers=12,
num_heads=6,
output_channels=3,
dtype=jnp.bfloat16,
attention_impl="auto",
)
objective = DiffusionObjective(
model, preset, inputs,
unconditional_prob=UNCONDITIONAL_PROB,
ema_decay=0.999,
solver=EulerAncestral(), guidance=CFG(3.0), steps=SAMPLE_STEPS,
)
trainer = Trainer(
objective, OptimConfig(learning_rate=LEARNING_RATE),
key=jax.random.key(SEED),
checkpoints=Checkpoints(RUN_DIR),
tracker=LocalTracker(f"{RUN_DIR}/tracking"),
)

fit trains until STEPS, prints the loss every 2,000 steps and writes a checkpoint at the end, and then we plot the loss from the tracker.

state = trainer.fit(data, steps=STEPS, log_every=2000, checkpoint_every=STEPS)
Training SimpleDiT from step 0 to 30000: 35.8M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 2000/30000  loss 0.06584  data_wait_ms 0.03893  grad_norm 0.09647  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.49  step_time_ms 132.6  samples_per_sec 482.6  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  1:01:51 left
step 4000/30000  loss 0.06888  data_wait_ms 0.03927  grad_norm 0.06400  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.50  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:58:25 left
step 6000/30000  loss 0.06451  data_wait_ms 0.03874  grad_norm 0.05582  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.53  step_time_ms 132.5  samples_per_sec 483.0  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:53:00 left
step 8000/30000  loss 0.07280  data_wait_ms 0.03918  grad_norm 0.05082  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.53  step_time_ms 132.5  samples_per_sec 482.9  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:48:37 left
step 10000/30000  loss 0.07597  data_wait_ms 0.03939  grad_norm 0.04776  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.53  step_time_ms 132.5  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:44:10 left
step 12000/30000  loss 0.06271  data_wait_ms 0.03939  grad_norm 0.04445  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.53  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:38:55 left
step 14000/30000  loss 0.06348  data_wait_ms 0.03958  grad_norm 0.04270  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.53  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:35:21 left
step 16000/30000  loss 0.07443  data_wait_ms 0.03920  grad_norm 0.04067  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.5  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:31:33 left
step 18000/30000  loss 0.06137  data_wait_ms 0.03897  grad_norm 0.04020  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.5  samples_per_sec 482.9  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:26:30 left
step 20000/30000  loss 0.07190  data_wait_ms 0.03934  grad_norm 0.03898  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:22:32 left
step 22000/30000  loss 0.06853  data_wait_ms 0.03947  grad_norm 0.03862  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:17:40 left
step 24000/30000  loss 0.06056  data_wait_ms 0.03947  grad_norm 0.03657  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.5  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:13:16 left
step 26000/30000  loss 0.06012  data_wait_ms 0.03952  grad_norm 0.03679  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:08:50 left
step 28000/30000  loss 0.05736  data_wait_ms 0.03922  grad_norm 0.03549  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.6  samples_per_sec 482.8  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  0:04:25 left
step 30000/30000  loss 0.06149  data_wait_ms 0.03932  grad_norm 0.03530  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 16.54  step_time_ms 132.5  samples_per_sec 482.9  mfu 29.5%  hlo_flops_util 31.3%  hfu 31.3%  accepted 100.0%  learning_rate 2.000e-04  accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552  chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0  truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0
Trained 30000 steps in 1:07:30: first step after 62.35 s, then 7.5 step/s
98.2% of the wall time in steps, final loss 0.06149
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]
rows = [row for row in rows if "train/loss" in row["scalars"]]
print("loss:", ", ".join(f"{row['scalars']['train/loss']:.4f} at {row['step']}"
for row in rows))
figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]},
x=[row["step"] for row in rows], x_label="step", y_label="loss")
loss: 0.0658 at 2000, 0.0689 at 4000, 0.0645 at 6000, 0.0728 at 8000, 0.0760 at 10000, 0.0627 at 12000, 0.0635 at 14000, 0.0744 at 16000, 0.0614 at 18000, 0.0719 at 20000, 0.0685 at 22000, 0.0606 at 24000, 0.0601 at 26000, 0.0574 at 28000, 0.0615 at 30000
loss against step0.060.0650.070.075100002000030000steploss0.06

The loss looks almost exactly like notebook 02’s: about 0.066 at step 2,000, and then jumping between about 0.057 and 0.076 for the rest of the run. The loss can’t show what the caption adds; the samples below can.

Because some captions were blanked during training, one model has actually learned two things: how to denoise a flower given its caption, and how to denoise a flower with no caption at all. At sampling time, we ask it both questions at every step and push the answer away from the uncaptioned one:

ϵ^=ϵuncond+s (ϵcond−ϵuncond)\hat{\epsilon} = \epsilon_{\text{uncond}} + s \, (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})

where:

  • ϵcond\epsilon_{\text{cond}} is the model’s prediction given the caption,
  • ϵuncond\epsilon_{\text{uncond}} is its prediction given the empty caption,
  • ss is the guidance scale.

With s=1s = 1 this is just the plain captioned prediction. The difference ϵcond−ϵuncond\epsilon_{\text{cond}} - \epsilon_{\text{uncond}} is basically the direction the caption pulls the image in, and the scale says how far we follow it. Larger scales follow the caption harder and give up some variety in return.

The guided prediction as the guidance scale grows from 0 to 6: at 0 it is the prediction without a caption, at 1 the captioned one, and past 1 it reaches further along the caption's pull.s = 0s = 1s = 0s = 1s = 2s = 3s = 4s = 6guidance scale

This is classifier-free guidance, from the paper “Classifier-Free Diffusion Guidance” by Ho and Salimans (2022), and CFG(scale) in Dew computes it.

But why “classifier-free”? Well, the guidance that came before it, in the paper Diffusion Models beat GANs on image synthesis, needed a separate classifier trained on noisy images, whose gradient pushed the sample towards the class we wanted. Here the same model plays both roles, so there is no classifier at all.

objective.pipeline(state) packages the trained model, the process, the text encoder and the EMA weights into a TextToImage pipeline. Calling it with a list of prompts returns the images.

pipe = objective.pipeline(state)
rows = []
for scale in GUIDANCE_SCALES:
out = pipe(PROMPTS * 4, steps=SAMPLE_STEPS, guidance=CFG(scale), solver=EulerAncestral(), key=1)
rows.append(np.asarray(out.images))

Each row below is one prompt, drawn from four different starting noises, and the picture steps through the guidance scales, 1, then 3, then 6.

# The pipeline returns the prompts in turn; group each prompt's images into a row.
frames = [uint8_pixels(images.reshape(4, len(PROMPTS), *images.shape[1:]).swapaxes(0, 1)
.reshape(images.shape)) for images in rows]
figures.images(frames, columns=4, size=96, rows=PROMPTS,
captions=[f"guidance {scale:g}" for scale in GUIDANCE_SCALES])
16 images at each of 3 pointsa red rosea yellow sunflowera white daisya purple irisguidance 1guidance 3guidance 6

At scale 1 the samples only loosely follow the prompts: one “yellow sunflower” is purple and another orange, one “white daisy” is peach, and the first “purple iris” is a green blur. At scale 3 every image has the colour its prompt asks for, and the sunflowers and daisies have the right shape too, dark or yellow centres and all. The irises are purple but look more like petunias. Scale 6 pushes further: the colours get more saturated, most of the sunflowers’ centres darker, and the four images in each row look more and more alike. The second sunflower loses its centre altogether in flat yellow. That’s what guidance gives up in return: variety, and some of the detail.

recipes/diffusion/train.py runs this setup from the command line and writes run.json next to the checkpoints, so that TextToImage.from_run(directory) can rebuild the pipeline later. For larger images, StableDiffusionVAE in dew.nn.autoencoders.sd_vae lets the model denoise small latents instead of pixels, which is the idea behind the paper “High-Resolution Image Synthesis with Latent Diffusion Models” by Rombach et al. (2022). Notebook 04 compares the samplers.