Learn image representations with I-JEPA
We train an encoder whose only product is a vector for each image, and the goal is that similar images get similar vectors. Why would we want that? Well, if the vectors are good, even a very simple classifier, or just looking up an image’s nearest neighbours, can tell the images apart.
We use I-JEPA, from the paper “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” by Assran et al. (2023). The encoder sees part of an image and has to predict the encoder’s own embeddings of the hidden parts. It never reconstructs pixels, and it needs no hand-made augmentations such as crops and colour jitter.
After training, we check the vectors in three ways: a linear probe and a k-nearest-neighbour probe that try to read the flower species out of them, and a look at each query image’s nearest neighbours.
Install Dependencies
Section titled “Install Dependencies”%pip install -q "dewml[cuda13,streaming,interop] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheel for dewml (pyproject.toml) ... done
Settings
Section titled “Settings”IMAGE_SIZE = 128PATCH_SIZE = 16GRID = (IMAGE_SIZE // PATCH_SIZE, IMAGE_SIZE // PATCH_SIZE)BATCH_SIZE = 128STEPS = 20000LEARNING_RATE = 5e-4EMB_FEATURES = 384NUM_LAYERS = 12NUM_HEADS = 6NUM_TARGET_BLOCKS = 4BLOCK_SCALE = (0.15, 0.2)CLASSES = 103 # the flower labels run from 1 to 102DATA_FILE = "data/06-flowers.parquet"RUN_DIR = "runs/06-jepa"SEED = 0from pathlib import Path
import jaximport numpy as np
from dew import figuresfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())[CudaDevice(id=0)]
Data pipeline
Section titled “Data pipeline”The Hub copy of Oxford Flowers is sorted by species. We shuffle it once and save a local Parquet copy, so a head holdout includes more species. Files reads that file directly through Arrow.
holdout(records=8 * BATCH_SIZE) reserves the first eight global batches of the shuffled source. The encoder never trains on them, and the probes score only those records. Both parts go through the images preset with no augment, which keeps a deterministic resize.
import datasets
Path(DATA_FILE).parent.mkdir(parents=True, exist_ok=True)flowers = datasets.load_dataset("pranked03/flowers-blip-captions", split="train")flowers.shuffle(seed=SEED).to_parquet(DATA_FILE)print(flowers)stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Dataset({
features: ['image', 'label', 'text'],
num_rows: 6552
})from dew.data import Files, Loading, presetsfrom dew.data.compile import compile as compile_datafrom dew.data.dataset import DataPartition
train, held = Files(DATA_FILE, format="parquet").holdout(records=8 * BATCH_SIZE)sized = {"batch": BATCH_SIZE, "rejects": False}data = compile_data(train.shuffle(seed=0).repeat().pipe(presets.images, IMAGE_SIZE, **sized), validation=held.pipe(presets.images, IMAGE_SIZE, **sized), read="memory", loading=Loading(workers=0, threads=16, read_buffer=64))
stream = data.val(DataPartition())try: val_labels = np.concatenate([batch["label"] for batch in stream])finally: stream.close()print("training images:", data.records, "| held-out images:", len(val_labels), "| species in the held-out set:", len(np.unique(val_labels)))training images: 5528 | held-out images: 1024 | species in the held-out set: 102
The mask
Section titled “The mask”Each training image gets NUM_TARGET_BLOCKS rectangles of patches to predict, with a random size and aspect ratio. The context the encoder sees is a random subset of the remaining patches. MultiBlockMask.for_grid works out the block sizes once for the grid, so every mask has the same number of tokens and the training step compiles only once.
Below is one sampled mask on a real flower. The encoder sees the context patches, the outlined blocks are the ones it has to predict, and the greyed-out patches are dropped.
from dew.objectives.jepa import MultiBlockMask
mask = MultiBlockMask.for_grid(GRID, num_targets=NUM_TARGET_BLOCKS, scale=BLOCK_SCALE)print("context tokens:", mask.num_context, "| target blocks:", mask.num_targets, "of", mask.block_area, "tokens each")
context_idx, target_idx = mask.sample(jax.random.key(SEED), 1)flower = next(iter(data.train(DataPartition())))["image"][0]side, rows = 240, GRID[0]patch = side / rowsdrawing = figures.Figure(side + 8, side + 8, seconds=8, label="a mask on a flower")drawing.image(4, 4, side, flower, photo=True)seen = set(np.asarray(context_idx[0]).tolist())blocks = np.asarray(target_idx).reshape(mask.num_targets, -1)hidden = set(range(rows * GRID[1])) - seen - set(blocks.reshape(-1).tolist())with drawing.group() as dropped: for index in sorted(hidden): r, c = divmod(index, GRID[1]) drawing.rect(4 + c * patch, 4 + r * patch, patch, patch, fill="surface", stroke="none", rx=0)drawing.appear(dropped, 0.6)for number, block in enumerate(blocks): with drawing.group() as outline: for index in block.tolist(): r, c = divmod(index, GRID[1]) drawing.rect(4 + c * patch, 4 + r * patch, patch, patch, fill="none", stroke="accent", rx=0, width=2) drawing.appear(outline, 1.6 + 0.8 * number)drawingcontext tokens: 16 | target blocks: 4 of 12 tokens each
Encoder, predictor and objective
Section titled “Encoder, predictor and objective”I-JEPA has three networks:
- the context encoder, a ViT that sees only the context patches;
- the target encoder, which sees the whole image and produces the embeddings to predict;
- the predictor, a narrower transformer that takes the context embeddings plus the positions of the hidden patches and guesses the target encoder’s embeddings there.
The target encoder is not trained. It is an exponential moving average of the context encoder, so the targets improve as the encoder improves. JepaObjective puts the three together and asks the trainer to keep that average, whose momentum grows from 0.996 to 1 over momentum_steps.
The loss is the squared distance between the predicted and the target embeddings. But couldn’t the encoder cheat, and give every image the same vector, so that every prediction is trivially right? It could, and this failure is called collapse. So the objective also reports repr_std, the spread of the embeddings across a batch. If it falls towards zero, the encoder is giving every image the same vector.
import jax.numpy as jnpfrom dew import Fieldfrom dew.objectives.jepa import JepaEncoder, JepaObjective, JepaPredictor
encoder = JepaEncoder(patch_size=PATCH_SIZE, emb_features=EMB_FEATURES, num_layers=NUM_LAYERS, num_heads=NUM_HEADS, dtype=jnp.bfloat16, attention_impl="auto")predictor = JepaPredictor(grid=GRID, emb_features=EMB_FEATURES, predictor_features=EMB_FEATURES // 2, num_layers=NUM_LAYERS // 2, num_heads=NUM_HEADS, dtype=jnp.bfloat16, attention_impl="auto")objective = JepaObjective(encoder, predictor, mask=mask, sample=Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)), momentum_steps=STEPS)
variables = jax.eval_shape(objective.init, jax.random.key(0))for name, tree in variables["params"].items(): print(f"{name}: {sum(x.size for x in jax.tree_util.tree_leaves(tree)) / 1e6:.2f}M parameters")context_encoder: 21.59M parameters predictor: 2.82M parameters
Training
Section titled “Training”from dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfig
trainer = Trainer(objective, OptimConfig(learning_rate=LEARNING_RATE), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR), tracker=LocalTracker(f"{RUN_DIR}/tracking"))state = trainer.fit(data, steps=STEPS, log_every=2000, checkpoint_every=STEPS)validation: 1024 records held out of train Training JepaEncoder from step 0 to 20000: 24.4M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 128, float32 parameters, bfloat16 compute train/mfu is unavailable for JepaObjective: JepaObjective has no model to count for train/mfu step 2000/20000 loss 0.1477 repr_cov_offdiag 0.4134 repr_std 0.6962 data_wait_ms 0.03871 grad_norm 0.2253 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.335 step_time_ms 33.43 samples_per_sec 3,829 hlo_flops_util 8.4% hfu 8.4% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:09:42 left step 4000/20000 loss 0.1604 repr_cov_offdiag 0.3615 repr_std 0.7104 data_wait_ms 0.03754 grad_norm 0.1870 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.341 step_time_ms 30.78 samples_per_sec 4,159 hlo_flops_util 9.1% hfu 9.1% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:06:34 left step 6000/20000 loss 0.1754 repr_cov_offdiag 0.2828 repr_std 0.6766 data_wait_ms 0.03672 grad_norm 0.1614 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.347 step_time_ms 24.29 samples_per_sec 5,269 hlo_flops_util 11.5% hfu 11.5% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:05:44 left step 8000/20000 loss 0.1770 repr_cov_offdiag 0.2153 repr_std 0.6684 data_wait_ms 0.03493 grad_norm 0.1557 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.21 samples_per_sec 5,288 hlo_flops_util 11.6% hfu 11.6% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:04:46 left step 10000/20000 loss 0.1709 repr_cov_offdiag 0.1948 repr_std 0.6456 data_wait_ms 0.03567 grad_norm 0.1462 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.26 samples_per_sec 5,277 hlo_flops_util 11.6% hfu 11.6% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:03:55 left step 12000/20000 loss 0.1786 repr_cov_offdiag 0.1784 repr_std 0.6309 data_wait_ms 0.03667 grad_norm 0.1348 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.55 samples_per_sec 5,215 hlo_flops_util 11.4% hfu 11.4% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:03:20 left step 14000/20000 loss 0.1702 repr_cov_offdiag 0.1564 repr_std 0.6133 data_wait_ms 0.03657 grad_norm 0.1332 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.28 samples_per_sec 5,273 hlo_flops_util 11.6% hfu 11.6% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:02:30 left step 16000/20000 loss 0.1731 repr_cov_offdiag 0.1465 repr_std 0.6039 data_wait_ms 0.03705 grad_norm 0.1261 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.38 samples_per_sec 5,251 hlo_flops_util 11.5% hfu 11.5% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:01:34 left step 18000/20000 loss 0.1601 repr_cov_offdiag 0.1370 repr_std 0.5932 data_wait_ms 0.03695 grad_norm 0.1186 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.359 step_time_ms 24.31 samples_per_sec 5,265 hlo_flops_util 11.5% hfu 11.5% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 0:00:47 left step 20000/20000 loss 0.1614 repr_cov_offdiag 0.1335 repr_std 0.6029 data_wait_ms 0.03512 grad_norm 0.1149 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 1.366 step_time_ms 24.40 samples_per_sec 5,245 hlo_flops_util 11.5% hfu 11.5% accepted 100.0% learning_rate 5.000e-04 accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528 chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0 truncated/files:parquet:data/06-flowers.parquet[1024:None] 0 Trained 20000 steps in 0:09:58: first step after 77.13 s, then 38.5 step/s 86.8% of the wall time in steps, final loss 0.1614
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]rows = [row for row in rows if "train/loss" in row["scalars"]]steps = [row["step"] for row in rows]for row in rows: print(f"step {row['step']:>6}: loss {row['scalars']['train/loss']:.4f}, " f"repr_std {row['scalars']['train/repr_std']:.3f}")figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]}, x=steps, x_label="step", y_label="loss")step 2000: loss 0.1477, repr_std 0.696 step 4000: loss 0.1604, repr_std 0.710 step 6000: loss 0.1754, repr_std 0.677 step 8000: loss 0.1770, repr_std 0.668 step 10000: loss 0.1709, repr_std 0.646 step 12000: loss 0.1786, repr_std 0.631 step 14000: loss 0.1702, repr_std 0.613 step 16000: loss 0.1731, repr_std 0.604 step 18000: loss 0.1601, repr_std 0.593 step 20000: loss 0.1614, repr_std 0.603
figures.curve({"repr_std": [row["scalars"]["train/repr_std"] for row in rows]}, x=steps, x_label="step", y_label="repr_std, the spread of the embeddings")The loss doesn’t fall the way the diffusion and language model losses did: it starts at about 0.15, rises to about 0.18 in the middle of the run, and ends near 0.16. Its targets aren’t fixed, because they come from the target encoder, which keeps changing as it averages in the context encoder’s weights, so the predictor is chasing a moving target. What we watch for instead is collapse, an encoder that maps every image to the same vector, which would make the prediction trivial. repr_std, the spread of the embeddings, would fall towards 0 if that happened. It drifts from about 0.7 to 0.6 and holds there. And the log’s repr_cov_offdiag, how strongly the embedding’s dimensions move together, falls from 0.41 to 0.13.
Probes on the held-out images
Section titled “Probes on the held-out images”objective.evaluate embeds a batch with the target encoder (the EMA weights) and averages each image’s patch embeddings into one vector. We collect those vectors for all 1,024 held-out images.
Each probe fits on half of the vectors and scores the other half. The linear probe is logistic regression; the k-NN probe gives each image the majority species of its 20 nearest neighbours. Now, a score means little on its own: we also want to know what the same probe scores when the vectors carry no species information at all. So we also run both probes with the labels shuffled. The gap between the real and the shuffled score is how much species information the vectors actually carry. For reference, chance on 102 species is about 1%.
from dew.objectives.base import Stepfrom dew.objectives.jepa import representation_healthfrom dew.objectives.jepa.probes import knn_probe_accuracy, linear_probe_accuracy
features, labels, held_out = [], [], []for batch in data.val(DataPartition()): scored = objective.evaluate(state.variables, batch, Step(step=state.step, key=jax.random.key(1), ema=state.averaged)) features.append(np.asarray(scored.features)) labels.append(np.asarray(scored.labels)) held_out.append(batch["image"])features, labels, held_out = np.concatenate(features), np.concatenate(labels), np.concatenate(held_out)print("held-out embeddings:", features.shape)
health = representation_health(features)print(f"repr_std {float(health['repr_std']):.3f} | repr_cov_offdiag {float(health['repr_cov_offdiag']):.4f}")
shuffled = np.random.default_rng(0).permutation(labels)for name, probe in (("linear probe", linear_probe_accuracy), ("k-NN probe", knn_probe_accuracy)): real = float(probe(features, labels, CLASSES)) control = float(probe(features, shuffled, CLASSES)) print(f"{name}: {real:.3f} with the real labels, {control:.3f} with shuffled labels")held-out embeddings: (1024, 384) repr_std 0.548 | repr_cov_offdiag 0.1108 linear probe: 0.336 with the real labels, 0.014 with shuffled labels k-NN probe: 0.221 with the real labels, 0.012 with shuffled labels
On the scoring half of the held-out images, the linear probe gets 33.6% right and the k-NN probe 22.1%, against 1.4% and 1.2% with shuffled labels, which is about chance. So the vectors carry real species information, and nobody ever told the encoder what a species is: all it learned was to predict the embeddings of hidden blocks from the visible ones.
The vectors in two dimensions
Section titled “The vectors in two dimensions”The vectors have 384 numbers each, so we can’t look at them directly. But we can project them onto the two directions along which they vary the most, their first two principal components, and colour three species. Here are the held-out vectors of the three most common species, first from a freshly initialized encoder, sliding to where the trained encoder puts them.
def embed(encoder_params, images): # Each image's patch embeddings, averaged into one vector. patches = objective.encode({"params": encoder_params}, images.astype(np.float32) / 127.5 - 1) return np.asarray(patches).mean(axis=1)
def plane(vectors): # The vectors' coordinates along their first two principal components. centred = vectors - vectors.mean(axis=0) return centred @ np.linalg.svd(centred, full_matrices=False)[2][:2].T
common = np.unique(labels, return_counts=True)species = common[0][np.argsort(common[1])[::-1][:3]]chosen = np.isin(labels, species)untrained = objective.init(jax.random.key(SEED))["params"]["context_encoder"]before = plane(embed(untrained, held_out[chosen]))after = plane(embed(state.averaged["params"]["context_encoder"], held_out[chosen]))before, after = before / before.std(), after / after.std()figures.points({f"species {s}": before[labels[chosen] == s] for s in species}, moved={f"species {s}": after[labels[chosen] == s] for s in species}, joined=False, captions=["before training", "after training"], x_label="first component", y_label="second component")Before training the three species are mixed together; species 89 leans to the left, but 51 and 77 overlap completely. After training, species 77 sits in a tight group on the left and 89 in one on the right. Species 51 doesn’t form one group: some of its images sit at the top, others along the bottom near 77, and a few near 89 on the right.
Nearest neighbours
Section titled “Nearest neighbours”The probes give one number each, but we can also look at the vectors directly. Each row below is one held-out query image (left) and its five nearest held-out images by cosine similarity. The row is labelled with the query’s species, and the species of its neighbours are printed above the grid.
normalised = features / np.linalg.norm(features, axis=-1, keepdims=True)queries = [0, 1, 2, 3, 4, 5]grid = []for query in queries: nearest = np.argsort(-(normalised @ normalised[query]))[1:6] print(f"query {labels[query]:>3}: neighbours", " ".join(f"{labels[i]:>3}" for i in nearest)) grid.extend(held_out[[query, *nearest]])figures.images([np.stack(grid)], columns=6, size=64, rows=[f"query {labels[query]}" for query in queries])query 40: neighbours 51 21 83 74 74 query 78: neighbours 78 69 96 80 41 query 31: neighbours 96 44 30 26 74 query 57: neighbours 57 57 37 88 40 query 37: neighbours 37 37 37 37 36 query 46: neighbours 46 46 90 46 89
Four of the six queries find their own species among their five nearest neighbours, and query 37 finds it four times. Where the neighbours are other species, they still look alike: query 40’s red flower brings up other red flowers, and query 31’s dark red one brings up a poinsettia, a poppy and a rose. The encoder has learned colour and shape well; telling apart species that look alike is harder.
Keeping the encoder
Section titled “Keeping the encoder”What we keep from a JEPA run is the EMA copy of the context encoder, without the predictor. save_params writes it as a safetensors file, and load_params reads it back. objective.encode embeds images in [-1, 1] with a given set of encoder variables (the weights under params, as Flax stores them).
from dew.interop import load_params, save_params
encoder_params = state.averaged["params"]["context_encoder"]save_params(encoder_params, f"{RUN_DIR}/encoder.safetensors")reloaded = load_params(f"{RUN_DIR}/encoder.safetensors")again = objective.encode({"params": reloaded}, held_out[:8].astype(np.float32) / 127.5 - 1)print("reloaded encoder output:", np.asarray(again).shape)reloaded encoder output: (8, 64, 384)
Where to go next
Section titled “Where to go next”The I-JEPA paper trains a ViT-H/16 for 300 epochs on ImageNet. recipes/jepa/train.py runs the same objective from the command line, and the jepa_video_encoder model with a factorized predictor does the same job on video clips.