Skip to content

Learn image representations with I-JEPA

We train an encoder whose only product is a vector for each image, and the goal is that similar images get similar vectors. Why would we want that? Well, if the vectors are good, even a very simple classifier, or just looking up an image’s nearest neighbours, can tell the images apart.

We use I-JEPA, from the paper “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” by Assran et al. (2023). The encoder sees part of an image and has to predict the encoder’s own embeddings of the hidden parts. It never reconstructs pixels, and it needs no hand-made augmentations such as crops and colour jitter.

One I-JEPA step on a flower. The context encoder sees the unmasked patches, the predictor guesses the embeddings of the hidden block, and the target encoder, an average of the context encoder's weights, reads the whole image and gives the embeddings to match.the image?context, and a hidden blockcontext encodersees the contextpredictor+ block's positionsguesswhole imagetarget encodersees everythingtargetlossaverage of weights

After training, we check the vectors in three ways: a linear probe and a k-nearest-neighbour probe that try to read the flower species out of them, and a look at each query image’s nearest neighbours.

%pip install -q "dewml[cuda13,streaming,interop] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simple
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheel for dewml (pyproject.toml) ... done
IMAGE_SIZE = 128
PATCH_SIZE = 16
GRID = (IMAGE_SIZE // PATCH_SIZE, IMAGE_SIZE // PATCH_SIZE)
BATCH_SIZE = 128
STEPS = 20000
LEARNING_RATE = 5e-4
EMB_FEATURES = 384
NUM_LAYERS = 12
NUM_HEADS = 6
NUM_TARGET_BLOCKS = 4
BLOCK_SCALE = (0.15, 0.2)
CLASSES = 103 # the flower labels run from 1 to 102
DATA_FILE = "data/06-flowers.parquet"
RUN_DIR = "runs/06-jepa"
SEED = 0
from pathlib import Path
import jax
import numpy as np
from dew import figures
from dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX call
print(jax.devices())
[CudaDevice(id=0)]

The Hub copy of Oxford Flowers is sorted by species. We shuffle it once and save a local Parquet copy, so a head holdout includes more species. Files reads that file directly through Arrow.

holdout(records=8 * BATCH_SIZE) reserves the first eight global batches of the shuffled source. The encoder never trains on them, and the probes score only those records. Both parts go through the images preset with no augment, which keeps a deterministic resize.

import datasets
Path(DATA_FILE).parent.mkdir(parents=True, exist_ok=True)
flowers = datasets.load_dataset("pranked03/flowers-blip-captions", split="train")
flowers.shuffle(seed=SEED).to_parquet(DATA_FILE)
print(flowers)
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Dataset({
    features: ['image', 'label', 'text'],
    num_rows: 6552
})
from dew.data import Files, Loading, presets
from dew.data.compile import compile as compile_data
from dew.data.dataset import DataPartition
train, held = Files(DATA_FILE, format="parquet").holdout(records=8 * BATCH_SIZE)
sized = {"batch": BATCH_SIZE, "rejects": False}
data = compile_data(train.shuffle(seed=0).repeat().pipe(presets.images, IMAGE_SIZE, **sized),
validation=held.pipe(presets.images, IMAGE_SIZE, **sized),
read="memory", loading=Loading(workers=0, threads=16, read_buffer=64))
stream = data.val(DataPartition())
try:
val_labels = np.concatenate([batch["label"] for batch in stream])
finally:
stream.close()
print("training images:", data.records, "| held-out images:", len(val_labels),
"| species in the held-out set:", len(np.unique(val_labels)))
training images: 5528 | held-out images: 1024 | species in the held-out set: 102

Each training image gets NUM_TARGET_BLOCKS rectangles of patches to predict, with a random size and aspect ratio. The context the encoder sees is a random subset of the remaining patches. MultiBlockMask.for_grid works out the block sizes once for the grid, so every mask has the same number of tokens and the training step compiles only once.

Below is one sampled mask on a real flower. The encoder sees the context patches, the outlined blocks are the ones it has to predict, and the greyed-out patches are dropped.

from dew.objectives.jepa import MultiBlockMask
mask = MultiBlockMask.for_grid(GRID, num_targets=NUM_TARGET_BLOCKS, scale=BLOCK_SCALE)
print("context tokens:", mask.num_context, "| target blocks:", mask.num_targets,
"of", mask.block_area, "tokens each")
context_idx, target_idx = mask.sample(jax.random.key(SEED), 1)
flower = next(iter(data.train(DataPartition())))["image"][0]
side, rows = 240, GRID[0]
patch = side / rows
drawing = figures.Figure(side + 8, side + 8, seconds=8, label="a mask on a flower")
drawing.image(4, 4, side, flower, photo=True)
seen = set(np.asarray(context_idx[0]).tolist())
blocks = np.asarray(target_idx).reshape(mask.num_targets, -1)
hidden = set(range(rows * GRID[1])) - seen - set(blocks.reshape(-1).tolist())
with drawing.group() as dropped:
for index in sorted(hidden):
r, c = divmod(index, GRID[1])
drawing.rect(4 + c * patch, 4 + r * patch, patch, patch, fill="surface",
stroke="none", rx=0)
drawing.appear(dropped, 0.6)
for number, block in enumerate(blocks):
with drawing.group() as outline:
for index in block.tolist():
r, c = divmod(index, GRID[1])
drawing.rect(4 + c * patch, 4 + r * patch, patch, patch, fill="none",
stroke="accent", rx=0, width=2)
drawing.appear(outline, 1.6 + 0.8 * number)
drawing
context tokens: 16 | target blocks: 4 of 12 tokens each
a mask on a flower

I-JEPA has three networks:

  • the context encoder, a ViT that sees only the context patches;
  • the target encoder, which sees the whole image and produces the embeddings to predict;
  • the predictor, a narrower transformer that takes the context embeddings plus the positions of the hidden patches and guesses the target encoder’s embeddings there.

The target encoder is not trained. It is an exponential moving average of the context encoder, so the targets improve as the encoder improves. JepaObjective puts the three together and asks the trainer to keep that average, whose momentum grows from 0.996 to 1 over momentum_steps.

The loss is the squared distance between the predicted and the target embeddings. But couldn’t the encoder cheat, and give every image the same vector, so that every prediction is trivially right? It could, and this failure is called collapse. So the objective also reports repr_std, the spread of the embeddings across a batch. If it falls towards zero, the encoder is giving every image the same vector.

import jax.numpy as jnp
from dew import Field
from dew.objectives.jepa import JepaEncoder, JepaObjective, JepaPredictor
encoder = JepaEncoder(patch_size=PATCH_SIZE, emb_features=EMB_FEATURES,
num_layers=NUM_LAYERS, num_heads=NUM_HEADS,
dtype=jnp.bfloat16, attention_impl="auto")
predictor = JepaPredictor(grid=GRID, emb_features=EMB_FEATURES,
predictor_features=EMB_FEATURES // 2, num_layers=NUM_LAYERS // 2,
num_heads=NUM_HEADS, dtype=jnp.bfloat16, attention_impl="auto")
objective = JepaObjective(encoder, predictor, mask=mask,
sample=Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)),
momentum_steps=STEPS)
variables = jax.eval_shape(objective.init, jax.random.key(0))
for name, tree in variables["params"].items():
print(f"{name}: {sum(x.size for x in jax.tree_util.tree_leaves(tree)) / 1e6:.2f}M parameters")
context_encoder: 21.59M parameters
predictor: 2.82M parameters
from dew import Checkpoints, LocalTracker, Trainer
from dew.config import OptimConfig
trainer = Trainer(objective, OptimConfig(learning_rate=LEARNING_RATE),
key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR),
tracker=LocalTracker(f"{RUN_DIR}/tracking"))
state = trainer.fit(data, steps=STEPS, log_every=2000, checkpoint_every=STEPS)
validation: 1024 records held out of train
Training JepaEncoder from step 0 to 20000: 24.4M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 128, float32 parameters, bfloat16 compute
train/mfu is unavailable for JepaObjective: JepaObjective has no model to count for train/mfu
step 2000/20000  loss 0.1477  repr_cov_offdiag 0.4134  repr_std 0.6962  data_wait_ms 0.03871  grad_norm 0.2253  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.335  step_time_ms 33.43  samples_per_sec 3,829  hlo_flops_util 8.4%  hfu 8.4%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:09:42 left
step 4000/20000  loss 0.1604  repr_cov_offdiag 0.3615  repr_std 0.7104  data_wait_ms 0.03754  grad_norm 0.1870  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.341  step_time_ms 30.78  samples_per_sec 4,159  hlo_flops_util 9.1%  hfu 9.1%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:06:34 left
step 6000/20000  loss 0.1754  repr_cov_offdiag 0.2828  repr_std 0.6766  data_wait_ms 0.03672  grad_norm 0.1614  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.347  step_time_ms 24.29  samples_per_sec 5,269  hlo_flops_util 11.5%  hfu 11.5%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:05:44 left
step 8000/20000  loss 0.1770  repr_cov_offdiag 0.2153  repr_std 0.6684  data_wait_ms 0.03493  grad_norm 0.1557  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.21  samples_per_sec 5,288  hlo_flops_util 11.6%  hfu 11.6%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:04:46 left
step 10000/20000  loss 0.1709  repr_cov_offdiag 0.1948  repr_std 0.6456  data_wait_ms 0.03567  grad_norm 0.1462  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.26  samples_per_sec 5,277  hlo_flops_util 11.6%  hfu 11.6%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:03:55 left
step 12000/20000  loss 0.1786  repr_cov_offdiag 0.1784  repr_std 0.6309  data_wait_ms 0.03667  grad_norm 0.1348  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.55  samples_per_sec 5,215  hlo_flops_util 11.4%  hfu 11.4%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:03:20 left
step 14000/20000  loss 0.1702  repr_cov_offdiag 0.1564  repr_std 0.6133  data_wait_ms 0.03657  grad_norm 0.1332  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.28  samples_per_sec 5,273  hlo_flops_util 11.6%  hfu 11.6%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:02:30 left
step 16000/20000  loss 0.1731  repr_cov_offdiag 0.1465  repr_std 0.6039  data_wait_ms 0.03705  grad_norm 0.1261  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.38  samples_per_sec 5,251  hlo_flops_util 11.5%  hfu 11.5%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:01:34 left
step 18000/20000  loss 0.1601  repr_cov_offdiag 0.1370  repr_std 0.5932  data_wait_ms 0.03695  grad_norm 0.1186  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.359  step_time_ms 24.31  samples_per_sec 5,265  hlo_flops_util 11.5%  hfu 11.5%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0  0:00:47 left
step 20000/20000  loss 0.1614  repr_cov_offdiag 0.1335  repr_std 0.6029  data_wait_ms 0.03512  grad_norm 0.1149  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 1.366  step_time_ms 24.40  samples_per_sec 5,245  hlo_flops_util 11.5%  hfu 11.5%  accepted 100.0%  learning_rate 5.000e-04  accepted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  attempted/files:parquet:data/06-flowers.parquet[1024:None] 5,528  chunks_per_record/files:parquet:data/06-flowers.parquet[1024:None] 0  truncated/files:parquet:data/06-flowers.parquet[1024:None] 0
Trained 20000 steps in 0:09:58: first step after 77.13 s, then 38.5 step/s
86.8% of the wall time in steps, final loss 0.1614
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]
rows = [row for row in rows if "train/loss" in row["scalars"]]
steps = [row["step"] for row in rows]
for row in rows:
print(f"step {row['step']:>6}: loss {row['scalars']['train/loss']:.4f}, "
f"repr_std {row['scalars']['train/repr_std']:.3f}")
figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]}, x=steps,
x_label="step", y_label="loss")
step   2000: loss 0.1477, repr_std 0.696
step   4000: loss 0.1604, repr_std 0.710
step   6000: loss 0.1754, repr_std 0.677
step   8000: loss 0.1770, repr_std 0.668
step  10000: loss 0.1709, repr_std 0.646
step  12000: loss 0.1786, repr_std 0.631
step  14000: loss 0.1702, repr_std 0.613
step  16000: loss 0.1731, repr_std 0.604
step  18000: loss 0.1601, repr_std 0.593
step  20000: loss 0.1614, repr_std 0.603
loss against step0.150.160.170.185000100001500020000steploss0.16
figures.curve({"repr_std": [row["scalars"]["train/repr_std"] for row in rows]}, x=steps,
x_label="step", y_label="repr_std, the spread of the embeddings")
repr_std, the spread of the embeddings against step0.60.650.75000100001500020000steprepr_std, the spread of the embeddings0.60

The loss doesn’t fall the way the diffusion and language model losses did: it starts at about 0.15, rises to about 0.18 in the middle of the run, and ends near 0.16. Its targets aren’t fixed, because they come from the target encoder, which keeps changing as it averages in the context encoder’s weights, so the predictor is chasing a moving target. What we watch for instead is collapse, an encoder that maps every image to the same vector, which would make the prediction trivial. repr_std, the spread of the embeddings, would fall towards 0 if that happened. It drifts from about 0.7 to 0.6 and holds there. And the log’s repr_cov_offdiag, how strongly the embedding’s dimensions move together, falls from 0.41 to 0.13.

objective.evaluate embeds a batch with the target encoder (the EMA weights) and averages each image’s patch embeddings into one vector. We collect those vectors for all 1,024 held-out images.

Each probe fits on half of the vectors and scores the other half. The linear probe is logistic regression; the k-NN probe gives each image the majority species of its 20 nearest neighbours. Now, a score means little on its own: we also want to know what the same probe scores when the vectors carry no species information at all. So we also run both probes with the labels shuffled. The gap between the real and the shuffled score is how much species information the vectors actually carry. For reference, chance on 102 species is about 1%.

from dew.objectives.base import Step
from dew.objectives.jepa import representation_health
from dew.objectives.jepa.probes import knn_probe_accuracy, linear_probe_accuracy
features, labels, held_out = [], [], []
for batch in data.val(DataPartition()):
scored = objective.evaluate(state.variables, batch,
Step(step=state.step, key=jax.random.key(1), ema=state.averaged))
features.append(np.asarray(scored.features))
labels.append(np.asarray(scored.labels))
held_out.append(batch["image"])
features, labels, held_out = np.concatenate(features), np.concatenate(labels), np.concatenate(held_out)
print("held-out embeddings:", features.shape)
health = representation_health(features)
print(f"repr_std {float(health['repr_std']):.3f} | repr_cov_offdiag {float(health['repr_cov_offdiag']):.4f}")
shuffled = np.random.default_rng(0).permutation(labels)
for name, probe in (("linear probe", linear_probe_accuracy), ("k-NN probe", knn_probe_accuracy)):
real = float(probe(features, labels, CLASSES))
control = float(probe(features, shuffled, CLASSES))
print(f"{name}: {real:.3f} with the real labels, {control:.3f} with shuffled labels")
held-out embeddings: (1024, 384)
repr_std 0.548 | repr_cov_offdiag 0.1108
linear probe: 0.336 with the real labels, 0.014 with shuffled labels
k-NN probe: 0.221 with the real labels, 0.012 with shuffled labels

On the scoring half of the held-out images, the linear probe gets 33.6% right and the k-NN probe 22.1%, against 1.4% and 1.2% with shuffled labels, which is about chance. So the vectors carry real species information, and nobody ever told the encoder what a species is: all it learned was to predict the embeddings of hidden blocks from the visible ones.

The vectors have 384 numbers each, so we can’t look at them directly. But we can project them onto the two directions along which they vary the most, their first two principal components, and colour three species. Here are the held-out vectors of the three most common species, first from a freshly initialized encoder, sliding to where the trained encoder puts them.

def embed(encoder_params, images):
# Each image's patch embeddings, averaged into one vector.
patches = objective.encode({"params": encoder_params}, images.astype(np.float32) / 127.5 - 1)
return np.asarray(patches).mean(axis=1)
def plane(vectors):
# The vectors' coordinates along their first two principal components.
centred = vectors - vectors.mean(axis=0)
return centred @ np.linalg.svd(centred, full_matrices=False)[2][:2].T
common = np.unique(labels, return_counts=True)
species = common[0][np.argsort(common[1])[::-1][:3]]
chosen = np.isin(labels, species)
untrained = objective.init(jax.random.key(SEED))["params"]["context_encoder"]
before = plane(embed(untrained, held_out[chosen]))
after = plane(embed(state.averaged["params"]["context_encoder"], held_out[chosen]))
before, after = before / before.std(), after / after.std()
figures.points({f"species {s}": before[labels[chosen] == s] for s in species},
moved={f"species {s}": after[labels[chosen] == s] for s in species},
joined=False, captions=["before training", "after training"],
x_label="first component", y_label="second component")
second component against first component02-202first componentsecond componentspecies 51species 77species 89before trainingafter training

Before training the three species are mixed together; species 89 leans to the left, but 51 and 77 overlap completely. After training, species 77 sits in a tight group on the left and 89 in one on the right. Species 51 doesn’t form one group: some of its images sit at the top, others along the bottom near 77, and a few near 89 on the right.

The probes give one number each, but we can also look at the vectors directly. Each row below is one held-out query image (left) and its five nearest held-out images by cosine similarity. The row is labelled with the query’s species, and the species of its neighbours are printed above the grid.

normalised = features / np.linalg.norm(features, axis=-1, keepdims=True)
queries = [0, 1, 2, 3, 4, 5]
grid = []
for query in queries:
nearest = np.argsort(-(normalised @ normalised[query]))[1:6]
print(f"query {labels[query]:>3}: neighbours", " ".join(f"{labels[i]:>3}" for i in nearest))
grid.extend(held_out[[query, *nearest]])
figures.images([np.stack(grid)], columns=6, size=64,
rows=[f"query {labels[query]}" for query in queries])
query  40: neighbours  51  21  83  74  74
query  78: neighbours  78  69  96  80  41
query  31: neighbours  96  44  30  26  74
query  57: neighbours  57  57  37  88  40
query  37: neighbours  37  37  37  37  36
query  46: neighbours  46  46  90  46  89
36 imagesquery 40query 78query 31query 57query 37query 46

Four of the six queries find their own species among their five nearest neighbours, and query 37 finds it four times. Where the neighbours are other species, they still look alike: query 40’s red flower brings up other red flowers, and query 31’s dark red one brings up a poinsettia, a poppy and a rose. The encoder has learned colour and shape well; telling apart species that look alike is harder.

What we keep from a JEPA run is the EMA copy of the context encoder, without the predictor. save_params writes it as a safetensors file, and load_params reads it back. objective.encode embeds images in [-1, 1] with a given set of encoder variables (the weights under params, as Flax stores them).

from dew.interop import load_params, save_params
encoder_params = state.averaged["params"]["context_encoder"]
save_params(encoder_params, f"{RUN_DIR}/encoder.safetensors")
reloaded = load_params(f"{RUN_DIR}/encoder.safetensors")
again = objective.encode({"params": reloaded}, held_out[:8].astype(np.float32) / 127.5 - 1)
print("reloaded encoder output:", np.asarray(again).shape)
reloaded encoder output: (8, 64, 384)

The I-JEPA paper trains a ViT-H/16 for 300 epochs on ImageNet. recipes/jepa/train.py runs the same objective from the command line, and the jepa_video_encoder model with a factorized predictor does the same job on video clips.