Skip to content
DewDew

dew.eval

Score image metrics behind dew.registry.metrics: metrics["fid"](), metrics["clip_score"](), metrics.psnr(), metrics.ssim(), metrics.clip(), each a factory returning a Metric the trainer scores an ImageGrid with.

fid(generated, reference) and clip_score(images, prompts) are the same numbers over image sets already in hand, with no trainer and no batch.

NameSummary
FIDScores FID over a pass, pooling statistics and taking one final distance.
ImageMetricAverages one image metric per image, or per frame for video, over a pass.
clipScore CLIP distance, mean(1 - cos(image, text)); lower is better.
clip_image_text_cosineReturn the per-image cosine between uint8 [N, H, W, 3] images and prompts.
clip_scoreScore CLIPScore of uint8 [N, H, W, 3] images against one prompt each.
clip_score_metricScore standard CLIPScore over a validation pass, the same number clip_score reports for the images and prompts the pass consumed.
fidMeasure FID between two sets of uint8 [N, H, W, 3] images.
framesReturn pixels in [-1, 1], with videos keeping their frame axis.
frechet_distanceReturn the Frechet distance between two multivariate gaussians.
peak_signal_noise_ratioPSNR = 10 log10(data_range^2 / MSE), per frame, as skimage defines it.
psnrMean PSNR in dB between the sampled frames and the batch’s, higher is better.
ssimMean SSIM between the sampled frames and the batch’s, higher is better, on the same [-1, 1] scale as psnr.
structural_similaritySSIM (Wang et al.

FID

Section titled “FID”

dataclass source

class FID(field: str = 'image', weights: str | None = None)

Scores FID over a pass, pooling statistics and taking one final distance.

The call gathers the sampled grid and the batch’s reference field. The features, the statistics and the distance are the ones fid runs, so a pass over 50,000 images a side reports the number fid reports, and weights names the extractor’s parameters there the same way.

FID.merge

Section titled “FID.merge”
def merge(accumulated: FIDStats, contribution: FIDStats) -> FIDStats

FID.finalize

Section titled “FID.finalize”
def finalize(accumulated: FIDStats) -> float

ImageMetric

Section titled “ImageMetric”

dataclass source

class ImageMetric(
    name: str,
    measure: Callable[[ImageGrid | VideoGrid, Batch], ArrayLike],
    reads: type = ImageGrid,
)

Averages one image metric per image, or per frame for video, over a pass.

measure: Callable[[ImageGrid | VideoGrid, Batch], ArrayLike]

One measurement per image or frame, never an already averaged scalar.

ImageMetric.merge

Section titled “ImageMetric.merge”
def merge(
    accumulated: tuple[float, int],
    contribution: tuple[float, int],
) -> tuple[float, int]

ImageMetric.finalize

Section titled “ImageMetric.finalize”
def finalize(accumulated: tuple[float, int]) -> float

clip

Section titled “clip”

function source

def clip(modelname: str = DEFAULT_MODEL, field: str = 'text') -> ImageMetric

Score CLIP distance, mean(1 - cos(image, text)); lower is better. It logs as val/clip_similarity; clip_score is the standard number for a new run.

clip_image_text_cosine

Section titled “clip_image_text_cosine”

function source

def clip_image_text_cosine(
    images: ArrayLike,
    input_ids: ArrayLike,
    attention_mask: ArrayLike,
    *,
    modelname: str = DEFAULT_MODEL,
) -> jax.Array

Return the per-image cosine between uint8 [N, H, W, 3] images and prompts.

The images go through the checkpoint’s own processor, so the embeddings are the ones the reference produces for these pixels and tokens.

clip_score

Section titled “clip_score”

function source

def clip_score(
    images: ArrayLike,
    prompts: Sequence[str],
    *,
    modelname: str = DEFAULT_MODEL,
    batch_size: int = 64,
) -> float

Score CLIPScore of uint8 [N, H, W, 3] images against one prompt each.

100 * mean(max(cos(image, prompt), 0)), higher is better; typical T2I models score around 25-35 on natural prompts. The images are scored batch_size rows at a time, and the prompts are tokenized the way a run’s batch carries them.

clip_score_metric

Section titled “clip_score_metric”

function source

def clip_score_metric(
    modelname: str = DEFAULT_MODEL,
    field: str = 'text',
) -> ImageMetric

Score standard CLIPScore over a validation pass, the same number clip_score reports for the images and prompts the pass consumed.

fid

Section titled “fid”

function source

def fid(
    generated: NDArray[np.uint8] | jax.Array | Iterable[ArrayLike],
    reference: NDArray[np.uint8] | jax.Array | Iterable[ArrayLike],
    *,
    batch_size: int = 64,
    weights: str | Path | None = None,
) -> float

Measure FID between two sets of uint8 [N, H, W, 3] images.

Each side is one array or an iterable of arrays, so a directory of samples can stream past in blocks of batch_size rows instead of being held at once. The value is the distance between the two populations passed in, which is FID-50k only at 50,000 images a side.

weights is the feature extractor’s parameters as a file, the way clip_score(modelname=) names a local CLIP: the InceptionV3 variables tree in safetensors, which tools/convert_inception_weights.py writes. Unset downloads the published checkpoint and converts it. Two distances are comparable only when both were measured with the same one, which is why every distance logs which it was. With the published weights, features and distance reproduce pytorch-fid 0.3.0’s (bilinear resize without antialiasing); tests/test_metrics.py holds the distance to 1e-5 relative.

frames

Section titled “frames”

function source

def frames(artifact: ImageGrid | VideoGrid) -> jax.Array

Return pixels in [-1, 1], with videos keeping their frame axis.

frechet_distance

Section titled “frechet_distance”

function source

def frechet_distance(mu_a, sigma_a, mu_b, sigma_b, eps=1e-06) -> float

Return the Frechet distance between two multivariate gaussians.

Runs once per consumed validation pass on the host through scipy. The matrix square root of the covariance product has no JAX equivalent.

peak_signal_noise_ratio

Section titled “peak_signal_noise_ratio”

function source

def peak_signal_noise_ratio(
    predictions: jnp.ndarray,
    targets: jnp.ndarray,
    data_range: float,
    per_example: bool = False,
) -> jnp.ndarray

PSNR = 10 log10(data_range^2 / MSE), per frame, as skimage defines it.

data_range is the dynamic range of the signal, 2.0 for [-1, 1] inputs and 255 for uint8. The mean over frames comes back unless per_example asks for the (N,) per-frame scores. Identical inputs give +inf.

psnr

Section titled “psnr”

function source

def psnr(
    data_range: float = 2.0,
    field: str = 'image',
    reads: type = ImageGrid,
) -> ImageMetric

Mean PSNR in dB between the sampled frames and the batch’s, higher is better.

The artifact is in [-1, 1] and the batch holds uint8 pixels, which are put on the objective’s scale, so both sides span the range that the default data_range of 2.0 describes. reads names the artifact type the trainer hands this metric; a video run passes VideoGrid.

ssim

Section titled “ssim”

function source

def ssim(
    data_range: float = 2.0,
    field: str = 'image',
    reads: type = ImageGrid,
) -> ImageMetric

Mean SSIM between the sampled frames and the batch’s, higher is better, on the same [-1, 1] scale as psnr. reads names the artifact type the trainer hands this metric; a video run passes VideoGrid.

structural_similarity

Section titled “structural_similarity”

function source

def structural_similarity(
    predictions: jnp.ndarray,
    targets: jnp.ndarray,
    data_range: float,
    per_example: bool = False,
) -> jnp.ndarray

SSIM (Wang et al. 2004) per frame, an 11x11 gaussian window of sigma 1.5 on each channel and the channels averaged.

data_range is the dynamic range of the signal, 2.0 for [-1, 1] inputs. The mean over frames comes back unless per_example asks for the (N,) per-frame scores. Identical inputs give 1.0.