dew.nn.categorical
The categorical distribution over a class axis: the one owner of its log-likelihood in Dew.
A model’s logits over a vocabulary, a classifier’s over its classes, a decision’s over a question’s
options and a sampler’s over its next token are each a categorical distribution over the last axis.
Every log-likelihood Dew takes of one composes on this module: the normalized log probabilities
(log_probs), a label’s cross entropy (cross_entropy), a soft target’s (soft_cross_entropy), a
divergence from a reference distribution (kl_divergence), the entropy (entropy), the proper scoring
rules’ charges of a forecast against a target distribution beside the logarithmic one (brier,
spherical, ranked_probability) and a target smoothed toward the even one (smoothed), the divergence
among a group’s members over the classes they share (group_divergence), and, for a head
that reads its vocabulary in blocks (dew.objectives.lm.chunked), the running partition it combines
block by block (rescaled), a label’s cross entropy from it (target_loss) and the log partition of
each segment of a ragged row (segment_log_partition), a support’s.
Each computes in at least float32 (dew.nn.precision.at_least_fp32): a bf16 log partition over a
vocabulary carries bf16’s 8-bit mantissa into every score, error in the second decimal of a log
probability near -10, while float64 logits stay float64. where masks the classes a row does not
hold, as a decision’s padded option slots: they take no probability, and a class of no probability adds
no term to a sum over the classes.
| Name | Summary |
|---|---|
brier | The quadratic score as a charge, sum_k (p_k - t_k)^2 over the last axis, between 0 and 2. |
cross_entropy | -log p[labels] over the last axis, labels of logits.shape[:-1]: the log partition less the label’s logit, each taken from the row’s maximum (target_loss), which reads the label’s logit without building the normalized row. |
entropy | -sum_k p_k log p_k over the last axis, a class of no probability (a -inf logit, or one where excludes) adding no term, in value or gradient. |
group_divergence | Each member’s share of its group’s generalized Jensen-Shannon divergence, and its share of the group, [..., M] arrays whose members sum to each group’s divergence and to one a group. |
kl_divergence | KL(q || p) over the last axis, p the categorical of logits and q that of the reference logits (Optax’s kl_divergence of the two), the classes where excludes adding no term. |
label_derivative | The derivative of -log p[labels] in its row’s logits, from the row’s probability over the last axis: each other class’s probability, and at the label label_entry’s. |
label_entry | The derivative of -log p in the label’s own logit, from the label’s share of its row and the others’ share summed: share - 1 while the label holds at most half the row, else minus the others’ sum (arXiv:2610.31642, eq. |
label_logit | Each row’s logit at its label over the last axis, 0 for a label outside the row’s columns, which picks none: a padding id past the vocabulary, or another tile’s or shard’s column. |
label_terms | |
log_probs | The normalized log probabilities over the last axis (jax.nn.log_softmax). |
partition | Each row’s maximum over the last axis, the sum of its exponentials scaled to it, and those exponentials: the terms every whole-row label loss takes its log partition from (target_loss; cross_entropy, dew.objectives.lm.chunked’s whole head). |
probabilities | The probabilities over the last axis (jax.nn.softmax). |
ranked_probability | The ranked probability score of classes that are ordered levels: the squared gaps between the forecast’s and the target’s cumulative distributions over the K - 1 cuts between a row’s K levels, sum_k (P_k - T_k)^2 / (K - 1) over the last axis, the levels where holds leading the row. |
rescaled | A sum of exponentials moved from its own maximum to the larger one, as a blockwise partition combines its blocks: where the two agree the shift is exactly 0 and the sum unchanged, its derivative still the moved sum’s, so two blocks whose maxima tie each move by their own maximum’s tangent less the larger’s; two sums whose maximum is -inf add as they are rather than turn NaN. |
segment_log_partition | The log partition [count] of each segment of a ragged row of logits [C], segments naming each entry’s (count for an entry of none), and whether each segment holds an entry: its maximum, held off the gradient as jax.nn.logsumexp holds it, plus the log of its exponentials’ sum scaled to it, a segment of no entry’s log partition 0. |
smoothed | targets over the last axis with share of their mass spread evenly over the classes where holds, (1 - share) t_k + share / K for a row of K classes (optax.smooth_labels), and nothing on the others; integer targets in the floating type a Python number promotes them to (float64 under x64). |
soft_cross_entropy | -sum_k t_k log p_k over the last axis against the distribution targets, a class of no target mass contributing nothing, whatever its log probability. |
spherical | The spherical score as a charge, `1 - (t . |
target_loss | A label’s cross entropy from its row’s maximum peak, the sum of exponentials scaled to it mass, and its label’s logit: log(mass) - (target - peak), as cross_entropy takes it, so neither term is rounded at the logits’ magnitude before the subtraction that cancels it does. |
token_log_probs | log_softmax(logits)[..., tokens], reduced in at least fp32 (cross_entropy’s negation). |
def brier( logits: jax.Array, targets: jax.Array, where: jax.Array | None = None,) -> jax.ArrayThe quadratic score as a charge, sum_k (p_k - t_k)^2 over the last axis, between 0 and 2.
cross_entropy
Section titled “cross_entropy”def cross_entropy( logits: jax.Array, labels: jax.Array, where: jax.Array | None = None,) -> jax.Array-log p[labels] over the last axis, labels of logits.shape[:-1]: the log partition less the
label’s logit, each taken from the row’s maximum (target_loss), which reads the label’s logit
without building the normalized row. Optax’s softmax_cross_entropy_with_integer_labels subtracts
the label’s logit from the whole log partition, maximum included, which rounds at the logits’
magnitude rather than the loss’s: mamba-tiny’s drawn tokens’ raw scores sat 1.47 times as far from
float64 as log_softmax’s, and 1.00 times so taken here. Its derivative is Barron, Gnanapragasam
and Tschannen’s (arXiv:2610.31642, eq. 7): each other class’s probability, and at the label minus
their sum, so the row’s derivative sums to zero. The label’s p - 1 collapses to 0 once its margin
over the other classes passes the dtype’s precision (eq. 4; 16.6 in float32), while the others’
probabilities stay until they underflow. A label outside the row takes the log partition, its
derivative the row’s probabilities (label_terms).
entropy
Section titled “entropy”def entropy(logits: jax.Array, where: jax.Array | None = None) -> jax.Array-sum_k p_k log p_k over the last axis, a class of no probability (a -inf logit, or one where
excludes) adding no term, in value or gradient.
group_divergence
Section titled “group_divergence”def group_divergence( logits: jax.Array, groups: jax.Array, classes: jax.Array, where: jax.Array | None = None,) -> tuple[jax.Array, jax.Array]Each member’s share of its group’s generalized Jensen-Shannon divergence, and its share of the
group, [..., M] arrays whose members sum to each group’s divergence and to one a group.
logits [..., M, K] are M members’ categoricals, each over its own K classes (where masks those
it does not hold); groups [..., M] names each member’s group among the M (-1: none); and
classes [..., M, K] each class’s shared class, in [0, K) (-1: none), as questions that ask
one thing with their options re-ordered or reworded share their options. A member’s answer is its
probabilities (probabilities) over its classes that map to a shared one, summed onto them; a
group’s divergence is the mean of each member’s KL divergence from the members’ mean answer, zero
exactly where they agree, at most the log of the member count, computed in at least float32 and
finite at any finite logits. A shared class of no probability adds no term, in value or gradient;
the gradient is autodiff’s, through the probabilities’ own derivative, which a confident member
keeps. A member in no group, or with no class that maps, has share zero and takes no part.
kl_divergence
Section titled “kl_divergence”def kl_divergence( logits: jax.Array, reference: jax.Array, where: jax.Array | None = None,) -> jax.ArrayKL(q || p) over the last axis, p the categorical of logits and q that of the reference
logits (Optax’s kl_divergence of the two), the classes where excludes adding no term.
Its derivative in logits, p - q, is taken from the two rows’ difference in log space, as Barron,
Gnanapragasam and Tschannen take it (arXiv:2610.31642, eq. 9), not by subtracting two softmaxes
(eq. 8), which near each other keep only their own rounding: logits equal to the reference’s have
derivative exactly zero, and near them it keeps the difference’s precision. A row where a class is
-inf in one and not the other takes p - q, a row holding no class 0; the derivative in
reference is autodiff’s.
label_derivative
Section titled “label_derivative”def label_derivative(probability: jax.Array, labels: jax.Array) -> jax.ArrayThe derivative of -log p[labels] in its row’s logits, from the row’s probability over the
last axis: each other class’s probability, and at the label label_entry’s. A label outside the
row one-hots to nothing, and its row’s derivative is the probabilities.
label_entry
Section titled “label_entry”def label_entry(share: jax.Array, others: jax.Array) -> jax.ArrayThe derivative of -log p in the label’s own logit, from the label’s share of its row and
the others’ share summed: share - 1 while the label holds at most half the row, else minus
the others’ sum (arXiv:2610.31642, eq. 7). Past a half share - 1 cancels, and rounds to 0 once
the share rounds to 1, where the others’ sum keeps it and the row’s derivative sums to zero in
any rounding; below a half 1 - share is at least a half and share - 1 rounds once, where the
others’ sum carries each class’s share of the partition’s rounding, which left an unconfident
label’s entry two to three times as far from float64. Every head’s label entry is this one rule
(label_derivative, dew.objectives.lm.chunked).
label_logit
Section titled “label_logit”def label_logit(logits: jax.Array, labels: jax.Array) -> jax.ArrayEach row’s logit at its label over the last axis, 0 for a label outside the row’s columns, which picks none: a padding id past the vocabulary, or another tile’s or shard’s column.
label_terms
Section titled “label_terms”label_terms = contracted('test_the_label_terms_hold_every_order')(declared_tangent(Protected(), _label_terms, _label_terms_tangent))log_probs
Section titled “log_probs”def log_probs(logits: jax.Array, where: jax.Array | None = None) -> jax.ArrayThe normalized log probabilities over the last axis (jax.nn.log_softmax).
partition
Section titled “partition”def partition(logits: jax.Array) -> tuple[jax.Array, jax.Array, jax.Array]Each row’s maximum over the last axis, the sum of its exponentials scaled to it, and those
exponentials: the terms every whole-row label loss takes its log partition from (target_loss;
cross_entropy, dew.objectives.lm.chunked’s whole head).
The maximum is finite, as jax.nn.logsumexp takes one: an infinite maximum shifts by 0, so a row
holding +inf scores +inf, and an all -inf row’s partition is -inf, never inf - inf.
probabilities
Section titled “probabilities”def probabilities(logits: jax.Array, where: jax.Array | None = None) -> jax.ArrayThe probabilities over the last axis (jax.nn.softmax). Their derivative is softmax’s,
p (dz - sum p dz), but at the most probable class, which takes the other classes’ derivative
summed, negated, so a row’s derivative sums to zero: softmax’s own subtracts two near-equal terms
there once its probability nears one, keeping only their rounding (as a label’s p - 1 does,
arXiv:2610.31642, eq. 4), while the others’ sum keeps their precision, in either mode.
ranked_probability
Section titled “ranked_probability”def ranked_probability( logits: jax.Array, targets: jax.Array, where: jax.Array | None = None,) -> jax.ArrayThe ranked probability score of classes that are ordered levels: the squared gaps between the
forecast’s and the target’s cumulative distributions over the K - 1 cuts between a row’s K levels,
sum_k (P_k - T_k)^2 / (K - 1) over the last axis, the levels where holds leading the row.
rescaled
Section titled “rescaled”def rescaled(maximum: jax.Array, total: jax.Array, larger: jax.Array) -> jax.ArrayA sum of exponentials moved from its own maximum to the larger one, as a blockwise partition combines its blocks: where the two agree the shift is exactly 0 and the sum unchanged, its derivative still the moved sum’s, so two blocks whose maxima tie each move by their own maximum’s tangent less the larger’s; two sums whose maximum is -inf add as they are rather than turn NaN.
segment_log_partition
Section titled “segment_log_partition”def segment_log_partition( logits: jax.Array, segments: jax.Array, count: int,) -> tuple[jax.Array, jax.Array]The log partition [count] of each segment of a ragged row of logits [C], segments naming
each entry’s (count for an entry of none), and whether each segment holds an entry: its maximum,
held off the gradient as jax.nn.logsumexp holds it, plus the log of its exponentials’ sum scaled to
it, a segment of no entry’s log partition 0. An entry’s log probability within its segment is its
logit less its segment’s log partition.
smoothed
Section titled “smoothed”def smoothed( targets: jax.Array, where: jax.Array | None = None, share: float = 0.0,) -> jax.Arraytargets over the last axis with share of their mass spread evenly over the classes where
holds, (1 - share) t_k + share / K for a row of K classes (optax.smooth_labels), and nothing on
the others; integer targets in the floating type a Python number promotes them to (float64 under
x64).
soft_cross_entropy
Section titled “soft_cross_entropy”def soft_cross_entropy( logits: jax.Array, targets: jax.Array, where: jax.Array | None = None,) -> jax.Array-sum_k t_k log p_k over the last axis against the distribution targets, a class of no target
mass contributing nothing, whatever its log probability. Its derivative in logits, p T - t for
target mass T, is p (T - t) - t (1 - p), each class’s 1 - p by the label’s rule (label_entry:
past a half, the other classes’ probability summed), and the class of the most target mass’s T - t
the other classes’ targets summed: a one-hot target’s derivative is cross_entropy’s, bit for bit
(arXiv:2610.31642, eq. 7), where p - t rounds to 0, and targets as confident as the logits keep its
sum 0.
spherical
Section titled “spherical”def spherical( logits: jax.Array, targets: jax.Array, where: jax.Array | None = None,) -> jax.ArrayThe spherical score as a charge, 1 - (t . p) / |p| over the last axis, between 0 and 1.
target_loss
Section titled “target_loss”def target_loss(peak: jax.Array, mass: jax.Array, target: jax.Array) -> jax.ArrayA label’s cross entropy from its row’s maximum peak, the sum of exponentials scaled to it
mass, and its label’s logit: log(mass) - (target - peak), as cross_entropy takes it, so neither
term is rounded at the logits’ magnitude before the subtraction that cancels it does.
token_log_probs
Section titled “token_log_probs”def token_log_probs(logits: jax.Array, tokens: jax.Array) -> jax.Arraylog_softmax(logits)[..., tokens], reduced in at least fp32 (cross_entropy’s negation).
tokens is logits.shape[:-1] of ids and the result has that shape.