Skip to content

dew.rl.distributions

DreamerV3’s output transforms and distributions on plain arrays (arXiv 2301.04104 and its code, danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, embodied/jax/nets.py and outs.py).

symlog compresses a target’s magnitude and symexp inverts it. A two-hot distribution over the bins symexp(linspace(-20, 20, 255)) predicts a scalar of any scale as its bins’ expectation, summed from both ends toward the middle so uniform probabilities predict zero exactly, and is trained by the cross-entropy of a target’s two-hot encoding, its weight split between its two neighbouring bins. A categorical latent mixes its softmax with 1% uniform (unimix) and is drawn one-hot, its gradient passed straight through the probabilities. A continuous policy’s normal reads its per-coordinate terms here (normal_log_density, normal_entropy), and a tanh-squashed one its log-Jacobian too.

NameSummary
normal_entropyEach coordinate’s entropy of a normal of scale std, before any sum.
normal_log_densityEach coordinate’s log-density of x under the normal N(mean, std), before any sum: a diagonal normal’s log-likelihood is their sum, a squashed one’s adds its transform’s log-Jacobian first (tanh_log_jacobian).
onehot_sampleA one-hot draw from the categorical log_probs, its gradient the probabilities’ (straight through).
symexpsign(x) (exp(|x|) - 1), symlog’s inverse.
symlogsign(x) log(1 + |x|).
tanh_log_jacobianEach coordinate’s log(1 - tanh(raw)^2), the log-Jacobian of squashing raw by tanh, in its stable form 2 (log 2 - raw - softplus(-2 raw)), finite where tanh(raw) rounds to one.
twohot_binsThe two-hot bins: symexp of count points evenly spaced over [-limit, limit], the negative half mirrored so the bins are exactly symmetric (embodied/jax/heads.py:132-144).
twohot_lossThe cross-entropy of softmax(logits) against target’s two-hot encoding over bins: its weight on the bins either side of it, each the other’s share of the distance between them, all of it on the first or last bin past the range (outs.py:311-323).
twohot_meanThe expectation of bins under softmax(logits), summed outward-in from both ends to the middle bin, so symmetric bins under uniform probabilities give exactly zero (outs.py:285-309).
unimix_log_probsA categorical’s log-probabilities, its softmax mixed with unimix of the uniform: without the mixture the log-softmax itself, finite wherever the logits are, where the log of a probability that underflowed would be -inf.

function source

def normal_entropy(std: jax.Array) -> jax.Array

Each coordinate’s entropy of a normal of scale std, before any sum.

function source

def normal_log_density(x: jax.Array, mean: jax.Array, std: jax.Array) -> jax.Array

Each coordinate’s log-density of x under the normal N(mean, std), before any sum: a diagonal normal’s log-likelihood is their sum, a squashed one’s adds its transform’s log-Jacobian first (tanh_log_jacobian).

function source

def onehot_sample(key: jax.Array, log_probs: jax.Array) -> jax.Array

A one-hot draw from the categorical log_probs, its gradient the probabilities’ (straight through).

function source

def symexp(x: jax.Array) -> jax.Array

sign(x) (exp(|x|) - 1), symlog’s inverse.

function source

def symlog(x: jax.Array) -> jax.Array

sign(x) log(1 + |x|).

function source

def tanh_log_jacobian(raw: jax.Array) -> jax.Array

Each coordinate’s log(1 - tanh(raw)^2), the log-Jacobian of squashing raw by tanh, in its stable form 2 (log 2 - raw - softplus(-2 raw)), finite where tanh(raw) rounds to one.

function source

def twohot_bins(count: int = 255, limit: float = 20.0) -> jax.Array

The two-hot bins: symexp of count points evenly spaced over [-limit, limit], the negative half mirrored so the bins are exactly symmetric (embodied/jax/heads.py:132-144).

function source

def twohot_loss(logits: jax.Array, target: jax.Array, bins: jax.Array) -> jax.Array

The cross-entropy of softmax(logits) against target’s two-hot encoding over bins: its weight on the bins either side of it, each the other’s share of the distance between them, all of it on the first or last bin past the range (outs.py:311-323).

function source

def twohot_mean(logits: jax.Array, bins: jax.Array) -> jax.Array

The expectation of bins under softmax(logits), summed outward-in from both ends to the middle bin, so symmetric bins under uniform probabilities give exactly zero (outs.py:285-309).

function source

def unimix_log_probs(logits: jax.Array, unimix: float = 0.01) -> jax.Array

A categorical’s log-probabilities, its softmax mixed with unimix of the uniform: without the mixture the log-softmax itself, finite wherever the logits are, where the log of a probability that underflowed would be -inf.