Skip to content

dew.data.processors

Turns raw audio into the arrays a batch carries.

This runs inside grain’s workers, so the device only ever sees ready tensors. transformers is imported when a step runs, not on import. Captions are tokenized by the condition encoder that reads them (dew.inputs).

NameSummary
SampledStates the sampling rate a feature extractor expects its audio at.
FeatureExtractorA Hugging Face audio feature extractor, as AutoFeatureExtractor loads one: it turns waveforms into the arrays its model reads, the first of which it names first.
sampling_rateThe rate a feature extractor reads its audio at: the one it states (Sampled), or 16 kHz.
LogMelA waveform’s log mel frames [frames, n_mels] float32, from transformers’ shared STFT and mel filters (audio_utils, which its feature extractors compute with): centred frames of a reflect-padded waveform under a periodic Hann window, HTK mel filters without a norm, the magnitude to power, floored at floor before the natural log.

class source

class Sampled(Protocol)

States the sampling rate a feature extractor expects its audio at.

Whisper’s and wav2vec2’s extractors do. One that states none is read at 16 kHz, the rate every released speech model here is trained on.

class source

class FeatureExtractor(Protocol)

A Hugging Face audio feature extractor, as AutoFeatureExtractor loads one: it turns waveforms into the arrays its model reads, the first of which it names first.

function source

def sampling_rate(extractor: FeatureExtractor) -> int

The rate a feature extractor reads its audio at: the one it states (Sampled), or 16 kHz.

dataclass source

class LogMel(
sampling_rate: int,
n_fft: int,
hop_length: int,
win_length: int,
n_mels: int,
f_min: float = 0.0,
f_max: float | None = None,
power: float = 1.0,
floor: float = 1e-07,
)

A waveform’s log mel frames [frames, n_mels] float32, from transformers’ shared STFT and mel filters (audio_utils, which its feature extractors compute with): centred frames of a reflect-padded waveform under a periodic Hann window, HTK mel filters without a norm, the magnitude to power, floored at floor before the natural log. That is torchaudio’s MelSpectrogram at its defaults and power, as MiMo-V2.6’s audio tokenizer reads it at 24 kHz and 128 mels (MiMo-Audio-Tokenizer’s mel_spectrogram, MiMoAudioTokenizer.mel).