Skip to content

dew.data.text

Tokenizers for language-model data, utf-8 bytes or any HF tokenizer, and the text record steps: tokenizing, chunking, dropping, normalizing and filtering text, and curating it through DataTrove’s filters (Language, Quality, Classifier, Decontaminated) and its MinHash deduplication (NearDuplicates, which Dataset.snapshot applies; the curation extra).

import dew.data stays cheap. Nothing here imports transformers at module scope, and ByteTokenizer needs nothing but numpy. HFTokenizer loads its tokenizer on first use, so a host without the hub cache still imports dew.data.text (and everything that re-exports it) fine.

NameSummary
RENAMED_TOKENIZERSTokenizer classes a release ships as remote code that are a transformers class under another name: load_tokenizer loads such a release with that class, and runs none of the release’s code.
PinnedTokenizerA Hugging Face tokenizer resolved once to immutable files (pin_tokenizer): the snapshot its identity hashes and the directory it reads its vocabulary and templates from, which are one revision by construction.
pin_tokenizerResolve name once (pin_files), with every file its loader can read hashed.
pin_filesResolve name once: a local directory, or a hub repo (optionally repo@revision) downloaded at the commit its revision names, with every file of patterns hashed: the snapshot an identity holds (its repo, the commit and each file’s hash) and the directory to read those files from, one commit by construction.
load_tokenizerThe HF tokenizer of a hub repo or local directory, loaded once per process, or once per contents, the digest of a directory’s files where a caller pinned them (PinnedTokenizer).
TokenArrayHolds ids as an array rather than as a list.
TokenizerWhat a text vocabulary offers, as ByteTokenizer and HFTokenizer do: text to ids, with or without the vocabulary’s special tokens, ids back to text, and the id a sequence starts with, or None.
OffsetsA vocabulary that says which characters each token of a text covers (offsets): a layout that encodes its prompt whole locates its spans by them (dew.decision.renderer).
ByteTokenizerEncodes text as one id per utf-8 byte, over a vocabulary of 256.
HFTokenizerA Hugging Face tokenizer, resolved from its hub name or local directory on first use.
SavedTokenizerA tokenizer object Dew names by the files it saves (tokenizer_name), such as a transformers tokenizer: a preset or a step given one reads it by that name.
tokenizer_nameA tokenizer as the name Dew’s steps carry: a name as it is, and a transformers tokenizer object saved once to a directory named for its files’ hash, which every worker reads and every run hashes alike.
tokenizeA record step that encodes the text in field to token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None.
TokenizeThe text in field as token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None.
ChunksA document’s token fields cut into pieces of at most length, each starting length - overlap after the last (flat_map), before pack, which refuses a document longer than its window.
DroppedThe text in field emptied with probability p, as classifier-free guidance trains its unconditioned branch.
NormalizedThe text in field in Unicode normal form form (unicodedata.normalize), so text written with composed, decomposed or compatibility characters (a ligature, a full-width digit) reads and tokenizes alike; other fields pass through.
MinCharsWhether the text in field holds at least count characters.
CuratedA record step over one of DataTrove’s document filters: the record where the filter keeps the text in field, else Rejected with the reason the filter names, or the step’s own.
LanguageThe text in field kept where DataTrove’s language identification scores it above min_score in one of languages (DataTrove’s codes; any language when none is named), over fastText’s lid.176 (backend="ft176") or GlotLID ("glotlid"), which DataTrove downloads on first use into its asset cache.
QUALITY_RULESThe heuristic quality rules Quality runs: Gopher’s (Rae et al.
QualityThe text in field kept where DataTrove’s rule passes it: Gopher’s, C4’s or FineWeb’s quality filter at DataTrove’s own thresholds, any of which settings replaces as (parameter, value) pairs of the filter’s constructor.
ClassifierThe text in field kept by a fastText classifier, DataTrove’s fastText filter over the model at model (a path or a URL, which DataTrove downloads on first use): where one of keep’s (label, score) pairs scores at least its score, or where none of remove’s does.
DecontaminatedThe text in field kept unless it shares an n-gram with one of the evaluation texts against, by DataTrove’s n-gram decontamination: both normalized, split into words and hashed as its filter reads documents, and a record with a shared n-gram Rejected("contaminated").
NearDuplicatesMinHash’s near-duplicate rule, as DataTrove applies it (Broder 1997; FineWeb’s defaults): a record’s text in field is shingled into n-word n-grams, buckets bands of hashes minhashes each sign it, and two records whose signatures agree in a whole band are near-duplicates, which for Jaccard similarity s happens with probability 1 - (1 - s^hashes)^buckets: about 0.72 at half odds by default.
near_duplicates_removedThe records each pass of records opens, one sealed snapshot’s in one order, with each group of near-duplicates (near) kept as one record by DataTrove’s four MinHash stages: signatures over a pass, band matches bucket by bucket, their clusters, and the filter that drops every record a cluster removes over a second pass.
export_tokensdataset’s documents in field tokenized by tokenizer and written to url as a committed token store (ArrayRecord shards, each record a document’s ids under “text”, its manifest naming the tokenizer, vocabulary and EOS), which dew.data.sources.TokenFiles(url) reads back and presets.lm(ds, None, ...) packs.
train_tokenizerA tokenizer of vocab_size tokens trained on the text in field of dataset’s records, the first records where given, by the algorithm, normalization and special tokens of base (a Hub repo or directory: a BPE tokenizer’s learns merges, a unigram’s pieces), saved at path, which it returns for tokenize(path) and the lm and sft presets to read.
tokenizer_forBuilds the tokenizer name asks for.

attribute source

RENAMED_TOKENIZERS: Mapping[str, str] = {'DreamTokenizer': 'Qwen2Tokenizer'}

Tokenizer classes a release ships as remote code that are a transformers class under another name: load_tokenizer loads such a release with that class, and runs none of the release’s code.

dataclass source

class PinnedTokenizer(snapshot: Mapping[str, object], directory: str)

A Hugging Face tokenizer resolved once to immutable files (pin_tokenizer): the snapshot its identity hashes and the directory it reads its vocabulary and templates from, which are one revision by construction.

def transformers() -> PreTrainedTokenizerBase

The transformers tokenizer of these files, whose chat template renders conversations, as transformers’ class cls where a reader names one (a CLIP tower’s slow CLIPTokenizer). A directory is read where it is, so the load reaches no hub, and it is the one load every reader of these files shares (load_tokenizer): keyed by their hashes, so a directory whose files changed since a load loads again.

function source

def pin_tokenizer(name: str, *, local_files_only: bool = False) -> PinnedTokenizer

Resolve name once (pin_files), with every file its loader can read hashed.

function source

def pin_files(
name: str,
patterns: Sequence[str],
*,
local_files_only: bool = False,
) -> tuple[Mapping[str, object], str]

Resolve name once: a local directory, or a hub repo (optionally repo@revision) downloaded at the commit its revision names, with every file of patterns hashed: the snapshot an identity holds (its repo, the commit and each file’s hash) and the directory to read those files from, one commit by construction. The Hub cache keeps a commit’s files in a directory named for the commit, which is the revision recorded; offline (HF_HUB_OFFLINE), or with local_files_only, the name resolves to the commit the local cache holds.

function source

def load_tokenizer(
name: str,
*,
revision: str | None = None,
local_files_only: bool = False,
contents: str | None = None,
) -> PreTrainedTokenizerBase

The HF tokenizer of a hub repo or local directory, loaded once per process, or once per contents, the digest of a directory’s files where a caller pinned them (PinnedTokenizer).

Every Dew path that needs an HF tokenizer loads it here, so one name at one revision is one object however many readers share it. A release whose tokenizer is a transformers class renamed in its own remote code loads as that class (RENAMED_TOKENIZERS); a reader that needs one class of transformers’ own names it (cls, a CLIP tower’s slow CLIPTokenizer, ledger 3013), a load of its own.

class source

class TokenArray(Protocol)

Holds ids as an array rather than as a list.

numpy’s and jax’s arrays both hand their ids over through tolist, and a decode takes either beside a plain sequence, because a sampler returns a row of a device array while a caller writes a list. A row has a length; the scalars ArrayLike also covers are one id and have none.

def tolist() -> list[int]

class source

class Tokenizer(Protocol)

What a text vocabulary offers, as ByteTokenizer and HFTokenizer do: text to ids, with or without the vocabulary’s special tokens, ids back to text, and the id a sequence starts with, or None.

def encode(text: str, *, add_special_tokens: bool = True) -> list[int]
def decode(ids: ArrayLike | Sequence[int]) -> str

class source

class Offsets(Protocol)

A vocabulary that says which characters each token of a text covers (offsets): a layout that encodes its prompt whole locates its spans by them (dew.decision.renderer).

def offsets(text: str) -> tuple[list[int], list[int], list[int]]

text’s ids without special tokens, and each token’s first character and the character after its last.

class source

class ByteTokenizer()

Encodes text as one id per utf-8 byte, over a vocabulary of 256.

It needs no training and no download, so it is the default for small corpora and for tests. Its eos id is 255, a byte that never occurs in utf-8 text, and it has no bos id. Decoding inverts encoding for any unicode text. Ids that are not valid utf-8, which a model can generate, decode with replacement characters.

name

The name a run records for this vocabulary, which tokenizer_for resolves back to it.

def dependencies() -> Mapping[str, object]
def encode(text: str, *, add_special_tokens: bool = True) -> list[int]

Return the text’s utf-8 bytes as ids.

add_special_tokens has no effect, because this vocabulary has no special tokens to insert.

def decode(ids: ArrayLike | Sequence[int]) -> str
def offsets(text: str) -> tuple[list[int], list[int], list[int]]

text’s bytes, each covering its character.

def document(text: str, *, eos: bool) -> list[int]

text as one document’s ids, its utf-8 bytes, ended by the EOS with eos where it holds any.

class source

class HFTokenizer(name: str, *, local_files_only: bool = False)

A Hugging Face tokenizer, resolved from its hub name or local directory on first use.

It resolves its name once (pinned), and its identity (dependencies) and every encoding read that one resolution, here and in any process it is pickled to, so a hub name that moves, or another revision of it a process already loaded, never encodes rows under the identity of this one. Because it resolves lazily, import dew.data.text (and import dew.data) does not import transformers or look anything up on the hub until a caller uses the tokenizer.

pinned: PinnedTokenizer

The name resolved once to immutable files (pin_tokenizer).

eos_id: int | None

The eos token’s id, or None for a tokenizer that declares none.

Raises TypeError when the tokenizer reports something other than one id.

bos_id: int | None

The id the tokenizer starts a sequence with, or None where it has none.

Raises TypeError when the tokenizer reports something other than one id.

def dependencies() -> Mapping[str, object]
def encode(text: str, *, add_special_tokens: bool = True) -> list[int]
def offsets(text: str) -> tuple[list[int], list[int], list[int]]

text’s ids and each token’s character span, by the fast tokenizer’s offsets; a slow tokenizer keeps none and is refused (TypeError).

def decode(ids: ArrayLike | Sequence[int]) -> str
def document(text: str, *, eos: bool) -> list[int]

text as one document’s ids, by the tokenizer’s own special tokens (its BOS where it adds one), ended by its EOS exactly once with eos where it holds any, and by none without, whether or not the tokenizer’s post-processor appends its own, as T5’s and an EOS-inserting Llama’s do. Raises ValueError for eos on a tokenizer without one.

def save_pretrained(directory) -> None

Write this tokenizer’s own files into an export directory.

The tokenizer writes tokenizer.json, tokenizer_config.json and its vocabulary itself, so the export copies no files by hand and the result loads in anything that reads the HF layout.

class source

class SavedTokenizer(Protocol)

A tokenizer object Dew names by the files it saves (tokenizer_name), such as a transformers tokenizer: a preset or a step given one reads it by that name.

def save_pretrained(save_directory: str, /) -> tuple[str, ...]

function source

def tokenizer_name(tokenizer: str | SavedTokenizer) -> str

A tokenizer as the name Dew’s steps carry: a name as it is, and a transformers tokenizer object saved once to a directory named for its files’ hash, which every worker reads and every run hashes alike.

function source

def tokenize(
tokenizer: str | SavedTokenizer = 'byte',
*,
field: str = 'text',
eos: bool | None = None,
) -> Tokenize

A record step that encodes the text in field to token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None. tokenizer is “byte” (Dew’s utf-8 vocabulary), a Hub repo or directory, or a transformers tokenizer object.

dataclass source

class Tokenize(tokenizer: str = 'byte', field: str = 'text', eos: bool | None = False)

The text in field as token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None.

vocabulary: ByteTokenizer | HFTokenizer

The tokenizer tokenizer names, which resolves its name once (HFTokenizer.pinned): the step’s identity and its encoding both read that one resolution, in this process and in any worker it is sent to.

costly: bool

Whether encoding costs more than reading the ids back from a cache (dew.data.pipeline.Costly): a learned tokenizer’s does, Dew’s bytes’ do not, as tools/benchmark_data.py caching measures (docs/design/data.md, section 8).

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Token ids of each document’s length under “text”, by tokenizer, vocab_size of them.

def dependencies() -> Mapping[str, object]
def map(element: Batch) -> Batch

dataclass source

class Chunks(length: int, *, fields: tuple[str, ...] = TOKEN_FIELDS, overlap: int = 0)

A document’s token fields cut into pieces of at most length, each starting length - overlap after the last (flat_map), before pack, which refuses a document longer than its window. With overlap=1 every token’s next-token target stays in one piece: pack to length with seq_len=length.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Pieces of the documents, each of fields up to length long.

def flat_map(element: Batch) -> Sequence[Batch]

dataclass source

class Dropped(field: str = 'caption', p: float = 0.1)

The text in field emptied with probability p, as classifier-free guidance trains its unconditioned branch.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class Normalized(
form: Literal['NFC', 'NFKC', 'NFD', 'NFKD'] = 'NFKC',
*,
field: str = 'text',
)

The text in field in Unicode normal form form (unicodedata.normalize), so text written with composed, decomposed or compatibility characters (a ligature, a full-width digit) reads and tokenizes alike; other fields pass through.

def map(element: Batch) -> Batch

dataclass source

class MinChars(count: int, field: str = 'text')

Whether the text in field holds at least count characters.

class source

class Curated(abc.ABC)

A record step over one of DataTrove’s document filters: the record where the filter keeps the text in field, else Rejected with the reason the filter names, or the step’s own. The filter is built once a process (curator), never pickled with the step, and the step’s identity records DataTrove’s release, which pins the filter’s rules and the models it names.

def curator() -> BaseFilter

The DataTrove filter the step runs.

def reads() -> str

What the filter reads beyond the step’s own fields, by its content (a model’s SHA-256), which the process’s filter is keyed by beside the step (_curator); none by default.

def dependencies() -> Mapping[str, object]

dataclass source

class Language(
languages: tuple[str, ...] = (),
min_score: float = 0.65,
backend: Literal['ft176', 'glotlid'] = 'ft176',
field: str = 'text',
)

The text in field kept where DataTrove’s language identification scores it above min_score in one of languages (DataTrove’s codes; any language when none is named), over fastText’s lid.176 (backend="ft176") or GlotLID ("glotlid"), which DataTrove downloads on first use into its asset cache.

def curator() -> BaseFilter

attribute source

QUALITY_RULES = ('gopher', 'c4', 'fineweb')

The heuristic quality rules Quality runs: Gopher’s (Rae et al. 2021), C4’s (Raffel et al. 2020) and FineWeb’s (Penedo et al. 2024), as DataTrove implements them.

dataclass source

class Quality(
rule: Literal['gopher', 'c4', 'fineweb'],
settings: tuple[tuple[str, float | int | bool | str | tuple[str, ...] | None], ...] = (),
field: str = 'text',
)

The text in field kept where DataTrove’s rule passes it: Gopher’s, C4’s or FineWeb’s quality filter at DataTrove’s own thresholds, any of which settings replaces as (parameter, value) pairs of the filter’s constructor. A rejected record is counted under the rule’s reason (gopher_short_doc, too_few_sentences, line_punct_ratio, …).

settings: tuple[tuple[str, float | int | bool | str | tuple[str, ...] | None], ...]

The filter’s constructor arguments, a sequence (Gopher’s stop_words) as a tuple.

def curator() -> BaseFilter

dataclass source

class Classifier(
model: str,
keep: tuple[tuple[str, float], ...] = (),
remove: tuple[tuple[str, float], ...] = (),
field: str = 'text',
)

The text in field kept by a fastText classifier, DataTrove’s fastText filter over the model at model (a path or a URL, which DataTrove downloads on first use): where one of keep’s (label, score) pairs scores at least its score, or where none of remove’s does. The model is read once a step, to a copy named by its SHA-256 (pinned), which the filter loads and the step’s identity records, so a model replaced at its path later reaches neither.

pinned: Path

The model’s bytes as this step first read them, under the system’s temporary directory by their SHA-256: a local file’s own, a URL’s as DataTrove downloads it once into its asset cache, which holds a local path’s first bytes too.

def reads() -> str
def curator() -> BaseFilter
def dependencies() -> Mapping[str, object]

dataclass source

class Decontaminated(against: tuple[str, ...], n: int = 12, field: str = 'text')

The text in field kept unless it shares an n-gram with one of the evaluation texts against, by DataTrove’s n-gram decontamination: both normalized, split into words and hashed as its filter reads documents, and a record with a shared n-gram Rejected("contaminated"). The evaluation texts are indexed once, with no evaluation harness, under the system’s temporary directory by what they hold.

def curator() -> BaseFilter

dataclass source

class NearDuplicates(
n: int = 5,
buckets: int = 14,
hashes: int = 8,
field: str = 'text',
)

MinHash’s near-duplicate rule, as DataTrove applies it (Broder 1997; FineWeb’s defaults): a record’s text in field is shingled into n-word n-grams, buckets bands of hashes minhashes each sign it, and two records whose signatures agree in a whole band are near-duplicates, which for Jaccard similarity s happens with probability 1 - (1 - s^hashes)^buckets: about 0.72 at half odds by default.

function source

def near_duplicates_removed(
records: Callable[[], Iterator[Batch]],
near: NearDuplicates,
) -> tuple[Iterator[Batch], int]

The records each pass of records opens, one sealed snapshot’s in one order, with each group of near-duplicates (near) kept as one record by DataTrove’s four MinHash stages: signatures over a pass, band matches bucket by bucket, their clusters, and the filter that drops every record a cluster removes over a second pass. Returns the records kept, lazily, and how many the clusters remove.

function source

def export_tokens(
dataset: Dataset,
url: str,
tokenizer: str,
*,
field: str = 'text',
eos: bool = True,
rows_per_shard: int = 100000,
read: Read = 'streamed',
provision: bool = False,
plan_version: int | None = None,
) -> Mapping[str, object]

dataset’s documents in field tokenized by tokenizer and written to url as a committed token store (ArrayRecord shards, each record a document’s ids under “text”, its manifest naming the tokenizer, vocabulary and EOS), which dew.data.sources.TokenFiles(url) reads back and presets.lm(ds, None, ...) packs. Each document is ended by the tokenizer’s EOS with eos, read by data plan plan_version (dew.data.plan.PLAN_VERSIONS; this Dew’s where None). Returns the store’s manifest.

function source

def train_tokenizer(
dataset: Dataset,
base: str,
path: str,
*,
vocab_size: int,
field: str = 'text',
records: int | None = None,
) -> str

A tokenizer of vocab_size tokens trained on the text in field of dataset’s records, the first records where given, by the algorithm, normalization and special tokens of base (a Hub repo or directory: a BPE tokenizer’s learns merges, a unigram’s pieces), saved at path, which it returns for tokenize(path) and the lm and sft presets to read. A vocab_size smaller than the base’s alphabet and special tokens, which its algorithm always keeps, is refused, and so is a base without a fast tokenizer, which has no algorithm to train.

function source

def tokenizer_for(
name: str,
*,
local_files_only: bool = False,
) -> ByteTokenizer | HFTokenizer

Builds the tokenizer name asks for.

byte is dew’s own utf-8 vocabulary; any other name is the HF tokenizer of that repo or local directory. Resolving names here alone keeps a training run and an export of what it trained reading one name one way. An export passes local_files_only, since writing a checkpoint out is no reason to reach the hub.