dew.data.images
Image record steps: decoding, filters, resizing, crops, captions and augmentation, on the host or on the accelerator.
| Name | Summary |
|---|---|
import_opencv | Import OpenCV in the thread that opens a loader, before its readers start. |
resize_image | image at size, a square side or (height, width); area interpolation where it shrinks along either side, cubic where it only grows. |
fetch | A record step (filter_map) that fetches the image at the record’s url field into to as bytes, in one attempt of at most timeout once its host is resolved (Fetch); a failed fetch is counted and dropped, a transient one fetched again on a later pass. |
DecodeImage | The image in field as RGB uint8 pixels under to, including prepared ArrayRecord pixels, labels and captions; bytes that are no image are counted and dropped. |
MinSide | Whether the pixels in field, [..., H, W, C], are at least size on their shorter side. |
MaxAspect | Whether the pixels in field are at most ratio times as long as wide. |
NotFlat | Whether the pixels in field hold more than one value. |
Interpolation | |
resized_to | image at height x width by OpenCV’s method interpolation; where None, area where it shrinks along either side and bicubic where it only grows. |
Resize | The pixels in field resized to size, a square side or (height, width), by OpenCV’s method interpolation, or by area down and bicubic up where None. |
Fit | The pixels in field fitted into a size square whole: the longer side resized to size (area down, bicubic up), the shorter in proportion, rounded, and the rest padded on both sides with fill, the odd pixel at the bottom or right. |
CenterCrop | The central size square of the pixels in field; a smaller image is first resized up so its shorter side is size. |
RandomCrop | A crop of scale of the area resized to size, optionally flipped and colour-jittered. |
Bucket | The pixels in field resized to cover the (height, width) of sizes nearest their aspect ratio and cut to it at the centre, with that size’s index under to, so records of one bucket stack. |
bucket | A record step that resizes the pixels in field to cover the (height, width) of sizes nearest their aspect ratio and cuts that size from the centre, writing its index under to; a batch grouped by to stacks images of one shape. |
RandomResizedCrop | torchvision’s RandomResizedCrop: a crop of scale of the area and ratio of aspect, drawn from the record’s generator, resized to a size square; after ten draws that do not fit, the central crop of the nearest aspect. |
RandomMask | An inpainting mask for each image in field, drawn as Diffusers’ inpainting trainer draws one (examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py’s random_mask at 86ef420b1, its default ratio=1): a size drawn uniformly up to ratio of each side, a centre where half of it stays within the image, and by a coin a rectangle or an ellipse there, drawn by PIL as that script draws it. |
Flip | The pixels in field mirrored left to right with probability p. |
ColorJitter | torchvision’s float ColorJitter of the pixels in field: brightness, contrast and saturation factors drawn from [1 - x, 1 + x], applied in a random order, rounded once. |
ImageProcessor | A Hugging Face image processor, as Processor calls it. |
Augmentation | An albumentations pipeline (Compose), as Albumentations seeds and calls it. |
Processor | A Hugging Face image processor’s pixel_values for the image in field, under to. |
Albumentations | An albumentations pipeline on the image in field, seeded from the record’s generator through Compose.set_random_seed. |
Torchvision | A torchvision v2 transform on the image in field (HWC uint8 in and out), seeded from the record’s generator. |
CAPTION_COLUMNS | The fields a record’s own caption is read from, the first it holds (Caption). |
Caption | A record’s caption and int32 numeric labels, from columns or a class template. |
RandomCaption | A caption template drawn from the record’s generator. |
caption | A record step that writes “caption”: template formatted with the record’s fields (“{label}” is the class name classes gives its label), or with none, the first of columns. |
ConfigureCaption | |
AugmentationKey | The key DeviceAugment augments the record with, drawn from the record’s generator. |
CAPTIONS | The field a diffusion objective keeps each batch’s captions under as host text (EncodeCaptions), which a metric or a reward reads with its own tokenizer (dew.eval.images.CLIPScore). |
EncodeCaptions | Each batch’s captions encoded by reader for the conditions that read them, or dropped without one; keep names a field to hold them beside as host text, which a metric reads against its own tokenizer (dew.eval.images.CLIPScore) and no compiled step takes (split_fields). |
EncodeFields | Each batch with the fields reader encodes from it (InputSpec.encode_fields), as EncodeCaptions does its captions. |
DeviceAugment | Crop, flip and jitter each image of the training batch inside the compiled step, from its record’s own key (AugmentationKey). |
import_opencv
Section titled “import_opencv”def import_opencv() -> NoneImport OpenCV in the thread that opens a loader, before its readers start.
The reader threads reach their first decode together, so each would
otherwise make the first import of cv2 at once. An import that fails in
one of them leaves the others the half-built module, which surfaces as
module 'cv2' has no attribute 'INTER_AREA' instead of the failure
itself. Imported here, a broken install raises its own error when the
loader opens. The import stays out of the module’s top so that reading
text never loads OpenCV.
resize_image
Section titled “resize_image”def resize_image(image: np.ndarray, size: int | tuple[int, int]) -> np.ndarrayimage at size, a square side or (height, width); area interpolation
where it shrinks along either side, cubic where it only grows.
def fetch( url: str = 'url', to: str = 'jpg', *, timeout: float = 10.0, max_bytes: int = 64 * 2 ** 20, threads: int | None = None,) -> FetchA record step (filter_map) that fetches the image at the record’s
url field into to as bytes, in one attempt of at most timeout once
its host is resolved (Fetch); a
failed fetch is counted and dropped, a transient one fetched again on a
later pass. threads fetches that many at once on the stage’s own pool, in
each loading worker, delivered in the order read; without it an indexed
read runs Loading.threads fetches at once in each worker. The bytes are
volatile: a published cache after the fetch keeps them for exact resume.
DecodeImage
Section titled “DecodeImage”class DecodeImage( field: str = 'jpg', to: str = 'image', *, at_least: int | None = None, rejects: bool = True,)The image in field as RGB uint8 pixels under to, including prepared ArrayRecord
pixels, labels and captions; bytes that are no image are counted and dropped. With
rejects=False, use map and fail on bad bytes. at_least bounds a reduced JPEG decode.
external: bool-
An image a record names by path is read from storage (
dew.data.pipeline.External).
DecodeImage.declared
Section titled “DecodeImage.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]RGB pixels under to, each image its own height and width, in place of the bytes in field.
DecodeImage.map
Section titled “DecodeImage.map”def map(element: Batch | bytes) -> Batch | RejectedMinSide
Section titled “MinSide”class MinSide(size: int, field: str = 'image')Whether the pixels in field, [..., H, W, C], are at least size on their shorter side.
MaxAspect
Section titled “MaxAspect”class MaxAspect(ratio: float, field: str = 'image')Whether the pixels in field are at most ratio times as long as wide.
NotFlat
Section titled “NotFlat”class NotFlat(field: str = 'image')Whether the pixels in field hold more than one value.
Interpolation
Section titled “Interpolation”resized_to
Section titled “resized_to”def resized_to( image: np.ndarray, height: int, width: int, method: Interpolation | None,) -> np.ndarrayimage at height x width by OpenCV’s method interpolation; where None, area where it
shrinks along either side and bicubic where it only grows. The one resize every image and video
step reads.
Resize
Section titled “Resize”class Resize( size: int | tuple[int, int], method: Interpolation | None = None, *, field: str = 'image',)The pixels in field resized to size, a square side or (height, width),
by OpenCV’s method interpolation, or by area down and bicubic up where None.
Resize.declared
Section titled “Resize.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Resize.map
Section titled “Resize.map”def map(element: Batch) -> Batchclass Fit(size: int, fill: int = 255, *, field: str = 'image')The pixels in field fitted into a size square whole: the longer side resized to size
(area down, bicubic up), the shorter in proportion, rounded, and the rest padded on both sides
with fill, the odd pixel at the bottom or right. A web image keeps all it shows, as the
retired online loader fitted it; Resize stretches it and CenterCrop cuts it instead.
Fit.declared
Section titled “Fit.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Fit.map
Section titled “Fit.map”def map(element: Batch) -> BatchCenterCrop
Section titled “CenterCrop”class CenterCrop(size: int, method: Interpolation = 'bicubic', *, field: str = 'image')The central size square of the pixels in field; a smaller image is
first resized up so its shorter side is size.
CenterCrop.declared
Section titled “CenterCrop.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The images size squares, as many channels as they held.
CenterCrop.map
Section titled “CenterCrop.map”def map(element: Batch) -> BatchRandomCrop
Section titled “RandomCrop”class RandomCrop( size: int, *, scale: tuple[float, float] = (0.8, 1.0), flip: bool = False, jitter: bool = False,)A crop of scale of the area resized to size, optionally flipped and colour-jittered.
RandomCrop.declared
Section titled “RandomCrop.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]RandomCrop.random_map
Section titled “RandomCrop.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchBucket
Section titled “Bucket”class Bucket( sizes: tuple[tuple[int, int], ...], method: Interpolation = 'area', field: str = 'image', to: str = 'bucket',)The pixels in field resized to cover the (height, width) of sizes nearest their aspect
ratio and cut to it at the centre, with that size’s index under to, so records of one bucket
stack.
Bucket.map
Section titled “Bucket.map”def map(element: Batch) -> Batchbucket
Section titled “bucket”def bucket( sizes: Sequence[tuple[int, int]], method: Interpolation = 'area', *, field: str = 'image', to: str = 'bucket',) -> BucketA record step that resizes the pixels in field to cover the (height, width) of sizes
nearest their aspect ratio and cuts that size from the centre, writing its index under to; a
batch grouped by to stacks images of one shape.
RandomResizedCrop
Section titled “RandomResizedCrop”class RandomResizedCrop( size: int, *, scale: tuple[float, float] = (0.08, 1.0), ratio: tuple[float, float] = (3 / 4, 4 / 3), method: Interpolation = 'bilinear', field: str = 'image',)torchvision’s RandomResizedCrop: a crop of scale of the area and
ratio of aspect, drawn from the record’s generator, resized to a size
square; after ten draws that do not fit, the central crop of the nearest
aspect.
RandomResizedCrop.declared
Section titled “RandomResizedCrop.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]RandomResizedCrop.random_map
Section titled “RandomResizedCrop.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchRandomMask
Section titled “RandomMask”class RandomMask(ratio: float = 1.0, *, field: str = 'image', to: str = 'mask')An inpainting mask for each image in field, drawn as Diffusers’ inpainting trainer draws one
(examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py’s random_mask at
86ef420b1, its default ratio=1): a size drawn uniformly up to ratio of each side, a centre where
half of it stays within the image, and by a coin a rectangle or an ellipse there, drawn by PIL as
that script draws it. It is written to to as [height, width, 1] float32: 1 inside, the region a
run paints, and 0 outside, the masked image’s pixels (dew.inputs.diffusion.latent_image_conditions).
A diffusion run over an inpainting pipeline reads it (DiffusionRunConfig).
RandomMask.declared
Section titled “RandomMask.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The facts with a mask of the images’ height and width beside them, where the facts size them.
RandomMask.random_map
Section titled “RandomMask.random_map”def random_map(element: Batch, rng: np.random.Generator) -> Batchclass Flip(p: float = 0.5, *, field: str = 'image')The pixels in field mirrored left to right with probability p.
Flip.declared
Section titled “Flip.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The images as their shape was: a mirror moves pixels, not the frame.
Flip.random_map
Section titled “Flip.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchColorJitter
Section titled “ColorJitter”class ColorJitter( brightness: float = JITTER.brightness, contrast: float = JITTER.contrast, saturation: float = JITTER.saturation, *, field: str = 'image',)torchvision’s float ColorJitter of the pixels in field: brightness,
contrast and saturation factors drawn from [1 - x, 1 + x], applied in a
random order, rounded once.
ColorJitter.declared
Section titled “ColorJitter.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The images as their shape was, uint8 still: the jitter changes their values alone.
ColorJitter.random_map
Section titled “ColorJitter.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchImageProcessor
Section titled “ImageProcessor”class ImageProcessor(Protocol)A Hugging Face image processor, as Processor calls it.
Augmentation
Section titled “Augmentation”class Augmentation(Protocol)An albumentations pipeline (Compose), as Albumentations seeds and calls it.
Augmentation.set_random_seed
Section titled “Augmentation.set_random_seed”def set_random_seed(seed: int) -> NoneProcessor
Section titled “Processor”class Processor( processor: ImageProcessor, *, field: str = 'image', to: str = 'pixel_values',)A Hugging Face image processor’s pixel_values for the image in field, under to.
Processor.map
Section titled “Processor.map”def map(element: Batch) -> BatchAlbumentations
Section titled “Albumentations”class Albumentations(transform: Augmentation, *, field: str = 'image')An albumentations pipeline on the image in field, seeded from the
record’s generator through Compose.set_random_seed. Each reading thread
seeds and calls its own copy of the pipeline, so no other thread’s seed,
from this stage or another holding the same pipeline, lands between the
two, and a record’s draw is the same at any thread count and after a
resume.
Albumentations.random_map
Section titled “Albumentations.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchTorchvision
Section titled “Torchvision”class Torchvision( transform: Callable[[torch.Tensor], torch.Tensor], *, field: str = 'image',)A torchvision v2 transform on the image in field (HWC uint8 in and
out), seeded from the record’s generator. torch draws from one generator a
process, so the transform runs one record at a time in each process, the
generator seeded inside that turn and restored after it.
Torchvision.random_map
Section titled “Torchvision.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchCAPTION_COLUMNS
Section titled “CAPTION_COLUMNS”CAPTION_COLUMNS = ('caption', 'text')The fields a record’s own caption is read from, the first it holds (Caption).
Caption
Section titled “Caption”class Caption( template: str | None = None, classes: tuple[str, ...] = (), columns: tuple[str, ...] = CAPTION_COLUMNS,)A record’s caption and int32 numeric labels, from columns or a class template.
Caption.declared
Section titled “Caption.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]A caption, a string of no fixed shape, in place of any the record held, beside the fields as they were.
Caption.map
Section titled “Caption.map”def map(element: Batch) -> BatchRandomCaption
Section titled “RandomCaption”class RandomCaption( template: str | None = None, classes: tuple[str, ...] = (), columns: tuple[str, ...] = CAPTION_COLUMNS, templates: tuple[str, ...] = (),)A caption template drawn from the record’s generator.
RandomCaption.random_map
Section titled “RandomCaption.random_map”def random_map(element: Batch, rng: np.random.Generator) -> Batchcaption
Section titled “caption”def caption( template: str | None = None, *, templates: tuple[str, ...] = (), classes: tuple[str, ...] = (), columns: tuple[str, ...] = CAPTION_COLUMNS,) -> Caption | RandomCaptionA record step that writes “caption”: template formatted with the
record’s fields (“{label}” is the class name classes gives its label),
or with none, the first of columns. With templates, a random step
instead draws one template per record; {} and {label} both name its class.
ConfigureCaption
Section titled “ConfigureCaption”class ConfigureCaption(caption: Caption)ConfigureCaption.declared
Section titled “ConfigureCaption.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]A caption, a string of no fixed shape, in place of any the record held, beside the rest.
AugmentationKey
Section titled “AugmentationKey”class AugmentationKey()The key DeviceAugment augments the record with, drawn from the record’s generator.
AugmentationKey.declared
Section titled “AugmentationKey.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]AugmentationKey.random_map
Section titled “AugmentationKey.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchCAPTIONS
Section titled “CAPTIONS”CAPTIONS = 'captions'The field a diffusion objective keeps each batch’s captions under as host text (EncodeCaptions),
which a metric or a reward reads with its own tokenizer (dew.eval.images.CLIPScore).
EncodeCaptions
Section titled “EncodeCaptions”class EncodeCaptions( reader: Callable[[Sequence[str]], Batch] | None = None, keep: str | None = None,)Each batch’s captions encoded by reader for the conditions that read them, or dropped without
one; keep names a field to hold them beside as host text, which a metric reads against its own
tokenizer (dew.eval.images.CLIPScore) and no compiled step takes (split_fields). A reader that
encodes a field of that name too is refused (ValueError), rather than one overwriting the other.
EncodeFields
Section titled “EncodeFields”class EncodeFields(reader: Callable[[Batch], Batch])Each batch with the fields reader encodes from it (InputSpec.encode_fields), as
EncodeCaptions does its captions.
DeviceAugment
Section titled “DeviceAugment”class DeviceAugment( size: int, *, crop_scale: tuple[float, float] = (1.0, 1.0), flip: bool = True, jitter: bool = False,)Crop, flip and jitter each image of the training batch inside the
compiled step, from its record’s own key (AugmentationKey).
DeviceAugment.declared
Section titled “DeviceAugment.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Each image cropped to a size square, its key spent.