Skip to content

dew.data.sources.av_utils

Frames and sound of a media file over a time window, which video.PickClip draws once for both.

moviepy is the reader because it is what the av extra installs and it reads from wheels alone; it drives ffmpeg, so nothing here holds a decoder open across records.

NameSummary
FPSThe frame rate a clip is read at where none is asked for (video.DecodeVideo).
media_infopath’s duration in seconds and whether it holds a video and an audio stream, from ffmpeg’s reading of the container, so a sound file answers too.
audio_windowThe sound of path from start for duration seconds, mono at sample_rate, float32 in [-1, 1], and whether the track holds each sample: silence and False before the track begins, whether start is before the file’s or the track starts late, and after it ends.
read_framescount frames of path at start + i / fps seconds, as uint8 [count, H, W, 3] RGB.

attribute source

FPS = 25.0

The frame rate a clip is read at where none is asked for (video.DecodeVideo).

function source

def media_info(path: str) -> tuple[float, bool, bool]

path’s duration in seconds and whether it holds a video and an audio stream, from ffmpeg’s reading of the container, so a sound file answers too.

function source

def audio_window(
path: str,
start: float,
duration: float,
sample_rate: int,
) -> tuple[np.ndarray, np.ndarray]

The sound of path from start for duration seconds, mono at sample_rate, float32 in [-1, 1], and whether the track holds each sample: silence and False before the track begins, whether start is before the file’s or the track starts late, and after it ends.

ffmpeg does the seek, the resampling and the downmix, through the binary moviepy drives. moviepy’s own audio reader asks for stereo at 44.1 kHz whatever the track is, and ffmpeg’s mono-to-stereo upmix takes 3 dB off a mono track; sampling that stream at another rate then picks the nearest 44.1 kHz sample, which aliases and is never sample-exact.

The samples are placed by the timestamp of the first one decoded, read a second before start: cut by ffmpeg’s own seek instead, an AVI’s track started 73 or 261 samples late in 2 of 80 windows and a Matroska one up to 464 samples off, and a track that starts after the video came back shifted to the window’s start. Placed, all 80 were sample-exact.

function source

def read_frames(path: str, start: float, count: int, fps: float) -> np.ndarray

count frames of path at start + i / fps seconds, as uint8 [count, H, W, 3] RGB.

Each frame is asked for by its own time: the reader seeks to the first and steps to the rest, where a subclip’s iterator enumerates times from a float duration and can come up one short.