graphragzen.preprocessing.preprocess.chunk

graphragzen.preprocessing.preprocess.chunk(inp, window_size=300, overlap=100)[source]

Chunk an sequence using a sliding window

Parameters:

inp (Iterable) – Iterable to chunk
window_size (int, optional) – size of the chunk window. Defaults to 300.
overlap (int, optional) – overlap between windows. Defaults to 100.

Returns:

(chunks, chunk lengths)

Return type:

tuple[list, list]