graphragzen.preprocessing.preprocess.chunk_documents

graphragzen.preprocessing.preprocess.chunk_documents(input, llm, column_to_chunk='content', results_column='chunk', id_column='chunk_id', window_size=300, overlap=100, method='tokens')[source]

Chunk documents based on number of tokens

Parameters:

  • input (Union[pd.DataFrame, List[str]]) – Containing the documents to chunk

  • llm (LLM)

  • column_to_chunk (str, optional) – Column to chunk. Defaults to ‘content’.

  • results_column (str, optional) – Column to write chunks to, Defaults to ‘chunk’.

  • id_column (str, optional) – Column to write chunk ID’s to. Can later be used to refence the source chunk. Defaults to ‘chunk_id’.

  • window_size (str, optional) – Number of tokens in each chunk, Defaults to 300.

  • overlap (str, optional) – Number of tokens chunks overlap, Defaults to 100.

  • method (str, optional) – What to chunk. Currently only ‘tokens’ is implemented meaning column_to_chunk is first tokenized and the tokens are chunked.

Returns:

All columns in the input dataframe are exploded with the chunks allowing referencing

Return type:

pd.DataFrame