graphragzen.llm.openAI_API_client.TikTokenTokenizer

class graphragzen.llm.openAI_API_client.TikTokenTokenizer[source]

tokenizes and detokenizes using tiktoken library

Methods

convert_tokens_to_string(tokens)

tokenize(content)

__init__(model_name)[source]
Parameters:

model_name (str)

convert_tokens_to_string(tokens)[source]
Parameters:

tokens (List[int])

Return type:

str

tokenize(content)[source]
Parameters:

content (str)

Return type:

List[int]