graphragzen.llm.openAI_API_client.ApiTokenizer

class graphragzen.llm.openAI_API_client.ApiTokenizer[source]

tokenizes and detokenizes using API endpoints

Methods

convert_tokens_to_string(tokens[, base_url])

tokenize(content[, base_url])

__init__(base_url)[source]
Parameters:

base_url (str)

convert_tokens_to_string(tokens, base_url=None)[source]
Parameters:

  • tokens (List[int])

  • base_url (str | None)

Return type:

str

tokenize(content, base_url=None)[source]
Parameters:

  • content (str)

  • base_url (str | None)

Return type:

List[int]