graphragzen.llm.base_llm.LLM
- class graphragzen.llm.base_llm.LLM[source]
Base class to communicate with local or remote LLM’s
Be carefull when using the same persistent cache file while switching or updating models or tokenizers. The LLM will search for cached llm-in -> llm-out in in the cache file and not re-process input.
Attributes
Methods
a_run_chat(chat[, max_tokens, ...])Runs a chat through the LLM asynchonously
check_cache(llm_input)Checks the hash(llm_in) -> llm_out cache and returns stored output if found.
format_chat(chat[, established_chat])format chat with the correct names ready for tokenizer.apply_chat_template
num_chat_tokens(chat)Return the length of the tokenized chat
print_streamed(stream[, timeit])Streams the generated tokens to the terminal and returns the full generated text.
run_chat(chat[, max_tokens, ...])Runs a chat through the LLM
tokenize(content)Tokenize a string
untokenize(tokens)Generate a string from a list of tokens
write_item_to_cache(llm_input, llm_output)If a persistent cache file exists, this function can be used to append llm output to it.
- async a_run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)[source]
Runs a chat through the LLM asynchonously
- Parameters:
chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …
max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to None
output_structure (Optional[Union[ModelMetaclass, dict]], optional) – Output structure to force. e.g. grammars from llama.cpp. When using a pydantic model, only the reference should be passed. Correct = BaseLlamaCpp(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp(“some text”, MyPydanticModel())
stream (bool, optional) – Placeholder for compatibility with sync version, not used.
kwargs (Any) – Any keyword arguments to add to the lmm call.
- Returns:
Generated content
- Return type:
str
- cache_persistent = False
- check_cache(llm_input)[source]
Checks the hash(llm_in) -> llm_out cache and returns stored output if found.
- Parameters:
llm_input (str) – To check in cache for existing cached output.
- Returns:
Union[str, None]
- Return type:
str | None
- context_size = 0
- format_chat(chat, established_chat=[])[source]
format chat with the correct names ready for tokenizer.apply_chat_template
- Parameters:
chat (List[tuple]) – [(role, content), (role, content)] - role (str): either “system”, “user” or “model” - content (str)
established_chat (List[dict], optional) – Already formatted chat to append to. Defaults to [].
- Returns:
[{“role”: …, “content”: …}, {“role”: …, “content”: …}]
- Return type:
List[dict]
- model_name: Any = None
- abstract num_chat_tokens(chat)[source]
Return the length of the tokenized chat
- Parameters:
chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …
- Returns:
number of tokens
- Return type:
int
- persistent_cache_file = ''
- print_streamed(stream, timeit=False)[source]
Streams the generated tokens to the terminal and returns the full generated text.
- Parameters:
stream (Iterator)
timeit (bool, optional) – If True display the number of tokens generated / sec. Defaults to False.
- Returns:
Generated text
- Return type:
str
- abstract run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)[source]
Runs a chat through the LLM
- Parameters:
chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …
max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to -1.
output_structure (Optional[Union[ModelMetaclass, dict]], optional) – Output structure to force. e.g. grammars from llama.cpp. When using a pydantic model, only the reference should be passed. Correct = BaseLlamaCpp(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp(“some text”, MyPydanticModel())
stream (bool, optional) – If True, streams the results to console. Defaults to False.
kwargs (Any) – Any keyword arguments to add to the lmm call.
- Returns:
Generated content
- Return type:
str
- abstract tokenize(content)[source]
Tokenize a string
- Parameters:
content (str) – String to tokenize
- Returns:
Tokenized string
- Return type:
List[str]
- tokenizer: Any = None
- abstract untokenize(tokens)[source]
Generate a string from a list of tokens
- Parameters:
tokens (List[str]) – Tokenized string
- Returns:
Untokenized string
- Return type:
str
- use_cache = True