graphragzen.llm.llama_cpp_models.BaseLlamaCpp

class graphragzen.llm.llama_cpp_models.BaseLlamaCpp

Loads a GGUF model using llama cpp python and it’s corresponding tokenizer from HF

Attributes

cache_persistent
chatnames
context_size
model_name
persistent_cache_file
tokenizer
use_cache

Methods

a_run_chat(chat[, max_tokens, ...])	Runs a chat through the LLM asynchonously
check_cache(llm_input)	Checks the hash(llm_in) -> llm_out cache and returns stored output if found.
format_chat(chat[, established_chat])	format chat with the correct names ready for tokenizer.apply_chat_template
num_chat_tokens(chat)	Return the length of the tokenized chat
print_streamed(stream[, timeit])	Streams the generated tokens to the terminal and returns the full generated text.
run_chat(chat[, max_tokens, ...])	Runs a chat through the LLM
tokenize(content)	Tokenize a string
untokenize(tokens)	Generate a string from a list of tokens
write_item_to_cache(llm_input, llm_output)	If a persistent cache file exists, this function can be used to append llm output to it.

__init__(model_storage_path, tokenizer_URI, context_size=8192, n_gpu_layers=-1, use_cache=True, cache_persistent=True, persistent_cache_file='./llm_persistent_cache.yaml')

Initiate a llama cpp model

Parameters:

• model_storage_path (str) – Path to the model on the local filesystem

• tokenizer_URI (str) – URI for the tokenizer

• context_size (int, optional) – Size of the context window in tokens. Defaults to 8192

• use_cache (bool, optional) – Use a cache to find output for previously processed inputs in stead of re-generating output from the input. Default to True.

• n_gpu_layers (int, optional) – Number of layers to offload to GPU (-ngl). If -1, all layers are offloaded. You need to install llama-cpp-python with the correct cuda support. Out of the box GraphRAGZen’s llama-cpp-python is the CPU version only. Defaults to -1.

• cache_persistent (bool, optional) – Append the cache to a file on disk so it can be re-used between runs. If False will use only in-memory cache. Default to True

• persistent_cache_file (str, optional) – The file to store the persistent cache. Defaults to ‘./llm_persistent_cache.yaml’.

Return type:

None

async a_run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)

Runs a chat through the LLM asynchonously

Parameters:

• chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

• max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to None

• output_structure (Optional[Union[ModelMetaclass, dict]], optional) – Output structure to force. e.g. grammars from llama.cpp. When using a pydantic model, only the reference should be passed. Correct = BaseLlamaCpp(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp(“some text”, MyPydanticModel())

• stream (bool, optional) – Placeholder for compatibility with sync version, not used.

• kwargs (Any) – Any keyword arguments to add to the lmm call.

Returns:

Generated content

Return type:

str

cache_persistent = False

chatnames: ChatNames = ChatNames(system='system', user='user', model='model')

check_cache(llm_input)

Checks the hash(llm_in) -> llm_out cache and returns stored output if found.

Parameters:

llm_input (str) – To check in cache for existing cached output.

Returns:

Union[str, None]

Return type:

str | None

context_size = 0

format_chat(chat, established_chat=[])

format chat with the correct names ready for tokenizer.apply_chat_template

Parameters:

• chat (List[tuple]) – [(role, content), (role, content)] - role (str): either “system”, “user” or “model” - content (str)

• established_chat (List[dict], optional) – Already formatted chat to append to. Defaults to [].

Returns:

[{“role”: …, “content”: …}, {“role”: …, “content”: …}]

Return type:

List[dict]

model_name: Any = None

num_chat_tokens(chat)

Return the length of the tokenized chat

Parameters:

chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

Returns:

number of tokens

Return type:

int

persistent_cache_file = ''

print_streamed(stream, timeit=False)

Streams the generated tokens to the terminal and returns the full generated text.

Parameters:

• stream (Iterator)

• timeit (bool, optional) – If True display the number of tokens generated / sec. Defaults to False.

Returns:

Generated text

Return type:

str

run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)

Runs a chat through the LLM

Parameters:

• chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

• max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to -1.

• output_structure (ModelMetaclass, optional) – Output structure to force, e.g. grammar from llama.cpp. This SHOULD NOT be an instance of the pydantic model, just the reference. Correct = BaseLlamaCpp.run_chat(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp.run_chat(“some text”, MyPydanticModel())

• stream (bool, optional) – If True, streams the results to console. Defaults to False.

• kwargs (Any) – Any keyword arguments to add to the lmm call.

Returns:

Generated content

Return type:

str

tokenize(content)

Tokenize a string

Parameters:

content (str) – String to tokenize

Returns:

Tokenized string

Return type:

List[str]

tokenizer: Any = None

untokenize(tokens)

Generate a string from a list of tokens

Parameters:

tokens (List[str]) – Tokenized string

Returns:

Untokenized string

Return type:

str

use_cache = True

write_item_to_cache(llm_input, llm_output)

If a persistent cache file exists, this function can be used to append llm output to it.

Parameters:

• llm_input (str)

• llm_output (str)

Return type:

None