graphragzen.llm.llama_cpp_models.Phi35MiniGGUF

class graphragzen.llm.llama_cpp_models.Phi35MiniGGUF[source]

Loads the GGUF version of a Phi 3.5 Mini model using llama-cpp-python

Attributes

cache_persistent

chatnames

context_size

model_name

persistent_cache_file

tokenizer

use_cache

Methods

a_run_chat(chat[, max_tokens, ...])

Runs a chat through the LLM asynchonously

check_cache(llm_input)

Checks the hash(llm_in) -> llm_out cache and returns stored output if found.

format_chat(chat[, established_chat])

format chat with the correct names ready for tokenizer.apply_chat_template

num_chat_tokens(chat)

Return the length of the tokenized chat

print_streamed(stream[, timeit])

Streams the generated tokens to the terminal and returns the full generated text.

run_chat(chat[, max_tokens, ...])

Runs a chat through the LLM

tokenize(content)

Tokenize a string

untokenize(tokens)

Generate a string from a list of tokens

write_item_to_cache(llm_input, llm_output)

If a persistent cache file exists, this function can be used to append llm output to it.
__init__(model_storage_path, tokenizer_URI, context_size=8192, n_gpu_layers=-1, use_cache=True, cache_persistent=True, persistent_cache_file='./llm_persistent_cache.yaml')[source]

Load the GGUF version of a Phi 3.5 Mini model using llama-cpp-python and it’s corresponding tokenizer.

Parameters:

  • model_storage_path (str) – Path to the model on the local filesystem

  • tokenizer_URI (str) – HuggingFace URI for the tokenizer

  • context_size (int, optional) – Size of the context window in tokens. Defaults to 8192

  • use_cache (bool, optional) – Use a cache to find output for previously processed inputs in stead of re-generating output from the input. Default to True.

  • n_gpu_layers (int, optional) – Number of layers to offload to GPU (-ngl). If -1, all layers are offloaded. You need to install llama-cpp-python with the correct cuda support. Out of the box GraphRAGZen’s llama-cpp-python is the CPU version only. Defaults to -1.

  • cache_persistent (bool, optional) – Append the cache to a file on disk so it can be re-used between runs. If False will use only in-memory cache. Default to True

  • persistent_cache_file (str, optional) – The file to store the persistent cache. Defaults to ‘./llm_persistent_cache.yaml’.

Return type:

None

async a_run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)

Runs a chat through the LLM asynchonously

Parameters:

  • chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

  • max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to None

  • output_structure (Optional[Union[ModelMetaclass, dict]], optional) – Output structure to force. e.g. grammars from llama.cpp. When using a pydantic model, only the reference should be passed. Correct = BaseLlamaCpp(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp(“some text”, MyPydanticModel())

  • stream (bool, optional) – Placeholder for compatibility with sync version, not used.

  • kwargs (Any) – Any keyword arguments to add to the lmm call.

Returns:

Generated content

Return type:

str

cache_persistent = False
chatnames: ChatNames = ChatNames(system='system', user='user', model='model')
check_cache(llm_input)

Checks the hash(llm_in) -> llm_out cache and returns stored output if found.

Parameters:

llm_input (str) – To check in cache for existing cached output.

Returns:

Union[str, None]

Return type:

str | None

context_size = 0
format_chat(chat, established_chat=[])

format chat with the correct names ready for tokenizer.apply_chat_template

Parameters:

  • chat (List[tuple]) – [(role, content), (role, content)] - role (str): either “system”, “user” or “model” - content (str)

  • established_chat (List[dict], optional) – Already formatted chat to append to. Defaults to [].

Returns:

[{“role”: …, “content”: …}, {“role”: …, “content”: …}]

Return type:

List[dict]

model_name: Any = None
num_chat_tokens(chat)

Return the length of the tokenized chat

Parameters:

chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

Returns:

number of tokens

Return type:

int

persistent_cache_file = ''
print_streamed(stream, timeit=False)

Streams the generated tokens to the terminal and returns the full generated text.

Parameters:

  • stream (Iterator)

  • timeit (bool, optional) – If True display the number of tokens generated / sec. Defaults to False.

Returns:

Generated text

Return type:

str

run_chat(chat, max_tokens=-1, output_structure=None, stream=False, **kwargs)

Runs a chat through the LLM

Parameters:

  • chat (List[dict]) – in form [{“role”: …, “content”: …}, {“role”: …, “content”: …

  • max_tokens (int, optional) – Maximum number of tokens to generate. Defaults to -1.

  • output_structure (ModelMetaclass, optional) – Output structure to force, e.g. grammar from llama.cpp. This SHOULD NOT be an instance of the pydantic model, just the reference. Correct = BaseLlamaCpp.run_chat(“some text”, MyPydanticModel) Wrong = BaseLlamaCpp.run_chat(“some text”, MyPydanticModel())

  • stream (bool, optional) – If True, streams the results to console. Defaults to False.

  • kwargs (Any) – Any keyword arguments to add to the lmm call.

Returns:

Generated content

Return type:

str

tokenize(content)

Tokenize a string

Parameters:

content (str) – String to tokenize

Returns:

Tokenized string

Return type:

List[str]

tokenizer: Any = None
untokenize(tokens)

Generate a string from a list of tokens

Parameters:

tokens (List[str]) – Tokenized string

Returns:

Untokenized string

Return type:

str

use_cache = True
write_item_to_cache(llm_input, llm_output)

If a persistent cache file exists, this function can be used to append llm output to it.

Parameters:

  • llm_input (str)

  • llm_output (str)

Return type:

None