llm

package
v0.0.0-...-9287f1f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 19, 2025 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package llm runs a LLM locally via llama.cpp, llamafile, or with a python server. It takes care of everything, including fetching gguf packed models from hugging face.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Client 

type Client struct {
	genai.Provider
}

Client wraps a Provider client and logs its calls.

func (*Client) GenStream 

func (c *Client) GenStream(ctx context.Context, msgs []genai.Message, opts ...genai.Options) (iter.Seq[genai.Reply], func() (genai.Result, error))

GenStream implements genai.Provider

func (*Client) GenSync 

func (c *Client) GenSync(ctx context.Context, msgs []genai.Message, opts ...genai.Options) (genai.Result, error)

GenSync implements genai.Provider

func (*Client) Unwrap 

func (c *Client) Unwrap() genai.Provider

type Conversation 

type Conversation struct {
	User         string
	SystemPrompt string
	Channel      string
	Started      time.Time
	LastUpdate   time.Time
	Messages     []genai.Message
	// contains filtered or unexported fields
}

Conversation is a conversation with one user.

type Memory 

type Memory struct {
	// contains filtered or unexported fields
}

Memory holds the bot's conversations.

func (*Memory) Forget 

func (m *Memory) Forget()

Forget forgets old conversations.

func (*Memory) Get 

func (m *Memory) Get(user, channel string) *Conversation

Get gets a previous conversations or returns a new one if it's a new conversation.

func (*Memory) Load 

func (m *Memory) Load(r io.Reader) error

Load loads previous memory.

func (*Memory) Save 

func (m *Memory) Save(w io.Writer) error

Save saves the memory for later reuse.

type Options 

type Options struct {
	// Backend is the name of the backend to run. It can be "llama-server" or "python" for genaipy.
	Backend string `yaml:"backend"`
	// Model specifies a model to use.
	Model PackedFileRef `yaml:"model"`
	// ContextLength will set the context length when using a locally managed "llamacpp".
	ContextLength int `yaml:"context_length"`
	// contains filtered or unexported fields
}

Options for NewLLM.

func (*Options) Validate 

func (o *Options) Validate() error

Validate checks for obvious errors in the fields.

type PackedFileRef 

type PackedFileRef string

PackedFileRef is a packed reference to a file in an hugging face repository.

The form is "hf:<author>/<repo>/HEAD/<file>"

HEAD is the git commit reference or "revision". HEAD means the default branch. It can be replaced with a branch name or a commit hash. The default branch used by huggingface_hub official python library is "main".

DEFAULT_REVISION in https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/constants.py

func MakePackedFileRef 

func MakePackedFileRef(author, repo, revision, file string) PackedFileRef

MakePackedFileRef returns a PackedFileRef

func (PackedFileRef) Author 

func (p PackedFileRef) Author() string

Author returns the <author> part of the packed reference.

func (PackedFileRef) Basename 

func (p PackedFileRef) Basename() string

Basename returns the basename part of this reference.

func (PackedFileRef) ModelRef 

func (p PackedFileRef) ModelRef() huggingface.ModelRef

ModelRef returns the ModelRef reference to the repo containing this file.

func (PackedFileRef) Repo 

func (p PackedFileRef) Repo() string

Repo returns the <repo> part of the packed reference.

func (PackedFileRef) RepoID 

func (p PackedFileRef) RepoID() string

RepoID returns the canonical "<author>/<repo>" for this repository.

func (PackedFileRef) RepoURL 

func (p PackedFileRef) RepoURL() string

RepoURL returns the canonical URL for this repository.

func (PackedFileRef) Revision 

func (p PackedFileRef) Revision() string

Revision returns the HEAD part of the packed reference.

func (PackedFileRef) Validate 

func (p PackedFileRef) Validate() error

Validate checks for obvious errors in the string.

type PackedRepoRef 

type PackedRepoRef string

PackedRepoRef is a packed reference to an hugging face repository.

The form is "hf:<author>/<repo>"

func (PackedRepoRef) ModelRef 

func (p PackedRepoRef) ModelRef() huggingface.ModelRef

ModelRef converts to a ModelRef reference.

func (PackedRepoRef) RepoID 

func (p PackedRepoRef) RepoID() string

RepoID returns the canonical "<author>/<repo>" for this repository.

func (PackedRepoRef) RepoURL 

func (p PackedRepoRef) RepoURL() string

RepoURL returns the canonical URL for this repository.

func (PackedRepoRef) Validate 

func (p PackedRepoRef) Validate() error

Validate checks for obvious errors in the string.

type Server 

type Server struct {
	HF    *huggingface.Client
	Model PackedFileRef
	URL   string
	// contains filtered or unexported fields
}

Server runs a llamacpp-server or python.

func New 

func New(ctx context.Context, cache string, opts *Options) (*Server, error)

New instantiates a llamacpp-server or python.

func (*Server) Client 

func (l *Server) Client() genai.Provider

func (*Server) Close 

func (l *Server) Close() error

Source Files

View all Source files

Directories

Path Synopsis
Package tools contains structures to generate function calls, tool calling from LLMs.
 Package tools contains structures to generate function calls, tool calling from LLMs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.