#Local LLM Backend Feasibility Analysis

#Summary

Adding local LLM support to llm9p is highly feasible and architecturally straightforward. The primary target is GPT-OSS (OpenAI's open-weight models), but the implementation covers any model served via the OpenAI-compatible /v1/chat/completions endpoint. The existing Backend interface already provides the right abstraction. A new OpenAIClient backend (~400-500 lines of Go) would enable llm9p to work with GPT-OSS (via Ollama, vLLM, or llama.cpp), as well as Llama, Mistral, Qwen, and any other model served by these platforms.

#Primary Target: GPT-OSS

GPT-OSS is OpenAI's first open-weight model release since GPT-2, released August 2025 under the Apache 2.0 license. It consists of two Mixture-of-Experts (MoE) models:

#Why GPT-OSS is a good fit for llm9p

• Strong tool calling: Outperforms o4-mini on TauBench -- important for llm9p's tool_use protocol
• 128K context: Matches Claude's context window, so the existing compaction logic works well
• Runs on consumer hardware: gpt-oss-20b needs only ~14 GB (Apple M-series or a single RTX 4090)
• Standard API: All serving backends expose it via OpenAI /v1/chat/completions -- no special handling needed
• Harmony format abstracted away: GPT-OSS uses a new token format called "Harmony" internally, but Ollama/vLLM/llama.cpp handle the conversion. The client just uses the standard chat completions API

#Serving GPT-OSS locally

# Ollama (simplest -- auto-downloads the model)
ollama pull gpt-oss:20b
# API at http://localhost:11434/v1

# vLLM (production, GPU servers)
vllm serve openai/gpt-oss-20b --tool-call-parser openai
# API at http://localhost:8000/v1

# llama.cpp (GGUF quantization, partial GPU offload)
llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja
# API at http://localhost:8080/v1

#Using GPT-OSS with llm9p (planned)

# GPT-OSS via Ollama
./llm9p -backend openai -openai-url http://localhost:11434/v1 -model gpt-oss:20b

# GPT-OSS via vLLM
./llm9p -backend openai -openai-url http://localhost:8000/v1 -model openai/gpt-oss-20b

#Current Architecture

llm9p uses a clean Backend interface (internal/llm/backend.go) with two implementations:

The SessionManager wraps any Backend and provides per-session isolation via the stateless AskWithRequest method. This means a new backend only needs to implement the Backend interface -- everything above it (sessions, filesystem, 9P protocol) works unchanged.

#The OpenAI Compatibility Ecosystem

Every major local LLM server now exposes an OpenAI-compatible /v1/chat/completions endpoint:

#Ollama

• Endpoint: http://localhost:11434/v1/chat/completions
• Streaming: Yes (SSE)
• Tool calling: Yes (Llama 3.1+, Mistral, Qwen 2.5)
• Models: Llama 3.x, Mistral, Phi, Qwen, Gemma, DeepSeek, etc.
• Setup: ollama run llama3.1:8b (auto-downloads)
• Key advantage: Simplest user experience; single binary, auto-manages models

#llama.cpp (llama-server)

• Endpoint: http://localhost:8080/v1/chat/completions
• Streaming: Yes (SSE)
• Tool calling: Yes (with --jinja flag)
• Models: Any GGUF model (quantized, runs on CPU or GPU)
• Setup: llama-server -m model.gguf --port 8080
• Key advantage: Also supports Anthropic Messages API at /v1/messages

#vLLM

• Endpoint: http://localhost:8000/v1/chat/completions
• Streaming: Yes (SSE)
• Tool calling: Yes (with --enable-auto-tool-choice)
• Models: HuggingFace models (full-precision or quantized)
• Setup: vllm serve meta-llama/Llama-3.1-8B-Instruct
• Key advantage: Highest throughput; PagedAttention, continuous batching

#LocalAI

• Endpoint: http://localhost:8080/v1/chat/completions
• Streaming: Yes (SSE)
• Tool calling: Yes (mature, production-ready)
• Models: GGUF, Safetensors, PyTorch, GPTQ, AWQ
• Setup: Docker: docker run -p 8080:8080 localai/localai:latest
• Key advantage: Also supports Anthropic API; most format-flexible

#LM Studio

• Endpoint: http://localhost:1234/v1/chat/completions
• Streaming: Yes (SSE)
• Tool calling: Yes (function calling support)
• Models: GGUF models via GUI
• Setup: Desktop app with built-in model browser
• Key advantage: GUI for non-technical users

#Implementation Strategy

#Recommended Approach: OpenAI-Compatible Backend

Create a new OpenAIClient in internal/llm/openai_client.go that implements the Backend interface using the OpenAI Chat Completions API. This is the right abstraction because:

1. Universal compatibility -- works with all 5+ local servers listed above
2. Well-defined API -- the OpenAI chat completions format is the de facto standard
3. Go library available -- github.com/sashabaranov/go-openai provides a mature, well-maintained Go client with streaming, tool calling, and custom base URL support
4. No vendor lock-in -- the same backend works for local models AND any OpenAI-compatible cloud provider

#Alternative Considered: Reuse Anthropic Client with Custom Base URL

Since llama.cpp and LocalAI now support the Anthropic Messages API, we could point the existing Client at a local server using option.WithBaseURL(). However, this is worse because:

• Only works with llama.cpp and LocalAI, not Ollama/vLLM/LM Studio
• The Anthropic Messages API is less universally supported than OpenAI
• Ties local LLM support to Anthropic-specific SDK behavior

#Implementation Outline

// internal/llm/openai_client.go
type OpenAIClient struct {
    client         *openai.Client
    mu             sync.RWMutex
    model          string        // e.g., "llama3.1:8b", "mistral"
    temperature    float64
    systemPrompt   string
    prefill        string
    messages       []Message
    lastTokens     int
    totalTokens    int
    thinkingTokens int
    streaming      bool
    streamChan     chan string
    streamDone     chan struct{}
}

func NewOpenAIClient(baseURL, apiKey, model string) *OpenAIClient {
    config := openai.DefaultConfig(apiKey)
    config.BaseURL = baseURL
    return &OpenAIClient{
        client:      openai.NewClientWithConfig(config),
        model:       model,
        temperature: 0.7,
        messages:    make([]Message, 0),
    }
}

The implementation follows the same pattern as the existing CLIClient:

• Conversation history managed as []Message
• Token counting estimated from response usage fields (most servers report these)
• Streaming via the SDK's CreateChatCompletionStream
• Tool calling via OpenAI function calling format

#Backend Interface Compatibility

All 18 methods of the Backend interface map cleanly:

#CLI Flag Design

# Ollama (default port)
./llm9p -backend openai -openai-url http://localhost:11434/v1 -model llama3.1:8b

# llama-server
./llm9p -backend openai -openai-url http://localhost:8080/v1 -model default

# vLLM
./llm9p -backend openai -openai-url http://localhost:8000/v1 -model meta-llama/Llama-3.1-8B-Instruct

# LM Studio
./llm9p -backend openai -openai-url http://localhost:1234/v1 -model local-model

# With API key (for cloud OpenAI-compatible providers)
OPENAI_API_KEY=sk-... ./llm9p -backend openai -openai-url https://api.openai.com/v1 -model gpt-4o

#New Dependency

github.com/sashabaranov/go-openai  (MIT license, ~8.6k stars)

This library supports:

• Custom base URLs (essential for local servers)
• Chat completions with streaming
• Function/tool calling
• Token usage reporting
• All model parameters (temperature, max_tokens, etc.)

#Effort Estimate

Total: ~700 lines of new code, mostly mechanical since it follows the existing CLIClient structure closely.

#Feature Gaps and Limitations

#Features that degrade gracefully with local models:

• Extended thinking: Not supported by local models. ThinkingTokens would be ignored (same as current API backend behavior).
• Prefill: Not natively supported by OpenAI API format. Would be simulated by prepending to response text (same as CLIClient).
• Tool calling: Depends on model capability. Works well with Llama 3.1+, Mistral, Qwen 2.5. Smaller models may not support it.
• Token counting: Most servers report usage, but some (older Ollama versions) may not. Fallback to character-based estimation.

#Features that work fully:

• Conversation history / multi-turn
• System prompts
• Temperature control
• Model switching (within what the server has loaded)
• Streaming
• Session isolation (handled by SessionManager, not the backend)

#Recommended Models for Testing

#Conclusion

Adding local LLM support is a well-scoped, low-risk enhancement. The Backend interface is already designed for exactly this kind of extension. GPT-OSS is the primary target -- it offers strong tool calling, 128K context, and runs on consumer hardware. The OpenAI-compatible API is the clear integration point since the entire ecosystem (including GPT-OSS serving via Ollama/vLLM/llama.cpp) has standardized on it. The sashabaranov/go-openai Go library provides everything needed.

The result would make llm9p usable with GPT-OSS and other open-weight models in fully offline/air-gapped environments, eliminate API costs for development and experimentation, and open the door to any model ecosystem.