llmman

llmman serve never checks credentials. OpenClaw still needs a non-empty apiKey on the provider entry so the provider counts as configured. This page uses LLMMAN_API_KEY=llmman-local with apiKey: "${LLMMAN_API_KEY}", mirroring the OLLAMA_API_KEY=ollama-local convention. A literal apiKey: "llmman-local" works too.

For llmman.hybrid/... and llmman.provider/... refs, llmman forwards the bearer OpenClaw presents to the hosted provider as that provider's API key. The one exception is the literal placeholder llmman, which tells the daemon to use its own key. See Hybrid inference for both patterns. A marker such as llmman-local would be sent to the hosted provider and rejected there.

A daemon started with LLMMAN_HOST=0.0.0.0 in its environment binds every interface with no auth. Point OpenClaw at a LAN host only inside a network you trust; there is no credential OpenClaw can send that llmman would enforce. A daemon reachable off loopback also refuses to spend its own hosted-provider key for callers that presented none.

${LLMMAN_API_KEY} resolves from the Gateway process environment or ~/.openclaw/.env. If the variable is missing, OpenClaw logs a config warning and treats the provider as unavailable, so put the export where the Gateway can read it. See Environment.

llmman pull qwen3.8
llmman serve
echo 'LLMMAN_API_KEY=llmman-local' >> ~/.openclaw/.env
openclaw models set llmman/qwen3.8

Use the Full config example for the provider entry, and set LLMMAN_CONTEXT_LENGTH=65536 in the daemon's environment if you want the server context to match it exactly.

The Hybrid config above: llmman.hybrid/qwen3.8,openai/gpt-5.6-luna as primary, LLMMAN_API_KEY set to the OpenAI key, LLMMAN_CONTEXT_LENGTH matching the pair's contextWindow. Add openai/gpt-5.6-luna to fallbacks so a stopped daemon does not block replies.

Local models served through a custom openai-completions provider use structured Tool Search automatically when unset. The explicit setting below pins that surface, keeping optional capabilities available while loading their schemas only when needed. Cap the context to what the host can run with LLMMAN_CONTEXT_LENGTH=32768 in the daemon's environment:

{
  agents: {
    defaults: {
      model: { primary: "llmman/qwen3.8" },
    },
  },
  tools: {
    toolSearch: { mode: "tools" },
  },
  models: {
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [
          {
            id: "qwen3.8",
            name: "Qwen3.8 (llmman)",
            reasoning: true,
            input: ["text", "image"],
            contextWindow: 32768,
            contextTokens: 32768,
            maxTokens: 4096,
          },
        ],
      },
    },
  },
}

Use compat.supportsTools: false only when the model or server reliably fails on tool schemas; it disables tool use entirely. For a deliberately narrower agent, prefer tools.profile or a per-agent tool policy.

Custom provider ids when running more than one daemon; each gets its own host, models, and timeout:

{
  models: {
    providers: {
      "llmman-fast": {
        baseUrl: "http://mini.local:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [{ id: "qwen3.5:9b", name: "qwen3.5:9b", input: ["text"], contextWindow: 32768 }],
      },
      "llmman-large": {
        baseUrl: "http://gpu-box.local:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 420,
        models: [{ id: "qwen3.8", name: "qwen3.8", reasoning: true, input: ["text", "image"], contextWindow: 131072 }],
      },
    },
  },
  agents: {
    defaults: {
      model: {
        primary: "llmman-fast/qwen3.5:9b",
        fallbacks: ["llmman-large/qwen3.8"],
      },
    },
  },
}

llmman can also pool several daemons itself: llmman config set aggregation.peers <host>,<host> (or LLMMAN_PEERS) makes one daemon forward to peers, so OpenClaw sees a single endpoint whose model list spans the group.

Model ids are whatever llmman resolves. Pull with the full reference and use the same string as the OpenClaw model id:

llmman pull hf.co/unsloth/Qwen3.5-0.8B-GGUF
llmman pull ghcr.io/myorg/private-model:v1

{
  models: {
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [
          { id: "hf.co/unsloth/Qwen3.5-0.8B-GGUF", name: "Qwen3.5 0.8B", input: ["text"] },
          { id: "ghcr.io/myorg/private-model:v1", name: "Private model", input: ["text"] },
        ],
      },
    },
  },
}

The agent ref is then llmman/hf.co/unsloth/Qwen3.5-0.8B-GGUF. Run llmman login <registry> first for private registries.

LLMMAN_CONTEXT_LENGTH is the server-side context (there is no flag). Semantics by backend:

• llama-server (GGUF): set, it is passed as --ctx-size for generation models; 0 means the trained context. Unset, llmman uses 262144 or the model's trained context if smaller, and on out-of-memory retries with the context halved down to a 16,384 floor.
• vLLM (safetensors): a positive value becomes --max-model-len; unset uses the vLLM default.
• mlx_lm.server: not forwarded.

LLMMAN_NUM_PARALLEL scales --ctx-size up by that factor so each slot keeps the full context.

On the OpenClaw side, contextWindow declares the model's window and contextTokens caps active input. Keep contextWindow at or below the server value; OpenClaw derives compaction and preflight thresholds from it. OpenClaw's contextWindow does not change llmman's hybrid byte budget; configure the daemon separately as described in Hybrid config.

{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "qwen3.8",
            contextWindow: 65536,
            contextTokens: 49152,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}

Qwen3.8 thinks by default; llmman returns the reasoning as reasoning_content, which OpenClaw's openai-completions adapter separates from the final text. Requests are proxied to llama-server, so chat_template_kwargs passes through. To turn thinking off for agent turns with a local Qwen model:

{
  agents: {
    defaults: {
      models: {
        "llmman/qwen3.8": {
          params: {
            chat_template_kwargs: { enable_thinking: false },
          },
        },
      },
    },
  },
}

For per-run or session control, declare the local Qwen model's thinking format:

{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "Qwen3.8 (llmman)",
            reasoning: true,
            input: ["text", "image"],
            compat: { thinkingFormat: "qwen-chat-template" },
          },
        ],
      },
    },
  },
}

With this declaration, openclaw agent --model llmman/qwen3.8 --thinking off, /think off, and openclaw infer model run --local --model llmman/qwen3.8 --thinking off --prompt "Reply with exactly: pong" --json map the thinking setting to chat_template_kwargs.enable_thinking. Without it, the generic proxy defaults do not send this control or reasoning_effort.

The lean infer model run path does not read the agent-level params recipe above; use the compatibility declaration and --thinking off for that probe. Do not combine a fixed enable_thinking agent param with per-run control, since the fixed param overrides the generated value. Apply Qwen-specific controls to a hybrid ref only if both its local and hosted backends accept them.

Models load on their first request, each in its own backend subprocess, and unload after five idle minutes. Tune with:

llmman ps shows loaded models with their context and expiry; llmman stop <model> unloads one now. A first request after startup or an idle unload pays the load cost, so set timeoutSeconds on the provider and raise LLMMAN_KEEP_ALIVE to keep the daemon warm for chat surfaces.

llmman probes CUDA, ROCm, Vulkan (Linux/Windows) or Metal (macOS) and downloads a matching llama-server release if none is on PATH. Override with LLMMAN_LLM_LIBRARY: cpu, cuda, cuda13, rocm, vulkan, or metal. Other knobs: LLMMAN_FLASH_ATTENTION (on/off/auto), LLMMAN_KV_CACHE_TYPE (f16, q8_0, q4_0), LLMMAN_SCHED_SPREAD for multi-GPU layer splitting, LLMMAN_IGPU_ENABLE to count integrated GPUs. On Linux, llmman serve --ociman docker|podman runs llama-server from the ghcr.io/ggml-org/llama.cpp images instead of a local binary. LLMMAN_DEBUG=1 prints the probe result.

llmman serves /v1/embeddings for GGUF embedding models, so memory search can use it through the generic openai-compatible embedding provider:

llmman pull embeddinggemma

{
  memory: {
    search: {
      provider: "openai-compatible",
      model: "embeddinggemma",
      remote: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
      },
    },
  },
}

Embedding models are capped to their trained context regardless of LLMMAN_CONTEXT_LENGTH. See Memory config for the remaining fields.

llmman also implements Ollama's native /api/chat, /api/tags, /api/show, and /api/ps, and OLLAMA_HOST=127.0.0.1:17434 ollama run <model> works against it. Prefer api: "openai-completions" from OpenClaw anyway: llmman's /api/show reports only completion and vision capabilities, so the bundled Ollama plugin's discovery would mark every llmman model compat.supportsTools: false. If you do point the Ollama plugin at http://127.0.0.1:17434 (no /v1), list models explicitly instead of relying on discovery.

llmman forwards message content and tool schemas to the backend without normalizing them, so compatibility depends on the selected engine and model. Structured content parts (text + image) and OpenClaw's full tool schema work with qwen3.8 on llama-server. If a different backend or model rejects them:

• messages[].content: invalid type: sequence, expected a string → set compat.requiresStringContent: true on the model entry. OpenClaw then flattens pure text content parts into plain strings.
• 400 JSON schema conversion failed → llama-server could not compile a tool schema into its grammar subset. Update OpenClaw first; if a third-party tool or MCP server contributes the offending schema, disable it for that agent, and use compat.supportsTools: false only as a last resort.

{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "qwen3.8",
            compat: {
              requiresStringContent: true,
            },
          },
        ],
      },
    },
  },
}

Because llmman is a non-native openai-completions endpoint, OpenClaw treats it as a proxy route: no service_tier, no Responses store, no prompt-cache hints, no OpenAI reasoning-compat payload shaping, no hidden OpenClaw attribution headers, and compat.supportsDeveloperRole is forced to false. Vendor-specific fields can be merged into the request body with agents.defaults.models["llmman/<model>"].params.extra_body.

Local llmman models are free, so set all costs to 0. The hosted half of a hybrid or llmman.provider/... ref is billed by that provider; llmman list --provider openai shows its per-million-token prices.

llmman serve is not running or is not reachable at the configured address. The default is 127.0.0.1:17434; if you set LLMMAN_HOST, update the OpenClaw baseUrl and healthUrl to match.

llmman serve
curl http://127.0.0.1:17434/api/version

models.providers.llmman.apiKey: *** env var "LLMMAN_API_KEY" in the config warnings means the substitution found no value. Add LLMMAN_API_KEY=llmman-local to ~/.openclaw/.env, or replace "${LLMMAN_API_KEY}" with the literal "llmman-local".

The model is not pulled, or the id in config does not match what llmman resolves. Compare against llmman list; a short name maps to docker.io/ai/<name>:latest, and an owner/repo name maps to hf.co/owner/repo.

llmman pull qwen3.8
llmman list

Large models can take minutes to load, especially on the first request after an idle unload. Raise models.providers.llmman.timeoutSeconds, warm the model with a first openclaw infer model run, and consider a longer LLMMAN_KEEP_ALIVE on the daemon.

llmman routed the request to the hosted half and found no usable key. Either the bearer OpenClaw sent was a marker (llmman-local) rather than a real key, or you used the llmman placeholder without giving the daemon its own OPENAI_API_KEY, or the daemon is bound off loopback and refuses to spend its own key. See Hosted-provider key.

Check the daemon log; every routed request records the side and the reason. Routing is by request body size against LLMMAN_CONTEXT_LENGTH (4 bytes per token) unless x-llmman-route is set. Lower LLMMAN_HYBRID_LOCAL_BYTES to overflow sooner, raise it to stay local longer, or pin a side with a provider headers entry as in Pinning a side.

Both probes are tool-free, so compat.supportsTools cannot change this failure. Check the configured base URL, model id, and LLMMAN_API_KEY, inspect the daemon and backend logs, and compare the two request payloads.

The agent turn adds a larger prompt and tool schemas. A 400 JSON schema conversion failed is llama-server rejecting a tool schema; update OpenClaw and check third-party tools or MCP servers. Otherwise enable Tool Search to defer schemas, confirm the server's actual context allocation, and use compat.supportsTools: false only as a last resort. See Smaller or stricter backends.

If schema errors are gone but the spawned llama-server still crashes on larger turns, treat it as an upstream llama.cpp or model limitation. Lower LLMMAN_CONTEXT_LENGTH, set LLMMAN_KV_CACHE_TYPE=q8_0 to reduce memory, or switch the backend or model.

Confirm the model's chat template supports tool calling and that the request reached /v1/chat/completions (not the Ollama or Anthropic surfaces via a proxy). If the model only calls tools when forced, set params.extra_body.tool_choice: "required" on that model ref as described in Local models.