QVAC
GuidesSDK9 min read

How to run Hermes, a self-improving personal AI agent, fully local with QVAC

Hermes, the self-improving personal agent from Nous Research, learns your persona, your memory, and your habits. That is a lot of data about you to hand to a cloud. Here is how to run the whole thing on a local model with QVAC, fully on-device, in six copy-paste steps.

Thomas Blanc
How to run Hermes, a self-improving personal AI agent, fully local with QVAC

What Hermes is, and why it is different

Most AI agents you have seen are task tools. You give them a job, they do it, they forget you. Hermes Agent, from Nous Research, is built on a different idea: an agent that is yours, that remembers you, and that gets better the longer you use it.

Three things make it stand out:

  • It has a persona. A single file, SOUL.md, defines how it talks. Edit it and the agent changes character on the next message.
  • It remembers. Conversations, facts, and context persist across sessions in a local memory, so it is a companion, not a one-shot.
  • It learns. Hermes has a library of skills (research, notes, GitHub, documents, smart home, and many more) and a background curator that creates and refines skills from experience. The agent that finishes your week is not the one that started it.

All of that personality, memory, and learned skill is data about you. Which raises the obvious question: where does the model behind it run?

Why local is the right default for a personal agent

When a model only answers questions, where it runs is mostly about privacy and costs. When a model is your personal agent, holding your persona, your memory, and skills it learned from your own habits, it becomes about ownership. If the brain lives in someone else's datacenter, then your assistant's memory of you, and everything it learns from you, flows through a system you do not control: its uptime, its terms, its access policy, all of which can change without you.

Running the model locally changes the property, not just the policy:

  • Your context stays on the machine. Your persona, your memory, the skills the agent builds: none of it has to leave your disk to reach the model.
  • It works offline. Once the model is downloaded, you can disconnect entirely and keep talking to your agent. Turn on airplane mode and it still answers.
  • No per-token bill. A personal agent you talk to all day, that runs scheduled jobs in the background, would meter up fast against a cloud API. Locally it costs nothing per token.
  • No silent model swaps. The weights are a file on your disk. Your agent's brain does not change under you between sessions.

The tradeoff is latency and raw capability: a model that fits on a laptop is smaller than a frontier cloud model. For a personal agent that lives with your data, that tradeoff is easy, and it keeps getting better as small models improve.

QVAC is the piece that makes the local half practical. It is an open-source SDK and CLI from Tether that runs models on your own device across every major GPU backend (NVIDIA, AMD, and Intel through Vulkan, Apple Silicon through Metal), and it ships an OpenAI-compatible server. That server is the bridge Hermes plugs into.

How it fits together

Two long-running pieces and you:

  1. The QVAC server holds the model in memory and answers OpenAI-compatible requests on a local port. Start it once and leave it running.
  2. Hermes runs the agent: persona, memory, skills, tools. You point it at the QVAC server once, then just talk to it.
  3. Your messages go to Hermes, which thinks using the local model and acts with its tools.

The model inference never leaves the machine. (Some skills, like web research, still use the internet for that specific tool, the same way a browser does. The brain stays on-device.)

Which model to run

Hermes is tool-heavy: it sends a large set of tools on every turn, so it pays to run a capable model. The setup below uses Qwen3.6-35B-A3B, a mixture-of-experts model: 35 billion parameters of knowledge, only about 3 billion active per token, so it stays fast for its size. To switch models, change the constant in the config file (step 2 of the setup).

Use case

Model

Recommended RAM

Storage

Config name

Lightest, runs almost anywhere

Qwen3.5-4B (4-bit)

8 GB

~3 GB

QWEN3_5_4B_MULTIMODAL_Q4_K_M

Balanced

Qwen3.5-9B (4-bit)

16 GB

~6 GB

QWEN3_5_9B_MULTIMODAL_Q4_K_M

Recommended for the full agent

Qwen3.6-35B-A3B MoE (4-bit)

36 GB

~22 GB

QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M

Be realistic: these are small local models, and tool-heavy agent work is where they struggle most. The 4B is for trying the setup, the 9B mainly adds reliability over the 4B rather than smarts, and the 35B-A3B MoE is the only one that drives the full toolset dependably, so it is what we recommend. It is a capable on-device assistant, not a frontier cloud model.

A GPU helps a lot but is not required. QVAC uses Apple Metal on Apple Silicon and Vulkan on NVIDIA, AMD, and Intel. Run qvac doctor to see what your hardware supports. On the 35B-A3B, expect the first turn to take around 20 seconds (Hermes sends a large system prompt plus its tools), and faster turns after. For a snappier agent, narrow the active skills.

Set it up

  1. 01

    Install the QVAC CLI

    One command. Brings the qvac tool and the SDK it runs on.
  2. 02

    Create the model config

    Makes a folder and writes one small config file. The two settings that matter are "tools": true and "toolsMode": "static": static is what lets an external client like Hermes run the model's tool calls. With "dynamic" they never execute, which looks like the model is broken.
  3. 03

    Start the QVAC server

    qwen3.6-moe is the alias you defined in step 2. Run this in the ~/qvac-hermes folder you just made and leave the terminal open, so QVAC auto-detects that config (which is also what sets toolsMode: static). The first run downloads the model once (about 22 GB for the MoE), so give it a few minutes. That is the whole command; run it elsewhere without the config and it errors that the alias is unknown. Ready when you see QVAC API server listening.
  4. 04

    Install Hermes Agent

    Open a new terminal. The installer sets up Python, the global hermes command, and its dependencies. On Windows, run this inside WSL (Ubuntu); the QVAC server from step 3 runs natively and Hermes reaches it over localhost either way.
  5. 05

    Point Hermes at QVAC

    Tell Hermes to use your local server as a custom OpenAI-compatible provider. qwen3.6-moe must match the alias in your config. The api-key is only a placeholder: the local QVAC server does not check it, but Hermes wants the field filled, so any non-empty value works.
  6. 06

    Talk to your local agent

    Opens an interactive chat. Try turning off your network and asking whether it runs in the cloud or on your machine. It keeps answering.
Install the QVAC CLI
npm install -g @qvac/cli
mkdir -p ~/qvac-hermes && cd ~/qvac-hermes
cat > qvac.config.json <<'EOF'
{
  "plugins": ["@qvac/sdk/llamacpp-completion/plugin"],
  "serve": {
    "models": {
      "qwen3.6-moe": {
        "model": "QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M",
        "type": "llamacpp-completion",
        "default": true,
        "preload": true,
        "config": { "tools": true, "toolsMode": "static", "ctx_size": 32768, "gpu_layers": -1, "reasoning_budget": 0 }
      }
    }
  }
}
EOF
Start the QVAC server
qvac serve openai --model qwen3.6-moe
Install Hermes Agent
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
Point Hermes at QVAC
hermes config set model.provider custom
hermes config set model.base_url "http://localhost:11434/v1"
hermes config set model.default qwen3.6-moe
hermes config set model.api_key qvac
Talk to your local agent
hermes

What you just built, and what to try next

You now have a personal AI agent running entirely on your own machine. A good first test is the obvious one: ask it whether it is in the cloud or local, then enable airplane mode and ask again. It keeps answering, because the inference was never leaving your device.

From there, the things that make Hermes Hermes all run on the same local model:

  • Give it a personality. Edit ~/.hermes/SOUL.md, then send a message. The persona is loaded fresh each time, so the change is immediate.
  • Let it remember. Tell it something about how you work, start a new session later, and it recalls it. The memory lives in ~/.hermes/.
  • Use a skill. Run hermes skills to see the library, then ask it for something real: take structured notes, organize a project on its task board, summarize a folder of documents. Pick local skills (notes, files, board) to stay fully offline; web skills will use the internet for that tool.
  • Let it improve. Over time the curator refines skills from what you do. That learning happens on your hardware and stays there.

For speed, narrow the active skills to what you actually use (hermes skills, hermes tools); a smaller toolset means a smaller prompt and faster turns on a local model.

Notes and limits

  • A model that fits on a laptop is not a frontier cloud model. Expect it to need clearer instructions and to be slower than a cloud API today. That gap is closing as local hardware speeds up and small models improve.
  • "Local" here means the model runs on your device. A given skill may still call the internet for its own job (web research, for example), the same as any app would. The intelligence, your persona, and your memory stay on the machine.
  • QVAC is Apache 2.0 and free. Hermes Agent is from Nous Research (github.com/NousResearch/hermes-agent). The model is downloaded once and cached locally. QVAC source and docs: github.com/tetherto/qvac and docs.qvac.tether.io.

Share

Latest articles

View all articles

Stay updated

Never miss a release

New versions, breaking changes and migration notes - straight to your inbox. No spam, unsubscribe anytime.

How to run Hermes, a self-improving personal AI agent, full… | QVAC