QVAC SDK v0.21.0
📦 NPM: https://www.npmjs.com/package/@qvac/sdk/v/0.21.0
QVAC SDK 0.21.0 sits on @qvac/[email protected]. A model can split across machines through llama.cpp's RPC backend, Ternary Bonsai 2 27B runs in QVAC's engine, tools can defer their schemas behind tool_search, and TTS/ASR pick up MOSS plus Parakeet Core ML on Apple. Mobile bundles can auto-install addon platform packages, and verifyBundle checks engines.bare against the Bare runtime each host actually runs. Catalog constant names for BitNet, Llama tool-calling, and Indic Parakeet change. assessModelFit now takes loadModel fields instead of a separate workload object, infers modelType from modelSrc when omitted, and the load-time fit probe runs on every engine that ships a fitter.
The committed range is ^0.21.0. tetherto-qvac-sdk publishes at the same version.
Table of Contents
Breaking Changes
assessModelFit uses loadModel parameters
Candidates are described the same way as a load: modelSrc, modelType, and modelConfig. The old model / workload / artifacts object is rejected.
Before:
await assessModelFit({
models: [
{
model: LLAMA_3_2_1B,
workload: { kind: 'llm', contextTokens: 4096 },
artifacts: [MMPROJ_F16]
}
]
})After:
await assessModelFit({
models: [
{
modelSrc: LLAMA_3_2_1B,
modelType: 'llamacpp-completion',
modelConfig: { ctx_size: 4096, projectionModelSrc: MMPROJ_F16 }
}
]
})modelType is optional and inferred from modelSrc when omitted, the same way loadModel infers it. Required only where the source names no engine. A load whose sources are all config fields may omit modelSrc. Audio workload.windowMs becomes modelConfig.duration_ms. workload.batch has no equivalent and is gone.
Flattened fit-probe projection
getLoadedModelInfo().fitProbe.projection is a flat byte breakdown. projection.devices and NativeProbeDevice are gone. Totals are summed across devices; deviceName names the first.
Before:
info.fitProbe?.projection?.devicesAfter:
info.fitProbe?.projection?.deviceBytes
info.fitProbe?.projection?.hostBytes
info.fitProbe?.projection?.weightsBytes
info.fitProbe?.projection?.contextBytes
info.fitProbe?.projection?.computeBytes
info.fitProbe?.projection?.deviceNameCatalog constant names
BitNet instructed TQ2_0 exports are retagged as base. Llama tool-calling 1B moves from Q4_K to Q4_K_M. Indic Parakeet Conformer CTC constants are replaced by the 600M GGUFs.
Before:
import { BITNET_0_7B_INST_TQ2_0 } from '@qvac/sdk'
import { LLAMA_TOOL_CALLING_1B_INST_Q4_K } from '@qvac/sdk'
import { PARAKEET_INDIC_CONFORMER_CTC_Q4_0 } from '@qvac/sdk'After:
import { BITNET_0_7B_BASE_TQ2_0 } from '@qvac/sdk'
import { LLAMA_TOOL_CALLING_1B_INST_Q4_K_M } from '@qvac/sdk'
import { PARAKEET_INDIC_CONFORMER_600M_Q4_0 } from '@qvac/sdk'New APIs
Clustered inference over RPC
startRpcServer starts a worker-owned llama.cpp RPC server. discoverRpcServers finds peers on a private topic and getRpcDeviceMap builds the devices string loadModel expects. One model splits across several machines, phones included, by layer or within a layer. Only the initiator needs the model file.
import {
startRpcServer,
stopRpcServer,
discoverRpcServers,
getRpcDeviceMap,
loadModel,
unloadModel
} from '@qvac/sdk'
const server = await startRpcServer({
host: '10.0.0.2',
allowNonLoopbackHost: true,
discoveryTopic: 'my-private-rpc-group'
})
const selected = (await discoverRpcServers({ topic: 'my-private-rpc-group' }))
.sort((a, b) => a.url.localeCompare(b.url))
.slice(0, 2)
const devices = getRpcDeviceMap(selected)
const modelId = await loadModel({
modelSrc,
modelConfig: {
device: 'gpu',
'rpc-servers': selected.map((s) => s.url).join(','),
devices: devices.map((d) => d.alias).join(','),
'split-mode': 'layer',
'tensor-split': devices.map(() => '1').join(',')
}
})
await unloadModel({ modelId })
await stopRpcServer({ serverId: server.serverId })Workers bind to loopback unless allowNonLoopbackHost is set. The channel is unauthenticated. Prebuilds use TCP; RDMA needs client and worker rebuilt on Linux with RDMA enabled. The current limit is 16 devices.
Deferred tool loading
A tool may set deferLoading: true (or an MCP client entry, to defer all its tools). The first prompt then carries only always-loaded tools and a built-in tool_search whose description lists the deferred tools by name and description. When the model calls tool_search, run executeToolSearch() and push its result as a tool message; the matching definitions are callable from the next turn. completion() stays stateless and reads the loaded set back from history, so keep that message. Tools without the flag are unchanged. qvac serve runs this loop itself.
import { completion, executeToolSearch, TOOL_SEARCH_NAME } from '@qvac/sdk'
const tools = [
{ name: 'get_weather', description: '...', parameters },
{
name: 'create_issue',
description: 'Open a new issue on a repository',
parameters,
deferLoading: true,
group: 'github'
}
]
const run = completion({ modelId, history, tools })
history.push({ role: 'assistant', content: (await run.final).raw.fullText })
for (const call of await run.toolCalls) {
const content =
call.name === TOOL_SEARCH_NAME
? executeToolSearch(tools, call.arguments, history)
: await runTool(call)
history.push({ role: 'tool', content })
}
// Defer every tool an MCP client exposes
completion({ modelId, history, mcp: [{ client, deferLoading: true, group: 'github' }] })New options: deferLoading and group on tools and on McpClientInput.
New exports: TOOL_SEARCH_NAME, executeToolSearch(tools, args, history), loadedToolNames(history), searchDeferredTools(deferred, query, limit?), buildToolSearchTool(deferred).
tool_search is reserved. generationParams.tool_choice cannot name a deferred tool; force tool_search or drop deferLoading from that tool.
MOSS TTS
tts-ggml 0.10.0 adds the MOSS engine. MOSS-TTS-v1.5 is directable (explicit pause and duration, Pinyin/IPA, optional voice cloning). MOSS-TTSD synthesizes 1 to 5 speakers in one pass. The Delay engine streams chunks so playback can start while generation continues.
import {
loadModel,
textToSpeech,
TTS_DELAY_LLM_MOSS_TTS_F16,
TTS_CODEC_DECODER_MOSS_TTS_F16,
TTS_CODEC_ENCODER_MOSS_TTS_F16
} from '@qvac/sdk'
const modelId = await loadModel({
modelSrc: TTS_DELAY_LLM_MOSS_TTS_F16,
modelType: 'tts',
modelConfig: {
ttsEngine: 'moss',
mossCodecDecoderModelSrc: TTS_CODEC_DECODER_MOSS_TTS_F16,
mossCodecEncoderModelSrc: TTS_CODEC_ENCODER_MOSS_TTS_F16,
referenceAudioSrc: '/voices/speaker-24k.wav',
streamChunkTokens: 25
}
})On Apple, Audio8 now stages its Core ML codec sidecar and reports codecSidecarLoaded / codecOnCoreml on synthesis stats.
ABot-World layer streaming
World-mode modelConfig.world accepts paramsBackend, maxVram, streamLayers, and the rest of the placement fields so an ABot-World graph can stream layers the same way other diffusion loads do.
Mobile host prebuilds and Bare runtime checks
ensureHostPrebuilds and bundleSdk({ installMissingPrebuilds: true }) add the Android/iOS platform packages the split speech addons need, pinned to each addon's version. qvac bundle sdk installs by default; --no-install skips it.
verifyBundle and qvac doctor compare each host's actual Bare runtime (from react-native-bare-kit on mobile) against engines.bare, and print the kit release or override to apply.
Loaded-model fit probe
getLoadedModelInfo can include fitProbe: verdict, a flat projection (deviceBytes, hostBytes, and the breakdown fields the engine filled), and the placement the fitter resolved. The probe runs on every engine that ships a fitter. assessModelFit model results can carry device and reasons. fitStubBudgetMs on config (default 40000) budgets the registry fetch of the weightless description.
BCI stream placement and diagnostics
bciTranscribe terminal frames can carry diagnostics. Streamed BCI segments expose windowStartTimestep so window-local timestamps can be placed on the stream timeline.
Features
Ternary Bonsai 2 27B (PTQ1_0 at 5.95 GB, PQ2_0 at 7.21 GB) loads through the qvac-fabric engine with the matching rotation applied at run time.
Parakeet GGUFs on Apple download their Core ML sidecars beside the weights. ACE-Step AudioGen packings and additional CosyVoice3 flow/hift/LLM constants are on the catalog.
Llama gpu_layers is unset by default so qvac-fabric can place layers to free device memory. Setting gpu_layers still pins the count and aborts that fit.
Hyperdrive model downloads stay inside the model cache. NMT translate() stats are per-request. Fused TTS export names come from registry tags. bare-runtime is ^1.30.3 so a retained lockfile cannot keep 1.24.x next to engines.bare >=1.30.3.
Bug Fixes
Loaded deferred tools run under the tool-call grammar instead of text parsing on the search-to-call step.
Model Changes
Added
AUDIOGEN_ACESTEP_5HZ_LM_0_6B_BF16
AUDIOGEN_ACESTEP_V15_BASE_Q4_K_M
BITNET_0_7B_BASE_TQ2_0
BITNET_1B_BASE_TQ2_0
BITNET_B1_58_3B_BASE_TQ2_0
LLAMA_TOOL_CALLING_1B_INST_Q4_K_M
PARAKEET_INDIC_CONFORMER_600M_F16
PARAKEET_INDIC_CONFORMER_600M_Q4_0
PARAKEET_INDIC_CONFORMER_600M_Q8_0
PARAKEET_TDT_1_1B_F16
PARAKEET_TDT_1_1B_Q8_0
TERNARY_BONSAI_2_27B_MULTIMODAL_PQ2_0
TERNARY_BONSAI_2_27B_MULTIMODAL_PTQ1_0
TTS_CODEC_DECODER_MOSS_TTS_F16
TTS_CODEC_ENCODER_MOSS_TTS_F16
TTS_COSYVOICE3_FLOW_COSYVOICE_BF16
TTS_COSYVOICE3_FLOW_COSYVOICE_FP16
TTS_COSYVOICE3_FLOW_COSYVOICE_Q4_0
TTS_COSYVOICE3_FLOW_COSYVOICE_Q8_0
TTS_COSYVOICE3_HIFT_COSYVOICE_FP16
TTS_COSYVOICE3_LLM_COSYVOICE_FUSED_Q8_0
TTS_COSYVOICE3_LLM_COSYVOICE_Q4_0
TTS_DELAY_LLM_MOSS_TTS_F16Removed
BITNET_0_7B_INST_TQ2_0
BITNET_1B_INST_TQ2_0
BITNET_B1_58_3B_INST_TQ2_0
LLAMA_TOOL_CALLING_1B_INST_Q4_K
PARAKEET_INDIC_CONFORMER_CTC_F16
PARAKEET_INDIC_CONFORMER_CTC_Q4_0
PARAKEET_INDIC_CONFORMER_CTC_Q8_0The Python client for QVAC: local-first, P2P AI inference (LLM completion,
embeddings, transcription, TTS, OCR, translation, diffusion, audio generation,
VLA, …) through the same worker and the same contract as the TypeScript
@qvac/sdk. Asyncio-native.
The package version tracks the @qvac/sdk version it speaks (e.g. 0.15.0), so
a given python-sdk release makes clear exactly which SDK it targets.
Install
The fastest path is a self-contained install — one command, no separate
worker, no Node.js. It pulls a per-platform wheel that bundles the QVAC worker
(the @qvac/sdk runtime) and the Bare runtime, from the matching GitHub
release (these wheels are far too large for PyPI, so they live as release
assets):
# Replace <version> with the release you want, e.g. sdk-v0.17.0:
pip install tetherto-qvac-sdk \
-f https://github.com/tetherto/qvac/releases/expanded_assets/sdk-v<version>pip selects the bundled wheel for your platform and pulls the pure-Python deps
(pydantic, bare-rpc, compact-encoding) from PyPI. Then
async with Client() as client: ... just works — the bundled worker is found
automatically, zero configuration. Optional extras: vla (numpy),
notebook (numpy + pandas).
Bundled wheels ship for darwin-arm64, linux-x64, linux-arm64, win32-x64. On any other platform pip falls back to the thin wheel below.
Thin install (any platform)
If there's no bundled wheel for your platform, or you'd rather run a shared system worker, install the thin client from PyPI and provide the worker separately:
pip install tetherto-qvac-sdkThen install the worker once (either route needs Node.js):
# via Python — fetches the exact worker version this package speaks and caches
# it under ~/.cache/qvac/worker/<version>:
python -m tetherto.qvac_sdk install-worker
# or via npm — install the @qvac/sdk version that MATCHES this package (they
# share a version; Client() warns on a mismatch):
npm install -g @qvac/sdk@<your tetherto-qvac-sdk version>Client() then finds the worker automatically (see "Worker resolution" below
for the full lookup order, including pointing QVAC_SDK_DIR at a locally-built
@qvac/sdk for development). The Python route pins the worker to the exact
version this package was generated against, so you never track it yourself.
Upgrading
To upgrade and keep the bundled worker, re-run the install with -f
pointing at the new release tag:
pip install -U tetherto-qvac-sdk \
-f https://github.com/tetherto/qvac/releases/expanded_assets/sdk-v<newversion>A plain pip install -U tetherto-qvac-sdk (no -f, or a stale tag) upgrades to
the thin PyPI wheel: pip applies no priority between locations and just
takes the highest version it can see, which on PyPI is the py3-none-any
wheel — you'd then need install-worker. To switch an already-installed thin
build to the bundled wheel at the same version, add --force-reinstall
(pip treats a version as satisfied regardless of which wheel is installed).
Quickstart
import asyncio
from tetherto.qvac_sdk import Client, load_model, completion, unload_model
from tetherto.qvac_sdk.models import LLAMA_3_2_1B_INST_Q4_0
async def main():
async with Client() as client:
t = client.transport
model_id = await load_model(t, model_src=LLAMA_3_2_1B_INST_Q4_0)
run = completion(
t,
model_id=model_id,
history=[{"role": "user", "content": "Explain quantum computing in one sentence"}],
)
async for event in run.events:
if event.type == "contentDelta":
print(event.text, end="", flush=True)
print()
await unload_model(t, model_id)
asyncio.run(main())More in examples/ — one runnable example per major capability
(completion events / tools / worker-orchestrated tools, cancel, embeddings,
translation, transcription, TTS, OCR, audio generation, registry queries, model
info, logging, VLA, plugins), mirroring packages/sdk/examples.
The public API (tetherto.qvac_sdk)
Everything you normally need is re-exported flat from tetherto.qvac_sdk; model constants
live in tetherto.qvac_sdk.models.
Client— starts/owns a worker connection;client.transportis passed to every call.- Ergonomic wrappers (kwargs + typed results):
load_model,unload_model,completion,translate,cancel,delete_cache,invoke_plugin/invoke_plugin_stream,model_registry_list/_search/_get_model. (Tool calling iscompletion(tools=...); the worker-orchestrated loop is an advanced path attetherto.qvac_sdk._completion.completion_orchestrate, not flat-public.) - Result types:
CompletionRun(.events,.final),CompletionFinal,ToolCall,TranslateRun. - Generated method stubs for every other contract method (
embed,transcribe,text_to_speech,ocr_stream,audio_gen_stream,audio_edit_stream,diffusion_stream,classify,get_model_info,download_asset, …), each taking a typed request model. - Request/response models + enums (
LoadModelRequest,ModelType, …), also available in full fromtetherto.qvac_sdk.schemas. - Errors (
QvacError,RPCError,InferenceCancelledError,ContextOverflowError, …) forexcept/isinstance. - Logging:
logging_stream,subscribe_server_logs,SDK_LOG_ID,SDK_ALL_LOG_ID. VLA:vla,vla_hparams,vla_preprocess_image,vla_pad_state. - Notebook facade:
tetherto.qvac_sdk.notebook.SyncClient— synchronous, numpy/pandas returns, live in-cell streaming.
The raw generated method stubs are also in tetherto.qvac_sdk.methods, and the pydantic
models in tetherto.qvac_sdk.schemas, if you prefer the explicit modules.
Audio generation currently uses the raw audio_gen_stream stub rather than an
ergonomic Python audio_gen() wrapper. Build an AudioGenStreamRequest, iterate
the progress and base64 PCM frames, and assemble the output audio. See
examples/audiogen.py. Source-driven editing
(audio_edit_stream with an AudioEditStreamRequest carrying the ordered
flow-edit / repaint operations) streams the same frame shape.
Intentional divergences from @qvac/sdk
Two top-level JS/TS facades are deliberately not ported; Python uses the ecosystem equivalent instead.
getLogger— use the standard libraryloggingmodule. Alogging.Handleris theLogTransportequivalent, and the log-stream surface that has no stdlib counterpart (logging_stream,subscribe_server_logs,SDK_LOG_ID,SDK_ALL_LOG_ID) is already exported above.profiler(process-wide aggregation +exportTable/exportJSON) — use the per-callprofiled_calland the__profilingenvelope helpers intetherto.qvac_sdk.profiling, aggregating the returnedProfilingReports yourself. The global profiler is not a Client-API capability and would only collect useful data after instrumenting the full streaming client.
Notebook / data science
For notebooks and REPLs, tetherto.qvac_sdk.notebook.SyncClient runs the async
client on a background thread so every call is plain and blocking (no await),
with numpy/pandas returns and live in-cell streaming. Needs the notebook
extra (pip install "tetherto-qvac-sdk[notebook]"):
from tetherto.qvac_sdk.notebook import SyncClient
from tetherto.qvac_sdk.models import EMBEDDINGGEMMA_300M_Q4_0
with SyncClient() as client:
m = client.load_model(model_src=EMBEDDINGGEMMA_300M_Q4_0)
vec = client.embed(m, "hello") # 1-D numpy array
df = client.embed_frame(m, ["a", "b", "c"]) # pandas DataFrame, indexed by textembed returns numpy arrays, embed_frame a DataFrame, completion streams
live and returns the text, and transcribe/text_to_speech round-trip audio
as numpy. Full examples: examples/notebook.ipynb
(Jupyter) and examples/notebook.py (script).
Configuring the SDK
Client(config=...) (and BareRpcTransport(config=...)) sends an SDK
config to the worker on connect — the same QvacConfig the TypeScript client
applies (cacheDirectory, loggerLevel, swarmRelays, per-device plugin
defaults, …):
async with Client(config={"cacheDirectory": "/data/qvac-models", "loggerLevel": "warn"}) as client:
...cacheDirectory (must be absolute) relocates where models are stored — handy
for a shared/mounted cache. As a shortcut, the QVAC_CACHE_DIR env var sets
cacheDirectory without code (used by CI to point at a warm model cache).
Worker resolution
Client() locates a worker by trying, in order: explicit worker_path /
bare_path (or QVAC_WORKER_PATH / QVAC_BARE_PATH); a self-contained
bundled wheel (tetherto/qvac_sdk/_bundle/, zero config); sdk_dir / QVAC_SDK_DIR
pointing at an installed @qvac/sdk; a worker fetched by python -m tetherto.qvac_sdk install-worker (into ~/.cache/qvac/worker/<version>, overridable with
QVAC_WORKER_HOME); then a global npm install -g @qvac/sdk. The fetched
worker is version-locked to this package.
Staying in sync with @qvac/sdk (drift avoidance)
The rule: generate what can be generated; put un-generatable behaviour in the worker; hand-write per language only what is irreducibly client-side, and never trust it to match — guard it.
- Generated from the one contract. The pydantic models + typed method stubs,
the model-type resolution maps (
_generated/model_type_maps.py), the error-code registries (_generated/error_codes.py), and the pinned SDK version (_generated/sdk_version.py) are all generated from../sdk/contract/**.generate.py --check(and the SDK'scontract:check) fail CI if either side drifts. - Behaviour lives in the worker.
translatesource-language detection and the completion tool-loop (completionOrchestrate) run in the worker, so Python and JS share one implementation instead of two that diverge (they did: Python once usedlingua, JS@qvac/langdetect-text). - Version lock-step. This package's version is the
@qvac/sdkversion it was generated against. - Conformance corpus.
../sdk/e2e/conformance/cases.jsonis run by both a JS runner andtests/test_conformance.py, so the two clients are diffed against the same cases.
The irreducibly-client-side code (typed error classes, numpy marshaling, stream assembly, the notebook facade) is the only hand-written surface mirroring the JS client, and it's covered by the conformance corpus + real-worker tests rather than trusted to match.
Development
python3 -m venv .venv
.venv/bin/pip install -e ".[gen,dev]"
.venv/bin/python3 scripts/generate.py # regenerate from ../sdk/contract
.venv/bin/python3 -m pytest # unit + (with a built worker) real-model e2eFormat, lint, typecheck, and the generation check (all run in CI):
.venv/bin/python3 scripts/generate.py --check
.venv/bin/python3 -m black --check src/tetherto/qvac_sdk scripts/ tests/
.venv/bin/python3 -m ruff check src/tetherto/qvac_sdk scripts/ tests/
.venv/bin/python3 -m mypy -p tetherto.qvac_sdk && .venv/bin/python3 -m mypy scripts testsReal-model tests spawn a worker (packages/sdk built via bun run build, or
QVAC_POC_SDK_DIR) and otherwise skip. generate.py runs black + ruff --fix --select I (with the package config) on its own output, so a fresh
regeneration already passes the checks above.
Stay updated
Never miss a release
New versions, breaking changes and migration notes - straight to your inbox. No spam, unsubscribe anytime.