Sub-Second Voice Typing Post-Processing with Local SLMs

Voice typing into developer terminals and coding agents frequently falls victim to acoustic misrecognitions: technical acronyms get mangled, domain terms turn into phonetically similar dictionary words, and grammatical homophones like two versus too lack contextual disambiguation.

Traditional regex and sed pipelines do not scale. They require endless manual curation, conflict with one another, and break valid numbers like two nodes. Conversely, routing every voice snippet to a cloud LLM gateway introduces 2+ seconds of latency, breaking conversational typing rhythm.

By placing a quantized 1.5B Small Language Model (SLM) running locally on CPU/iGPU immediately behind a local Whisper speech-to-text engine, we achieve context-aware grammatical cleanup in under 350 milliseconds.

The Architecture

The pipeline divides labor cleanly between acoustic decoding and semantic repair:

1. Acoustic Front-End (Whisper Medium via Vulkan GPU): Takes microphone audio and outputs fast, raw phonetics (~200ms).
2. Post-Processing Pipe (Local SLM): Feeds raw text into a locally hosted model via standard completions API.
3. Typing Injector (ydotool / clipboard): Injects cleaned text directly into the focused window.

[ Mic Audio ] -> [ Whisper STT ] -> [ Local Qwen 1.5B via llama-server ] -> [ OS Injector ]
                     (~200ms)                    (~100ms)

Why sed Rules Fail

A naive phonetic replacement table breaks legitimate language patterns:

# Naive sed replacement:
s/\b(two)\b/too/gI

# Input:
"we have two nodes and you two make mistakes"
# Output:
"we have too nodes and you too make mistakes" # Broken!

An SLM understands that two nodes denotes a cardinal number, while you two in the context of an address means adverbial too.

Latency Benchmark: Local SLM vs Gateway

We benchmarked three backends on the same real-world developer dictation:

A 2.5-second pause turns voice dictation into a waiting game. At ~300ms, the response appears almost instantaneously upon releasing the push-to-talk hotkey.

The Few-Shot Completion Pattern

Instruct models often try to converse when given standard chat templates. To ensure deterministic string transformation with zero conversational drift, use few-shot completions with a strict stop token:

const FEW_SHOT_PREFIX = `You are a speech transcription post-processor for a software engineer.
Clean the user speech input. Fix typos, slang, and phonetic speech errors:
- "two" -> "too" (adverbial: "you too", "me too", "too much"). NEVER change numbers ("two nodes").
- "jet" / "jat" -> "chat"
- "task" -> "tsq" (when referring to tasks or task CLI)
- "gitlix" -> "gitleaks"
Output ONLY the corrected text. Never converse. Never add commentary.

Input: mark this done in task right
Output: mark this done in tsq right

Input: we have two nodes and you two make mistakes with jet messages
Output: we have two nodes and you too make mistakes with chat messages

Input: `;

The Bun Post-Processor

Here is the lightweight TypeScript runner wired directly to stdin/stdout:

#!/usr/bin/env bun

interface CompletionResponse {
  choices?: Array<{ text?: string }>;
}

const ENDPOINT = "http://127.0.0.1:8082/v1/completions";
const TIMEOUT_MS = 2000;

const input = (await Bun.stdin.text()).trim();
if (!input) process.exit(0);

let result = input;
try {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), TIMEOUT_MS);

  const res = await fetch(ENDPOINT, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "qwen2.5-1.5b-instruct",
      prompt: `${FEW_SHOT_PREFIX}${input}\nOutput:`,
      temperature: 0.0,
      max_tokens: Math.max(input.split(/\s+/).length * 2, 40),
      stop: ["\n", "Input:"]
    }),
    signal: controller.signal
  });

  clearTimeout(timeout);

  if (res.ok) {
    const data = (await res.json()) as CompletionResponse;
    const cleaned = data.choices?.[0]?.text?.trim();
    if (cleaned) result = cleaned;
  }
} catch {
  // Fail-safe: fall back to raw input on timeout or failure
}

process.stdout.write(result);

Update: The Visual Feedback Problem on GNOME Wayland

Even with sub-350ms semantic cleanup, voice typing remains awkward without immediate visual feedback. When holding a push-to-talk key, how do you know the microphone is active and not hung on an audio buffer?

Voxtype provides an official companion waveform visualizer (voxtype-osd-gtk4), but deploying it on Ubuntu 24.04 under GNOME Wayland uncovers two specific desktop compositor hurdles:

1. Missing libgtk4-layer-shell Dependency

Upstream packages distribute voxtype-osd-gtk4 linked dynamically against libgtk4-layer-shell.so.0. Ubuntu 24.04 does not package this library in its standard universe repositories. Running the binary yields:

error while loading shared libraries: libgtk4-layer-shell.so.0: cannot open shared object file: No such file or directory

The resolution is extracting the shared library directly from the official Debian package (libgtk4-layer-shell0) into ~/.local/lib and pointing LD_LIBRARY_PATH to it.

2. The GNOME Mutter Layer-Shell Reality

Once loaded, voxtype-osd-gtk4 attempts to initialize a Wayland layer surface via the wlr-layer-shell protocol:

voxtype-osd-gtk: it appears your Wayland compositor does not support the Layer Shell protocol
voxtype-osd-gtk: Failed to initialize layer surface

GNOME Mutter strictly rejects wlr-layer-shell. Because GTK4 removed absolute window positioning APIs like gtk_window_move(), the unmanaged window falls back to origin (0, 0)—the top-left corner of the primary display.

Choosing Between Native Wayland and X11 Emulation

There are two practical workarounds on multi-monitor GNOME desktops:

For low latency and crisp HiDPI rendering across mixed DPI monitors, the native Wayland path anchored at (0, 0) proves the most reliable daily driver.

Source

• llama.cpp Server
• Qwen 2.5 1.5B Instruct GGUF
• gtk4-layer-shell
• Debian libgtk4-layer-shell0 Package