Benchmarks · updated 2026-11-17

Measured in public. Including where we lose.

Every OpenFlow number on this page comes from the app's own engine running on a Mac mini (Apple M4 Pro), on test sets anyone can download, scored by code anyone can run. Third-party numbers are quoted from their own publications.

Speech recognition

Word error rate on public test sets

The same public sets Wispr reports for its Canto model, plus Voice Arena's public conversational set. Lower is better. "Live" is exactly what you get while you talk (sliding-window streaming); the full-context pass re-reads the whole dictation in one go.

Test setClipsAudioOpenFlow live (streaming)OpenFlow full-context passWispr Canto (published)
LibriSpeech test-clean2,6205.4 h1.80%1.70%1.7%
FLEURS en-US6471.8 h5.12%4.65%3.6%
Common Voice (Open ASR test split)16,33427.0 h–8.95%9.2%
LibriSpeech test-other2,9395.3 h3.60%3.22%–
Voice Arena MonsoonASR-en-US (public)3820.8 h5.73%5.08%–

Side by side with Wispr's published Canto table

Test setOpenFlow (on-device)Wispr CantoGemini 3.1 ProGemini 3.5 TranscribeGPT TranscribeAssemblyAIDeepgram Nova-3
LibriSpeech1.70%1.7%1.8%2.1%1.8%1.7%2.9%
FLEURS4.65%3.6%3.5%3.5%3.2%2.9%8.6%
Common Voice8.95%9.2%8.8%10.7%10.7%8.7%26.0%

On real dictations (the developer's own 83 recordings, scored against Wispr Flow's transcripts of the same audio), the full-context pass cut word errors by 16-33% on dictations up to 15-25 seconds (up to 15 s: 3.31% to 2.21%). OpenFlow 1.3 uses it for dictations up to 20 seconds, where key-up to text is 112 ms median (up to 241 ms) instead of a few ms; longer dictations use streaming.

Wispr's numbers are from their Canto post (they did not say which LibriSpeech split, or which Common Voice version and subset). Ours: LibriSpeech test-clean, FLEURS en_us test, the Common Voice test split used by the Open ASR Leaderboard; Whisper English normalizer for both reference and output; no dictionary or context.

Voice Arena public conversational set (MonsoonASR-en-US)

Normalized WER, public set · lower is better
Resonant-1 (Reson8)
3.62%
Nemotron 3.5 Transcribe (NVIDIA, 30B)
3.79%
Scribe v3 (ElevenLabs)
3.83%
MAI-Transcribe-2 (Microsoft)
3.85%
Qwen3-ASR-1.7B (Alibaba)
3.93%
Gemini 3.5 Transcribe (Google)
4.3%
Parakeet-TDT-0.6B (NVIDIA)
4.42%
Universal-3.6 Pro Realtime (AssemblyAI)
4.68%
OpenFlow (on-device)
5.08%
Nova-3 Streaming (Deepgram)
5.4%
GPT-4o Mini Transcribe (OpenAI)
6.13%

Other systems: Voice Arena's published public-set column (voicearena.com, US English). OpenFlow: our run of the same 382 clips with the same normalizer family. Voice Arena ranks on a private set we can't access; this is the public half.

Dictation quality

OpenFlow Dictation Bench v1

Built like Voice Arena's Dictation Bench: would you send the typed text as is? 1,105 dictations: 725 synthetic cases across AI prompts, code, email, work chat, messaging, notes and long form (micro to long, easy to hard: fillers, self-corrections, numbers, emails, spoken lists), voiced by 28 voices with room noise on harder cases, plus 380 real human recordings from Voice Arena's public set. Three judges from three labs (Anthropic Claude Haiku 5.5, DeepSeek V4.1 Flash, Qwen 3.8 Flash); a dictation counts as zero-edit when at least two agree it could be sent as is. The judges are strict (a US spelling in a Canadian's email counts), and the cases are hard on purpose (invented names, self-corrections, numbers), so absolute rates are low; compare systems.

OpenFlow 1.5 (17 Nov, Apple-silicon Macs): a bigger clean-up model (LFM2.5 2.6B, LoRA rank 256), trained on targets rewritten with a stricter clean-up guide (self-corrections keep only the final value, speech-planning talk removed, list lead-ins kept). In the shipped engine: 37.4% zero-edit vs 33.1% for 1.4, all three judges agree (38.4 / 34.4 / 40.5 vs 34.0 / 31.3 / 36.1); real conversational speech 35.3% vs 28.4%. Key-up median 49 ms on the same 30 recordings as 1.4. Identical training runs differ by about 1 point, so smaller differences on this page are noise. Screen context (OpenFlow reads the names around where you dictate, on the device) is measured by giving each dictation its own names plus 5 unrelated names, as a busy screen would: 41.9% zero-edit for 1.5 (1.4: 37.8%). These rows assume the names are visible, which is true when replying to a thread and not when dictating something new. Windows now uses a 1.2B model (36.2% vs 30.0% for the 350M it replaced, measured on the same transcripts outside the engine); Android keeps the 350M.

SystemZero-edit rateWords to fixSynthetic (725)Real speech (380)Hard cases
Reference texts (sanity check)100.0%0.00%100.0%–%100.0%
OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen)41.9%8.42%44.3%37.5%39.9%
OpenFlow 1.4 + screen context (simulated, as above)37.8%10.34%42.8%28.2%38.3%
OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model)37.4%9.15%38.5%35.3%34.5%
OpenFlow 1.4 universal Mac + screen context (simulated, as above)36.0%10.56%40.1%28.2%32.5%
OpenFlow 1.4 (previous Mac version, model v3)33.1%11.25%35.6%28.4%31.5%
OpenFlow 1.4 universal Mac, portable 1.2B model v331.2%11.02%32.6%28.8%28.9%
OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above)31.2%11.75%33.7%26.6%24.7%
1.4 engine with the 1.3 clean-up model (v2): isolates the new model30.7%11.50%30.9%30.3%24.7%
OpenFlow Android 0.4.4 and Windows, 350M model v328.9%12.06%30.1%26.6%23.4%
OpenFlow 1.3 as first published (streaming only, no punctuation fix)26.8%12.34%28.8%22.9%23.0%
OpenFlow portable 1.2B, model v225.1%12.89%26.9%21.6%20.5%
Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only)23.7%12.37%32.7%6.6%26.8%
OpenFlow Android 350M, model v222.7%17.88%22.5%23.2%19.1%
Whisper large-v3-turbo, no clean-up16.9%26.90%8.3%33.4%0.0%
Parakeet TDT 0.6B, no clean-up12.9%29.74%8.0%22.1%0.0%
OpenFlow 1.2 (on-device, public model v1)12.1%20.86%16.7%3.4%7.7%

Zero-edit rate by domain

SystemAI promptCodeEmailWork chatMessagingNotesLong formReal speech
OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen)58.357.834.140.944.538.325.037.5
OpenFlow 1.4 + screen context (simulated, as above)47.058.237.646.136.743.318.328.2
OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model)52.253.624.734.840.833.316.735.3
OpenFlow 1.4 universal Mac + screen context (simulated, as above)48.753.227.140.941.736.720.028.2
OpenFlow 1.4 (previous Mac version, model v3)41.751.828.234.831.736.711.728.4
OpenFlow 1.4 universal Mac, portable 1.2B model v341.750.918.826.134.230.813.328.8
OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above)40.948.224.734.831.133.310.026.6
1.4 engine with the 1.3 clean-up model (v2): isolates the new model36.548.218.826.132.531.710.030.3
OpenFlow Android 0.4.4 and Windows, 350M model v335.749.122.427.829.228.35.026.6
OpenFlow 1.3 as first published (streaming only, no punctuation fix)36.550.916.521.725.029.211.722.9
OpenFlow portable 1.2B, model v236.046.411.825.223.323.313.321.6
Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only)49.656.420.040.012.522.521.76.6
OpenFlow Android 350M, model v236.534.516.516.519.219.26.723.2
Whisper large-v3-turbo, no clean-up16.59.10.05.210.07.56.733.4
Parakeet TDT 0.6B, no clean-up13.99.12.45.210.85.86.722.1
OpenFlow 1.2 (on-device, public model v1)29.631.810.613.93.315.08.33.4

Each judge on its own

Systemclaude-haiku-5.5deepseek-v4.1-flashqwen3.8-flash
Reference texts (sanity check)100.0%100.0%100.0%
OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen)42.1%39.0%46.1%
OpenFlow 1.4 + screen context (simulated, as above)37.9%35.0%40.6%
OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model)38.4%34.4%40.5%
OpenFlow 1.4 universal Mac + screen context (simulated, as above)35.6%34.1%39.1%
OpenFlow 1.4 (previous Mac version, model v3)34.0%31.3%36.1%
OpenFlow 1.4 universal Mac, portable 1.2B model v331.0%29.3%34.0%
OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above)30.7%28.9%33.4%
1.4 engine with the 1.3 clean-up model (v2): isolates the new model30.8%29.3%34.1%
OpenFlow Android 0.4.4 and Windows, 350M model v328.8%27.5%31.3%
OpenFlow 1.3 as first published (streaming only, no punctuation fix)25.3%28.5%27.8%
OpenFlow portable 1.2B, model v224.0%26.6%25.1%
Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only)23.0%30.2%24.4%
OpenFlow Android 350M, model v220.3%23.7%23.1%
Whisper large-v3-turbo, no clean-up16.7%16.0%19.1%
Parakeet TDT 0.6B, no clean-up12.7%13.5%14.2%
OpenFlow 1.2 (on-device, public model v1)11.4%17.9%12.7%

For context, Voice Arena's own Dictation Bench (private recordings, played into installed Mac apps) reports: Wispr Flow 79.4%, Gemini 3.5 Transcribe 75.8%, Willow 75.7%, Superwhisper 59.8%, Apple Dictation 34.6%. Those are not directly comparable to the numbers above (different recordings and judges). OpenFlow has not yet been tested by Voice Arena.

Speed

Key release to text

OpenFlow transcribes and cleans up while you are still talking, using speculative decoding (draft tokens copied from the transcript and verified in one pass: 3.1-4.5 tokens per forward, 1.79x faster on the Mac GPU) and reusing the work at key-up. New in 1.3: for dictations up to 20 seconds, the final text comes from one pass over the whole recording (no window stitching), computed at each pause. It was ready in time for 3 of the 11 short recordings below; otherwise it runs after you let go, which is why 1.3's median is 51 ms instead of 1.2's 2 ms. We chose accuracy: on our own real dictations it cut word errors by 16-33%. 1.4 (new clean-up model, same engine) measured 54 ms median over two runs of the same recordings (55 and 53 ms). The portable engine (Windows, universal Mac) now also finishes during pauses: on a 4-thread laptop CPU, key-up median 0 ms and p90 278 ms on 64 Dictation Bench clips replayed in real time, down from 481 ms and 823 ms.

System (key release to final text)Medianp90Max
OpenFlow 1.4, 30 recordings (2 runs pooled)54 ms170 ms243 ms
OpenFlow 1.4, those up to 20 s (full-context pass) (2 runs pooled)124 ms187 ms243 ms
OpenFlow 1.3 (same engine, model v2)51 ms189 ms241 ms
OpenFlow 1.2 (published, same 30 recordings)2 ms142 ms185 ms
Wispr Flow (reported by phonon on its own clips; not measured by us)520 ms––

Method: 30 real dictations (our own recordings, not published), replayed into the shipped Apple-silicon package in 100 ms pieces at real speed on a Mac mini (M4 Pro), GPU otherwise idle. The clock starts when the last audio arrives (key release) and stops when the final text is ready. The Wispr Flow figure is reported, not measured by us.

Memory

Dreaming: memory that predicts what you'll say

While your Mac is idle, OpenFlow "dreams": a local model rereads your dictation history and rewrites a small layered memory (names and the ways they get misheard, habits, ongoing topics, per-app style). The memory picks which names to listen for and suggests what you might say next. Measured on the developer's own dictation history in date order: memory built only from earlier dictations, tested on later ones. Nothing leaves the device.

On 95 later dictations, the hand-made dictionary already contained 22% of the names and jargon used; with the dreamed memory's 40 picks added, 50%.

Next-sentence predictorSame intent as what was saidRelated or better
Same model, no memory4%46%
Dreaming memory + small local model0%23%
Habit phrases only (no model)3%18%
Method

Reproduce every number

python scripts/fetch_public_bench.py                  # public test sets, cached as 16 kHz audio
python scripts/public_bench.py ls_clean,fleurs,cv,monsoon offline,stream
python scripts/gen_dictation_bench.py && python scripts/build_dictation_bench_audio.py synth && python scripts/build_dictation_bench_audio.py real
python scripts/run_dictation_bench.py openflow && python scripts/judge_dictation_bench.py openflow,parakeet,whisper,reference
python scripts/judge_dictation_bench.py --report

Hardware: Apple M4 Pro, 48 GB. Speech model: NVIDIA Parakeet TDT 0.6B v2 (MLX, fp16). Clean-up: OpenFlow's public LFM3-1.7B fine-tune (4-bit), trained only on public and synthetic data. Judges: Claude Haiku 5.5, GPT-5.6 Luna, Gemini 3.5 Flash Lite, DeepSeek V4.1 Flash, Qwen 3.8 Flash (reasoning off). Limits: the synthetic track uses TTS voices (cleaner than real microphones); LLM judges are not humans; English only. FLEURS is scored against its normalized transcription column: the raw column puts words in parentheses, which the standard Whisper normalizer deletes (that alone turns 4.65% into 6.35%).