Word error rate on public test sets
The same public sets Wispr reports for its Canto model, plus Voice Arena's public conversational set. Lower is better. "Live" is exactly what you get while you talk (sliding-window streaming); the full-context pass re-reads the whole dictation in one go.
| Test set | Clips | Audio | OpenFlow live (streaming) | OpenFlow full-context pass | Wispr Canto (published) |
|---|---|---|---|---|---|
| LibriSpeech test-clean | 2,620 | 5.4 h | 1.80% | 1.70% | 1.7% |
| FLEURS en-US | 647 | 1.8 h | 5.12% | 4.65% | 3.6% |
| Common Voice (Open ASR test split) | 16,334 | 27.0 h | – | 8.95% | 9.2% |
| LibriSpeech test-other | 2,939 | 5.3 h | 3.60% | 3.22% | – |
| Voice Arena MonsoonASR-en-US (public) | 382 | 0.8 h | 5.73% | 5.08% | – |
Side by side with Wispr's published Canto table
| Test set | OpenFlow (on-device) | Wispr Canto | Gemini 3.1 Pro | Gemini 3.5 Transcribe | GPT Transcribe | AssemblyAI | Deepgram Nova-3 |
|---|---|---|---|---|---|---|---|
| LibriSpeech | 1.70% | 1.7% | 1.8% | 2.1% | 1.8% | 1.7% | 2.9% |
| FLEURS | 4.65% | 3.6% | 3.5% | 3.5% | 3.2% | 2.9% | 8.6% |
| Common Voice | 8.95% | 9.2% | 8.8% | 10.7% | 10.7% | 8.7% | 26.0% |
On real dictations (the developer's own 83 recordings, scored against Wispr Flow's transcripts of the same audio), the full-context pass cut word errors by 16-33% on dictations up to 15-25 seconds (up to 15 s: 3.31% to 2.21%). OpenFlow 1.3 uses it for dictations up to 20 seconds, where key-up to text is 112 ms median (up to 241 ms) instead of a few ms; longer dictations use streaming.
Wispr's numbers are from their Canto post (they did not say which LibriSpeech split, or which Common Voice version and subset). Ours: LibriSpeech test-clean, FLEURS en_us test, the Common Voice test split used by the Open ASR Leaderboard; Whisper English normalizer for both reference and output; no dictionary or context.
Voice Arena public conversational set (MonsoonASR-en-US)
Other systems: Voice Arena's published public-set column (voicearena.com, US English). OpenFlow: our run of the same 382 clips with the same normalizer family. Voice Arena ranks on a private set we can't access; this is the public half.
OpenFlow Dictation Bench v1
Built like Voice Arena's Dictation Bench: would you send the typed text as is? 1,105 dictations: 725 synthetic cases across AI prompts, code, email, work chat, messaging, notes and long form (micro to long, easy to hard: fillers, self-corrections, numbers, emails, spoken lists), voiced by 28 voices with room noise on harder cases, plus 380 real human recordings from Voice Arena's public set. Three judges from three labs (Anthropic Claude Haiku 5.5, DeepSeek V4.1 Flash, Qwen 3.8 Flash); a dictation counts as zero-edit when at least two agree it could be sent as is. The judges are strict (a US spelling in a Canadian's email counts), and the cases are hard on purpose (invented names, self-corrections, numbers), so absolute rates are low; compare systems.
OpenFlow 1.5 (17 Nov, Apple-silicon Macs): a bigger clean-up model (LFM2.5 2.6B, LoRA rank 256), trained on targets rewritten with a stricter clean-up guide (self-corrections keep only the final value, speech-planning talk removed, list lead-ins kept). In the shipped engine: 37.4% zero-edit vs 33.1% for 1.4, all three judges agree (38.4 / 34.4 / 40.5 vs 34.0 / 31.3 / 36.1); real conversational speech 35.3% vs 28.4%. Key-up median 49 ms on the same 30 recordings as 1.4. Identical training runs differ by about 1 point, so smaller differences on this page are noise. Screen context (OpenFlow reads the names around where you dictate, on the device) is measured by giving each dictation its own names plus 5 unrelated names, as a busy screen would: 41.9% zero-edit for 1.5 (1.4: 37.8%). These rows assume the names are visible, which is true when replying to a thread and not when dictating something new. Windows now uses a 1.2B model (36.2% vs 30.0% for the 350M it replaced, measured on the same transcripts outside the engine); Android keeps the 350M.
| System | Zero-edit rate | Words to fix | Synthetic (725) | Real speech (380) | Hard cases |
|---|---|---|---|---|---|
| Reference texts (sanity check) | 100.0% | 0.00% | 100.0% | –% | 100.0% |
| OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen) | 41.9% | 8.42% | 44.3% | 37.5% | 39.9% |
| OpenFlow 1.4 + screen context (simulated, as above) | 37.8% | 10.34% | 42.8% | 28.2% | 38.3% |
| OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model) | 37.4% | 9.15% | 38.5% | 35.3% | 34.5% |
| OpenFlow 1.4 universal Mac + screen context (simulated, as above) | 36.0% | 10.56% | 40.1% | 28.2% | 32.5% |
| OpenFlow 1.4 (previous Mac version, model v3) | 33.1% | 11.25% | 35.6% | 28.4% | 31.5% |
| OpenFlow 1.4 universal Mac, portable 1.2B model v3 | 31.2% | 11.02% | 32.6% | 28.8% | 28.9% |
| OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above) | 31.2% | 11.75% | 33.7% | 26.6% | 24.7% |
| 1.4 engine with the 1.3 clean-up model (v2): isolates the new model | 30.7% | 11.50% | 30.9% | 30.3% | 24.7% |
| OpenFlow Android 0.4.4 and Windows, 350M model v3 | 28.9% | 12.06% | 30.1% | 26.6% | 23.4% |
| OpenFlow 1.3 as first published (streaming only, no punctuation fix) | 26.8% | 12.34% | 28.8% | 22.9% | 23.0% |
| OpenFlow portable 1.2B, model v2 | 25.1% | 12.89% | 26.9% | 21.6% | 20.5% |
| Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only) | 23.7% | 12.37% | 32.7% | 6.6% | 26.8% |
| OpenFlow Android 350M, model v2 | 22.7% | 17.88% | 22.5% | 23.2% | 19.1% |
| Whisper large-v3-turbo, no clean-up | 16.9% | 26.90% | 8.3% | 33.4% | 0.0% |
| Parakeet TDT 0.6B, no clean-up | 12.9% | 29.74% | 8.0% | 22.1% | 0.0% |
| OpenFlow 1.2 (on-device, public model v1) | 12.1% | 20.86% | 16.7% | 3.4% | 7.7% |
Zero-edit rate by domain
| System | AI prompt | Code | Work chat | Messaging | Notes | Long form | Real speech | |
|---|---|---|---|---|---|---|---|---|
| OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen) | 58.3 | 57.8 | 34.1 | 40.9 | 44.5 | 38.3 | 25.0 | 37.5 |
| OpenFlow 1.4 + screen context (simulated, as above) | 47.0 | 58.2 | 37.6 | 46.1 | 36.7 | 43.3 | 18.3 | 28.2 |
| OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model) | 52.2 | 53.6 | 24.7 | 34.8 | 40.8 | 33.3 | 16.7 | 35.3 |
| OpenFlow 1.4 universal Mac + screen context (simulated, as above) | 48.7 | 53.2 | 27.1 | 40.9 | 41.7 | 36.7 | 20.0 | 28.2 |
| OpenFlow 1.4 (previous Mac version, model v3) | 41.7 | 51.8 | 28.2 | 34.8 | 31.7 | 36.7 | 11.7 | 28.4 |
| OpenFlow 1.4 universal Mac, portable 1.2B model v3 | 41.7 | 50.9 | 18.8 | 26.1 | 34.2 | 30.8 | 13.3 | 28.8 |
| OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above) | 40.9 | 48.2 | 24.7 | 34.8 | 31.1 | 33.3 | 10.0 | 26.6 |
| 1.4 engine with the 1.3 clean-up model (v2): isolates the new model | 36.5 | 48.2 | 18.8 | 26.1 | 32.5 | 31.7 | 10.0 | 30.3 |
| OpenFlow Android 0.4.4 and Windows, 350M model v3 | 35.7 | 49.1 | 22.4 | 27.8 | 29.2 | 28.3 | 5.0 | 26.6 |
| OpenFlow 1.3 as first published (streaming only, no punctuation fix) | 36.5 | 50.9 | 16.5 | 21.7 | 25.0 | 29.2 | 11.7 | 22.9 |
| OpenFlow portable 1.2B, model v2 | 36.0 | 46.4 | 11.8 | 25.2 | 23.3 | 23.3 | 13.3 | 21.6 |
| Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only) | 49.6 | 56.4 | 20.0 | 40.0 | 12.5 | 22.5 | 21.7 | 6.6 |
| OpenFlow Android 350M, model v2 | 36.5 | 34.5 | 16.5 | 16.5 | 19.2 | 19.2 | 6.7 | 23.2 |
| Whisper large-v3-turbo, no clean-up | 16.5 | 9.1 | 0.0 | 5.2 | 10.0 | 7.5 | 6.7 | 33.4 |
| Parakeet TDT 0.6B, no clean-up | 13.9 | 9.1 | 2.4 | 5.2 | 10.8 | 5.8 | 6.7 | 22.1 |
| OpenFlow 1.2 (on-device, public model v1) | 29.6 | 31.8 | 10.6 | 13.9 | 3.3 | 15.0 | 8.3 | 3.4 |
Each judge on its own
| System | claude-haiku-5.5 | deepseek-v4.1-flash | qwen3.8-flash |
|---|---|---|---|
| Reference texts (sanity check) | 100.0% | 100.0% | 100.0% |
| OpenFlow 1.5 + screen context (simulated: the dictation's names + 5 unrelated names on screen) | 42.1% | 39.0% | 46.1% |
| OpenFlow 1.4 + screen context (simulated, as above) | 37.9% | 35.0% | 40.6% |
| OpenFlow 1.5 (Apple-silicon Mac, 2.6B clean-up model) | 38.4% | 34.4% | 40.5% |
| OpenFlow 1.4 universal Mac + screen context (simulated, as above) | 35.6% | 34.1% | 39.1% |
| OpenFlow 1.4 (previous Mac version, model v3) | 34.0% | 31.3% | 36.1% |
| OpenFlow 1.4 universal Mac, portable 1.2B model v3 | 31.0% | 29.3% | 34.0% |
| OpenFlow Android 0.4.4 and Windows + screen context (simulated, as above) | 30.7% | 28.9% | 33.4% |
| 1.4 engine with the 1.3 clean-up model (v2): isolates the new model | 30.8% | 29.3% | 34.1% |
| OpenFlow Android 0.4.4 and Windows, 350M model v3 | 28.8% | 27.5% | 31.3% |
| OpenFlow 1.3 as first published (streaming only, no punctuation fix) | 25.3% | 28.5% | 27.8% |
| OpenFlow portable 1.2B, model v2 | 24.0% | 26.6% | 25.1% |
| Experiment, not shipped: our speech recognition + Gemini 3.8 Flash clean-up (text only) | 23.0% | 30.2% | 24.4% |
| OpenFlow Android 350M, model v2 | 20.3% | 23.7% | 23.1% |
| Whisper large-v3-turbo, no clean-up | 16.7% | 16.0% | 19.1% |
| Parakeet TDT 0.6B, no clean-up | 12.7% | 13.5% | 14.2% |
| OpenFlow 1.2 (on-device, public model v1) | 11.4% | 17.9% | 12.7% |
For context, Voice Arena's own Dictation Bench (private recordings, played into installed Mac apps) reports: Wispr Flow 79.4%, Gemini 3.5 Transcribe 75.8%, Willow 75.7%, Superwhisper 59.8%, Apple Dictation 34.6%. Those are not directly comparable to the numbers above (different recordings and judges). OpenFlow has not yet been tested by Voice Arena.
Key release to text
OpenFlow transcribes and cleans up while you are still talking, using speculative decoding (draft tokens copied from the transcript and verified in one pass: 3.1-4.5 tokens per forward, 1.79x faster on the Mac GPU) and reusing the work at key-up. New in 1.3: for dictations up to 20 seconds, the final text comes from one pass over the whole recording (no window stitching), computed at each pause. It was ready in time for 3 of the 11 short recordings below; otherwise it runs after you let go, which is why 1.3's median is 51 ms instead of 1.2's 2 ms. We chose accuracy: on our own real dictations it cut word errors by 16-33%. 1.4 (new clean-up model, same engine) measured 54 ms median over two runs of the same recordings (55 and 53 ms). The portable engine (Windows, universal Mac) now also finishes during pauses: on a 4-thread laptop CPU, key-up median 0 ms and p90 278 ms on 64 Dictation Bench clips replayed in real time, down from 481 ms and 823 ms.
| System (key release to final text) | Median | p90 | Max |
|---|---|---|---|
| OpenFlow 1.4, 30 recordings (2 runs pooled) | 54 ms | 170 ms | 243 ms |
| OpenFlow 1.4, those up to 20 s (full-context pass) (2 runs pooled) | 124 ms | 187 ms | 243 ms |
| OpenFlow 1.3 (same engine, model v2) | 51 ms | 189 ms | 241 ms |
| OpenFlow 1.2 (published, same 30 recordings) | 2 ms | 142 ms | 185 ms |
| Wispr Flow (reported by phonon on its own clips; not measured by us) | 520 ms | – | – |
Method: 30 real dictations (our own recordings, not published), replayed into the shipped Apple-silicon package in 100 ms pieces at real speed on a Mac mini (M4 Pro), GPU otherwise idle. The clock starts when the last audio arrives (key release) and stops when the final text is ready. The Wispr Flow figure is reported, not measured by us.
Dreaming: memory that predicts what you'll say
While your Mac is idle, OpenFlow "dreams": a local model rereads your dictation history and rewrites a small layered memory (names and the ways they get misheard, habits, ongoing topics, per-app style). The memory picks which names to listen for and suggests what you might say next. Measured on the developer's own dictation history in date order: memory built only from earlier dictations, tested on later ones. Nothing leaves the device.
On 95 later dictations, the hand-made dictionary already contained 22% of the names and jargon used; with the dreamed memory's 40 picks added, 50%.
| Next-sentence predictor | Same intent as what was said | Related or better |
|---|---|---|
| Same model, no memory | 4% | 46% |
| Dreaming memory + small local model | 0% | 23% |
| Habit phrases only (no model) | 3% | 18% |
Reproduce every number
python scripts/fetch_public_bench.py # public test sets, cached as 16 kHz audio python scripts/public_bench.py ls_clean,fleurs,cv,monsoon offline,stream python scripts/gen_dictation_bench.py && python scripts/build_dictation_bench_audio.py synth && python scripts/build_dictation_bench_audio.py real python scripts/run_dictation_bench.py openflow && python scripts/judge_dictation_bench.py openflow,parakeet,whisper,reference python scripts/judge_dictation_bench.py --report
Hardware: Apple M4 Pro, 48 GB. Speech model: NVIDIA Parakeet TDT 0.6B v2 (MLX, fp16). Clean-up: OpenFlow's public LFM3-1.7B fine-tune (4-bit), trained only on public and synthetic data. Judges: Claude Haiku 5.5, GPT-5.6 Luna, Gemini 3.5 Flash Lite, DeepSeek V4.1 Flash, Qwen 3.8 Flash (reasoning off). Limits: the synthetic track uses TTS voices (cleaner than real microphones); LLM judges are not humans; English only. FLEURS is scored against its normalized transcription column: the raw column puts words in parentheses, which the standard Whisper normalizer deletes (that alone turns 4.65% into 6.35%).