How much can you trust a local transcript?

One of the recordings we tested contains “I couldn’t do this one.” Phonon returned “I can do this one.” The transcript reads smoothly, but it changes what the speaker meant.

That is an awkward problem if you’re choosing a transcription model. You want a useful way to compare them, but a score cannot tell you whether the word a model missed was “um,” someone’s surname, or the “not” in a decision.

We build MeetingStow, which transcribes meetings on your Mac. We wanted to compare the four engines in the app: Apple SpeechAnalyzer, Qwen3-ASR, Phonon-2, and Parakeet Ultra. Ultra made the fewest word errors in our test, while Apple took the least time. Looking through the actual transcripts helped us understand what those results did and didn’t tell us.

The comparison uses 26 public clips: six English meeting excerpts and twenty Italian recordings of people reading sentences. That is enough to find useful examples and check our implementation, but it is a small sample. We’ll explain the terms first, then walk through the results. If you’d rather inspect the outputs yourself, the recordings’ references, transcripts, scores, and reproduction tools are available below.

Is speech-to-text the same thing as TTS?

It’s easy to mix up the names. Speech-to-text, or STT, takes audio and produces written words. You’ll also see it called automatic speech recognition, or ASR. Text-to-speech, or TTS, starts with written words and generates a spoken voice. A transcription app uses the first; a tool that reads an article aloud uses the second.

This article measures speech-to-text. If you’re choosing a TTS voice, you’ll want to listen for pronunciation, natural pacing, and how it handles names and numbers, as well as how quickly speech starts. Our transcription scores don’t measure those things. Apple has separate APIs for the two jobs: SpeechAnalyzer for recognition and AVSpeechSynthesizer for speaking text.

What happens between recording a call and reading the notes?

Recognizing the words is only one part of making a transcript you can use. These are the other terms you’ll encounter in the comparison:

Voice activity detection

Usually shortened to VAD. It estimates which parts of a recording contain speech. An app can use that information to skip silence or choose where to split the audio. A poorly placed cut can lose part of a word.

Diarization

Works out who spoke when, usually with labels such as “Speaker 1” and “Speaker 2.” It doesn’t, by itself, tell you a person’s name. Recognizing someone from a saved voice profile is a separate task.

Alignment

Places recognized words on the audio timeline. That gives you timestamps you can use to replay a sentence. A transcript can have the right words and still send you to the wrong point in the recording.

Summarization

Turns the transcript into shorter notes, decisions, or action items. It is a separate model task. We didn’t score recaps here, so a good transcription result is not a measurement of summary quality.

You can therefore get several different things wrong: the words, the speaker labels, the timestamps, or the summary. If you need to verify a client’s deadline, having the right words attached to a useful timestamp matters more than a tidy paragraph.

Why can two apps using the same model behave differently?

The model contains learned patterns for recognizing speech, stored as numerical weights. The runtime is the software that runs the model. The app also decides how to prepare the audio, split it into chunks, and assemble the results. When we say “engine” below, we mean that implemented recognition path, not just a model name.

You’ll see labels such as FP16 and Q8_0 in the method table. FP16 uses 16-bit floating-point numbers; Q8_0 is an 8-bit quantization format. These describe how model weights are stored. Those choices can affect memory, speed, and output. Comparing the versions an app actually uses is more informative than assuming every implementation of a model behaves alike.

“Local” means the recording is processed on the device rather than sent to a remote service for recognition. The model may still need an initial download. It doesn’t tell you whether recognition is accurate, whether the app is fast on your hardware, or whether a backup service copies the resulting files.

What did we compare?

We ran the four engines on an Apple M2 Max with 64 GB RAM, running macOS 26.5.2. The app was a signed Debug build of MeetingStow 0.5.27 (build 35), with its native MLX package optimized. We used the app’s transcription methods so that audio preparation and transcript assembly would be included in the comparison.

We computed speaker intervals once for each clip and supplied the same intervals to every engine. None received the reference words, a list of names, or vocabulary hints. We checked the audio files against the original fixture hashes so every run used the same recordings. See the AMI documentation and FLEURS dataset card for the source recordings’ scope.

What does a word error rate of 13.94% mean?

Word error rate, or WER, counts three kinds of mistake: substituting one word for another, leaving a word out, or adding a word that wasn’t in the reference. Add those errors and divide by the number of words in the human reference. For example, one substitution and one missing word in a ten-word reference give a WER of 20%.

Lower is better, but the percentage isn’t a count of sentences understood correctly. It doesn’t say how serious the errors were, and it can exceed 100% if the model adds enough words. Our English result for Ultra is 108 errors across 775 reference words, which gives 13.94%.

Here are the four engines under the same scoring rules. Use the language buttons to switch samples; the timings are for processing all the clips after warmup.

In English, Ultra makes 108 errors out of 775 words, seven fewer than Phonon and sixteen fewer than Qwen. Its lead over Phonon is only 0.90 percentage points. Dropping one English clip at a time leaves Ultra with the lowest aggregate WER in 5 of six subsets. That tests sensitivity to these clips; it does not make six excerpts representative of meetings in general.

In Italian, Ultra makes 8 errors out of 493 words, compared with Qwen’s 11. Three errors separate them. English conversation and Italian read speech are different tasks: the lower Italian percentages do not establish that any engine is “better at Italian.”

How long would you actually wait for the transcript?

The transcription stage is quick on this machine: Ultra takes 4.99 seconds for the six minutes of English audio, and Apple takes 4.32 seconds. But we also spent 5.34 seconds separating the speakers. Adding that stage brings Ultra’s total to 10.33 seconds, before saving or creating a recap. Once recognition is this fast, work elsewhere in the app becomes a noticeable part of the wait.

These are warm runs: the model is already loaded and has processed a clip in each language. A cold start would include loading and initialization; a first-time setup may also include downloading model files. We haven’t combined those into a startup benchmark. And processing a finished recording faster than its duration doesn’t tell you how soon live captions would appear while someone is speaking.

We made two measured passes per clip. Every repeated word-error count matched, although one Apple Italian name changed spelling without changing the score. Both outputs and their timings are in the download. Two passes are useful as a consistency check, but they don’t tell us how speed varies over many runs.

What does a word error sound like?

The counts become more useful when you listen to the recording beside the output. Here is the sentence from the opening, with its surrounding words. Phonon changes the negation; the other three engines preserve it.

If you were writing notes from the Phonon version, you could come away with a different understanding of what the speaker could do. We didn’t run this through a summarizer, so we can’t say how a recap model would handle it. The transcription error is already enough to warrant checking the audio.

The same Italian sentence contains a harmless style difference: the reference writes 15 metri, while Phonon writes quindici metri. Both mean “fifteen meters.” That deserves different treatment from changing 2017 into 207.

Other production problems remain outside this test: clipping, quiet microphones, music, long silence, echo, and sentences that switch languages. Plausible but unsupported text is also a failure to investigate. We did not run a silence or hallucination stress test, so this article assigns no hallucination rate to any engine.

Why can the same model get a different score?

Should “fifteen” and “15” count as different words?

We rescored Italian with the same limited rules on both sides: split letter–digit boundaries, expand integer digits to Italian cardinal words, and expand numeric m to metri. They forgive 15 metri versus quindici metri, but not a change from 2017 to 207.

Apple falls from 24 to 23 errors, Qwen from 11 to 10, and Phonon from 59 to 51. Ultra stays at 8. Formatting explains some errors, but it does not change the ordering on these recordings.

What happens if Qwen runs without Apple’s help?

Our Qwen path uses Apple evidence to influence speech windows and text decisions, as well as to place text on the timeline. We reran Qwen on all 26 clips with that evidence disabled, using the app’s voice-activity-based fallback. This changes several pipeline decisions together, so it cannot isolate the effect of alignment alone.

Without Apple evidence, English changes from 124 to 116 errors; Italian changes from 11 to 14. The same pipeline change helps one sample and hurts the other.

That is why we’ve described the implementation alongside the model name. Giving Qwen the same recording with a different set of speech windows and correction rules changes its output, even though the underlying model is the same.

We would need word, timestamp, and speaker-label evaluations together before using this result to change Apple assistance in production. A better word count alone cannot settle that choice.

Which model should you try?

If your recordings resemble these samples, we’d start with Parakeet Ultra. It made seven fewer English errors than Phonon and three fewer Italian errors than Qwen. It also preserved the negation and date shown above. Those are reasons to try it on your own recordings, rather than evidence that it will always be best.

Apple is worth trying if you want a draft quickly and have macOS 26. It finished first in both samples, with more word errors. Whether that tradeoff is acceptable depends on what you’re doing with the text. Searching a recording for a topic and sending a client a written commitment demand different levels of checking.

We’d compare a few recordings that contain the things you need the model to get right: your colleagues’ names, technical terms, accents, or people talking over one another. Listen to the unclear passages and check the output. Include the time you spend correcting it when you decide whether a model is fast enough for your work.

We haven’t included Whisper, cloud services, or every other local model. These are the four engines already integrated into MeetingStow. The results help us choose among those implementations; they don’t settle the wider market.

What do we still need to test?

The next useful step is more conversation audio, particularly Italian meetings, with longer recordings and a wider range of speakers. We also need separate checks of speaker labels and timestamps. The current mixed-audio fixtures don’t include speaker-label ground truth or separate microphone tracks, so they can’t validate those parts of the app or its recent microphone-filtering changes.

MeetingStow has used Parakeet Ultra by default for new installs since version 0.5.25, while preserving existing engine and language preferences. This comparison checks that choice against a fixed set of clips. It also gives us recordings to return to when the app changes, so we can check whether an apparent improvement actually kept the words we care about.

Where can you read more?

If you came here looking for text-to-speech, Ariya Hidayat’s Kokoro walkthrough is about generating a voice from text. It includes an audio demo, setup commands, and a small timing experiment. That is a different task from recovering the words in a recording.

For recognition, Inscribe’s SpeechAnalyzer comparison and the author’s explanation of the method examine Apple’s APIs and Whisper on LibriSpeech. Oliver Wehrens’s Whisper timing experiment shows how much an optimized runtime can change a comparison. Their workloads differ from ours, so their percentages and times shouldn’t be combined with this table.

Simon Willison’s attempt to transcribe a city-council meeting with Gemini is also useful reading. He checks the generated text against the recording and finds incorrect timestamps and summarized speech where he asked for a transcript. It’s a concrete example of why checking a fluent output still takes work.

Can you check the results yourself?

The JSON includes human references, both outputs per engine, per-clip error counts and timings, audio hashes, and the supplementary analyses. The CSV contains aggregate scores. Every figure is available as SVG and PNG, and the PDF contains this version of the article.

If you want to rescore the saved transcripts, download the JSON and Python analysis above. You don’t need to load any speech models for that. To make new measurements through the app, the repository workflow is:

python app/Benchmarks/Transcription/benchmark.py prepare \
  --output app/.benchmark-data/transcription --italian-count 20
app/scripts/build-app.sh debug
MEETINGSTOW_RUN_TRANSCRIPTION_BENCHMARK="$PWD/app/.benchmark-data/transcription/fixtures.json" \
MEETINGSTOW_TRANSCRIPTION_BENCHMARK_OUTPUT="$PWD/app/.benchmark-data/transcription/native.json" \
  "app/.build/MeetingStow Dev.app/Contents/MacOS/MeetingStowNative"
python app/Benchmarks/Transcription/benchmark.py score \
  app/.benchmark-data/transcription/native.json --expected-engine-count 4 \
  --hardware-label 'Apple M2 Max' --output /tmp/native-results.json

The repository’s app/Benchmarks/Transcription/README.md has dependency setup and the Qwen alternative command. You can also download the standalone analysis to rescore the published outputs and recreate the charts without running a recognizer or accessing our repository. Build and run provenance records the binary and source hashes.

The measurements were collected on October 19 with MeetingStow 0.5.27. The earlier October 5 article covered three engines. This revision adds explanations; it uses the same saved outputs and scores.

Dataset attribution. English audio and references: AMI Consortium / University of Edinburgh, via the FluidInference AMI mirror and AMI 1.6.2 annotations. Italian audio and references: Google FLEURS, Conneau et al. Dataset material is credited under CC BY 4.0. The English player is an extracted excerpt; the Italian player retains the selected recording. Transcript highlights and scores are our analysis.