A little over a month ago, I spent a couple days playing with AI speech recognition models and Xcode to create a live captioning MacOS and iOS app. Yes, Apple already makes one (called nothing other than Live Captions); but I have my own ideas and wanted to see if I could implement it, too. It gave me an excuse to play around in Xcode and see what it was like.

I created a floating caption window using whisper.cpp (a port of OpenAI’s Whisper in C++) to create a MacOS program that could create live captions and save them as transcripts. This worked both from sound input locally from YouTube videos, for example, and externally from sounds occurring around the computer (or whatever device it is using as an audio input).

The difficult part with live captions has always been maintaining both speed and accuracy. Opting for small chunk sizes and using the smaller models of whisper.cpp resulted in eerie hallucinations where complete silence got transcribed as “thank you” and “okay” due to how OpenAI’s Whisper models were trained on video subtitles often ending with those phrases. I found the best results by using the large-v3-turbo-q5_0 model, increasing chunk sizes from 500ms to 5 seconds, switching from greedy to beam search decoding, and adding an RMS energy gate to reduce hallucinations.

For the iOS app, my next step is to combine Apple’s SFSpeechRecognizer (or their newer SpeechAnalyzer) with whisper.cpp to see if/how that makes it more accurate and fast. It would initially process speech with SFSpeechRecognizer/SpeechAnalyzer and then use whisper.cpp, so that it would give you some context quickly while also giving you accurate captioning shortly thereafter. Unfortunately, I haven’t finished the iPhone app. My oldest child enjoyed helping me test it with SFSpeechRecognizer, but when it came time to add in whisper.cpp functionality, I became very fed up with Xcode’s cursed UI that didn’t save .plist changes while I was personally crumbling underneath the vast distractions and stresses of summer vacation. So, I have placed my captioning app idea on the back burner for now.