"I’ll probably write an elisp function to send the output to GPT and insert the results into the buffer"
I suggest taking a look at my LLM command-line tool. It's great for cobbling together these kinds of things, because you can use whatever shell integration your environment has to pipe things to it.
cat bad-dictation.txt | \
llm -m gpt-4-turbo --system '
You are going to correct for text that
has been produced by voice recognition
software. Rewrite any text provided to
you. The text will not have punctuation,
so please add where needed. If the text
has the word period, then insert a
period, and do not insert another one
just before or after. If the text has
“new line” or “newline”, insert a newline
character and start a new paragraph. If
the text says “comma”, insert a comma. If
the text has a number spelled out,
replace it with the actual number. So
“five thousand” becomes “5000”. Also, if
something seems off in the text, it was
probably due to a misrecognized word.
Please correct for it.' > out.txt
I wrote it in December, when I was none the wiser. I've already got an elisp function (using whisper now for the backend - which has some cons to it).
In fact, it's fairly common for me to use the combination of Whisper and GPT to compose emails, etc. I should do a followup post on my setup.
Interestingly, Simon, I also submitted another story[1] where I refer to your shot-scraper. You're the gift that keeps on giving (especially given that my first programming job was with Django in 2008!)
BTW I’ve heard from researchers from a big GAFAM (can’t name) that they actually use gpt to do this on the ground truth and spot that correction and do a second pass of human labelling on the ground truth to have better dataset.
If you prefer an existing webapp over an elisp function, https://huggingface.co/spaces/ndurner/oai_chat. Choose Whisper as the model, upload your 25 MB chunks, hit Send, choose GPT-4 Turbo, ask it to clean up, hit Send. Then, hit the Download button (hidden away on the very bottom).
Helpful fact: Whisper works on 16 KHz sampling behind the scenes, so can make your recording smaller by downsampling to 22 KHz, mono. AAC is supported, and commenters to the web say Whisper is pretty robust so it doesn‘t have do be hi-fi - just so that you can make a split at the beginning of the QA session perhaps, if you can‘t fit it into one 25 MB chunk right away.
I've tried using LLMs to restore and clean up raw unpunctuated transcripts, but they tend to hallucinate new words. And chunking is an issue since long transcripts need to be split according to the LLM input context size. But where do we split if we don't have the punctuation yet?
In Scribe I chose instead to restore punctuation marks using a token-classifier (here a DistilBert running in your browser) https://www.appblit.com/scribe
I take a lot of long voice notes thinking out loud during my walks. I use this prompt when I give the Whisper output to GPT-4. I've found it to be pretty reliable:
> Punctuate the following transcript of a voice note I took. Insert periods, commas, and paragraph breaks where appropriate. Remove filler words such as 'right?' 'you know', and 'uh'. But do retain my original wording! Do not paraphrase my sentences beyond recognition: this is not a rewriting task! The transcript now follows:
As I mention in another comment, I actually do now use Whisper. In fact, I just spent the last few hours using Whisper and GPT to write some drafts.
Whisper has a very serious downside: It often ignores quite a bit of relevant audio, and I dealt with this many times in the last few hours, to the point where I'm considering finding an alternative.
Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.)
But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls.
I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR?
I had a look recently, and what I found was:
- Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate).
- Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data.
- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.
- Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.
“- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.”
That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?
I suggest taking a look at my LLM command-line tool. It's great for cobbling together these kinds of things, because you can use whatever shell integration your environment has to pipe things to it.
Should be easy to call that from elisp, and you can then install plugins to have it talk to other models like Claude or Llama 3: https://llm.datasette.io/en/stable/plugins/directory.html