I think we should circle back and fix this properly. I discovered the issue far too late into whisper development and we opted to mark the feature as experimental rather than fix.
The problem is the fix isn’t simple, but I think it’s worthwhile to figure it out.