FYI llama.cpp can do that for a "while" https://github.com/ggerganov/llama.cpp/pull/1773

Somebody is also working on a whisper.cpp version, which is maybe even more interesting because if you have grammar you can speak not only JSON but also a code (or anything)