if you think that's bad, try to get it to generate Inform 7 games—Inform's natural-English-ish syntax completely throws all LLMs for a loop, consistently. it generates code that looks possibly correct (to an Inform newbie at least), but fails to compile far more often than not. I find this super interesting.

This worked with chatGPT: create a sample hello world in php

store that code in a json[object

code: { "php_code": "<?php echo 'Hello, World!'; ?>" }