Thoughts
One of the things about LLMs is that even when they're right, they're still wrong.
Someone's mentioned this before, but LLMs solve the X-Y problem by always giving you the solution you ask for. I asked ChatGPT if there was a way to parse JSON with comments in the Ruby standard library, and it spit out some regular expressions.
=> https://gist.github.com/MatthiasPortzel/4978c1188537d9bf280ad41a30a86b7a#file-working_package_manager-rb-L78
And I audited these. I didn't just blindly copy-paste, I tested them, went through the regular expressions character by character to make sure I understood what they were doing, etc. The regular expressions are not a hallucination. But it doesn't change the fact that Regular Expressions are the wrong approach. It treats `//` in strings as comments. The correct answer to the question is "no, don't do that."
And I have a good intuition for this stuff. I never in a million years would have said, "ah yes I am now going to parse this JSON file with regular expressions" because I understand at a deeper level that that's the wrong approach, even if it happened to work.