So far, my experiences with large language modes for non-toy projects has been mostly negative.

If someone has done something a lot like it before, and written about it online, then ChatGPT and its sisters can probably do it, but otherwise, no.

I had an interesting case when I was trying to control some DMX lighting and I had a friend who's very GPT savvy and came up with a series of refinements, using GPT and... ah, I forget, one other.

The first code almost worked, but actually created two things that were kind of right but could not be connected. The second one seemed even more likely to be right when I first saw it, but simply made up a missing class. The third one used an existing class that had nothing to do with anything and then it went downhill from there.

Further clean sessions had similar results. The interesting part was that I by then had solved my problem which was mostly contained in a fairly short Gist on GitHub, but going back and trying again with this knowledge only got to that program with really blatant hints.

I was blown away when I first started my experiments - here it was writing complete, well-formed programs at a prompt! But then it didn't actually save me any time at all in writing actual application code.

I have worked with people like this, full of energy who keep spitting out bad code. They are wearing. They contribute a negative amount to the group. But at least with a human you might be able to educate them on principles.

I still feel large language models might be promising in the future, but you completely undervalue correctness of the code, and the concept of correctness is completely orthogonal to what these models do.