Intelligent Machines interviewed Stephen Wolfram on Episode 608. Stephen used an analogy between the output of LLMs and building a drywall stone wall. I quoted it here. This analogy gives a great visualization why LLMs create results that are inherently irreproducible.
Interestingly, Stephen’s analogy also makes a compelling case for the superiority of the Wolfram Language and their Notebook Assistant for LLM code generation. You tell in prose what you want the Notebook Assistant to do; it generates Wolfram Language code. At this point, the programmer runs that bit of code and judges its fitness. The programmer either accepts the code, asks the AI to generate code again (perhaps with a different prompt), or manually edits the code to do what the programmer intended.
If the consistency of a human programmer is like an AI, this would make sense. Stephen is saying that LLMs are fundamentally different in what they’re doing than any human. Furthermore, he thinks the entire LLM code generation procedure is vastly superior for creating production code in the Wolfram Language.
Again, this is not the mechanism that Stephen is suggesting. He’s suggesting that each member of a team be assisted with LLM code generation, and that the team member individually assimilate the LLM code into the code base. Read his analogy: Wolfram is suggesting this mechanism would generate code where the “rock wall” never gets too “high”. I’m not saying Stephen is correct, but the model he’s proposing is far saner than simply having an AI being “part of the team”.
Wolfram’s approach to LLM code generation may be the best thing going today.