And sometimes, to guide users towards a better understanding of what they were actually asking for. Having had involvement in documenting business processes, I’ve lost count of the number of times users said things like “I don’t know, we never thought of that”, or “can that happen?”, or “I’ll have to discuss that and get back to you” in response to questions about what they thought was a finished requirements specification.
It takes time and patience to discover what people mean rather than what they said.
During the process of defining requirements I would ask business clients to describe a process and then ask “are there any exceptions?” The senior people would say ‘none’, but the closer you got to the ‘coal face’ the more you found there were lots of exceptions and special cases. It always took a long time to obtain agreement on which ones were to be retained and which were unimportant. Again, a totally human process of description, discussion, negotiation, and haggling.
Of course, if you are building a package rather than a custom solution this is much easier. In that case the designer is the dictator and the customer has no choice.
I always loved Barbara Liskov’s course 6.170 (Software Engineering) at MIT. Final project had a series of deadlines for the teams implementing the project. The earliest was for the requirements specification. That ended up being the most important one for the project.
We had a project of Reversi: “A minute to learn, a lifetime to master”. That was changed in the specification document to, “A lifetime to implement.” When I scanned the document, there was one obvious question: how were the TAs – the customer – going to test each implementation rapidly? The answer was pretty obvious: they were going to repeatedly use the “load saved game” function, and see how our Reversi engine calculated the very next move. Specifically, they were going to load pathological game states into our apps, and see how our engine coped – or melted down. A cynic might have asked my question more simply: “How is The Man going to s…w with us?”
In short, the most important part of an implementation was the “load game” function. It needed to examine the saved game state and reject to load if the saved game state was pathological. Our team worked very hard to define a viable/pathological game state and – per our amendments – gave an error if any loaded game state was pathological.
The fun started after that first deadline had passed. We explained to the other teams what we had guessed the test phase would look like – and why our amendments were brilliant. You could tell immediately when they understood what we (and other teams) had done – and what the TAs were going to do to them. Many of the saved games created by the TAs could not possibly happen in a real game, so it was a challenge for the teams to have appropriate behavior at that point. This hits right on target for @Clayton 's “we never thought of that” comment. Doing something appropriate was possible, but it was much more difficult than simply rejecting pathological game state earlier. Final testing was pretty boring for us, because only a handful of the TA test cases would even load. We smiled.
Whether or not students succeeded in anticipating problems up front, I suppose it was a good learning experience for all. Their example was far simpler than anything you would encounter in the real world, but general principles are broadly applicable.
LOL…love reading the comments above.
I LOVED the first part of the show. Finally, some balance. Someone to call BS when BS needs to be called. I thought “maybe there’s some hope for this show (and network) after all.”
Intelligent Machines interviewed Stephen Wolfram on Episode 608. Stephen used an analogy between the output of LLMs and building a drywall stone wall. I quoted it here. This analogy gives a great visualization why LLMs create results that are inherently irreproducible.
Interestingly, Stephen’s analogy also makes a compelling case for the superiority of the Wolfram Language and their Notebook Assistant for LLM code generation. You tell in prose what you want the Notebook Assistant to do; it generates Wolfram Language code. At this point, the programmer runs that bit of code and judges its fitness. The programmer either accepts the code, asks the AI to generate code again (perhaps with a different prompt), or manually edits the code to do what the programmer intended.
If the consistency of a human programmer is like an AI, this would make sense. Stephen is saying that LLMs are fundamentally different in what they’re doing than any human. Furthermore, he thinks the entire LLM code generation procedure is vastly superior for creating production code in the Wolfram Language.
Again, this is not the mechanism that Stephen is suggesting. He’s suggesting that each member of a team be assisted with LLM code generation, and that the team member individually assimilate the LLM code into the code base. Read his analogy: Wolfram is suggesting this mechanism would generate code where the “rock wall” never gets too “high”. I’m not saying Stephen is correct, but the model he’s proposing is far saner than simply having an AI being “part of the team”.
Wolfram’s approach to LLM code generation may be the best thing going today.