It seems simply plugging a LLM into a search machine doesn’t help it to produce accurate results.
LLMs, like GPT 3.5, used in ChatGPT were trained up to 2021 on “the Internet”. MS Bing AI (GPT 4) and Google’s Bard are additionally plugged into their respective search engines, but that isn’t really helping them to come up with accurate and up to date answers, it seems, according to research.
“One might think connecting the search engine and ChatGPT is a perfect solution, but the reality is more challenging because of the limited accuracy of search results,” Hongyin Luo, a postdoctoral associate at MIT’s Computer Science & Artificial Intelligence Laboratory, told The Register.
A search engine is essentially a keyword indexed database, not an authoratative source.
Great read. Using this software to run search queries is like playing baseball with a hockey puck. All the “AI” marketing tricks people into thinking these LLMs are something they’re not, and they end up being used incorrectly. Tesla did the same thing with the “Autopilot” branding, albeit in a much more dangerous circumstance.
Anecdotally, I sent a query into the Bing LLM about wardrobe on the show “Cheers” which generated a completely fabricated response. While watching, I noticed that the neckties worn by actors seem to have squared off bottoms rather than the typical pointed bottoms - not something I’d ever seen before, I thought maybe they were cut short on purpose.
I was curious if this was some kind of easter egg but couldn’t find anything using traditional search. I queried the Bing LLM about it and a response was generated that the costume director mandated cutting of all ties worn by characters in order to create a jovial, lax atmosphere on set. It provided reference links for this fact, however when reviewing the links there was no mention of this whatsoever that I could find.
I queried the LLM again and requested specifically where in the provided references was this fact was stated. This generated a response saying something along the lines of “actually, you’re right, the ties were not cut on purpose.” When I queried to ask why the first response was fabricated, Bing gave me the boot and killed the session .
I think the second to last sentence from that article sums up the hype around pseudo AI:
The researchers noted that they cannot explain why a search result is trustworthy or not
When the software can explain the why, then I’ll get excited. Until then, I’m convinced these LLMs are not even in the same galaxy as true AI, and are simply the next step in computer interface design. First was the CLI, then the GUI, now the Language User Interface. Which, imho, is pretty cool and a massive leap forward.
.