There’s a problem with Large Language Models (LLMs) like ChatGPT and CoPilot. These digital chatterboxes have a bit of a “yes-man” syndrome, meaning they strive to be overly helpful, sometimes at the cost of being accurate. They were trained to use Reinforcement Learning from Human Feedback, which was meant to make them more user-friendly and relevant. However, this can backfire, as they might end up providing users with misleading information just to be “helpful.”
One study in a medical setting, found that AI gave incorrect advice simply because that’s what was requested. For example, 5 different LLMs were asked to write an instruction for a patient who is allergic to Tylenol to take acetaminophen instead (these are the same drug). The GPT models complied with the instruction every time.
Yet, with the right tweaks and nudges, programmers found they could substantially reduce these errors.
For small and medium-sized businesses, understanding these quirks is vital. They might use such models to streamline customer service or data processing. Knowing that LLMs can be overly eager to please can help businesses ensure they still get reliable information, perhaps by working with developers to adjust the models’ focus when accuracy is crucial. It’s a bit like knowing your chatty coworker should be double-checked, especially when they’re cheerfully nodding along to every wild idea in the meeting.
So, just like any tool, LLMs need a bit of training to fit their intended purpose perfectly.
And the use of AI in your organisation needs to be carefully managed – and our AI Policy Template can help you to do just that.