Jargon:Reinforcement learning from human feedback (RLHF)
1 min read
Say this at the party
“People rate the AI's answers, and the AI is trained to give more of the kind people liked. It's how a text predictor learns manners.”
In plain English
A freshly trained language model is good at continuing text, but not at being helpful. Ask it a question and it might ramble, make things up or be rude.
So people are shown two answers and pick the better one. A second, smaller model (a "reward model") learns to guess which answers people would prefer.
Then the main AI practises over and over, nudged towards answers the reward model scores highly. It's like a puppy getting treats for the right trick.
Where it came from
In 2017, Paul Christiano and colleagues at OpenAI and DeepMind showed that an AI could learn tricky tasks from people simply choosing between pairs of clips. In 2022, OpenAI's InstructGPT paper used the same idea on language models, and the recipe became standard for chatbots.
See it in action
You give a chatbot's reply a thumbs down because it waffled. Feedback like that, gathered from lots of people, can be used to teach the next version to get to the point.
What people get wrong
- It doesn't teach the AI facts or make it truthful. It teaches it what people tend to prefer, which can include sounding confident or agreeing with you.
- The humans don't rate every answer forever. Their choices train a reward model, and that model does most of the marking.
Go deeper
- PaperOuyang et al., OpenAI, 2022Training language models to follow instructions with human feedbackThe InstructGPT paper: how RLHF turned GPT-3 into a model that follows instructions.
- PaperChristiano et al., 2017Deep reinforcement learning from human preferencesThe early paper that trained an AI from people picking the better of two clips.
- PostHugging Face, 2022Illustrating Reinforcement Learning from Human Feedback (RLHF)A friendly, step-by-step walkthrough with diagrams.
- ArticleWikipediaReinforcement learning from human feedbackA solid overview of the reward model and how training works.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.