Jargon:Multimodal
1 min read
Say this at the party
“An AI that understands more than words. You can show it a photo, play it a sound or hand it a video, as well as type to it.”
In plain English
A "mode" is a kind of information: text, pictures, sound, video. Early chatbots only did text.
A multimodal AI can take in several kinds at once and connect them. It can read the words in a photo, describe what's in a picture, or answer a question about a chart.
Some can also reply in more than one way, such as speaking out loud or making an image.
Where it came from
Researchers have combined text, images and sound for years. A big step came in 2021, when OpenAI's CLIP learned to match pictures with captions from 400 million image and text pairs. In 2023, OpenAI described GPT-4 as a multimodal model that could accept images as well as text.
See it in action
Your boiler shows a strange error light. You snap a photo, send it to a chatbot and ask what it means. It reads the display and suggests what to check first.
What people get wrong
- Seeing a picture isn't the same as understanding it like you do. Multimodal AIs can still misread charts, miscount objects or make things up about an image.
- Multimodal doesn't mean it does everything. Many can take in images but only reply in text.
Go deeper
- PaperRadford et al., OpenAI, 2021Learning Transferable Visual Models From Natural Language SupervisionThe CLIP paper: teaching AI to link images and words at huge scale.
- PaperOpenAI, 2023GPT-4 Technical ReportIntroduces GPT-4 as a model that accepts both images and text.
- ArticleWikipediaMultimodal learningA plain overview of how AI combines text, audio, images and video.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.