Multimodal AI

A text-only model reads and writes words. A multimodal model takes in — and sometimes produces — several data types at once: read a chart, watch a clip, listen to a recording, and reason over them alongside text. It works by mapping each modality into a shared representation, so “the dog in this photo” and the word “dog” live in compatible space.

This is what lets AI act on the messy, non-text world most real work lives in: a support agent reading a screenshot of an error, a coding tool seeing a design mock, a robot interpreting a camera feed. As agents move off the chat box and into real tasks, multimodality stops being a demo feature and becomes table stakes — which is why frontier models increasingly ship multimodal by default.