Perplexity
Perplexity measures how surprised a language model is by a piece of text — lower means the model found it more predictable. It's a quick intrinsic gauge of how well a model fits a dataset, but it says little about whether the model is actually useful or correct.
Perplexity scores how well a model predicts a sample of text: it’s based on the probability the model assigned to the actual next tokens, so a lower perplexity means the text was less surprising to the model. It’s cheap to compute and handy for comparing language models on the same dataset or tracking a model during training.
Its limit is that predicting text well isn’t the same as being helpful, truthful, or good at your task. A model can have low perplexity and still hallucinate, miss instructions, or fail your use case — and perplexity isn’t comparable across different tokenizers or datasets. So treat it as an intrinsic health check, not a measure of quality. For “is this good enough to ship,” you need task-level evaluation, not perplexity.
From the conversation
-
Architecting AI Agents: The Shift from Models to Systems | Aishwarya Srinivasan - Building an AI-Native Startup | GrowthX's Marcel Santilli
- I Started r/AI_Agents and Now I'm Launching a VC Fund
- Practical Lessons for GenAI Evals | Chip Huyen & Vivienne Zhang
- The State of AI: Open-Source Models & Enterprise Trust | May Habib