OpenAI Chief Scientist Jakub Pachocki argues that tracking model perplexity (prediction error) during pretraining serves as a fundamental and enduring metric for measuring AI intelligence and predicting the emergence of new capabilities. To help establish shared industry safety standards and transparent scientific baselines, OpenAI is disclosing its internal pretraining perplexity measurements. Pachocki warns that future drops in perplexity will likely unlock transformative capabilities and self-improvement in AI research faster than human safety monitoring can adapt, necessitating cooperative industry limits.

Perplexity is a fundamental concept in language modeling. It measures how well a probability model predicts a sample. In the context of AI, it serves as a key metric for evaluating the quality and progress of language models.