LLM production console — a model drift & MLOps teaching simulation
The model's weights are frozen at deployment, but the world keeps moving: new products, new APIs, new slang, new regulations. The blue curve (training data) stays put while the teal curve (live traffic) slides away. The bigger the gap, the more often the model answers from a world that no longer exists.
Accuracy needs labeled data, which is slow and expensive to collect. The drift score (PSI) compares input distributions only — no labels needed — so it acts as an early-warning smoke detector. Watch how the PSI alert usually fires before accuracy visibly craters.
Notice that a fresh deployment starts slightly behind: the pipeline trains on data collected when it started, and the world moved during training. That's the MLOps treadmill — monitor → collect → train → evaluate → deploy → monitor — and why teams automate it instead of retraining by hand.