OpenAI built its own eval dashboard. Now it's retiring it and pointing developers to the tool it bought.

On June 3, 2026, OpenAI told developers its Evals platform is being deprecated. Existing evals become read-only on October 31. The dashboard and API shut down on November 30. The whole notice is a short table and a link to a migration guide.

The guide is the interesting part. It points developers to Promptfoo, the open-source eval and red-teaming tool OpenAI agreed to acquire on March 9, 2026. So OpenAI built an eval product inside its own API console, bought an outside one, and is now retiring the one it built. Promptfoo's technology is also headed into Frontier, OpenAI's platform for building and running AI agents, and OpenAI says the open-source project will continue.

I've written before that evals became the new acceptance criteria, the executable "definition of done" for software that doesn't behave the same way twice. That post was about what an eval is. This one is about where it lives.

Promptfoo is a CLI and a YAML config file. You define prompts, test cases, and assertions in that file, run it locally or in CI, and check it into the same repo as the feature it tests. OpenAI's dashboard lived in a separate web app. The scoring idea in both is roughly the same: a dataset, a grader, a pass bar. My read is that the difference that mattered was location. One sat in the pipeline engineers already run. The other sat one browser tab away.

Running QA on LLM training data, I saw a version of this. A quality bar people have to open a second tool to see gets checked on launch day and forgotten by the next sprint. The one that sits where the work happens is the one people actually hold themselves to.

The migration itself is a second lesson. In July, a developer on OpenAI's forum said the guide described an export menu that didn't exist in the interface. OpenAI support clarified that the export option gives you JSONL results, not a runnable Promptfoo config. The guide now says plainly that migration doesn't need an export feature. You rebuild each eval by hand in a Promptfoo config. 180 days of notice sounds generous until you're rebuilding every eval yourself.

So treat the eval definition as source code, and keep it out of someone else's dashboard. Version the dataset and the rubric next to the feature they gate. Run the eval the way you'd run a unit test, in the pipeline that blocks a merge. And keep it in a format you can run without the vendor who hosts it today.

That said, a hosted eval builder still earns its place for a first pass: sketching a rubric, getting a quick pass rate before anyone commits to a schema. The mistake is treating that convenience as the system of record. Once an eval becomes the bar a team won't ship without clearing, it needs the same durability as the code it's gating.

The eval that survives is the one still running after the vendor changes its mind.