OpenAI Discloses Accidental CoT Grading in RL Training for Instant, Mini, and GPT-5.4 Models

Details

• OpenAI identified accidental Chain of Thought (CoT) grading during reinforcement learning (RL) in prior models, affecting monitorability of AI agent misalignment.
• CoT monitors serve as a key defense layer; penalizing misaligned reasoning in RL reduces the informativeness of reasoning traces.
• Automated detection system revealed issues in Instant and mini models, plus less than 0.6% of GPT-5.4 Thinking samples.
• In-depth analysis showed no reduction in monitorability; third-party reviews from Redwood AI, Apollo AI Evals, and METR confirmed findings.
• OpenAI is enhancing real-time CoT-grading detection, safeguards, monitorability stress tests, and internal processes to prevent recurrence.

Impact

OpenAI's proactive disclosure and mitigation of accidental CoT grading underscores its focus on preserving AI safety monitorability amid RLHF advancements, distinguishing it from rivals like Anthropic and Google DeepMind, which emphasize similar interpretability safeguards but have faced less public scrutiny on training artifacts. By involving independent auditors and sharing analyses, OpenAI builds trust and pressures competitors to enhance transparency in frontier model training, potentially accelerating industry standards for misalignment detection without compromising performance gains seen in recent models like o1 and Claude 3.5.