GPT-4o and Co. get it wrong more often than right, says OpenAI study
OpenAI’s best model, o1-preview, achieved only a 42.7 percent success rate. GPT-4o followed with 38.2 percent correct answers, while the smaller GPT-4o-mini managed just 8.6 percent accuracy.
Anthropic’s Claude models performed even worse. Their top model, Claude-3.5-sonnet, got 28.9 percent right and 36.1 percent wrong. However, smaller Claude models more often declined to answer when uncertain – a desirable response that shows they recognize their knowledge limitations.
