#ai-evaluation

#ai-evaluation

Some lawyers have learnt that the hard way, and have been fined for filing AI-generated court briefs that misrepresented principles of law and cited non-existent cases. The same is true in other fields. For example, AI models can pass the gold-standard test in finance - the Chartered Financial Analyst exam - yet score poorly on simple tasks required of entry-level financial analysts (see go.nature.com/42tbrgb).

What if the secret to unlocking AI's full potential wasn't in the technology itself but in how we use it? After spending over 200 hours teaching AI to write, Nate B Jones discovered that the biggest mistakes aren't about algorithms or software limitations, they're about human misunderstanding. Too often, we assume AI can read between the lines of vague instructions or magically produce brilliance without guidance. The result? Generic, uninspired content that misses the mark.

ChatGPT maker OpenAI has released a new evaluation, dubbed GDPval, to measure how well its AIs perform on "economically valuable, real-world tasks across 44 occupations." "People often speculate about AI's broader impact on society, but the clearest way to understand its potential is by looking at what models are already capable of doing," the company wrote in an accompanying blog post. "Evaluations like GDPval help ground conversations about future AI improvements in evidence rather than guesswork, and can help us track model improvement over time," OpenAI added.

Result ambiguity can come in different forms. The lack of agreement among raters is the most common one, known as Inter Rater Reliability (IRR).