"The researchers used the Measuring Massive Multitask Language Understanding (MMLU) benchmark to test persona-based prompting and found that when the LLM is asked to decide between multiple-choice answers, the expert persona underperforms the base model consistently across all four subject categories."
"For alignment-dependent tasks, like writing, role-playing, and safety, personas do improve model performance. However, for pretraining-dependent tasks like math and coding, using the technique produces worse results."
Persona-based prompting, where AI is instructed to act as an expert, has mixed results according to research. While it enhances performance in alignment-dependent tasks such as writing and safety, it negatively impacts pretraining-dependent tasks like math and coding. Researchers from USC found that this technique does not add expertise or facts to the model's training data, and can hinder its ability to retrieve factual information. Testing with the MMLU benchmark showed lower accuracy for models using expert personas compared to base models.
Read at Theregister
Unable to calculate read time
Collection
[
|
...
]