How much does the wording of a prompt matter when AI scores students’ answers? In our paper, we investigated this using three open-weight language models and two English-language short-answer scoring benchmarks. We compared human-written prompts with prompts automatically refined using ProTeGi, which analyses scoring errors and proposes improved instructions. Across 135 paired comparisons, optimisation delivered a median improvement of around two F1 points. Beyond improving performance, it also revealed patterns in effective prompt design: optimised prompts used more words that make logical relationships explicit, such as “if” and “because,” and shorter noun phrases. These features were also associated with better scoring performance, while more directive verbs were associated with poorer results. Our findings suggest that clear logical structure and simpler phrasing are promising principles for prompt design, and that automatic prompt optimisation can help us understand what makes instructions effective for educational AI.
