September 16, 2026
Artificial Intelligence (AI) can increasingly evaluate complex educational practices, including classroom teaching. But assigning a score is only one part of the challenge. If an AI system rates the quality of a teacher’s feedback, teachers and other stakeholders also need to understand why the system arrived at that judgment.
This raises an important question for explainable AI in education: Do the explanations generated by AI systems actually reflect what drives their decisions, or do they merely sound plausible?
A cross-institutional research team including Ivo Bueno, our EdTech Fellow Dr. Babette Bühler and EdTech Director Prof. Dr. Enkelejda Kasneci from the Technical University of Munich and the Munich Center for Machine Learning, together with Philipp Stark (Lund University), Ulrich Trautwein (University of Tübingen), Dorottya Demszky (Stanford Graduate School of Education), and Heather Hill (Harvard Graduate School of Education), investigated this question by comparing two approaches to explaining automated teaching-quality scores: SHAP-based explanations and LLM-generated sentence rankings. Their findings show that explanations which appear convincing are not necessarily faithful to the model’s actual decision-making.
The Study
The researchers used the NCTE dataset, which contains more than 1,600 transcripts of 45- to 60-minute elementary mathematics lessons. These lessons were divided into 15-minute segments, resulting in 6,005 segments that had been evaluated by expert observers using the Classroom Assessment Scoring System (CLASS).
The study focused on the CLASS dimension Quality of Feedback (QoF). This dimension captures how effectively teachers provide meaningful feedback, scaffold student thinking, encourage metacognition, elaborate on students’ responses, and clarify misunderstandings. Each classroom segment receives a score between 1 and 7, ranging from low or absent feedback to consistently high-quality feedback.
The researchers compared two families of AI models:
- Fine-tuned pretrained language models (PLMs) included variants of BERT, ALBERT, RoBERTa, and DeBERTa. These models were specifically trained on the teaching-quality data.
- Large language models (LLMs) included several versions of Llama, Mixtral, Qwen, and Mistral. Unlike the PLMs, these models were not trained specifically for this task. Instead, they were prompted with instructions and examples to produce a Quality of Feedback score. Only open-source LLMs were selected, allowing local deployment and reducing privacy concerns associated with sensitive classroom data.
But the study went beyond asking which models produced the most accurate scores. The researchers also wanted to know which sentences within a classroom transcript actually influenced those scores.
For the fine-tuned models, they used SHAP, an explainability method that estimates how much individual pieces of an input contribute to a model’s prediction. In this study, each sentence received a SHAP value indicating its estimated contribution to the final teaching-quality score.
The LLMs were asked to perform a comparable task: identify and rank the ten sentences they considered most influential for the score.
The researchers then tested these explanations rather than simply accepting them at face value. Starting with the highest-ranked sentence, they progressively removed influential sentences from a transcript and asked the model to score it again. If removing a supposedly important sentence meaningfully changes the prediction, this provides evidence that the explanation is faithful to what the model actually uses. If the score barely changes, the selected sentence may sound relevant without actually driving the prediction.
Key Findings
Fine-tuned models were more accurate, but compressed their scores
The fine-tuned PLMs generally produced more accurate Quality of Feedback scores than the prompted LLMs. Their average error was approximately one point on the seven-point CLASS scale, meaning that predictions are typically within one level of the expert annotation, with DeBERTaV3 large achieving the strongest overall results.
However, this accuracy came with an important limitation: the fine-tuned models tended to concentrate their predictions around the middle of the scale. For example, DeBERTaV3 large never predicted the extreme scores of 1 or 7, and several models largely collapsed their predictions into the 3-5 range.
The LLMs were less accurate overall, but their predictions covered a substantially broader portion of the 1-7 scale. This reveals a trade-off: fine-tuned models offered greater numerical accuracy, whereas LLMs better preserved score variability.
Plausible explanations were not necessarily faithful explanations
The strongest distinction emerged when the researchers examined why models produced their scores.
SHAP-based sentence rankings consistently identified sentences whose removal produced meaningful changes in model predictions. In contrast, sentences selected by LLMs generally caused smaller and less stable prediction changes, even when those sentences were removed from the same LLM that had identified them as important.
This means an LLM can point to a sentence and present it as central to its judgment even though changing or removing that sentence has relatively little effect on the resulting score.
The two explanation approaches also selected surprisingly different evidence. Across the LLMs, the average overlap between the top ten sentences chosen by SHAP and by the LLMs corresponded to only around one or two sentences per transcript. Their agreement on the ordering of important sentences was similarly weak.
SHAP explanations transferred better across different AI models
The researchers also asked a more demanding question: Does an explanation remain meaningful when applied to another type of model?
To test this, sentences selected by LLMs were removed before the fine-tuned models rescored the transcripts, and vice versa.
Removing LLM-selected sentences generally produced only small and inconsistent changes in the fine-tuned models’ predictions. By contrast, removing sentences identified through SHAP frequently produced substantial changes even when the modified transcripts were scored by LLMs.
In particular, removing just the most influential SHAP-selected sentence often caused an immediate shift in an LLM’s score. This suggests that SHAP was identifying aspects of classroom dialogue that were relevant across different model architectures rather than merely exposing quirks of one particular model.
A random sentence-removal comparison further supported this conclusion: removing randomly selected sentences produced prediction changes close to zero. The observed effects therefore cannot simply be explained by shortening or disrupting the transcripts.
Takeaway Message
When AI is used to evaluate complex educational practices, a convincing explanation should not automatically be treated as a faithful one.
LLMs can produce explanations that appear intuitive and pedagogically reasonable, yet this study shows that the evidence they identify does not consistently correspond to the information actually driving model predictions. SHAP-based explanations, in contrast, produced larger, more stable changes when their identified sentences were removed and transferred more reliably across different model architectures.
For educational technology, this distinction is particularly important. Automated teaching-quality assessment could eventually support teacher reflection and make individualized feedback more scalable, but transparency cannot simply be added by asking an LLM to explain an AI-generated score. Explanations themselves need to be evaluated.
At the same time, explainability does not make automated judgments objective. The study is based on text transcripts and one Quality of Feedback dimension, while actual classroom interactions also include visual, temporal, and vocal information. Human annotations can themselves contain subjective judgments or biases. For these reasons, the authors stress that automated scoring should support rather than replace professional judgment, particularly in high-stakes educational contexts.
Moving from “What score did the AI assign?” to “What actually caused the AI to assign this score?” is therefore an essential step toward more transparent and responsible educational AI.
References
- Bueno, I., Bühler, B., Stark, P., Fütterer, T., Trautwein, U., Demszky, D., Hill, H., & Kasneci, E. (2026). From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment. Findings of the Association for Computational Linguistics: ACL 2026, 7590-7606. https://aclanthology.org/2026.findings-acl.375/
