September 23, 2026

Classroom observation can provide teachers with valuable feedback on their instructional practice. Yet systematically observing lessons is difficult to scale: trained observers must watch classroom interactions, identify relevant teaching practices, and annotate what happens, often second by second. This makes detailed feedback costly and time-intensive.

Could artificial intelligence help automate part of this process?

A research team led by Ivo Bueno and Ruikun Hou, with researchers from the Technical University of Munich, the Munich Center for Machine Learning, the University of Tübingen, Gordon College, the University at Albany, the University of Virginia, and the Leibniz Institute for Knowledge Media, investigated whether modern AI models can automatically recognize what teachers and students are doing and saying in authentic classrooms. Rather than treating automated recognition as feedback in itself, the researchers position it as a technical foundation from which interpretable and actionable teacher feedback could eventually be developed.

The Study

The researchers worked with classroom recordings from elementary mathematics and English Language Arts lessons. Their dataset contained 164 hours of video from 195 lessons, with human-produced transcripts available for 68 lessons. The videos had been annotated every second for 24 instructional activities, such as small-group activity or presentation with technology. Transcript turns were annotated using 19 discourse labels, including practices such as open-ended questioning and elaborated feedback.

This created two related but technically different problems. The video models needed to identify instructional activities from what was visually happening in the classroom, while the language models needed to recognize instructional discourse from what teachers and students were saying.

The researchers therefore developed two parallel AI pipelines rather than combining video and text from the outset. For classroom video, they compared a zero-shot multimodal large language model with fine-tuned vision-language and video models that were adapted to the classroom data. For transcripts, they compared zero- and few-shot LLM prompting with a fine-tuned transformer classifier. The models also incorporated short amounts of context, e.g., preceding video or earlier conversational turns, to determine whether context helps AI interpret instructional practice. 

Importantly, the problem was not a simple one-label classification task. Several instructional activities can occur simultaneously, and some practices are much rarer than others. For example, teacher writing occurred in around 4% of the video data, whereas teacher standing occurred in more than half. The researchers therefore also investigated techniques designed specifically for imbalanced, multi-label data.

Key Findings

Fine-tuning outperformed general-purpose prompting
Across both video and transcript analysis, models specifically adapted to the classroom task performed better than approaches relying primarily on prompting.

For video, the strongest model was the fine-tuned X-CLIP ViT-L/14, reaching a macro-F1 score of 0.577. The zero-shot multimodal model Qwen2.5-VL reached 0.423. The specialized X-CLIP model also surpassed the BaS-Net result of 0.469  previously reported  by Foster et al. (2024) on the same dataset and labels.

The difference was particularly pronounced for classroom discourse. According to Table 2, zero- and few-shot Llama models produced macro-F1 scores ranging from 0.060 to 0.265, whereas the best fine-tuned DeBERTaV3 configuration reached 0.460. The strongest configuration used Asymmetric Loss together with additional weighting for positive examples, helping the classifier deal with rare instructional practices. 

The result suggests that simply giving a general-purpose LLM instructions about classroom practices is not enough for this complex recognition task. Domain-specific adaptation remains important, particularly when categories overlap and examples are unevenly distributed.

Context matters
Instructional practices do not occur in isolation, and the results showed that giving the models some information about what happened immediately beforehand improved recognition.

For video, using the target one-second clip alone produced a macro-F1 of 0.558. Adding one preceding second increased performance to 0.577. Extending the context further to three preceding seconds, however, slightly reduced performance to 0.567. 

Context had an even larger effect on discourse recognition. When the text classifier saw only the target utterance, its macro-F1 was 0.392. Including the two preceding utterances raised this to 0.460. 

More context is therefore not automatically better. The findings point instead toward a short, relevant contextual window: enough information to interpret an action or statement, but not so much that unrelated classroom activity introduces noise.

Automated classroom analysis is promising, but not deployment-ready
The results show that AI can recognize a meaningful range of instructional activities and discourse patterns in authentic classroom recordings. However, the researchers emphasize that the current performance levels are not sufficiently reliable for high-stakes evaluation or autonomous judgments about teaching quality.

Recognition also remained uneven across categories. Rare and interaction-heavy practices were particularly difficult to detect. Real-time use presents another challenge: the best-performing video model processed roughly nine clips per second on a high-end GPU, creating practical cost and infrastructure questions for deployment in schools. The models were also trained only on elementary mathematics and English Language Arts classrooms, meaning their ability to generalize to other subjects, age groups, and populations still needs to be tested.

What This Means for Practice

The immediate opportunity is not an AI system that “judges” a teacher. Instead, automated recognition could eventually help transform lengthy classroom recordings into structured information teachers can reflect on.

For example, a system might summarize how much lesson time involved small-group discussion versus extended teacher explanation, or identify recurring transitions between instructional activities. Such summaries could make patterns visible that are difficult for teachers to notice while simultaneously managing a classroom. The researchers describe these kinds of concrete, interpretable signals as a possible basis for future actionable feedback. 

At the same time, classroom recordings contain highly sensitive data, often involving minors. The study therefore used open-source models that can be deployed locally rather than requiring classroom data to be sent to external commercial services. Even with local processing, questions around privacy, consent, algorithmic bias, transparency, and the ability to opt out remain essential. The authors emphasize that teachers should retain responsibility for interpreting feedback and making professional judgments. 

Takeaway Message

This study demonstrates an important step toward scalable, AI-supported classroom observation. Specialized models can already identify a range of instructional activities from video and discourse practices from transcripts, and their performance improves when they are trained specifically for the educational setting and provided with relevant contextual information.

But recognition is only the first layer. The greater educational challenge is turning automatically detected classroom events into feedback that is reliable, pedagogically meaningful, interpretable, and useful to teachers.

For now, these systems are therefore best viewed not as replacements for human observation or professional judgment, but as a potential supporting infrastructure for teacher reflection, professional development, and large-scale research on teaching practice.

References

• I. Bueno et al., “Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data,” 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA, 2026, pp. 6806-6817, doi: 10.1109/WACV61042.2026.00658.

• J. K. Foster et al., “Automated classification of activities in classroom videos”. Computers and Education: Artificial Intelligence, 6, 2024.