September 04, 2026

Assessing student-created models such as UML diagrams is difficult to scale. Unlike programming exercises, where automated tests can often determine whether a solution works, modeling tasks are open-ended, graphical, and highly contextual. In large software engineering courses, manually reviewing every diagram can therefore become time-consuming, while differences between graders may also lead to inconsistent feedback.

A new study by Maximilian Sölch and Stephan Krusche from the Technical University of Munich explores how large language models (LLMs) could support this process. Rather than replacing human graders, the researchers developed a human-in-the-loop assessment system that generates contextualized feedback suggestions directly within the existing grading workflow. Human graders remain responsible for reviewing, adapting, or rejecting the generated feedback before an assessment is finalized.

The Approach

The researchers extended Athena, an open-source automated feedback system integrated with the learning platform Artemis, to support modeling exercises. Students create their models using Apollon, the modeling editor embedded in Artemis. When a submission is assessed, Athena can generate draft feedback that is returned directly to the grading interface.

The feedback process consists of three main stages: formatting, predicting, and parsing. First, Athena collects information about the exercise, such as the problem statement, an optional example solution, and instructor-defined grading criteria. The student model is then passed to an LLM, which generates feedback suggestions. Finally, Athena links these suggestions back to the corresponding elements in the original diagram. This means that feedback about a particular class, relationship, or other model element can appear directly next to that element. Human graders can then accept, modify, or discard each suggestion.

Making diagrams understandable to LLMs
A central challenge is that LLMs primarily operate on text, while UML models are graphical. Feeding the underlying editor data directly to the model is also problematic because the JSON representation used by Apollon is verbose and closely tied to the editor’s internal structure.

To address this, the researchers introduce ApollonUML, a domain-specific textual representation of modeling diagrams. ApollonUML describes diagram elements, their attributes, methods, relationships, and hierarchies in a compact but explicit form. Importantly, Athena preserves mappings between this textual representation and the original diagram elements, allowing generated feedback to be anchored precisely within the student’s model.

Aligning AI feedback with instructor expectations
The system does not simply ask an LLM whether a diagram is “good” or “bad.” Instead, it incorporates the structured grading instructions defined by instructors in Artemis. These function much like a rubric: they specify criteria, associated points, predefined feedback, and, where relevant, examples.

Including these criteria in the LLM prompt helps ensure that generated feedback reflects the learning goals and grading expectations of the particular exercise rather than generic modeling advice. Athena can also generate structured grading instructions from an exercise description when no rubric has been provided, although this functionality was not evaluated in the study because all four evaluated exercises already contained instructor-defined criteria.

The Evaluation

To examine how well the approach works at scale, the researchers conducted a retrospective observational validation study using authentic submissions from an undergraduate software engineering course with up to 2,000 students.

The evaluation covered 3,368 student submissions across four exercises, including both UML class and activity diagrams with different levels of complexity. The existing human assessments served as the comparison baseline. These were produced by trained student tutors who had previously completed the course, each grading individually without double marking. Athena generated a new assessment for every submission using GPT-4o, deployed through a private Azure OpenAI instance, and the researchers compared the resulting scores with those originally assigned by human graders.

Importantly, this was not a study in which graders actively used the AI suggestions during assessment. Instead, the researchers generated the LLM feedback retrospectively and compared it with existing human grading. The study therefore evaluates how well the system can approximate grading in authentic educational data, rather than directly measuring how much time graders save when using it.

Key Findings

LLM grading showed moderate alignment with human grading
Across all 3,368 submissions, human graders awarded an average score of 78.90%, while the LLM averaged 64.68%. The mean absolute difference between LLM and human scores was 20.21 percentage points, with a median absolute difference of 18.75 percentage points.

Overall, the LLM tended to grade more harshly than human graders, assigning scores that were on average 14.22 percentage points lower. However, this pattern depended on student performance: the LLM tended to be more generous with weaker submissions and harsher with stronger submissions.

The authors therefore do not argue that the system is accurate enough to independently determine grades. Instead, the level of alignment supports using LLM assessments as drafts that human graders can inspect and refine.

More complex diagrams were harder to assess
Performance varied substantially between exercises. The closest alignment occurred for a comparatively simple activity-diagram exercise, where the mean absolute error was 12.59 percentage points. The largest discrepancy occurred for the most structurally complex exercise, with an error of 25.35 percentage points.

The more complex exercise contained substantially more elements and relationships on average. This suggests that increasing structural complexity makes it harder for the LLM to reason reliably about the complete diagram.

Extreme errors exposed important failure cases
The researchers separately inspected the 26 submissions where LLM and human scores differed by at least 70 percentage points. Eleven cases involved substantial LLM undergrading, seven involved overgrading, and eight were categorized as cases in which both the human and LLM assessments were problematic.

One particularly important failure involved empty submissions. Because the prompt contained both the student’s submission and an instructor example solution, the LLM occasionally confused the two. In several cases, it effectively assessed content from the example solution as though it belonged to the student and consequently awarded a high score to an empty model. Only one of six empty submissions examined in this analysis received the correct zero score.

This failure is technically significant but also relatively actionable: empty submissions could be detected before an LLM is called, and clearer separation between student work and example material in the prompt could reduce the risk of confusing the two.

AI assessment can also reveal inconsistencies in human grading
Not every disagreement was caused by the LLM. During the qualitative analysis, the researchers also encountered submissions with visible modeling problems that nevertheless received full credit from human graders.

This points to another potential role for LLM-supported assessment: rather than merely accelerating grading, AI-generated assessments could help identify grading outliers and support calibration between human graders.

Feedback generation was fast and inexpensive
From an operational perspective, the system was able to generate feedback relatively quickly. Median processing times ranged from approximately 3.7 to 6.7 seconds per submission, depending on the exercise. The average LLM cost remained below two US cents per submission.

These results suggest that the computational cost of providing LLM-generated feedback is sufficiently low for deployment in large university courses.

Takeaway Message

This study suggests that the most promising role for LLMs in modeling assessment is not autonomous grading, but scalable assistance for human graders.

Athena demonstrates how this can be implemented in practice: student diagrams are transformed into an LLM-readable representation, instructor rubrics guide the assessment, and generated feedback is returned directly to the existing grading interface. Because feedback can be attached to individual diagram elements, graders can inspect AI suggestions in context and decide whether to accept, revise, or reject them.

At the same time, the results show why human oversight remains essential. LLM assessments still exhibit substantial score deviations, particularly for complex diagrams, and prompt-design issues can produce severe errors. The current evidence is therefore not sufficient to justify fully automated grading in high-stakes settings.

For educational practice, the opportunity lies in changing where human effort is spent. Instead of writing every piece of feedback from scratch, graders could begin with AI-generated suggestions and concentrate on verifying, correcting, and contextualizing them. LLMs may additionally provide a second perspective that helps uncover inconsistent human assessments. The paper therefore illustrates a shift from replacing graders toward building systems in which AI and human judgment complement one another.

With median generation times of only a few seconds and costs below two cents per submission, the technical conditions for scaling such support are already promising. The next step is to examine the system in actual grading practice: how frequently human graders accept, modify, or reject its suggestions, and ultimately whether this changes feedback quality, grader workload, and student learning.

References

• Maximilian Sölch and Stephan Krusche. 2026. Scaling Assessment of Student Models with LLMs: Integrating Feedback into Practice. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEET ’26), April 12–18, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3786580.3786985