Poor-quality requirements often throw sprint plans off track because unclear user stories lead to rework and lengthy clarification loops. LLM-based review of user stories is a promising approach that enables teams to identify quality gaps earlier. Instead of discovering problems such as missing acceptance criteria only during sprint planning, teams can identify them as early as the refinement phase. Typical signs include stories that are difficult to estimate, too large for a sprint, or hard to test. As a result, teams spend valuable time clarifying details instead of implementing features. In this article, we demonstrate how Large Language Models close this gap. We introduce the DeepQuali approach, analyze the benefits of automated quality checking based on INVEST criteria, and share practical experiences regarding the reliability and acceptance of this AI-powered feedback.
Read the article in English: Improving User Story Quality Using LLMs
In the DeepQuali research project , we developed an approach for LLM-based evaluation of user story quality. Here, a Large Language Model (LLM) supports backlog refinement, evaluates stories based on quality checklists such as INVEST and a team’s Definition of Ready, and provides structured feedback with ratings, explanations, identified issues, and suggestions for improvement. In an initial evaluation with two medium-sized software companies, experts particularly appreciated the approach’s clear explanations, as these made the quality assessment easier to understand. In addition, they found the concise summary of implementation readiness helpful, as it accelerated the assessment of stories’ maturity. At the same time, limitations became apparent, such as when relevant project context was missing from individual evaluations. This occasionally led to false positives.
How the LLM-based review of user stories works
From backlog entry to structured feedback
DeepQuali implements an LLM-based review process for user stories that evaluates backlog entries based on defined quality rules. To ensure consistent results, the content of the stories is converted into a structured format before being passed to the LLM, and a prompt instructs the model to generate a structured response.
This makes it easier to review the output during refinement meetings. Additionally, teams can compare multiple stories more easily because the results are grouped according to rules. Since issues are already linked to specific criteria, teams can translate them into targeted adjustments more quickly. Furthermore, this structure helps in delegating tasks to the right people.
Output of the LLM-based review of user stories
The output is designed to be action-oriented and consistent. Therefore, it typically includes one assessment per rule along with a brief explanation. It also lists specific issues associated with each rule and assigns a severity level for prioritization, as not every issue blocks implementation. Finally, the approach provides suggestions for improvement, allowing teams to move directly from diagnosis to implementation.
Implementation Readiness of User Stories in LLM-Based Review
Additionally, DeepQuali generates a summary assessment indicating whether a story is ready for implementation. This provides teams with a quick signal regarding implementation readiness, helping them decide whether a story can be included in sprint planning.
In the evaluation context, this implementation readiness was tied to several practical criteria, as teams rely on clear benchmarks during planning. For example, stories had to include clear acceptance criteria and a sufficient description. Furthermore, they had to conform to the team template and include a effort estimate, ensuring that the assessment of implementation readiness aligns with common agile planning practices.
INVEST Criteria in User Story Quality Assessment
INVEST is a widely used guideline for writing high-quality user stories [1]. In the DeepQuali approach, this checklist supports the LLM-based review of user stories, with each letter representing a distinct quality attribute.
- Independent
A story should be implementable without depending on stories that have not yet been completed. This reduces dependencies and helps teams avoid downtime during a sprint. - Negotiable
A story should describe the goal rather than a fixed solution, so that the team can continue to discuss various implementation options. - Valuable
The story should offer clear value to users or stakeholders, as this facilitates prioritization and focuses development on real benefits. - Estimable
The team must be able to estimate the effort required. If this is difficult, the story often lacks context or a clear definition of its scope. - Small
The story should fit into a sprint. If it is too large, it should be broken down into smaller units that still deliver value. - Testable
Success must be verifiable. Therefore, acceptance criteria and test scenarios should be clearly defined.
Definition of Ready in User Story Quality Assurance
A “Definition of Ready” describes the minimum conditions a user story must meet before it is included in a sprint, because teams want to avoid including incomplete stories in their planning. Many teams therefore expand this checklist beyond the basic story format.
The evaluation considered criteria such as the quality of acceptance criteria, the presence of a effort estimate, the completeness of the template, prerequisites, and the description of test scenarios. Since these rules reflect local development practices, the LLM-based review of user stories can evaluate stories according to the team’s own standards.
Reliability of LLM-Based User Story Quality Assessment
Quality assurance requires consistent results. That is why DeepQuali prioritizes stability over creativity. If an LLM delivers different results for the same story, teams quickly lose confidence. Therefore, the approach reduces randomness in model configuration and enforces structured output. This makes repeated reviews predictable and easier to interpret.
Using LLM-Based User Story Review in Backlog Refinement
A practical application pattern is to treat the LLM’s results as a checklist rather than a final decision. For example, teams can conduct a brief preliminary review before refinement to identify missing acceptance criteria or unclear benefits early on.
Subsequently, teams can perform a second review before sprint planning, as this confirms whether a story is ready for implementation, thereby reducing sprint risks. In practice, it is also useful to distinguish between two levels of feedback. First, the tool lists problems and explanations. It then suggests possible improvements, allowing teams to retain control over the final decision.
Evaluation of the LLM Approach for Reviewing User Stories
Study Design
Two medium-sized German companies with two agile projects from different sectors participated in the evaluation. Experts reviewed a small selection of user stories and compared their own assessments with the LLM’s evaluations.
Since the prototype was not directly integrated into development tools, the experts interacted with the approach through guided workshops and surveys. Consequently, the results primarily provide initial insights into practical usability.
Greater Agreement on Implementation Readiness Than on Detailed Assessments
The experts did not always agree on the detailed criteria evaluations. However, agreement was higher for the “Small” criterion, and there was also more frequent consensus on the overall assessment of implementation readiness. As a result, summarizing implementation readiness in discussions proved easier to agree upon.
This observation underscores an important reality of agile development, as the quality of user stories can be subjective. An LLM can therefore support consistency but cannot replace team alignment.
Where the approach was most helpful
The experts particularly appreciated the explanations provided by the approach, as these made it clear why a story received a specific rating. Furthermore, they helped the teams decide what should be improved first.
The summary of implementation readiness also proved useful because it provides a quick overview of the stories’ quality. This allows teams to decide more quickly whether a story is ready for implementation. Brief explanations combined with a clear indication of implementation readiness can therefore be more helpful than complex rating schemes.
Where difficulties arose
Three practical limitations emerged during the evaluation. First, some checks require backlog context, so a lack of context can impair the assessment of dependencies. Second, false positives can undermine confidence, which is why fewer but more precise results are preferable. Third, expectations regarding the size of stories vary between companies, so the interpretation of the “Small” criterion must align with local writing conventions.
Key Takeaways for Practice
- Use LLMs to identify user stories with missing acceptance criteria and unclear value early on.
- Focus on explanations and suggestions for improvement rather than just ratings.
- Provide contextual information whenever possible, especially regarding dependencies.
- Adapt the quality checklist to your team’s writing conventions.
- Integrate the approach into backlog tools to support daily workflows.
Conclusion: AI as a Compass in Refinement
DeepQuali demonstrates how an LLM can support the quality assurance of user stories when it follows explicit quality rules and provides structured feedback. For agile teams, this approach offers practical advantages because reviews become faster, decisions regarding implementation readiness are clearer, and suggestions for improvement are easier to implement.
Nevertheless, LLM feedback should be understood as a support for human evaluation and not as the ultimate truth, because human judgment remains indispensable. Domain knowledge, project context, and team conventions continue to determine when a user story is actually ready for implementation.
References
[1] Wake, B. »INVEST in Good Stories, and SMART Tasks.«
