Where the Diagnostic Loop Lost Quality
An evaluation of a multi-turn, LLM-based diagnostic assistant for distributed-systems incidents, showing where quality holds across iterations and where the diagnostic loop begins to lose precision.
This article examines a multi-turn, LLM-based diagnostic assistant for distributed-systems incidents and asks a narrow question: where does quality actually start to degrade inside the diagnostic loop? The system is not designed to produce a one-shot answer from a single user query. Its job is to maintain a diagnostic frame that can become more specific as an investigation moves forward (diagnostic model).
That diagnostic frame includes a current problem understanding, a competing interpretation, two or more active hypotheses, one discriminating check, and guidance for how to interpret the result of that check. The check is not just the next arbitrary action. It is supposed to help separate the live hypotheses by showing which of them remains viable after one concrete observation.
The assistant starts from an initial user-reported problem and produces the first version of that frame. When a new observation arrives, it should not restart the investigation from scratch. It should update the same frame: refine the problem understanding, strengthen, weaken, or replace active hypotheses, preserve the most important competing line of reasoning, and propose the next check that best separates the updated hypotheses.
That makes evaluation harder than ordinary final-answer grading. A continuation can fail even when the text still sounds plausible. The system may understand the new observation but fail to update its hypotheses cleanly, or it may keep the right hypotheses while repeating a low-leverage next check. The practical question behind this article is therefore not just whether the assistant gives good answers, but where the diagnostic loop stops getting sharper.
Why Final Answer Quality Is Not Enough
If we only score the final answer, too much of the system disappears from view.
Before the model can produce a useful diagnostic response, the system has to do several other jobs well. It has to turn the user query into a compact diagnostic structure, retrieve relevant precedent material from incident reports, retrieve theory chunks from a distributed-systems book that can explain the mechanism behind the incident pattern, assemble a prompt that gives the model the right practical and theoretical context, and then produce a diagnostic frame rather than a generic answer.
When the user later provides a new observation, the problem changes again. At that point the system is no longer being judged only on whether the text sounds good. It is being judged on whether it updates the diagnostic frame correctly: whether it keeps the right hypotheses alive, weakens the wrong ones, and proposes the next check that actually helps separate them.
A good-looking response can hide failures in any of these earlier layers. The model may sound plausible even when the query was structured poorly, the wrong chunks were retrieved, the prompt context was incomplete, or the continuation failed to sharpen the diagnostic frame.
That is why this project uses a layered evaluation surface rather than one generic answer-quality score.
How The Eval Engine Works
The eval engine is an offline system (evaluation story). It does not evaluate the assistant during the live user interaction. Instead, it runs over runtime history after execution.
The runtime is organized around three nested units: run, iteration, and step. A run is one evolving diagnostic session. An iteration is one bounded execution pass inside that session. A step is one concrete unit of work inside that pass. The important evaluation boundary is the iteration: each iteration produces or updates the diagnostic frame, so it is the natural unit for judging whether the system made a good diagnostic move.
The eval engine does not ask only whether the final response sounded good. It asks whether a particular iteration produced a good diagnostic frame given the current input and context, and whether a continuation iteration updated that frame correctly after a new observation.
This matters because the assistant is not a one-shot system. A run may continue as new observations arrive, but each continuation still has to do one bounded piece of work: interpret the new input, update the current frame, and propose the next discriminating check. That is why the iteration, not the run as a whole, is the main evaluation subject.
The eval layer combines three metric surfaces.
Judge-based metrics evaluate the semantic quality of the system’s diagnostic work at the iteration level. They cover how the system frames the problem, how well the selected prompt chunks support that frame, how good the diagnostic response is, and how well the diagnostic frame is updated after a new observation.
Runtime gold metrics evaluate structured outputs and retrieved chunks against expected targets from the golden set.
Runtime diagnostics are supporting signals derived from runtime outputs, gold comparisons, and retrieval statistics. They do not provide the main quality verdict, but they help explain why a metric moved and whether a failure came from runtime selection, prompt construction, or model behavior.
Together, these layers let the eval run report answer three different questions at once: did the system make a good diagnostic move, did the runtime supply the right material for that move, and where in the loop did quality begin to degrade?
What We Measure On The Initial Iteration
The first iteration is evaluated as the first diagnostic move on a new problem report.
At this stage, the system is judged on three main surfaces.
Query structuring
The first surface is query structuring. This stage does not drive retrieval: semantic search still runs on the initial user query. Query structuring extracts diagnostically important concepts from the user input into a structured form. That structure is then used to assess the completeness and shape of the user input, and part of it is passed into the final prompt.
Some structured fields are backed by a controlled vocabulary built from incident material. For those fields, the golden dataset contains expected targets and the runtime computes direct gold-backed quality metrics. Other fields remain useful parts of the structured output, but they are not covered by the same runtime gold surface.
The most important runtime gold metrics here are:
| Metric | What it measures |
|---|---|
macro_precision_soft | Whether the selected terms fall inside an acceptable semantic target set |
macro_recall_strict | Whether the canonical expected terms were covered |
grounded_strict_recall | Whether the canonical expected terms were both selected and grounded well enough in the user query |
The report also shows a harsher contract-style signal, runtime_query_structuring_core_success_rate, but that is a stricter compliance diagnostic rather than the main quality view.
The judge layer then asks two different questions:
| Judge suite | What it asks |
|---|---|
query_structuring_field_boundary_correctness | Were the extracted concepts placed into the right structured fields? |
query_structuring_grounding_conservatism | Were the selected terms grounded tightly enough in the user query, or did the system overreach? |
These two views answer different questions. Runtime gold metrics ask whether the selected terms matched expected targets. The judge layer asks whether the structured output was disciplined enough to be useful downstream.
Chunk pack quality
The second surface is chunk selection for generation. By this point, the system has already retrieved precedent and theory material around the current problem.
The most important runtime gold metrics here come from retrieval evaluation:
| Metric | What it measures |
|---|---|
recall_strict | Whether the expected target chunks were retrieved at all |
rr_strict | How early the first expected target appeared in the ranking |
nDCG | How well the ranking preserved the overall relevance ordering of the expected targets |
In the report, these metrics are shown separately for different chunk categories. They evaluate the retrieved chunk sets and rankings before selection into the prompt budget.
After retrieval, many retrieved chunks are mapped into diagnostic roles through their tags, and the result is compressed into a hard budget of six selected chunks. This is the object that the judge suites evaluate.
For the selected six-chunk context, the judge layer asks two different questions:
| Judge suite | What it asks |
|---|---|
evidence_pack_role_fit | Did the selected chunks actually serve the intended diagnostic roles inside the six-chunk prompt budget? |
evidence_pack_sufficiency | Did that six-chunk pack give the model enough context to produce a good diagnostic frame? |
Again, the two layers answer different questions. Runtime gold metrics ask whether the expected chunks were retrieved and ranked well before selection. The judge layer asks whether the selected six-chunk context was diagnostically well-formed and sufficient for generation.
Final answer quality
The third surface is the final diagnostic response itself.
The judge suites for this surface are:
| Judge suite | What it asks |
|---|---|
final_no_root_cause_claim | Did the response avoid collapsing too early into a single claimed root cause? |
final_first_check_discriminates | Did the response propose one check that actually helps separate the live hypotheses? |
final_hypothesis_source_alignment | Were the hypotheses aligned with the precedent and theory material that supported them? |
final_alternative_context_handling | Did the response preserve and handle competing context appropriately instead of flattening it away? |
final_result_interpretation_usefulness | Did the response explain how the result of the proposed check should update the diagnosis? |
This is the initial-response slice of quality. It tells us whether the system can make a useful first diagnostic move before any continuation input arrives.
What We Measure On Continuation Iterations
A continuation iteration is evaluated differently. The system is no longer judged as a first answer to a new problem. It is judged as an update over:
- prior diagnostic state
- one new observation
- one updated diagnostic response
That changes what “good behavior” means.
The updated diagnostic response is still evaluated by the same final-answer suites used on the initial iteration. On top of that, continuation iterations are evaluated with continuation-specific suites that focus on how the diagnostic frame was updated after the new observation.
The continuation-specific suites then ask four update-specific questions:
| Judge suite | What it asks |
|---|---|
continuation hypothesis update discipline | Did the system update its hypotheses in a disciplined way after the new observation? |
continuation problem understanding update | Was problem_understanding updated faithfully to reflect the new observation? |
continuation next check progression | Did the next proposed check actually progress the diagnosis rather than repeating earlier work? |
continuation observation resolution context recovery | Was a short or referential observation reconstructed into a useful standalone observation using prior context? |
Here, observation resolution means turning a short or referential new observation into a usable standalone statement by recovering the missing context from the previous diagnostic state.
Together, these suites separate what would otherwise look like one broad continuation score into distinct update behaviors.
Where Quality Was Lost
The goal of this evaluation is not only to say whether the system is good overall. It is to localize where quality starts to degrade inside the diagnostic loop.
We do that in three passes: first across pipeline stages, then across initial and continuation iterations, and finally across different parts of the continuation update itself.
At the stage level, the current runs show a fairly stable pattern. Retrieval is strong, chunk selection is strong, and final answer quality is strong. Query structuring remains somewhat mixed. Continuation is the main mixed stage that remains.
The continuation slice is more informative because it separates three different questions:
- how usable continuation responses remained at the product level;
- how well the diagnostic frame was updated after a new observation;
- how well the new observation itself was resolved against prior context.
This tells us whether the loop is failing at understanding the new input or later, during the frame update.
The latest eval run makes that visible in the numbers (run report). This batch covered 5 runtime runs and 15 evaluated iterations in total: for each of the 5 questions, 1 initial iteration and 2 continuation iterations. The continuation slice therefore contains 10 continuation iterations. The suite scale is 0|1|2, where 0 is a hard fail, 1 is a partial or borderline pass, and 2 is a strong pass. Unless stated otherwise, the continuation scores below are averages over those 10 continuation iterations.
At the product and aggregate level, the key continuation signals are:
| Metric | Built from | Latest value | What it means |
|---|---|---|---|
usable continuation response rate | continuation iterations where continuation hypothesis update discipline >= 1, continuation problem understanding update >= 1, continuation next check progression >= 1, and the final-answer suites used in the usable-response gate also score at least 1 | 0.60 | Only 60% of continuation iterations still met the minimum product bar for a usable updated response: acceptable problem_understanding, acceptable hypothesis update, an acceptable next check, and useful result interpretation |
continuation update judge score | mean of continuation hypothesis update discipline, continuation problem understanding update, continuation next check progression over continuation iterations | 1.73 | update logic is generally strong overall, but not as strong as observation resolution |
continuation input judge score | mean of continuation observation resolution context recovery over continuation iterations | 1.90 | the new observation is usually resolved well against prior context |
The continuation suites show where the gap comes from:
| Continuation suite | Mean score | Hard fails (score = 0) | Strong passes (score = 2) | What it tells us |
|---|---|---|---|---|
continuation hypothesis update discipline | 1.80 | 0 / 10 | 8 / 10 | hypothesis updates are usually strong, but not uniformly at the strongest level |
continuation problem understanding update | 1.80 | 1 / 10 | 9 / 10 | problem_understanding is usually updated correctly to reflect the new observation, but one hard fail shows this can still break cleanly |
continuation next check progression | 1.60 | 2 / 10 | 8 / 10 | next-check progression is the clearest weak point: it has the lowest mean score and the highest number of hard fails |
continuation observation resolution context recovery | 1.90 | 0 / 10 | 9 / 10 | observation resolution is one of the strongest continuation behaviors |
These numbers point to a clear conclusion. The loop usually carries the new observation forward successfully, but it is less reliable at turning that observation into a sharper next diagnostic move. The main weakness is not in reading the continuation input. It is more about two follow-on failures:
- not always advancing to the best next discriminating check;
- not always narrowing or re-ranking hypotheses cleanly enough;
That is a much more actionable diagnosis than “continuation quality is low.”
Conclusion
This evaluation did not show a system that collapses everywhere after the first turn. It showed a more specific pattern. The loop usually carries a new observation forward successfully, and observation resolution remains one of the strongest continuation behaviors. Quality starts to degrade later, when the updated diagnostic frame has to be turned into the next best discriminating move.
That makes next-check progression the most important weakness uncovered by the current eval. The system often understands more than it can convert into the best next discriminating step. That is exactly the kind of failure a one-shot score would blur away, and exactly why iteration-aware evaluation matters for multi-turn, LLM-based diagnostic assistant.