Why We Built Proactive Briefings Instead of Another Dashboard

The dashboard problem

The engineering metrics dashboard has become the default answer to a real problem: how do you give engineering leaders visibility into risk without adding meetings to their calendar? The dashboard promises visibility on demand. The practical reality is that demand rarely materializes until after an incident.

We have talked to dozens of engineering managers who have Koalr, LinearB, or Jellyfish dashboards open in a pinned tab. Most of them check it reactively — after a bad deploy, during a retrospective, when a VP asks why MTTR spiked last week. The dashboard is excellent for those conversations. It is not where risk gets caught before it becomes an incident.

The pattern we kept seeing was this: the information was in the system. The high-risk PR had been scored. The CODEOWNERS gap had been flagged. The SLO burn rate was elevated. But nobody was looking at the dashboard that Monday morning when the deploy queue was filling up.

The design constraint: no alert fatigue

The obvious answer to "the dashboard doesn't get checked" is more alerts. Add a PagerDuty rule for high-risk PRs. Slack-notify on every score above 70. This is the wrong answer. Alert fatigue is already endemic in engineering teams, and adding more low-signal notifications makes engineers trust the channel less, not more.

The design constraint for the briefing was: one message per week, per engineering manager, surfacing only the signals that changed materially. Not every high-risk PR — only the pattern shift. Not every CODEOWNERS gap — only when coverage has dropped enough to matter. Not raw scores — a narrative that tells you what to do with them.

This forced a different architecture than a notification system. A notification system fires on threshold breaches. A briefing synthesizes a week of data into a coherent picture of what the risk landscape looks like now versus what it looked like last week.

What goes into the briefing

The weekly risk briefing is generated by Claude from a structured data payload containing the week's deploy activity. The inputs to the synthesis are:

• →Risk score distribution. How many deploys scored in the safe, moderate, high, and critical ranges this week versus last week. The absolute numbers matter less than the direction.
• →High-risk concentrations. Which services are contributing disproportionately to high-risk scores. A spike in payments-service risk is more actionable than a diffuse increase across 20 services.
• →Signal-level drivers. Which of the 33 signals are contributing most to elevated scores this week. Change entropy up? CODEOWNERS coverage down? Coverage delta deteriorating? Each has a different remediation path.
• →MTTR and incident context.Whether MTTR improved or deteriorated this week, and whether any incidents co-occurred with high-risk deploys — which feeds the model's accuracy signal.
• →Positive signals. Teams or services that had notably low risk scores this week. Surfacing what is working creates a reinforcement mechanism, not just a problem log.

Why LLM synthesis, not templates

The briefing could have been a templated report. Pull the top 3 highest-risk services, list the most common signal contributors, format as bullet points. This would have been faster to build and easier to predict.

We chose LLM synthesis because the value of the briefing comes from narrative coherence — the ability to say "payments-service and auth-service are both elevated this week, and both have CODEOWNERS gaps as the primary driver, which suggests a governance issue rather than a change volume issue." A template cannot make that connection. It can surface the two data points separately, but it cannot synthesize the pattern.

The synthesis also allows the briefing to be appropriately calibrated to context. A week where MTTR improved and risk scores are down is a different briefing than a week where three high-risk deploys shipped on a Friday before a bank holiday weekend. The LLM generates the right emphasis for the actual situation.

Severity classification: critical, warning, info

Each briefing card is classified as critical, warning, or info. This is not determined by the LLM — it is a deterministic classification based on the underlying metrics before synthesis:

• criticalHigh-risk score concentration above threshold, or incident co-occurrence with high-risk deploys in the same week. Requires action before the next release window.
• warningA signal trending in the wrong direction that has not yet produced incidents but warrants monitoring. Coverage drift, emerging CODEOWNERS gaps, MTTR regression.
• infoA clean week, or a positive signal worth reinforcing — a team that has maintained low risk scores for three consecutive weeks, or model accuracy trending above target.

The classification is shown first in the briefing so the reader can triage at a glance. An engineering manager receiving a Slack digest at 9am on Monday should be able to determine within 10 seconds whether this week requires immediate action or a quick scan.

What we learned from use

The most consistent feedback we have received from engineering managers using the briefing is that it changed how they start their Monday. Not dramatically — it takes 90 seconds to read — but it means they arrive at the first standup already knowing whether there is a risk concentration to address.

The second most common feedback is about specificity. The briefing names services, names signals, and names the engineers whose PRs are driving elevated scores. Vague reporting ("risk is elevated this week") does not produce action. Specific reporting ("payments-service has the highest change entropy in 90 days, and three of the five contributors this week had no prior file-level expertise in the modified paths") does.

The briefing does not replace the dashboard. For deep investigation, for quarterly review, for explaining a trend to a VP, the dashboard is still the right tool. What the briefing does is ensure that the information in the system gets to the right people at the right time — before the deploy queue fills up, not after the incident report.