Deploy Risk Score: What It Is, How It Is Calculated, and How to Use It

What Is a Deploy Risk Score?

A deploy risk score is a single number — typically on a 0 to 100 scale — that estimates the probability that a given deployment will cause a production incident, service degradation, or rollback. It is calculated at merge time or pre-deploy time from observable properties of the change: the code itself, the author, the files affected, the test coverage, the deployment timing, and dozens of other signals.

The core value proposition is prediction before the fact. Change failure rate — the DORA metric that tracks what percentage of deployments result in incidents — is measured after deployment. It tells you how your past deployments performed. A deploy risk score tells you, before this specific deployment goes out, how likely it is to fail.

That prediction window is where the operational value lives. A team that knows a particular PR carries high deploy risk can make an informed decision: deploy during a maintenance window instead of peak traffic hours, require an additional reviewer with expertise in the affected files, add a targeted integration test for the specific code path being changed, or defer the change until the risk factors can be addressed. None of those options are available once the incident has already started.

Deploy risk scores are not a replacement for good engineering practices — testing, review, staged rollouts. They are a forcing function that makes the risk embedded in each change visible to the people making deployment decisions, rather than leaving that risk implicit and unacknowledged.

The Signals That Drive It

The predictive accuracy of a deploy risk score depends almost entirely on the quality and breadth of its signal set. A score based on two or three signals will have high false positive and false negative rates. A score based on a comprehensive, validated signal set can achieve meaningful predictive power — typically 70–85% correlation with actual incident occurrence at high risk tiers.

Koalr's deploy risk model uses 23 validated signals, organized into five categories:

Change Complexity Signals

• Change entropy: A measure of how scattered the change is across the codebase. A PR that modifies 40 files across 12 different modules has high change entropy — not because any single change is complex, but because the blast radius of a mistake is large. Research consistently finds change entropy to be one of the strongest individual predictors of post-deployment incidents.
• PR size (lines changed): Larger PRs have more surface area for bugs. Beyond approximately 400 lines of change, review thoroughness degrades and defect escape rate increases.
• File churn rate: Files modified more than three times in the past 30 days are actively unstable areas of the codebase. Changes to high-churn files carry elevated risk because they often indicate incomplete refactoring or ongoing architectural uncertainty.
• Number of services affected: Changes that cross service boundaries introduce integration risk on top of unit-level risk.

Author and Review Signals

• Author file expertise: How much prior experience does the author have with the specific files being changed? An engineer modifying the payments service for the first time carries statistically higher risk than the engineer who has touched those files 50 times.
• CODEOWNERS bypass: PRs merged without review from the designated code owner for the affected files have significantly higher incident correlation than those that followed the ownership chain.
• Review cycle count: A PR that went through four rounds of review before approval has embedded more uncertainty and more code changes than one approved on the first pass.
• Time since last similar change: If the affected module has not been changed in six months, institutional knowledge about its behavior may be stale. Dormant areas of code carry elevated risk when disturbed.

Test and Coverage Signals

• Test coverage delta: Is this PR adding, maintaining, or reducing coverage of the lines it modifies? Net-negative coverage delta on the changed lines is a leading indicator of escaped defects.
• Test flakiness rate: If the test suite covering the changed files has a high flakiness rate, the tests are providing less actual signal about correctness than their existence implies.
• Missing integration tests: PRs that add cross-service behavior without corresponding integration test coverage are structurally undervalidated.

Schema and Infrastructure Signals

• DDL migrations: Database schema changes — especially additive ones like new non-nullable columns or index modifications on large tables — carry deployment risk that is qualitatively different from application code risk. DDL migrations can cause locks, timeouts, or dual-write failures that are not visible in code review.
• Dependency version changes: Updating a dependency, especially a major version bump, introduces external change that the author cannot fully audit. Dependency changes are correlated with elevated incident rates because the surface area of change extends beyond the repository.
• Dependency vulnerability score: Introducing a new dependency with known CVEs raises both security and operational risk.
• Infrastructure-as-code changes: Changes to Terraform, Kubernetes manifests, or other IaC files have blast radii that extend beyond the application layer.

Deployment Context Signals

• Deployment timing: Deployments on Friday afternoons, before holidays, or outside core business hours have historically higher incident rates — partly because support coverage is thinner and rollback response times are longer.
• Deploy queue depth: How many other changes are deploying at the same time? A high-concurrency deploy window makes it harder to isolate which change caused an incident if one occurs.
• Recent incident proximity: If the service had an incident in the past 48 hours and is still in a recovery window, new deployments carry elevated risk.
• AI-authored code percentage: The fraction of the diff that was AI-generated (estimated from Copilot telemetry or code patterns) affects risk modeling because AI-generated code has different defect distribution characteristics than human-written code — more syntactically correct but potentially less semantically appropriate.

Historical Pattern Signals

• Author historical incident rate: What fraction of this author's past deployments have resulted in incidents? This is used as a calibration signal, not a punitive one — it helps weight other signals appropriately for individual contributors.
• Module historical incident rate: Some parts of the codebase are structurally more fragile than others — legacy code, frequently modified services, areas with low test coverage. Historical incident rate by module is a strong prior for future risk.
• SLO burn rate: Is the service currently burning through its error budget faster than normal? Deploying into an already-degraded SLO window compounds risk.
• Rollback frequency: How often have recent deployments to this service been rolled back? High rollback frequency signals structural fragility in the deploy pipeline.

How Scores Are Aggregated: Weighted Ensemble

Twenty-three individual signals need to be combined into a single 0–100 score. The aggregation method matters enormously — a naive average of normalized signal values will not produce a score with meaningful predictive power.

Koalr uses a two-stage approach. In the first stage, each signal is normalized to a 0–1 scale based on historical distributions for that organization. What constitutes"high" change entropy depends on your codebase — a PR touching 10 files might be large for a tightly scoped service and small for a monolith. Normalization is organization-specific, which is why risk scores improve in accuracy as more organizational deployment history is accumulated.

In the second stage, normalized signals are aggregated using a weighted ensemble. Signal weights are initialized from research-validated defaults (change entropy and CODEOWNERS bypass carry the highest default weights) and adjusted over time using the organization's own incident outcomes as a training signal. A signal that correlates strongly with incidents in your organization gets upweighted; a signal that fires frequently but does not correlate with actual incidents gets downweighted.

The ML model underlying the aggregation is a gradient-boosted tree ensemble — the same class of model used in production by fraud detection and credit risk systems, which share similar properties: binary outcome prediction, high-dimensional feature space, strong non-linear interactions between features. The model does not require large amounts of data to produce useful predictions; even 50–100 historical deployments with known outcomes is enough to begin calibration.

Score Tiers: Low, Medium, High

The 0–100 score maps to three operational tiers. The thresholds are adjustable per organization, but the defaults are calibrated against incident correlation data.

The incident correlation percentages above represent the fraction of deployments at each tier that resulted in a production incident requiring remediation, based on aggregated deployment outcomes. High-risk deployments are not guaranteed to fail — the 47% correlation means roughly half of them proceed without incident. But that is still twelve times the incident rate of low-risk deployments.

The appropriate response to each tier is organizational policy, not an automatic block. Blocking high-risk deployments entirely would create an incentive to game the score downward and would prevent legitimate urgent fixes from reaching production. The goal is informed decision-making: high-risk changes deploy with eyes open and appropriate safeguards, not that they are prevented from deploying.

How to Use Risk Scores in Your Workflow

PR Check Integration

The most natural integration point is as a GitHub status check on open PRs. The risk score appears alongside CI checks — test results, lint, build — so reviewers see the risk assessment at the same time they are reviewing the code. A high risk score triggers a discussion in review: which signals are driving it, and can any of them be addressed before merge?

This integration surfaces risk at the moment when it is cheapest to address. A missing test that is pushing the coverage delta signal into negative territory takes ten minutes to add before merge. The same gap costs hours to diagnose after a production incident.

Merge Gates

Merge gates use the risk score as a blocking condition. A score above a configurable threshold (typically 70–80) requires an explicit override approval before merge. The override is not a back door — it is a documented decision that this deployment is proceeding at elevated risk, made by a named approver with full visibility into the signal breakdown.

Merge gates work best when they are rare by design. If every third PR is triggering a gate, the threshold is set too low and engineers will route around it. Gates should be calibrated so they fire on genuinely high-risk changes — schema migrations, large cross-service refactors, dependency major version bumps — not on routine feature work.

Deploy Windows

Deploy windows use risk scores to inform scheduling. Low-risk deployments can go out anytime, including during off-peak hours via automated pipelines. Medium-risk deployments are routed to standard deploy windows during business hours. High-risk deployments are flagged for scheduled maintenance windows with full on-call coverage available.

This tiered scheduling approach addresses one of the most common sources of avoidable incidents: high-risk changes deployed at low-attention times (Friday afternoons, holiday weeks) because the team had not made the deployment timing decision explicit. Risk score + deploy window policy makes the timing decision automatic rather than discretionary.

Canary and Blue/Green Triggers

High-risk deployments can automatically trigger a canary rollout strategy rather than a full deploy. The first 5% of traffic goes to the new version; if error rates and latency are stable after a defined bake period, the rollout proceeds. If metrics degrade, the canary is automatically reverted. This strategy contains the blast radius of a failed high-risk deployment to a small fraction of traffic.

False Positives and How to Tune Signal Weights

Any scoring system will produce false positives — high-risk scores on deployments that proceed without incident — and false negatives — low-risk scores on deployments that cause incidents. Both are worth managing, and the right balance depends on your organization's risk tolerance.

False positives erode trust. If engineers consistently see high risk scores on deployments that cause no problems, they will stop taking the score seriously — the same way alert fatigue causes on-call engineers to tune out noisy monitors. The fix is signal weight tuning: identify which signals are firing most frequently on changes that do not result in incidents, and reduce their weight in the aggregation model.

The most common sources of false positives are signals that are technically valid risk indicators but are not calibrated to your organization's specific context. Change entropy, for example, is a strong general predictor of risk — but a team that routinely makes large, well-tested refactors across many files will see it fire frequently without corresponding incidents. The calibration fix is to raise the change entropy threshold for your organization based on your historical distribution, or to downweight it relative to signals that are better calibrated.

False negatives — incidents that were not predicted — are addressed by expanding the signal set and by ensuring that post-incident analysis feeds back into the model. Every incident should be examined for which signals were present but not weighted highly enough. Over time, this creates a feedback loop where the model improves from organizational learning rather than requiring manual tuning.

Signal Override Policy

For specific well-understood scenarios — a routine dependency bump that the security team has reviewed, a DDL migration that has been validated in staging — teams can configure signal overrides that exclude specific signals from the score calculation for tagged PRs. Overrides require explicit justification and create an audit trail, so they are a deliberate exception rather than a catch-all escape hatch.

Risk Score vs. Change Failure Rate: How They Relate

Change failure rate (CFR) — the percentage of deployments that result in a service degradation — is one of DORA's four key metrics. Deploy risk score and CFR are related but not the same thing, and understanding the relationship helps you use both correctly.

CFR is a lagging indicator: it measures outcomes across all past deployments. It tells you the historical reliability of your deployment pipeline. If your CFR is 15%, roughly one in seven deployments has caused an incident over the measurement period.

Deploy risk score is a leading indicator: it estimates the probability of failure for a specific upcoming deployment, before it happens. A score of 75 on a particular PR does not tell you that you will definitely have an incident — it tells you that this deployment has the characteristics associated with incidents in your organization's history.

The two metrics are complementary. CFR gives you the system-level answer — how reliable is our deployment pipeline overall? Risk score gives you the change-level answer — how risky is this specific deployment? Together, they cover both organizational health (CFR) and operational decision-making (risk score).

When used together, a natural improvement loop emerges. High CFR triggers a review of recent deployments. Which ones caused incidents? What were their risk scores at merge time? If the incidents consistently came from deployments with scores above 60 that were pushed through anyway, the intervention is clearer merge gate enforcement. If the incidents came from deployments that scored low but caused incidents, the intervention is signal set expansion — something important is not being measured.

Elite DORA teams — those with change failure rates below 5% — consistently use a combination of leading risk indicators and lagging outcome tracking. The risk score prevents incidents from happening; CFR measures how often prevention fails. Both are necessary for a mature deployment risk program.