How to Block High-Risk Deploys with GitHub Check Runs

How GitHub Check Runs Work

A Check Run is a status report posted to a specific commit SHA by an external service (typically a GitHub App). It has a name, a status (queued, in_progress, completed), a conclusion (success, failure, neutral, cancelled, timed_out, action_required), and an optional details URL that links to your service for more information.

Check runs appear in the PR merge status area — the same section that shows CI test results. When a branch protection rule marks a specific check run name as a required status check and that check run fails, GitHub blocks the merge button. This is the mechanism that makes risk gating enforcement automatic rather than advisory.

Step 1: Create a GitHub App

You need a GitHub App (not a personal access token) to post check runs because check runs are tied to the App's identity, which appears in the PR UI. Create a GitHub App in your organization settings with the following permissions:

• Checks: Read & Write
• Pull requests: Read
• Contents: Read
• Members: Read (for CODEOWNERS lookup)

Subscribe to the following webhook events: pull_request (opened, synchronize, reopened) and check_run (requested). Install the app on your repository or organization.

Step 2: Handle the PR Webhook and Score the Change

import hmac
import hashlib
from flask import Flask, request, jsonify
import jwt
import time
import requests

app = Flask(__name__)

GITHUB_APP_ID = "your-app-id"
GITHUB_PRIVATE_KEY = open("private-key.pem").read()
WEBHOOK_SECRET = "your-webhook-secret"

def get_installation_token(installation_id):
    """Exchange JWT for installation access token."""
    now = int(time.time())
    payload = {
        "iat": now - 60,
        "exp": now + 600,
        "iss": GITHUB_APP_ID,
    }
    token = jwt.encode(payload, GITHUB_PRIVATE_KEY, algorithm="RS256")
    resp = requests.post(
        f"https://api.github.com/app/installations/{installation_id}/access_tokens",
        headers={
            "Authorization": f"Bearer {token}",
            "Accept": "application/vnd.github.v3+json",
        }
    )
    return resp.json()["token"]

@app.route("/webhook", methods=["POST"])
def handle_webhook():
    # Verify signature
    sig = request.headers.get("X-Hub-Signature-256", "")
    expected = "sha256=" + hmac.new(
        WEBHOOK_SECRET.encode(), request.data, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(sig, expected):
        return jsonify({"error": "invalid signature"}), 401

    event = request.headers.get("X-GitHub-Event")
    payload = request.json

    if event == "pull_request" and payload["action"] in ["opened", "synchronize"]:
        pr = payload["pull_request"]
        installation_id = payload["installation"]["id"]
        token = get_installation_token(installation_id)

        # Post initial "in progress" check run
        create_check_run(token, payload, status="in_progress")

        # Score the PR (your risk calculation here)
        risk_score = calculate_risk_score(pr, token)

        # Post final check run result
        complete_check_run(token, payload, risk_score)

    return jsonify({"ok": True})

Step 3: Post the Check Run with Risk Score

def create_check_run(token, payload, status="in_progress"):
    repo = payload["repository"]["full_name"]
    sha = payload["pull_request"]["head"]["sha"]
    requests.post(
        f"https://api.github.com/repos/{repo}/check-runs",
        headers={
            "Authorization": f"token {token}",
            "Accept": "application/vnd.github.v3+json",
        },
        json={
            "name": "Deploy Risk Score",
            "head_sha": sha,
            "status": status,
            "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ"),
        }
    )

def complete_check_run(token, payload, risk_score):
    repo = payload["repository"]["full_name"]
    sha = payload["pull_request"]["head"]["sha"]
    pr_number = payload["pull_request"]["number"]

    THRESHOLD = 75  # Block merge if risk score >= 75

    if risk_score >= THRESHOLD:
        conclusion = "failure"
        summary = f"Deploy risk score {risk_score}/100 exceeds threshold ({THRESHOLD}). "
        summary += "Mandatory expert review required before merge."
    elif risk_score >= 50:
        conclusion = "neutral"
        summary = f"Deploy risk score {risk_score}/100. Elevated risk — additional review recommended."
    else:
        conclusion = "success"
        summary = f"Deploy risk score {risk_score}/100. Change is within normal risk bounds."

    requests.post(
        f"https://api.github.com/repos/{repo}/check-runs",
        headers={
            "Authorization": f"token {token}",
            "Accept": "application/vnd.github.v3+json",
        },
        json={
            "name": "Deploy Risk Score",
            "head_sha": sha,
            "status": "completed",
            "conclusion": conclusion,
            "completed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ"),
            "output": {
                "title": f"Risk Score: {risk_score}/100",
                "summary": summary,
                "text": build_risk_details(risk_score, payload),
            },
            "details_url": f"https://app.koalr.com/prs/{repo}/{pr_number}",
        }
    )

Step 4: Configure Branch Protection

Once your GitHub App is posting "Deploy Risk Score" check runs, add it as a required status check on your main or release branch:

1. Repository Settings → Branches → Branch protection rules → Edit main
2. Check "Require status checks to pass before merging"
3. Search for and add "Deploy Risk Score"
4. Optionally: also check "Require branches to be up to date before merging"

With this configured, any PR where your check run returns failure will have its merge button disabled — GitHub will show "All required checks must pass before merging." Engineers cannot merge a high-risk PR without either:

• Reducing the risk score (improving the change), or
• An administrator overriding the branch protection (which is audited)

The Reviewer UX: What Engineers See

When a PR is blocked by a high risk score, the reviewer sees three things in the GitHub PR interface:

The check run status: A red X next to "Deploy Risk Score" in the status checks section, with the label "Risk Score: 82/100 — Expert review required."

The details link: A link to your risk scoring platform (e.g., Koalr) that shows the breakdown — which signals contributed to the high score and what actions would reduce it.

The check run output: Expanded inline in the PR, a summary of the risk factors: "Author has no prior commits to 3 of 4 changed files. Coverage decreased by 8%. 1 DDL migration detected."

This UX is significantly more actionable than an advisory risk badge. The engineer knows exactly what the scoring system found objectionable and what they can do about it.

Setting the Right Blocking Threshold

The threshold for blocking versus warning is a judgment call that depends on your team's risk tolerance and current CFR. Some guidelines:

Start with a high threshold (85) and lower it as your team calibrates to the signal. The goal is not to block most PRs — it is to create a forcing function for the specific changes that are statistically most likely to cause incidents.