Model Risk Governance

model risk governance targets three output risks of generative models: ① jailbreak & prompt injection — jailbreaks use crafted prompts (role-play, fictional scenarios, multilingual variants) to bypass safety alignment, while prompt injection smuggles malicious instructions into user or external content to trigger unauthorized actions (consistently #1 in the OWASP LLM Top 10); defense is two-sided: input-side (rules, classifiers, similarity search, instruction delimiters) and output-side (format constraints, second-pass verification); ② output abuse detection — content moderation on model output: harmful-content classifiers, PII re-scanning, code/link allowlists and explicit confirmations for sensitive operations; ③ red teaming — before release, adversarial prompt suites (jailbreak template banks, multilingual and encoded variants, tool-calling lures) systematically hunt weaknesses, quantifying attack success rates and reporting severity; ④ audit & traceability — recording who ran which model version to produce what output, keeping request/response logs with version IDs for accountability and regulatory evidence.

any production system exposing LLM capabilities, especially high-risk tool-calling/agent scenarios; interviews ask about jailbreak vs injection, the most effective defense layer, red-team design and which log fields to keep for traceability.

an ungoverned model passes evaluations yet misbehaves in the wild — safety tests all green, but a one-line jailbreak works, web content injects instructions, and outputs look normal while leaking privacy or violating policy; systematic governance replaces luck with a repeatable defense: two-sided detection forces injection to beat two layers, red teaming finds and patches attack surfaces before release, and complete audit logs trace any disputed output back to the exact model version and call chain for compliance.

Jailbreak vs prompt injection — the difference? Typical attack patterns of each?

What ranks #1 in the OWASP LLM Top 10, and why does it outrank traditional application-layer vulnerabilities?

How is output abuse detected — harmful-content classifiers, PII re-scanning, link/code allowlists, and what each solves?

How to design red teaming? Which attack categories to cover, and how to quantify success rates and track fixes?

How to implement audit and traceability? Which log fields are needed to support accountability and regulatory evidence?