A hallucination eval for an AI career/interview coach — 30 test cases across six failure modes, scored against two system-prompt versions of the same underlying model. The variable is the prompt, not the model.
Both versions are the same "AI Career Coach" persona run on the same underlying model — only the system prompt changes. V1 (naive) is friendly and is explicitly told to "avoid being wishy-washy," which rewards confident specificity with no instruction for what to do when it doesn't actually know something. V2 (grounded) keeps the same helpful persona but adds explicit rules: distinguish known facts from things it can't verify, say so when unsure instead of inventing a precise-sounding figure, correct false premises instead of agreeing with them, and refer out to a professional for anything with legal/immigration consequences.
30 cases across six categories, five each: company-specific facts, salary/market stats, ambiguous/underspecified questions, adversarial false-premise prompts, out-of-scope visa/legal questions, and control questions that should pass regardless of prompt. Facts used as ground truth (Google's APM process, Anthropic's PM interview stages, PM salary ranges, F-1/STEM OPT unemployment limits) were verified against public sources — Glassdoor, levels.fyi, igotanoffer.com, and USCIS — not invented for the exercise.
Each response was graded PASS / PARTIAL / FAIL against the pass/fail criteria defined per case before generating responses, to avoid grading to the outcome. PARTIAL covers responses that move in the right direction but don't fully commit to it — e.g. softly validating an exaggerated claim before pivoting, instead of naming it as exaggerated. Aggregate rates on this page count PARTIAL as 0.5.