Case Study
Interview signal decays the moment the call ends. This captures it before it does.
An original system design, informed by my work on interview intelligence at Employ Inc — not a description of Pillar's proprietary implementation, UI, or data. Business impact figures below are reasoned estimates, not measured production numbers.
A hiring manager runs a 45-minute structured interview while simultaneously taking notes — which means the notes are necessarily incomplete, because attention is split between listening, asking good follow-ups, and writing. Within 24–48 hours (often longer, once the rest of the week's calendar hits), they open the recording or raw transcript and reconstruct their evaluation from memory: scrubbing to the "tell me about a time..." answer, replaying it at 1.5x, manually copying two or three quotes into a scorecard field per competency, then assigning a 1–4 rating that's really a blend of the transcript, their notes, and how they felt about the candidate by the end of the call.
This repeats per interviewer, per candidate. For one open role, a realistic funnel is 5 candidates through 3 relevant interview rounds each — 15 scorecards, at roughly 20–25 minutes of write-up per scorecard once quote-hunting and note cleanup are included. That's on top of the interview itself, done by people whose actual job isn't writing scorecards.
| Friction | Root Cause | Impact |
|---|---|---|
| Quote loss over time | Real-time notes are incomplete by construction — the interviewer is also running the conversation | By debrief, interviewers can't produce the specific evidence behind their score, so ratings collapse into "vibes" |
| Recency/primacy bias in comparison | Humans over-weight the most recent or most emotionally memorable candidate when comparing several for one role | Strong early candidates get unfairly discounted; weak-but-recent candidates get overrated |
| Inconsistent rubric application | No two interviewers decompose "communication" or "ownership" the same way without an enforced structure | The same competency label means different things depending on who scored it, so aggregate scores aren't comparable |
| Scorecard write-up as a bottleneck | 20+ minutes of manual write-up per interview, per interviewer | Interviewers rush or skip scorecards under sprint pressure, especially for later-round candidates |
potential_bias_signal and a human reviews the raw transcript directly.Error handling: if the transcript segment relevant to a competency is missing, garbled below diarization confidence threshold, or the candidate's response doesn't address the competency at all, the model outputs score: null, confidence: "low", flag: "insufficient_signal" — a missing score is always preferable to a fabricated one.
| Case | Input | Expected Output | What a Bad Result Reveals |
|---|---|---|---|
| Clear | The Ownership example above | Score 4, confidence high, 2–3 verbatim quotes, flag none | Paraphrasing instead of quoting, or hedging to a 3 on unambiguous evidence, means the model is under-confident on clear signal |
| Ambiguous / low-signal | "I'm definitely a team player, I think collaboration is super important" — no concrete example | Confidence low, flag insufficient_signal, no score forced | Scoring this 3–4 on positive sentiment alone means the model is pattern-matching on tone instead of requiring evidence — the most dangerous failure mode here |
| Multi-signal / contradictory | Candidate's "ownership" story reveals they bypassed their manager and caused friction with a peer team | Ownership scored honestly, and the friction surfaced as relevant to Collaboration — not silently resolved in the candidate's favor | Only extracting the flattering half of the story reveals a selection bias toward positive-sounding evidence |
| Edge case | Call dropped after 6 minutes, before any competency questions were reached | insufficient_signal on every competency, no scores, no invented summary | Summarizing technical difficulty as a soft-skills read ("stayed calm under pressure") reveals the model fabricating signal from irrelevant content |
| Metric | Before | After | Reasoning |
|---|---|---|---|
| Scorecard write-up time per interview | ~22 min | ~6 min | Quote-hunting and re-listening is the dominant time cost; editing a pre-filled, quote-attached draft is materially faster than writing one from scratch |
| Time-to-scorecard availability | ~24–48 hrs (batched) | <15 min | Generation runs immediately once the transcript is ready, closing the recall-decay gap that causes quote loss |
| Contradictory scores caught before debrief | ~0% (surfaced informally, if at all) | ~100% (structurally flagged) | A capability change, not an incremental one — every scorecard pair is checked automatically instead of relying on someone noticing mid-conversation |
The math, shown: 5 candidates × 3 relevant interview rounds = 15 scorecards per open role. At 22 min → 6 min, that's 16 minutes saved × 15 = 240 minutes (4 hours) saved per hiring cycle — before counting the time saved in debrief meetings that no longer re-litigate half-remembered impressions.
Risk: AI competency scoring could encode or amplify bias if not carefully designed. A rubric-anchored score is only as fair as the rubric and the model's application of it. Score distributions get audited per interviewer and per candidate demographic slice on a regular cadence, not just at launch; the bias-language flag in Section 5 exists specifically to pull suspect content out of the automated path entirely rather than let the model quietly "handle" it.
Risk: over-trusting AI-generated scorecards could reduce interviewer rigor over time. If a pre-filled draft is good enough often enough, interviewers may stop critically checking it — the same complacency risk as autocomplete. The tool is positioned and measured as a drafting aid, not a scoring authority; edit rates are tracked, and a sustained near-zero edit rate on a given interviewer is itself a flag worth investigating, not a success metric.
Design choice — what stays human, and why: final hiring decisions, any borderline scoring call (defined by the confidence threshold in Section 5), and anything bias-flagged. The system is built to produce evidence and a first-pass read, never a verdict.
Dependency: quality is bounded by the rubric, not the model. A vague or poorly-anchored rubric produces vague, poorly-anchored scores no matter how good the extraction is — garbage rubric in, garbage signal out. This is why Prerequisite #2 isn't optional.