Designing Human Oversight for AI-Assisted Workspaces

01The Challenge & Question

The product team designed a conceptual workspace where AI bots handle tasks while a human reviews and approves the output. However, the concept had little direct evidence from the agents and supervisors who would use it.

I was brought in to answer a critical question: What specific data points does a human need to see to confidently approve or override an AI's work, and does this theoretical model survive real-world operational chaos?

02Strategic Decisions That Shaped the Work

Separating raw user data from researcher interpretation

To protect the integrity of the data, I explicitly partitioned my interim report. I categorized my synthesis into direct user inputs versus "researcher interpretation."

The Strategy: By framing my insights as high-probability hypotheses rather than definitive user feature requests, I prevented stakeholders from prematurely anchoring on unverified solutions.

Prioritizing intellectual honesty over a "perfect" narrative

Only two of our four primary research goals were fully met. I discovered that the presence of a facilitator in the room caused participants to alter how they validated the AI's work, skewing our understanding of "confident approval."

The Strategy: Instead of over-indexing on compromised data, I flagged the limitation to the product team, stakeholders and the UX director in the readout and proposed an unmoderated, facilitator-free testing track for the next phase.

03Key Discoveries & Operational Insights

• The human-in-the-loop model works, but accuracy is non-negotiable. Six out of seven sessions validated the core loop (bot proposes, human edits/approves). However, users made it clear that accuracy is a zero-sum game; they will completely abandon the tool if the AI introduces errors.
• Context-switching is a bigger pain point than AI capability. Agents routinely juggle four to six disconnected systems to resolve a single customer interaction. The research proved that workflow consolidation drove far more user excitement than any individual standalone AI feature.
• The AI-to-human handoff is the highest-value friction point. When a bot fails and escalates a ticket to a live agent, agents require a concise synopsis of what occurred, what was attempted, and where the logic broke. Without this context, customers are forced to repeat themselves.
• Supervisors fear blind trust and "rubber-stamping." Supervisors expressed concern that agents would blindly click "approve" to hit speed metrics. They explicitly requested behavioral coaching signals and quality-assurance views rather than punitive tracking features.

04Product Impact & Outcomes

My research heavily steered the definition of the V2 prototype. Four of the five opportunity areas I identified were greenlit, designed, and evaluated with 8 enterprise users across 5 accounts:

• Escalation Handoff Summary: Built to pass bot-to-human context seamlessly.
• Persistent Context Surface: Designed to keep critical customer data visible across tasks.
• In-Context Knowledge Delivery: Surfaced proactive documentation based on live conversation needs.
• Consolidation Layer: Unified multiple disparate legacy tools into a singular interface.

The Open Question: The fifth area—oversight-quality views for supervisors to mitigate the rubber-stamping problem—was deferred due to product scope. This remained the primary strategic area I was targeting for the next phase of development.

05Methodological Constraints

To ensure our team designed responsibly, I explicitly documented our sample boundaries across all executive readouts:

• Frontline Agent Gaps: Phase A relied on 7 sessions across 5 accounts, but included only one direct frontline agent and one group session for the higher-education vertical.
• V2 Validation Limitations: The subsequent prototype validation phase lacked direct frontline agent interviews.
• Mitigation: I leveraged insights from a parallel concept-testing study on Grid Layouts to cross-reference and de-risk our structural layout decisions.

06What I'd Do Differently

I would reach frontline agents sooner. Phase A rested on one frontline agent, so the agent-side findings were directional, and recruiting two or three more was the first thing I would do. I would widen the higher-education sample beyond one group session. And I would run the facilitator-free test earlier, because two of the four research goals were only partly met when people reacted to a facilitator in the room.