Case study
Choosing a Layout for an AI Copilot: Three Rounds of Concept Testing
- Project scope
- Cross-modal AI copilot layout (3 comparative evaluation rounds).
- My role
- UX Researcher (Experimental design, participant recruitment, moderation, analysis and synthesis).
- Timeline
- Jun to Jul 2026 (three testing rounds)
In short
Problem: The team was leaning toward one copilot layout, with no evidence it would hold up during live calls and chats.
Result: How popular a layout was depended on what it was compared with, so I advised against choosing on that data. The hybrid direction agents asked for carried into Next Gen.
I developed a multi-round concept testing framework to identify the optimal structural layout for an AI copilot during high-stress, live agent customer calls and chats. My analysis uncovered an "opponent effect" in user preferences, demonstrating that perceived utility depended on the comparison set. This insight is why I advised the team against choosing a layout on that data.
01The Challenge & Question
The product team was leaning toward a single layout concept, but lacked empirical evidence that it could support agents during real-time phone and chat interactions.
I evaluated five distinct structural layout hypotheses: three tailored for live phone calls and two optimized for live chat. I needed to determine which structural model minimizes cognitive load and maximizes utility for agents multitasking under pressure.
02Strategic Decisions
Multi-Round Concept Testing Framework
Rather than depending on a one-off preference survey, I designed a flexible, three-stage evaluation process to rigorously test the concepts.
- Round 1 (N=22): A side-by-side evaluation of all 5 concepts to establish baseline comprehension.
- Rounds 2 & 3 (N=11, N=16): Target head-to-head matchups designed to break the tie left by the first round.
Uncoupling comprehension from user desirability
All five layout concepts scored high on a 7-point comprehension scale (ranging between 5.3 and 6.0). However, the dense baseline layout scored third in comprehension but received zero votes for preference.
The Strategy: I showed the product team that comprehension was not a factor in preference. The core usability blocker was visual clutter, not a lack of understanding.
Acting as a data guardrail against premature optimization
When two leading concepts emerged, they only faced each other once, in a small sample of 11 agents.
The Strategy: I advised the team against choosing a foundational product layout based on thin data, advocating instead for systematic validation.
03Key Discoveries: The "Opponent Effect"
The most significant finding was that user preference for the single-feed chat concept was not absolute; its vote share moved with its competitive pairing:
| Testing phase | Competitive pairing | Vote share for single-feed |
|---|---|---|
| Round 1 | Up against 4 alternative concepts | 36% (8 of 22 votes) |
| Round 2 | Head-to-head with a 3-panel phone layout | 18% (2 of 11 votes) |
| Round 3 | Head-to-head with a step-by-step phone layout | 63% (10 of 16 votes) |
The Synthesis
The core concept never changed. The large swing suggests the multi-panel layout structure wasn't inherently flawed; rather, the specific three-panel execution was simply a stronger, highly skimmable layout. That is why I cautioned against reading these results as "single-feeds always beat multi-panels," or the reverse.
What Held Across the Rounds
- AI Guidance Is Crucial: Across all rounds, features like real-time sentiment analysis, auto-flagged issues, and suggested resolutions made agents feel more confident during live calls.
- The Structural Tradeoff (my interpretation): Multi-panel views offered rapid scannability but risked causing visual claustrophobia. Conversely, single-feed layouts felt familiar and readable but suffered from high scroll-fatigue, creating a risk that agents would miss critical, time-sensitive information.
- Micro-UI Friction Causes Macro-Confusion: Minor issues—such as non-distinct tag color palettes, ambiguous labels regarding what data customers could see, and unlabeled icons—caused real confusion for agents, regardless of the overarching layout.
04The Strategic Pivot & Impact
While this standalone layout track was ultimately deprioritized by business leadership, the insights were far from lost. We carried this data into the design of our high-priority Next-Gen Agent Workspace initiative.
Instead of forcing a binary choice between a single feed or a multi-panel layout, I used this evidence to champion a hybrid direction directly requested by our users: pairing the clean, low-clutter aesthetics of a single feed with the highly actionable, glanceable guidance of a multi-panel architecture. I also recommended shipping the micro-UI fixes regardless of which layout was chosen.
05Methodological Boundaries & Self-Critique
To maintain absolute transparency with hiring panels and stakeholders, I treat these multi-round head-to-heads as directional rather than definitive due to several structural realities:
- Cross-Modal Matchups: The head-to-head rounds paired phone-specific concepts against chat-specific concepts, which introduces variable noise.
- Sample Size & Cohorts: Rounds 2 and 3 utilized localized sample groups (11 and 16 agents).
- Mitigation: I presented these results as directional, not a final, closed verdict, and as a proof-of-concept for the hybrid layout direction that was slated for development as of my last knowledge.
06What I'd Do Differently
I would design the finalists to meet more than once, with a larger sample, from the start. I would keep each head-to-head in the same format (phone against phone, chat against chat) so a swing in votes could be traced to the layout and not the format. And I would confirm up front whether the same agents take part in every round.