Studio Transition: Benchmarking a Critical Desktop-to-Web Migration

01The Challenge & Question

The product organization was actively working toward retiring its legacy desktop scripting studio in favor of a modern web version. While the team knew anecdotally that customers were struggling with the transition, they lacked an objective, standardized metric to measure the severity of the friction.

I set out to answer a multi-layered question: How wide is the usability gap between the two architectures, which specific user cohorts feel it most acutely, and what structural gaps are driving the deficit?

02Strategic Decisions That Shaped the Work

Implementing a rigorous, within-subjects research design

To reduce confounding variables, I structured the survey so that the exact same individuals evaluated both product versions.

The Strategy: This approach supported reading the 30+ point drop as a genuine disparity in user experience between the architectures, rather than demographic or sample variance.

Leveraging a lean, validated industry standard (UMUX)

I deployed a four-item Usability Metric for User Experience (UMUX) framework, normalizing the results to a widely understood 0–100 scale.

The Strategy: This kept the survey short for respondents. I paired this with targeted open-text fields to immediately capture the qualitative context behind the quantitative scores.

Comparative benchmarking across internal and external cohorts

I intentionally sampled two highly distinct user groups: external customers and internal users, mostly engineers with deep scripting expertise.

The Strategy: Both expert internal power users and external customers landing in nearly the same place on the web application suggests the friction isn't only an enablement or onboarding issue.

03Key Discoveries & Quantitative Benchmarks

The benchmark scores revealed a clear performance gap when compared with the reference range of roughly 60 to 70 cited in the readout for enterprise software:

Web scores strictly exclude 14 customer respondents who had not yet initiated their web transition.

Core Migration Risk Signals

• High Friction for Early Adopters: Among customers who actively attempted the migration, 57% rated the process as difficult or very difficult.
• Low Migration Velocity: 22% of the customers surveyed had not yet started the transition process at all, a friction point for the planned deprecation timeline.
• Tenure is Not a Buffer: Correlation testing found that an agent or admin's tenure was not significantly correlated with migration difficulty.
• Feature Parity vs. Enablement: Qualitative analysis revealed that missing core functionality was the primary reason users regressed to the desktop application. However, a secondary "enablement gap" around training was identified—suggesting that some friction could be mitigated via documentation independently of the core engineering roadmap.

04The Strategic Pivot & Impact

While immediate engineering resources were ultimately shifted toward high-priority AI initiatives, we positioned this quantitative framework as a use case for our emerging Design Ops practice.

The study provided product managers with risk-mapped evidence of our product gaps and confirmed several of their pain points. Furthermore, the plan was to use this study as an example of how to measure and compare experiences, and as a template for future benchmarks of product transitions, deprecations, and legacy migrations.

05Methodological Boundaries & Self-Critique

To maintain absolute data integrity for future testing cycles, I called out several structural realities in our baseline snapshot:

• Expert Skew: The internal group was heavily weighted toward expert power-users, which likely inflated the baseline scores for the legacy desktop application.
• The Untested Cohort: The 14 customers who had not yet logged into the web version were excluded from web calculations, meaning the web average may lean slightly high compared to a true market-wide deployment.

06What I'd Do Differently Next Time

In future benchmarking phases, I would standardize the data capture schema early to prevent mixed numeric and text encoding issues in the raw data export, proactively over-sample the "not-yet-transitioned" customer segment, and run the qualitative interview track in parallel with the survey to speed up time-to-insight.