Improving Admin Center setup and deployment times
Revising the Viva product suite's setup and management dashboard to improve setup times by creating a consistent experience
Overview
The Viva app management dashboard in the Microsoft admin center allows admin users to setup, deploy, and manage the Viva apps purchased by an admin’s company
The driving principle of the dashboard is to help customers ensure all app features are fully setup and running. Tasks for users include things like giving proper admin permissions; audience and role assignment; content sources; and other app specific steps
Solution
An updated setup dashboard that reports status and a consolidated and more standardized setting configuration process
Results
Reduction of setup and deployment times
Constraints
Working within a rigid design system
6 week design and ship timeline (+ 2 weeks of A/B testing)
The problem:
Based on submitted feedback and interview data, users were having difficulty with the app setup process. Without a way to check app status and complete the setup process in one place, users were missing key steps and task-switching.
This could cause low usage rates and impact customer retention due to a lengthy time-to-value perception.
01. Finding the cause of the problem
Auditing the design
Using data gathered from user interviews, I ran a product audit to identify specific areas where the problems were occurring
Notable findings:
1
Lack of information that left users in the dark about what is complete, in progress or even needs to be started
2
The look and feel of setup pages could vary wildly and sometimes occur outside of the Admin Center almost entirely
3
Users had a general uncertainty about what steps needed to be done first or if the steps needed to be done at all
02. lightweight cross-functional workshopping
Collaborating on ideas
Design approach
I used the audit results to help support a collaborative session with my PM and engineering partners. We first discussed what our key solutions should encompass.
From there, we generated some early ideas via a crazy 8s session and then dot-voted to narrow down our top three. This helped to build alignment in the group about potential approaches.
Sorting ideas
To decide on the areas where we could most easily create impact, I guided the team to sort ideas from the workshop into three categories of potential approaches:
1
Notify
Ideas that focused on improving how users were informed of any action needed
2
Organize
Approaches that could present info in a more consistent way to streamline the process
3
Consolidate
Ways to bring all setup steps into the same admin center dashboard
The hypothesis
By surfacing relevant info upfront and providing a consistent experience across apps, users will finish setup faster and be confident that they completed their task.
This will improve setup times and setup success rates. As a result, app usage and customer churn rates will also improve across the Viva product suite.
03. Iterating
Exploring design options
As I explored designs I wanted to make sure I kept in mind the following:
A minimal disruption to users typical workflow
Staying within Microsoft Admin Center's rigid design system guidelines
Creating a scalable, flexible design for different app setup needs and future updates
Design Flow A
Separate settings out in a table similar to the landing page
Surface the priority of each setting on the app setup page
Place each setting's individual steps in a fly-out component
Design Flow B
Surface setting setup upon entering page
Sort each setting into a guided wizard
Mark "priority" steps with a relevant icon
04. Guiding direction with Quantitative data
A/B/C testing to validate our decisions
Measuring success
To ensure that we gained value from the test, my project PM and I met to establish what would define a “successful” test
1
Success rate
Users in the non-control groups should be able to setup or configure an app successfully more often than the control
2
Setup time improvement
Metric should have a noticeable improvement of at least 5 minute setup time per app over the control version
3
Negative qualitative notes
Did we receive any noticeably negative qualitative feedback from users during the test
Testing
Once the new designs were built by engineering, they were pushed to production where user traffic was split across the 3 options (control vs. the a/b variations). The experiment ran for close to 3 weeks.
I worked with engineering to make sure we frequently and prominently surfaced our feedback widget to gather qualitative data in addition to quantitative. I also worked closely with a data science team to help finalize the metrics and understand the outcomes.
The test stats
Over the 3 week testing period our experiment had approximately 5,000 admin users
Results
20
Minute decrease (Version A)
Version A resulted in users decreasing their setup time by over 10 minutes. A 60% reduction in setup time per app!
2.5
Minute decrease (Version B)
Version B resulted in users decreasing their setup time by 2.5 minutes. While positive, users did better with version A.
98%
Success rate
Users had a success rate when setting up any app of over 98%!
Based on the results from the test, we determined that the version that focused on showing status of all apps upfront provided the best time improvement for users and also returned more positive qualitative feedback.
05. Collected qualitative data pushes for updates
Iterating from feedback
Following user feedback
While the results from the status and table version already showed our designs were successful, some users had mentioned that the sudden shift from the old design confused them.
Finding a solution
We hypothesized that the shift in design may have strayed too far off from a user's existing mental model for setup in the admin center.
To counter this, we also added in an onboarding process to guide users through the new system.
This shift in product strategy helped to anticipate adoption risks and mitigate them through design.
Key impacts
Task time reduction
Setup success rate increase
Time-to-value improvements
Post-launch, customers gave qualitative feedback that they felt value quicker from Pulse.
Project takeaways
Areas to improve on
While we had strong metrics to understand the strength of the designs presented in A/B testing, it would have been worth setting up methods to gain additional qualitative feedback. This would have allowed us to better measure usability.
Next steps - Implementing opportunities to promote premium plans
While not within the scope of the project, I identified an opportunity to leverage unused white space for promotion premium features (known as "add-ons").
This would not only make a user aware of available features that would improve their experience, but also take could work to notify users if a "add-on" was purchased and needed setup.
Unfortunately, this idea was tabled after I moved to a different product team, but at the time, it was noted by product leadership as an idea that could boost sales and adoption rates, while also increasing value customers received from their Viva apps!
NEXT PROJECT
