Enhancing Search with AI

Alight Worklife's ineffective search function caused user frustration and increased support calls. Stakeholders needed a solution to enhance user experience and explore monetization opportunities. I was responsible for the creation of a research plan that aligned with user needs and business objectives, resulting in a redesigned search experience with the AI-powered chatbot, Ask Lisa, which improved efficiency, boosted user satisfaction, and generated new revenue streams.

Overview

Alight Solutions

Role: Senior UX Researcher

Research Methods: Concept Testing, 2x2 counterbalanced within-subjects Usability Testing, Secondary Research, CSAT analysis

Tools: UserZoom, Figma, The Baymard Institute

Sector: Wellbeing, Human Resources, SaaS

Context

Alight Worklife, an employee benefits platform, serves Fortune 100 clients and their employees, but faced consistent negative feedback regarding the inefficiencies of its search function. Users were frequently presented with long lists of unrelated PDFs from their search inquiry, leading to frustration, negative AppStore reviews, and a rise in customer service calls. Business stakeholders across departments requested a solution that would not only improve search functionality, but also explore personalized recommendations as a potential revenue driver. My role involved translating these business needs into a strategic research and testing plan that ensured the AI-powered Ask Lisa chatbot could meet user expectations while driving business outcomes.

Problem Statement

Users are frustrated with the current search function, which provides disorganized and irrelevant results, leading to an overwhelming experience and dissatisfaction. The business needs a solution that not only improves the user experience, but also creates revenue opportunities through personalized chatbot recommendations. The challenge is to design a chatbot-driven search experience that aligns with user expectations, resolves usability issues, and delivers measurable business value.

Research Goals

  • Identify if the proposed chatbot-driven search experience provided value to users.

  • Uncover usability issues with the redesigned search experience.

  • Evaluate when a chatbot should appear and how many personalized recommendations users found useful.

  • Explore potential monetization through personalized chatbot recommendations.

Research Approach

After identifying business needs, I crafted a comprehensive research plan that combined stakeholder insights, user research, and secondary analysis to set a new standard for chatbot interaction.

Key Insights:

  • Initial Assessment: I mapped the current search flow for both desktop and mobile platforms, analyzed thousands of CSAT comments to identify the most problematic areas, and consulted with client teams to understand their client’s experience with the search function and AI tools.

  • Unknowns to Explore: I tested whether the chatbot redesign aligned with user’s mental models of search results and identified any usability challenges, including determining optimal chatbot appearance timing and the ideal number of recommendations.

Hi-fidelity mockup of a dual-sided search results page with Ask Lisa displaying a direct search result.

Concept Testing Insights

Through rounds of concept testing, I evaluated how users interacted with the chatbot and being presented with a dual-sided search results page, which displayed both chatbot responses and a list of documents.

  • Findings:

    • Over half of the users were familiar with chatbot-driven results, but confusion arose when balancing the dual-screen layout, prompting further refinement of the visual layout to better emphasize the chatbot responses.

    • Additionally, frustration increased when users couldn’t find a direct answer, especially when prompted to rephrase their search queries.

Hi-fidelity mockup of Ask Lisa displaying 3 recommendations.

Usability Testing Insights

I used a 2x2 counterbalanced within-subjects design to test chatbot timing (5 seconds vs. 10 seconds) and the number of recommendations (2 vs. 3) across different experiences.

  • Findings:

    • Users found 1-2 recommendations to be more useful than 3 recommendations, especially in mobile environments, as 3 recommendations overwhelmed them in small chat windows.

    • Users were split on the chatbot appearance timing, with secondary research from the Baymard Institute suggesting a 30-second delay improved engagement.

Tradeoffs

During the redesign process, several key tradeoffs were carefully considered to balance user experience and business goals:

User Expectation vs. Chatbot Integration:

While rerouting users from traditional search to a chatbot had the potential to significantly improve search accuracy, it also risked alienating users who were more accustomed to standard search results layout pages. The tradeoff was between improving the efficiency of search results and potentially disrupting established user expectations. By conducting thorough testing, we aimed to minimize user confusion and ensure the chatbot felt intuitive within the user flow.

Timing vs. Engagement:

Choosing the optimal moment for the chatbot to appear was another important tradeoff. Introducing the chatbot too early could overwhelm users, while too late might result in missed opportunities for engagement. Through testing, we weighed the balance between offering timely assistance and giving users space to explore the page, ultimately settling on a delay that best aligned with user preferences and research-backed best practices.

Personalization vs. Overload:

Providing personalized recommendations offered significant potential for increasing engagement and monetization, but too many recommendations risked overwhelming users, especially in mobile environments. The tradeoff here was between maximizing the value of each interaction without cluttering the user experience. Testing showed that 1-2 personalized suggestions struck the right balance, enhancing engagement while maintaining a clean, user-friendly interface.

These tradeoffs were essential to ensuring that the chatbot both met user needs and supported business objectives, resulting in a solution that optimized the search experience without sacrificing usability or overwhelming users.

Hi-fidelity mockup of Ask Lisa displaying 2 recommendations in a mobile experience.

Business Impact

  • Enhanced User Satisfaction: The AI-powered chatbot improved search accuracy and relevance, reducing frustration, lowering call center volume, and achieving a 90% self-service rate.

  • Increased Engagement: Engagement with the chatbot increased by 37% after implementing personalized recommendations that users found relevant, offering new revenue streams for Alight.

  • Streamlined Usability: Simplifying the chatbot interface and reducing the number of recommendations improved search navigation and user satisfaction, particularly on mobile devices.

  • Cost Savings for Employees: The AI-powered recommendation capabilities provided automated decision support and choice optimization for benefits and care, saving employees an average of $500 in premium expenses annually.

Outcome

The redesigned search experience with Ask Lisa went beyond a feature update. It aligned business goals with user needs, improving search efficiency, driving user engagement, and creating new revenue opportunities. The chatbot has since become a cornerstone of Alight’s strategic vision, laying the foundation for future product advancements.