The Comment Analysis tool is a single component from the larger Survey Results Dashboard redesign. Some of the greatest qualitative engagement insights come from open-ended comments. In order to create the right experience for our users, we partnered with a company that specialized in Natural Language Processing. Their AI capabilities helped us take the long, open-ended comments and break them up into thematically linked comment fragments. It also enabled us to rank the sentiment of these fragments from -20 to +20, most negative to most positive.
The previous Comments dashboard lacked the basic functionality to conduct proper analysis. In fact, our analytics showed that most users went straight to exporting their comments into Excel. We found this unacceptable considering the Comments dashboard was an additional cost to our clients, and wanted the new tool to aid in their analysis.
Users wanted to understand if comment themes were company-wide problems, or if specific demographics or business units were having issues. They wanted to track how themes compared to past surveys, and understand how the themes varied across key demographics like location or discipline.
The comments were extremely disconnected compared to the rest of the survey results. There was only one access point to Comments and it would take the user to an entirely separate dashboard. We wanted the new survey results dashboard to feel like a holistic experience with relevant qualitative and quantitative data
To better integrate the comments with the survey results, we wanted to create a dedicated Comments Component to display on the survey results dashboard. We wanted this section to serve as: 1) an overview of the themes, 2) a tool to conduct theme analyses and comparisons, and 3) a jump-off point for the detailed Comments Dashboard when users were ready to dig in and read through specific comments.
The Comments Dashboard is where users can read through the comments. We wanted to create a filtration system so users could narrow the comments by theme, sentiment, question, or any of the various demographics. We also wanted to ensure they could do a keyword search and save comments to refer to later.
I love to start on a whiteboard. Whiteboarding forces me to stay high-level and get ideas out. Plus, the kid in me loves to play with the chunky markers.
Once I have a few ideas going, I’ll move to the computer and start wireframing or playing with visual explorations, depending on the audience or conversations I plan to facilitate.
I led a weekly design meeting with stakeholders, architects, front-end developers, and the product owner and business analyst that would be working on this feature. On these calls we discussed design concepts and iterated based on business requirements, technical constraints, and end-user needs.
Once we had a team consensus on our favorite (and feasible) Comments Analysis experiences, we began usability testing to validate the designs. I partnered with a UX Researcher and we collaborated on the script. I also created the prototype and presented the results to our stakeholders.
We tested the experience with six Cathy (HR business professional) users. While both managers and HR business professionals will use the tool, Cathy’s have access to the most complex functionality — which were the areas we had the most questions and concerns with.
Overall, the test went well and users were very excited about the new and improved experience. However, there were a few areas of concern:
In the Comments Component, I wanted to avoid displaying the comment themes in a typical word cloud as they can be overwhelming and difficult to interact with. Instead, I opted for word bubbles. The size of the bubbles conveys the volume of comments per theme while the color conveys the sentiment.
When users click on a comment bubble, a detailed popup displays for that theme with sentiment distribution, demographic groups that commented frequently, and the most prevalent sub-themes. From there, they can decide if they want to read comments for that theme or explore others. Users can also use the Breakdown view to see the top themes for a demographic, or use the Benchmark view to see how the frequency and sentiment of their comments themes compares to internal, external, or historical benchmarks.
The comments dashboard is where users can read through comment fragments. To indicate the sentiment, we display a colored dot next to each fragment. We decided to use the left column for filters like themes, sub-themes, sentiment, comment questions, and demographics. This was inspired by typical eCommerce experiences. Additionally, the vertical approach was more flexible — some clients only use a few demographics while others could use 30 or more, and it would have been challenging to accommodate the latter in our earlier horizontal filter explorations.
When users click on a comment fragment, the entire comment will appear with metadata about the employee that commented (as long as there are enough comments from those groups to maintain confidentiality). We added the re-categorization functionality in the ‘more’ button that appears when users hover over a comment. Additionally, users can ‘star’ comments, to allow them to refer to later.
In addition to including the Comments component on the Survey Results Dashboard, we wanted to sprinkle comment insights throughout in more quantitative-driven components as well. In the Strengths and Opportunities section, we include contextual comments in the detailed popup. We also show related comment themes in the Topic detail popups, and display the number of comments related to the various categories in our Benchmark component.
