The Supervisor's Playbook Usability Test
The supervisor's playbook was a new kind of product for us, so we wanted to getting feedback was important. While this process was a bit harder than anticipated due to administrative uncertainty in the federal government, we still managed to get great feedback and insight to influence the product.
My Team
-
UX Researcher (Me)
-
Instructional Designers
-
Graphic Designer
-
Project Manager
-
Product Manager
Methods
-
Experimental Design
-
Heuristic Evaluation
-
Qualitative Research
-
Usability Test Facilitation
-
Usability Test Analysis
Time
-
Q1 2025
The Goal
The goal was to get some feedback on a prototype of the Supervisor's Playbook, a new type of product that we were working on. We wanted to know if the way we were designing the product was helpful to the new supervisors, and if it was something that fit into their workflows.
The Process
Meet with the team to discuss research priorities.
Ask for survey feedback from students in current classes.
Conduct usability tests to get feedback on functional aspects of the playbook
Analyze feedback & create actionable feedback for the next iteration
The Product
The Supervisor's playbook is a tool that is for new supervisors who need to learn how to be a better and more efficient leader. This tool is essentially an eLearning which consists of a calendar that they are able to integrate into their personal calendar, and several eLearning modules about how to do the various tasks that are required of them.
This is different for us! We typically do training in a more traditional sense, where the student attends a class. This was more of a guided hands-on learning experience (with the added benefit of helping them manage their supervisory tasks), heavily focused on outcomes and we wanted to make sure we were getting things right!
Step 1: Unmoderated Feedback
We wanted to get the Supervisor's Playbook Prototype in front of some people so that we could hear some general feedback. We embedded some survey questions within the prototype to get feedback as they used the tool. These questions were mainly about specific functionality of the tool or the content.
After they had the chance to view the tool for a few days, we sent them another more general survey, focused primarily on questions regarding overall likes and dislikes, how (and if) they would see themselves using the complete tool, etc.
Distributing the Prototype
We had a very specific audience for this tool, new supervisors. Luckily for us, we already teach some classes focused on new supervisor skills, and we were able to distribute the survey to the participants of those classes. I was about to pat myself on the back, getting participants was easier than ever, and with such a difficult audience too.
My parade was rained on
Unfortunately, the federal government began to face uncertainty, and so did our chances of getting lots of survey participants to answer our questions about the Supervisor's Playbook.
Just like the people in this photo, I would have to make due with the conditions I was given.
We got results, but not as much as we were hoping.
Federal government employees had bigger things to worry about than our supervisor's playbook, so we had to take what we could get. We got some great pieces of feedback, especially on the embedded questions, but I decided I'd incorporate some of the questions in the survey into the usability testing. it would be nice to hear a more detailed explanation of how users actually see using this product at work, now that it was in a prototype form and not just a concept (we had done a bit of this in the discovery phase, but now we had something concrete for them to use).
Step 2: Usability Testing
Recruiting
I gave myself a bit more time than usual to find participants here. It was already a niche group, new supervisors in the government, and it was also a hard time to recruit in general, as could be seen by the survey results.
I created a post on User Interviews (a recruiting panel I've enjoyed working with in the past) and sent out a post.
I managed to recruit three new supervisors, but I could not for the life of me recruit two more. I had some interest from a couple of managers (which in the federal government means someone who has at least one supervisor who reports up to them).
I decided that since the manager is likely providing some amount of support to the new supervisor in their learning experience, they would likely be the next best thing to a new supervisor. I checked with the stakeholders that this was okay and recruited two managers who had managed a new supervisor before.
This setback ended up being a good thing!
The managers gave some of the most helpful feedback of all. Since they had managed new supervisors before, they understood what types of things they would find the most helpful.
In fact, I learned at many agencies, the manager is responsible for helping supervisor's get up to speed and learn how to supervise. The managers were thrilled to be able to have a tool that helped supervisors learn to do tasks, give them reminders, and give them an overview. This was a discovery I likely wouldn't have made if I had not talked to the managers about this product. This discovery was especially useful to Sales and Marketing, as this knowledge lets them put a different spin on this product, if you can encourage manager's to use it, all the new supervisors they work with will use the tool (multiple customers for the price of one!).
The managers were also able to tell me the areas where new supervisors were often the most confused, and where in the tool we might want to give a bit more support. This was largely the data we wanted to gather from the surveys and the manager's feedback combined with the survey results we did gather provided us with actionable improvements we could make to the product.
Between the managers and the new supervisors we were also able to identify usability issues including:
-
The navigation of the tool was confusing
-
The calendar integration had a few aspects users found confusing
-
The product needed a better introduction and overview on how best to use it (as often employees might be assigned a tool without much explanation from management)
Overall, stakeholders ended up being thrilled that we included the managers in the participants of the usability test, as they found that data to be most valuable. So, sometimes problems end up being for the best!
Step 3: Translating Insights into Actions
If there's one thing I've learned as a UX Researcher, it's that giving people actionable insights is the most important part of doing the research.
Some of the data (largely from the surveys and manager feedback) I passed directly to the instructional design team. This included things like suggesting we add sections covering how to support team's mental health and conflict, etc. These were things that supervisors and their managers both mentioned that they struggled with the most.
Other pieces of insights, specifically the usability issues, I designed simple solutions to assist with the issue. For example, the navigation wasn't as clear to the users as it could be, so I redesigned it using best practices and presented the sketch to the group. By presenting simple solutions to the small problems, I was able to help solve some of the usability issues in a way that followed principles of user-centered design.
Reflection
This project was one that I had imagined would be relatively simple from the start. However, administrative changes in the federal government, including mass layoffs of federal employees put a bit of a damper in my initial efforts at recruiting. I was able to adapt; however, and my plan B ended up being more successful than my plan A would have been. While this isn't always the case, it does help show that flexibility is one of the most important aspects a UX researcher can have, because there's always hiccups and problems in projects, that's just a fact of life, and being able to deal with them is vital.