Secret Student
Want a lot of insights fast? Be the customer!
I was able to evaluate the wholistic customer experience by attending my company's classes. I got to get to know our customers and the customer experience by participating in it first-hand.
This is a good project for you to look at if you:
Want to see what I do when I need to get a lot of insight fast and on a budget.
My Role
-
User Experience Researcher team of 1
Methods
-
Ethnographic Research
-
Guerilla Research
-
Journey Mapping
-
Creating Heuristics
-
Research Roadmapping
The Outcomes
-
We established better foundational knowledge on our customers
-
Grew my knowledge of our products
-
Insights that contributed to which projects we would prioritize
-
A research roadmap for the year, based on what the largest pain-points identified
-
A standard evaluation form used by Product
Background
I was a few months into my role as a UX Researcher at Management Concepts, and I was tasked with planning our research roadmap. I needed to prioritize research and come up with ideas for projects.
I had to prioritize what would have the largest impact on our users, but I didn't know our users well yet. Also, as a new staff member, I didn't feel my grasp of the actual class experience was good enough (I had never taken one of our classes, after all.)
Problem Solving
The Problem
The Ask: Plan a research roadmap for the next year by identifying high-impact projects
What I needed to complete the ask:
-
A better understanding of the users
-
What their journey looked like
-
What their pain points were
-
-
A better understanding of the products
-
Overall, I needed a better foundation to build future research insights off of
Extra Challenges: No budget, every stakeholder had a different opinion of what was most important (as we had very little formal research at this point to back up any assumptions).
I needed to make scaffolding, but I didn't even have a foundation built yet.
My First Idea
The first approach I thought of was to conduct some user interviews with customers about their experiences with us. This wouldn't work though...
The problems with this approach:
-
That didn't help me get to know our products better
-
I didn't have any room in my budget for participant incentives :(
-
I needed to understand more than just what the pain points were, I needed to get to know our customers and their journey with us
My Next Idea
My next thought was to break it up into a few projects, after all, there was a lot we wanted to know. I was trying to gain a better foundation for my research by understanding the customer, their journey, and their products, as well as plan future research.
But let's be real... that wasn't an ideal solution because:
-
I needed a foundation fast
-
I wanted to start conducting the impactful research ASAP
-
Pure foundational research wasn't going to help me win stakeholder support
My next idea was to break the project into pieces, but that wasn't ideal either.
My Solution
I needed to get my hands dirty and take some of our classes.
There were benefits to this approach:
-
I could get a better understanding of the products, the students, and their journey
-
I would get some valuable data
-
I'd have a much better understanding about what would have the greatest impact on our customers
I enrolled in a variety of classes, from a good representation of subjects and course types (different course lengths, purposes (such as test prep, workshops, technical skills, etc.).
I was deliberate as I attended the courses to squeeze every last drop of data I could out of the experience:
-
I created and used an evaluation from to document the feedback and observations
-
I observed the other students in the course, giving me a deeper understanding of the Learner Experience
-
I asked students for feedback
This mixed-methods approach was a bit of a combination of methods
-
Longitudinal Ethnographic Research
-
Because I was participating in training courses fully immersed as a student to observe user behaviors, learning patterns, and engagement
-
-
Contextual Inquiry
-
During the courses, and sometimes after, I'd ask students I'd talked to in class about their experiences in the course (and any pain points they had)
-
-
First Person UX Evaluation
-
Using an evaluation form I'd created I evaluated the experience, almost like a longer-term Heuristic Evaluation
-
-
Secret Shopper
-
While this is traditionally done with a hired participant, I was putting myself in that role as a student and reporting on the experience
-
I'm a big proponent of the idea that to be a good UX researcher, you need to have used your products before, the more the better. In my case, I hadn't gotten the chance to attend one of my company's training courses yet, until now!
Creating the Evaluation
To keep the evaluation as unbiased as possible, and to help us measure the learning experience as well as the user experience during the class, I decided to create an evaluation tool to use when evaluating the course.
Putting to use my Learning Experience Concentration and chatting with a few of our Instructional Designers, I created an evaluation tool in FigJam to use during the class.
The evaluation tool was essentially a guided heuristic evaluation, with heuristics based on learning experience principles that we wanted to measure for our classes. We also created sections on the evaluation for specific elements that our courses had, like exercises, scenarios, the Learning Management System, course materials, the virtual classroom, etc.
In the tool, it asked you to take some time after each lesson to reflect on the heuristics and write down feedback based on your observations of the other students in the class and also your own experience.
The Evaluation Tool's Impact
The product managers were very interested in the evaluation tool, the domain managers (the people responsible for managing the courses and content of those courses in each subject area) began using my form in their evaluations as well, giving us a standard set of criteria we evaluated courses based upon.
Conducting the Research
I enrolled into a variety of courses including:
- Persuasive Speaking
- Briefings and Presentations
- Negotiating Contracts
- Data Visualizations and Modeling
- Introduction to Project Management
- PMP Test Prep
- Diversity, Equity, Inclusion, and Accessibility
As I attended the classes, I filled out the evaluation tool (which included writing down any problems that occurred, any pain points I saw with students, and any other areas for improvement). This data was helpful on an individual course level, but also helped me get a feeling for what our products were like.
I also made sure to talk with the students in the classes. I wanted to get to know our students, so I could better understand what they needed and act as a better advocate for user needs. They were happy to tell me about what they thought of the classes and gave me a lot of great feedback.
An Opportunity I Took Advantage of
Many of the participants I talked to were happy to conduct research with me in the future.
These participants were much easier to recruit than the times I'd just gone to our classes and asked for participants, since they had gotten to know me better, as a classmate, but also as someone who listened to their feedback and complaints.
It helped me get a start on building my research panel of customers.
The Findings and the Projects they Inspired
Painpoint Findings
-
Our Learning Management System was difficult to navigate
-
Students needed a better format for their materials
-
Students wanted more of the hands on learning aspects of the training, with more of a focus on being able to apply what they're learning to their jobs (aka outcomes based learning)
-
Our virtual classroom experience was frustrating
-
The registration process for classes was frustrating
Projects Inspired by these findings:
-
Updating the Learning Management System
-
Creating the Digital Guide to start replacing PDFs of participant materials
-
Updating the student experience on the website
Student Experience Findings
I was able to immerse myself in the student experience, giving me a much better feel for the products I was working on and how they functioned in the real world.
Projects Inspired from these Findings: The course evaluation tools I'd completed helped inform the wholistic journey map I created that mapped the entire student experience, starting when they first decided they needed to take a class, through the purchase process, through the classroom experience, and then through the after class parts. Smaller details of each section of the journey were informed by other projects, but the larger journey came from this (and helped me identify what those more specific projects would need to be).
Getting to Know Our Students
I was able to talk to a lot of students, who were in various government roles. There was a wide variety of students, but I learned:
-
What was important to them
-
Why they were taking the classes
-
What they wanted to get out of classes
-
What they thought of different learning methods
A project this inspired: This specifically helped push me into exploring more about student motivations for classes, we had some students in it to upskill themselves, whereas other students were fulfilling a requirement, either for their agency or for a certification. This led to us conducting a more extensive project exploring student motivations for classes, leading to the generation of several student personas.
Finally, I got to know users on a personal level, they were no longer the concept of the user, they were actual people.
Some Self Reflection
Writing this down is making me reflect that there was a lot going on with this project. I had a lot of questions, plus I was taking the classes.
Keep in mind, however, that these classes took place over several days, with time for me to work on the reflection aspects, and I didn't take these classes back to back, I spread them out over several months.
However, it was a lot. I needed to get a lot of information to get on my feet as a researcher, there was a lot I didn't know. I didn't know our products well, I didn't know our users well, and I needed to figure out what was the most important things to research were.
I absolutely milked this project for everything it was worth. I do that with most projects. Working on a UX team of 1 with a small budget, I have to get every insight I can. I have to be clever and adaptable. If I see an opportunity, I don't have the luxury of pursuing it later.
I 100% would do it this way again. This project was the foundation I build everything upon. My knowledge about what to prioritize, my knowledge about our customers, my knowledge about our products, started with this project.
To this day, I make sure I take a class every few months, not only to keep the student experience fresh in my mind, but also to see how new innovations have changed that experience. (Also it's nice from a professional development standpoint as well! I've learned a lot about analytics and project management, as well as soft-skills from our classes.)
Sharing the Findings
The Pain Points: This project left me with many findings, the first stage was sharing the major pain-points. I created a list of these pain points and presented them to stakeholders. I kept it simple, with the focus being the top five issues and evidence to support them. From there we created the research roadmap to focus on researching these issues further and looking for solutions.
The Results of Individual Course Evaluation: Using the evaluation tool had given a good amount of detailed feedback about specific courses, as well as the feedback on our courses in general. This feedback was provided to domain managers, who were responsible for updating courses, that way, when the course I had evaluated was updated, that feedback could be taken advantage of.
The Data on Who our Students were: this was shared via the creation of personas, who were then presented about for a variety of relevant audiences who would be especially interested in them.
The Foundation was built
With this project complete, I had the solid foundation I needed to build upon creating research within our organization. This project gave me the information and base I needed to have a better understanding of what we didn't know. Did I have more questions after the project than I did before the project? Yes! However, I'm a researcher (and a naturally curious person), so that's par for the course.
My Next Steps
With my strong foundation established, my next step was answering those questions! You can explore the rest of my portfolio to see some of those projects.
This project is one that I found valuable, and an exercise I find myself repeating periodically. Every so often, I take one of our classes, because the better I understand our products and users, the better I can do as a UX researcher.