How to integrate user interviews and user tests in product development

Written by

Talent Garden

Don't you want to read? Try listening to the article in audio mode 🎧

User Experience (UX) has become a buzzword and mega-trend. Innovation and project/product managers, designers and developers are looking for ways to analyze and improve their users journey throughout all contact points with a product. An important role for UX is user research, and this article covers two broad user research methods and proposes how to integrate them into product development lifecycles: User Tests, also sometimes known as UX Tests or Usability Tests, as well as User Interviews. What is a User Test? A user test is a structured and pre-defined set of tasks within a new interactive application, e.g. a new enterprise or consumer platform, which pre-selected potential users try to complete, while they are observed and asked about their experience. The tasks should cover user goals, which refers to the intention of what a user wants/needs to do with an application. A common goal in a Customer Relation Management-system for example is to identify potential leads for upselling activities. Commonly, a moderator leads users through the test protocol, this is referred to as moderated User Test. User Tests play a crucial role in UX & Design, as the goal is to uncover potential problems related to usability and user acceptance early in the development process. What is a User Interview? While in a User Test, an interactive prototype with some fidelity (more about prototype fidelity below) is used to test goals or tasks a user can perform, an interview is more centered around opinions and needs of a user. A user interview consists of a semi-structured interview guide with a set of questions and flexibility for the interviewer to ask additional or new questions, depending on the answer of a user. Being able to communicate in a flexible manner with potential users is a way of creating authentic and “feel-good”atmospheres, in which users tell you openly and freely about their experiences. When to use a User Test and when to use a User Interview? In development projects and life cycles, typically you choose either User Tests or User Interviews for different purposes and at different times: User Interviews are often conducted at the beginning of development, with the goal of understanding users, their problems, already ideating solutions or to test assumptions/hypotheses. User Tests are used when a prototype of any fidelity becomes available. In software applications, prototypes range from low-fidelity scribbles, paper prototypes and mock-ups to medium- and high-fidelity click-prototypes and functional Front-Ends. What is the best time to conduct User Interviews and Tests? Your development project should have a (rough) timeline, depicting at which points in development you should involve users to ask for their opinion or to test the UX of a workflow before finalization. It makes sense to interview users or test your prototypes for any kind of product, technology, service or platform, even the ones on the market can profit off optimization! For a certain project, a unique timeline may be needed, however common sense tells us that especially early in the beginning user involvement pays off more than when done mainly later in the process! What do I need to consider for the planning of user activities? There are three parts that you need to consider for the planning of your user interview or test: For user interviews: Which content do you want to cover? What are your assumptions about the users that you already have? What do you actually know (for sure?) about your users? What do you only believe to know? For user tests: How can users access the prototype? Is it a website? Is it a link to a click-prototype? Test this thoroughly, because you don’t want to risk the validity of your results by having a flawed technical setup. The setting: Do you interview/test on-site - which is preferred for more in-depth settings, smartphone applications, hardware and ecological validity (= how “real” the context is in which you study the users) - or do you have a remote setup? Go through your test users’ journey several times and make sure your setting does not get “in the way” of obtaining useful and unbiased results and insights! The protocol: For interviews you need a set of questions, ordered in a semi-structured way, allowing the interviewer to follow a natural flow of conversation with the user, adding or removing questions if felt out of place. A core set of questions, which all users are asked, enables a way of aggregating the results and summarizing it (see “tagging” below). For user tests you need a set of tasks, representing natural user goals (that you have - for example - derived in User Interviews before building the prototype) and a working prototype and a flawless setting to test it: Make sure the devices needed are charged, connected to internet, all applications are in place and working. Metrics are part of your interview or test protocol (see above). This is the data related to your desired outcome to measure, e.g. number, kind and severity of usability problems in a user interface (UI). You can collect data through observation, interview questions and UX questionnaires: The most commonly used - also often criticized for its age and development process - UX instrument is the System Usability Scale (SUS), a 10 item questionnaire measuring perceived usability from 0 to 100, offering data to compare (benchmark) your results to other, similar applications (https://measuringu.com/interpret-sus-score/). A new tool on the market is rapid UX score (https://user-feedback.at/rapid-ux-score-projektpage), building on the latest science and measuring usability, perceived aesthetics and user experience for a broad range of applications. Users to Interview and to test your prototype: This is probably the hardest part in user research, acquiring and incentivizing authentic and “real” users and obtaining high-quality, helpful and guiding insights. We’ll have a deeper look into it now. How can I find users to interview or to test? In organizations that have direct access to their desired business or end users, users could be contacted directly, if according to the EU General Data Protection Regulation (GDPR) a documented consent for this inquiry exists. If not, or other reasons make this an invalid option, you have two other options:

You build a user panel by yourself.
You rely on external specialised partners and agencies to recruit users for your projects.

The advantage of building a panel by yourself is that - after a while of time and investments - you might have a reasonable number of users to invite for interviews and tests, and even repeat user tests with optimised prototype UI with the same users to see how they respond to the changes! The disadvantage is that you have to invest resources in time for administration, community building, management and continuous engagement of your panel. If you rely on partners that recruit the users for you, you don’t have the hassle to care about panel management, but you might not be able to invite the same users twice to a project and will have to factor in the costs for these services - too. No possibility, time or budget to find users? If both of these options are out of questions due to budget or time reasons, here is one guerrilla method to find users for interviews or tests: Place ads on Social Media through easy-to-access ad managers: Older users are more easily to reach on Facebook than on Instagram, a good place to find gamers is Reddit, a lot of expert communities have their digital “home” on Twitter. You get the idea. You don’t need to spend a fortune: Start with 50 Euro for a predefined target group, explain clearly in your ad what your goal is (offer an incentive!), link to a landing page and contact people via EMail or Telephone. Don’t forget to ask for consent in a clearly understandable manner. You might - or might not - be surprised that (with some luck) even for seemingly special user groups a critical mass to perform interviews and user tests might be reached. What is the optimal number of users to interview/test? This is a hot debated question in academia and industry and is not easy to answer, but let’s start with some basic things: If you want or must generalize findings from a sample of users to a large group of users, your users have to be well selected or your sample size needs to be larger. Earlier in the process, you might need fewer users to identify more obvious usability problems than less obvious ones, that you only uncover with higher fidelity-prototypes. If you want a quick number to start: 8-12 is a good sample size to start, both for user interviews and user tests. Be aware that this is a good sample size for one user group, that you need to pre-define and iterate according to the results that you get in interviewing activities. If you have more user groups, you need to multiply that number. In any case, don’t try to go down a sample size of 5. How do I analyse the results of a user interview or test? Observations from user tests can be classified into problems of minor or major severity for usability, user flow or UX. The same problems encountered by more than one participant should capture your attention. Data from user interviews can be analysed by “tagging” the raw data that is transcribed from the interview. You identify themes that appear in the raw data, and tag each appearance over all users with a tag describing the theme. You count the tags and have a summary, which you can use as a structure to explain the more subtle findings from your interviews. What to keep in mind In every development project of new technologies, products or services, most often UX decides over buying and usage behavior of users. UX can be addressed by incorporating user interviews and tests through different stages of your development project. User interviews are suited for understanding users or generating ideas, User Tests are suited for task-based encounters of real users with a prototype. Finding users is often the hardest part in user research, so plan for it accordingly. Try to start with 8 users!

Article updated on: 09 August 2023

Written by

Talent Garden, Digital Skills Academy

How to integrate user interviews and user tests in product development

Don't Waste Your Talent. Turn It Into a Career With a Course That Fits Your Needs!

TALENT GARDEN S.P.A.

How to integrate user interviews and user tests in product development

Don't Waste Your Talent. Turn It Into a Career With a Course That Fits Your Needs!

Keep reading

UX Design: A complete guide to plan an efficient User Experience

User Research: the key to successful project development

Product Metrics: What they are, and how to use them?

10 Ways to Utilize User Experience Analytics to Improve Website Engagement