4 mistakes researchers make when administering incentives
So, your organization launched a study to gauge user experience (UX) and, to improve participation, you offered an incentive in the form of financial compensation.
Now it's time to administer incentives. And, unfortunately, there are a bunch of pitfalls that you can experience at this stage, from mis-delivering gift cards and vouchers, to sending money in the wrong currency, to typing in the wrong email or code and having to pay for those losses out of pocket.
Even when all goes well, it can be a time-consuming process to make sure participants receive their rewards properly and are able to redeem them, all the while carefully tracking who's received what.
Offering the right kind and amount of incentives—in a timely fashion, to the right person, in the way you promised—is key to prevent the process from becoming a serious (and expensive) problem. Here are some of the top mistakes to avoid, with ideas about how to do it better.
Table of contents
Don't just reward participants with a free subscription...or nothing at all
Don't forget to consider the tax, legal, and ethical implications for you and for the participants
Don't just reward participants with a free subscription…or nothing at all
You've probably already worked out this aspect of incentives, but it bears repeating: if you don't compensate study participants for their time—or do it in a way that feels fair to them—you won't get as much feedback. You also won't get the high quality information you're looking for.
"When somebody gives you something for free, it usually means they’re complaining," explains Holly Cole, ReOps CEO. The point of compensation is to get access to that higher-quality feedback.
When UX researchers do promise participants that they'll be compensated for their time, then only offer free access to their own company's product, it can feel like a bait-and-switch (particularly since participants might be expecting a gift card or voucher unless explicitly told otherwise).
"I see startups do that," explains Brad Orego, Head of Research at Auth0. "You’re making them do work and then rewarding them with your product? I don’t know about that.”
If you must resort to that incentive structure, communicating with participants well ahead of time is key.
It's important to avoid looking "cheap" when offering incentives, too.
Orego remembers a debate he and UX researchers had over a structure that only offered compensation to the first 100 participants who filled out a survey.
"It’s not good for our brand to offer an incentive where it looks like we don’t have enough money to do our research. Either you pay everyone or you pay no one. I don’t think there’s anything in between,” he says.
Don't offer inappropriate or thoughtless gifts
Similarly, an incentive structure needs to be built in a way that mitigates payment problems for both the UX researchers and the participants.
This essentially necessitates steering clear of payment structures that just don't work: for example, "depending on your company and depending on the companies of your participants, they may not be able to take an actual cash or gift card reward," says Orego.
According to Lowell Reade, senior research program manager at Duolingo, one of the major incentive no-nos he's learned is to avoid sending out physical gift cards.
Aside from the annoyance of buying and managing them (especially if you accidentally over-ordered and now have no idea what to do with them), "The last thing you want to do is give the researcher another thing to worry about or another step they have to take during the interview," he says.
Think about simplicity and efficacy. Does your system require that UX researchers follow up with each participant after the fact to ask how they'd like to receive their voucher? Or what currency they need to receive it in? That's complex and time consuming.
Do they have to hand participants physical gift cards at the end of the interview (and what happens if they forget)? Try to problem-solve ahead of time.
Don't treat all of your study participants the same
Your study participants may live in different countries, have different backgrounds, and bring varying skill levels to the table.
Depending on the study, you may not want to pay the same amount for different levels of expertise, says Cole.
"It might be totally appropriate to pay someone in the United States $100 for an hour of intense conversation about your college plans or university plans. But if you’re talking to someone in India about their university plans for an hour, it’s probably not appropriate to pay them $100," she says.
"You need to know the conversion and the appropriate amount to pay them there, without having to ask every single person and then do a custom compensation for each person," she adds. At the same time, it will be important to make sure an incentive system is fair and adheres to the structure you've set up—which, again, can be time-consuming.
Don't forget to consider the tax, legal, and ethical implications for you and for the participants
Even if it means putting in the legwork well in advance of the study, it's worth it to explain the tax ramifications to participants before they sign on.
For example, even though the situation is rare, "In the U.S., if you receive $600 in a year, it becomes taxable income. It varies by country and can be very complex. We include simple language in our recruitment emails letting people know there may be tax implications, and that we won’t manage that for them,” says Orego.
Another important aspect to consider is the legal and/or ethical requirements based on the participants' professional and personal backgrounds.
There are, for example, problems with potential bias for government workers, particularly those in leadership roles, says Cole.
Journalists and other media professionals may have to steer clear of conflicts of interest. If you're giving out Amazon gift cards, make sure none of your participants are Amazon employees.
You might not be able to incentivize internationally, depending on the work your organization does.
Even though it can be a hassle to work with your internal compliance, finance, and legal team(s), being proactive will lead to the best possible outcome.
"They want to work with you, but they have no idea what you do. They don't have the context," says Orego. "We give them a one page research agreement form…and it has very plain English language on it. Build those relationships with those partners that you're going to have to deal with. The last thing you want to have happen is that you're ready to get some research thing going, and then somebody from legal or finance or whatever finds out about [it] at the last minute. [Then] you have this big fire drill."
How to create an incentive system that actually works
If you've gone through this process before, you'll know that you may need to budget more than you think you need to adequately provide incentives (and handle potential mistakes that happen along the way, along with additional resources that may be needed).
In an anonymous survey with the ReOps (research operations) Slack community, consisting of 12,000 members around the world, respondants listed out a number of best practices that helped them avoid incentive horror stories:
Make a highly detailed plan well in advance
Identify the maximum number of people needed and cap the study once that number is reached
Carefully, properly vet study participants well in advance
Optimize workflows to ensure timely payment (and reduce hassle)
Triple-check contact details like email addresses to avoid misdelivery
Be extremely careful when incentives are in the form of physical gift cards by sending them directly to participants
Have cross-accountability with Finance and/or Legal team members as incentives are being paid
Be receptive to feedback about what worked and what didn't, both internally and externally
Change the incentivization system and/or structure based on results, as needed
The process can be a tricky balance.
But when it works, and UX researchers have what they need to make improvements to their product, the hassle is well worth it.
If not, it's an expensive mistake.
"The whole point of good user research is to know what to say no to. If you don’t have good solid research that’s balanced with not just quantitative research—A/B studies, all these other things—but with qualitative research as well, you end up going down a rapidly narrowing path that takes you in the wrong direction,” says Cole.
