Top mistakes research ops people make administering incentives
Ideally, administering incentives would be an easy, frictionless part of a research ops person’s day-to-day. But we don’t live in an ideal world.
A lot of times, incentive payments don’t go as planned. And things happen. Inconvenient, sometimes calamitous things that – worst case – damage relations with study participants.
We asked over 14k research operations experts part of the ReOps Slack community about the issues they’ve run into when administering incentives. We compiled their responses here so you don’t have to make the same mistakes.
Four big things stood out as being consistent challenges:
Choosing the right incentive amount
Avoiding user input errors
Getting international payments right
Slogging successfully through manual processes
Incentivizing for success
Here’s a couple best practices we gleaned from our survey results:
Common mistake #1: choosing the right incentive amount
A bunch of our respondents stumbled initially when trying to figure out how much to pay their research participants.
“I think the hardest challenge is figuring out the right amount for each study,” wrote one respondent. “Getting it wrong is probably the biggest mistake we have made.”
“We didn’t have a good way to decide the amount of the incentives, so sometimes we choose the wrong amount. For example, for user research with a high-income target, we chose our standard amount that wasn't enough to interest the target.”
Identifying the appropriate amount to pay research participants is tricky. No one wants to underpay.
“We tried to recruit a niche specialist group but didn't offer a high enough incentive for the group. Normally we offer ~ £20 - £30 per hour for the general public, but this was a highly paid specialism, so it wasn't worth their time for such a low incentive.”
But you also don’t want to offer a garish amount of money that way overshadows the ask.
It leads to an overblown research incentive budget, and it may exert undue influence on research participants.
“Not offering the right incentive; assuming people want a lot more than they actually do.”
Some research ops folks also gave participants an incentive, and then found out the participant wasn’t the right person to talk to.
“Promising an incentive and finding out the interviewee was not a good fit/didn't actually qualify for the study.”
Finally, some research ops people wrongly assumed that participants would be interested in donating their incentive to charity.
Common mistake #2: user input errors
This category comprises our highest number of responses.
“I've accidentally sent multiple incentives to the same person, sent incentives to the wrong participants, sent incentives to the wrong email, and entered the wrong name of the participant.”
Vouchers caused a lot of issues. They’re processed differently in certain regions, which can complicate a participant's ability to cash in.
“We've entered the wrong voucher code in the excel sheet. In some cases, we also have processed the same Amazon Gift Voucher to two users as we purchase the vouchers in bulk(that are sent by email).”
Then, there are the times when every step of the incentive payment process goes wrong from start to finish.
“We sent our the vouchers too late (e.g. 14 days instead of 1-3 days); we didn't document the purpose of the voucher properly for our accounting; we didn't get the email of the interview participant (because the survey tool we tried out the first time didn't support it and we didn't know) so we couldn't send out the voucher; we sent an email campaign but then couldn't track back the survey responses so we couldn't determine who participated and who not so we couldn't send the voucher. We ended up paying everyone a voucher.”
We also got a lot of “forgot to write the incentive down,” “noted the wrong email,” and “made the instructions unclear on the timeframe to receive the incentive.”
User input errors are especially common when multiple teams are doing independent research ops projects, or when several layers of stakeholders are involved in a project.
Common mistake #3: international payment problems
Dealing with an incentive program that spans several countries also causes a lot of problems for people.
“I accidentally incentivized internationally, which was a no-no.”
Several respondents said they paid out incentives more than once with the wrong currency. Others sent unredeemable vouchers.
“I didn't make it clear enough in the past that participants could only receive UK/US Amazon gift cards.”
“The process we use is pretty automated, but we've sent a specific country's gift cards to participants outside of that country before.”
In scenarios where compensation had to be tailored to the user’s location, a couple of respondents noted how much those changes complicated their workflows.
Common mistake #4: annoying manual processes
Doing things by hand that could otherwise be automated by an incentives payment platform also lead to a ton of issues.
“We had a multi-step process in which I (the researcher) would have to create a spreadsheet with the participant's email, date of the study, type of study (survey, interview, etc), and what they were owed. I then sent this doc to someone in our finance team and asked them to keep an eye on it as the sessions were completed. There were several mistakes that would happen in this process, including people not getting paid, people getting paid the wrong amount, and people getting paid weeks after the study was completed. It was really frustrating and unprofessional.”
The longer the manual process, the more likely research ops teams were to make mistakes.
“Because the manual process is so long, there are so many spots where mistakes can happen. In Airbase, I could request the wrong credit card amount. When I input emails from my Google form into Amazon, I could put the wrong email. Participants themselves can mistype their email in the Google form. The email is the thing I'm most worried about, to be honest.”
Some participants were double-paid because of these mistakes. Others never received their incentive.
And a lot of our research ops respondents noted the arduous work of downloading and uploading lists and maintaining their own spreadsheets.
Additional incentive mistakes
There were a few notable responses that didn’t fall into our four primary categories. In one case, the delivery of a portion of the incentive was done during user testing.
“We were asking the participant to place an order during the session using half the amount we promised as an incentive to test the food order tracking experience, as real as it can get. Some expected the full amount, when we thought we clearly communicated how the session will go.”
One respondent hit a snag when they overestimated their cash transfer system software.
“Ordering too many physical Visa gift cards. Not doing usability testing first on my company's cash transfer system so the payments revealed accessibility issues.”
Getting incentives right
We also received a solid number of success stories among our survey responses.
“We have developed a framework of different incentive levels for different specialisms or user groups to help understand incentive levels and maintain consistency.”
Some teams developed incentive ranges as they came to understand their research participants better.
“We looked closely at our incentive amounts and created a range depending on the type of user we are interviewing or testing with. For example, the range is higher for B2B customers than for B2C.”
Tracking was a must-do for research ops folks who needed and benefited from the accountability.
“Switched to a system that has tracking and confirmation of incentive receipt. Also, centralized distribution of incentives to one person through one system.”
Overall, developing a framework to determine what to pay different research participants and implementing an incentive payment platform that issues and tracks incentives helped research ops professionals eliminate mistakes across their workflows.
