How Duolingo uses research incentives to maintain its No. 1 spot on app charts
Research powers Duolingo, the world’s most popular learning app.
With more than 500 million users, the Pittsburgh-based company has the world's largest collection of language-learning data at its fingertips and a pool of potential UX research participants as large as the Atlantic. Duolingo capitalizes on that potential by meticulously testing every aspect of the app and uncovering insights about people’s love for language.
Increasingly that research takes place across the globe, which presents challenges for Lowell Reade, the company’s lone UX research program manager. He’s charged with ensuring the company’s research follows the “three Es.”
“I make sure all of our research is efficient, effective, and ethical,” Reade said.
Duolingo is, indirectly, a Tremendous customer. They distribute incentives through Ethnio, a research participation management platform, that integrates with Tremendous.
Reade sat down with us to discuss:
Duolingo’s approach to UX research;
How Duolingo incentivizes research participants; and
Challenges of international research.
Ian: You're a research program manager. What does that entail?
Lowell: I joined four years ago as the company’s first UX researcher and transitioned into research ops about a year and a half ago as a senior UX research program manager. I’m responsible for putting processes in place and providing resources that make it easy for anyone in our company to conduct research.
A big part of what I do is empowering non-researchers to do their own research. I make sure not only researchers but also product managers, designers, and other stakeholders have access to tools and templates that make research easy for them to do — and that it’s done efficiently and ethically.
Ian: What kind of research does your team do?
Lowell: Today we have a team of 10 that includes the head of the research team, Hannah Pileggi, eight UX researchers, and me as the one program manager on the team. The majority of the research our team does is product focused. That could be testing features on the current product as well as new concepts we're exploring.
Each researcher on our team is embedded on a specific team around different product areas. For example, our learning area focuses on all of the learning features in the app; the monetization area focuses on features for our paid subscribers; the growth area focuses on retention and engagement aspects of the app that bring new learners in and keep them learning.
We also do some work to understand our audience more. We want to understand who our learners are, why they’re learning, and how we can make their experience better and reach them better.
Ian: What are the advantages of an embedded team vs. a centralized team?
Lowell: Transitioning to an embedded model really transformed our impact because we’re able to build relationships with product managers, designers, and engineers on product teams. When you start a new project, you already know the context. That helps our researchers move faster. It helps us be even stronger as researchers. So it’s cleared the path for even more hiring.
When I was at Facebook, we had a similar structure.
Ian: What are you focused on right now?
I'm excited for the continued growth of our team and for the team to do more and more international research. We just opened an office in Berlin that will be focused on more international learner growth.
Ian: What challenges do you confront with international research and how are you addressing them?
Lowell: More than half of our learners are learning English. While many of them are located in the United States, we also have learners spread all over the world. Even if we're conducting a study fully in English with English speakers, they might be non-native English speakers who speak English well enough to participate in an interview, or they might be expats living in a different country.
We’re limited in our ability to conduct research in other languages, though we’re adding new capabilities. For example, we use interpreters for many of our interviews so that the entire product team can participate. We also have an amazing localization team that can localize things like emails and any kinds of survey questions or other sorts of content that we need to send to other learners.
It’s also a challenge to recruit participants in some areas because email isn’t as widely used as it is in the U.S. We haven’t really cracked that one.
Ian: Sending incentives to different countries can present challenges. How have you approached that?
Lowell: International compensation is really important to us and something that we struggled with in the past.
What’s very cool about Tremendous is that it auto-detects a person’s location, or it gives them the option to specify where they are, and then automatically presents the right options for that region, in the local currency.
Ian: What else are you focused on with incentive payments?
Lowell: One thing that we haven't really dealt with is changing incentive values based on where people live. When we do localized studies in a specific country, we do often get guidance from partners in local markets to determine those incentives. But in general, we have set standards that we use for everybody. That’s something we might need to revisit.
It's a big question for us, especially as we expand our international research.
Ian: You used to pay participants with Amazon gift cards that you tracked manually. How has the switch to a payments platform helped your team?
Lowell: Part of what I focus on in my role is how can I make life as easy as possible for the researcher. There are so many moving pieces to conducting research. There’s everything from making sure your discussion guide is ready, making sure that you have the right prototypes that are working correctly, and so on.
The last thing you want to do is give the researcher another thing to worry about or another step they have to take during the interview. You don’t want to make them remember, “Oh, don't forget to hand the participant a gift card,” or, if it's remote, “Don't forget to ask them how they want to redeem it or what currency they need.”
Having such a flexible platform gives the researcher one less thing to worry about. All they have to do is submit the person's email and the person gets to pick what kind of gift card they want or what kind of incentive they want to receive in their local currency.
Ian: Duolingo went public last year. How did that impact your team?
Lowell: Becoming a public company changed a lot of things in terms of what our team needed to track and report to finance.
We created a whole new process for providing the paper trail of who's requesting incentives, who they're being sent to, how we're determining how much to send, and things like that. I provide monthly reports to our finance team on what each team spends on incentives.
The Tremendous platform makes it super easy to track and report who’s sending money and how much they’re sending. It gives me everything I need to pass onto finance. It also records information about who’s receiving the reward, so we're able to make sure rewards are going to the right people.
All of this was a huge challenge when we were sending physical gift cards and digital Amazon cards.
Ian: Duolingo uses charitable donations as a research incentive. Is that popular with your participants?
Lowell: It’s very popular.
The Duolingo user base are some of the nicest and most thoughtful people that I've met in my role as a researcher. They're genuinely really great people to get to spend time with. A lot of participants say, “You don't even have to pay me. I don't need an incentive.” Of course, we still send them anyway.
It’s super cool for us to be able to provide flexibility to a participant to choose their own incentive. For some of them, they're participating in research because they want to make Duolingo better, not for the financial incentive. And so they pass that on to others that might benefit more from it.
Ian: What product updates can we expect that are informed by your team’s research?
Lowell: I'm excited for some upcoming changes to the learning app. They’re going to help people learn more effectively and also in a more fun way.
Currently the way the app is set up, you have lots and lots of choice in terms of what lessons to do next or where to progress down what we call the “tree,” which is sort of the main Duolingo interface. We've found that amount of choice to be overwhelming.
The redesign, which was informed by our learning scientists and our curriculum designers, makes this a linear path.
