100

Studies in the Time of 5

Impact

Reduced churn: “Enabled the team to make the right move” and ship a key Claude Code feature that cut drop off
Scaled output: Now, researchers do “100 studies” instead of “five or six” in the same amount of time
Streamlined workflow: We “condensed [our research process] into one step instead of two”

Company Background

Anthropic is an AI safety company building reliable, interpretable, and steerable AI systems. Its products include Claude, a trusted virtual collaborator used worldwide, and Claude Code, an agentic coding tool for developers.

Jane Justice Leibrock is Anthropic’s Head of User Experience Research. Her team’s research informs product decisions across the company, from which bugs to fix to where the next strategic opportunity lies.

The Challenge

“As a researcher, what you’re looking for is to be trusted,” Leibrock said. “Lack of having both a qualitative and a quantitative component means there’s some confidence in your results, but it can be a little harder to make sure they can be acted upon.”

Before Listen, collecting credible qualitative and quantitative evidence meant running separate, sequential studies. First, in-depth interviews to identify what topics matter, then a survey to quantify them once the interview findings were in. The sequence was a bottleneck.

At Anthropic’s pace, that forced a tradeoff: “a compressed timeline where maybe you’re going to miss your deadline,” she said, “or just do one study where you really would’ve rather done a multi-step thing.”

With Listen, both methods run in one study and on the same participants: “your research process is condensed into one step instead of two.”

Listen has spread across Anthropic: first Product Marketing, then UXR, Marketing Insights, and beyond. “I’m fairly sure we’re the biggest users now,” Leibrock said.

The Study: Understanding Churn on Claude Code

During Claude Code’s research preview, one of the signals the team wanted to understand better was churn. It became Leibrock’s first Listen study.

Users said they valued that Claude Code was terminal-first. But the same users asked for IDE integration.

“From the beginning, we’re getting strange mixed messages,” Leibrock said. “It would be easy for the team to say, ‘No, we have strong conviction. We’re going to keep following it.’”

Instead, Listen allowed the team to dig deeper. “If we’d only had structured questions, we would’ve been stuck,” she said. “But open-ended questions to the same participants meant that we understood where that was coming from.”

The result

“The crux of it was context switching: people not wanting to go back and forth between their code editor and the terminal,” Leibrock said. “That enabled the team to make the right move to extension for VS Code.”

The team kept running churn studies even after they shipped the extension. “We’re able to, in the middle of this continuous study, introduce new sections where we start to explore these new topics, understand them this round and then ask better questions next round,” she said.

"It's kind of like a self-healing study. It finds the new things it needs to understand, and then helps you measure those better going forward."

What changed with Listen

Increased insight-to-action: Running qual and quant in one motion produces more robust findings teams act on. “It’s taking them from a signal to one that we think we can rely on,” Leibrock said.

Scaling researchers’ time. “Think about it as 100 studies for the price, at least in terms of time, of five or six. Our researchers’ time is one of the scarcest commodities. So if we can take the same person’s research skills and insights and quickly scale that up in terms of the scope, that’s very valuable.”

Breadth from a single study. “We can cover a lot more segments than we otherwise would have,” Leibrock said. “Within the same study we can compare them against each other.”

Research across Anthropic. User researchers are embedded with product teams and work with team leads, designers, engineers, and PMs. Those teams now run Listen studies of their own.

“I’m very regularly pointing folks that I don’t have time to own a study for to Listen Labs,” Leibrock said. “It’s something that I have confidence they’ll be able to figure out how to use, have a good experience with, get good quality insights from.”

The UXR team also built their own plugin so anyone at Anthropic can create Listen studies from Claude Code. “You can also get the results from it directly,” she added, “and it just allows us to blend all that with all the other workflows that we have.”

Why it matters

“We’re moving really fast as an organization,” Leibrock said, “and getting input from users is what makes sure that we’re just moving in the right direction.”

For a company whose mission is about people, that input isn’t optional. “Humans are the whole point,” she said. “That’s why we’re all here.”