Real Enough? Synthetic Users, Synthetic Data, and the Limits of Simulation

Welcome back to Think First: Perspectives on Research with AI. This installment takes on what may be the most divisive topic in UX research right now.

Some see synthetic users and AI-moderated research as the future of scale; others see them as a fundamental misunderstanding of why we talk to people in the first place. We brought together three researchers who sit at very different points on that spectrum – and let them disagree.

Mike Brzozowski, a Principal UX Researcher in Microsoft’s Core AI group, sees the risks of replacing real user empathy in the product development process completely while finding them useful in narrow cases. Zack Spolar, a Senior UX Researcher, is the builder-optimist. He works on a team creating large-scale synthetic data grounded in real users. And Connor Joyce, a UX Researcher and behavioral scientist who has advised a startup in this space, is the pragmatist whose answer to almost everything is a thoughtful “it depends.”

Where they land

We opened each conversation the same way: where do you land on synthetic users and data?

Mike has long been interested in scaling qualitative research – “there’s something really powerful about being able to learn a lot about a few people,” and an opportunity to stretch that to “a few more people.” He sees AI-moderated interviews occupying an intriguing middle space “between rich, one-on-one interviews that are human moderated and unmoderated surveys.” But his enthusiasm is measured: “There’s great potential for us to think about these tools as ways of enriching the connection that we form with users, as long as we don’t use it to replace the connection that we form with users instead.”

The tension in the question, Zack says, “is exactly why the team I sit within undertook the venture that we did.” His group looked at the synthetic data and synthetic users available inside and outside Microsoft, saw “a lack of depth in user fidelity,” and set out to fix it – bringing together engineering, research, design, and applied science to build synthetic data “that’s really deep, robust, and has this fidelity that feels like it truly understands the the user within the contexts that surround them.” Get that right, he argues, and you “unlock the future of scale” for research and prototyping with LLMs at the core.

Conor refuses both extremes. “‘It depends’ is a trite statement for a researcher,” he says, “but I mean it when I say it.” Anyone claiming a research team can be replaced by agents is “misguided, and they will ultimately make poor decisions based on faulty evidence.” But anyone refusing to learn the tools “is also going to become outdated very quickly,” because stakeholders’ expectations are shifting fast: “Code is created quicker, products are built faster… Research has to do the same.” These are “real tools that have real places” – sometimes the right call, sometimes not.

Two tools, often lumped together

Conor’s most useful move is to pull apart two things that often get lumped together. “AI-moderated interviewers and synthetic users – I really want to break them apart,” he says, “because they are very different tools,” and bundling them is part of why some people reject the whole idea. An AI-moderated interview still has a human in the loop: it’s an agent asking the questions, but a real person answering. That, he argues, is “much more advanced” and further along toward operational scale. It shines at upgrading what used to be unmoderated studies – a survey that can finally “ask a follow-up question, just like a person would” – and it never tires: “It listens well, it doesn’t get distracted; it can take a 7 a.m. call.” Synthetic users, where no human is in the loop at all, are “much trickier” and, in his view, still experimental.

Mike is drawn to that same middle space from the skeptic’s side. He’s interested in an AI-powered screener that could “reach out to more people to start and get more perspectives,” then decide whom to follow up with in depth. But he keeps circling the relationship at stake. What happens, he asks, when someone signs up for a study with a vendor they trust “and instead of talking to a human, you’re getting relegated to an AI interviewer”? The unspoken message risks being “we really care about your opinion, but not enough to actually spend our time talking to you.” His tentative answer is honesty about the frame: present it as “an AI-powered survey” or a screener for a larger study, not a substitute for the interview itself. Conor is less troubled by the same worry: if a participant might feel misled, “I say you will be interacting with an AI moderator,” and let them opt in – “it’s as simple as that.”

Zack draws a parallel distinction on the data side – between a synthetic user and synthetic data. A synthetic user is a persona-like depiction of an individual the model can voice. Synthetic data is bigger: “large-scale data that is in the shape of a real organization.” Personas alone, he argues, don’t give a model “the full context of the user” – so his team bundles in “their files, chats, emails, who they are outside of work,” the whole “organizational makeup,” to reflect how an LLM actually encounters a person. For large-scale evals, prototyping, and simulated environments, he says, that organizational depth is what matters.

The bias problem

No one dodged the bias question – but they answered it from opposite premises.

“Synthetic users have no lived experience to draw on,” Mike says – so for the hardest cases, bias is close to disqualifying. LLMs are trained on internet data “heavily biased towards Western, educated, industrialized populations,” with far less signal about underrepresented groups. Ask one to play a specific kind of person and it reaches for plausible-sounding stereotypes. His analogy is an improv show: the actor asks the audience for a profession, someone yells “proctologist,” and “the actor on stage has no idea what it’s actually like” – so they lean on what gets laughs. LLMs, tuned to deliver what we expect, do the same. Which is why, he notes, the hard-to-reach users a team most wants to understand are exactly the ones a synthetic stand-in represents worst.

Zack agrees the bias is real – and treats it as the design problem his team exists to solve. The trouble, he says, is that synthetic data tends to get “built around the mean user or the average organization,” when what research actually needs is the edge cases: “how your product will impact somebody who is unlike the average user.” So his pipeline deliberately inserts findings from real research to introduce those edge cases, aiming for data that “reflects real users in their real workflows and their real nuances” rather than an average composite. Done well, he argues, this doesn’t perpetuate bias but “actually reduces bias and allows us to represent users who are not the average.”

Connor, a behavioral scientist by training, names the mechanism: regression to the mean. A large model isn’t one person, it’s “all the things they’ve been trained on,” so it “falls towards stereotypes rather than individual nuances” and acts “hyper-rational,” which real people are not. Tell it you’re a 33-year-old tech worker in Seattle and “it’s not Conor – it doesn’t have the person that grew up in Chicago, who lived in Philadelphia, who now is in Seattle, who has a dog.” Fine-tune on a hundred characteristics and “every other characteristic is going to just regress towards that mean.” The averages can help; the trouble is “we don’t know exactly when it is helpful versus when knowing the extremes is actually what’s showing us what needs to be done.”

Where synthetic actually helps

Ask where the tools do earn their keep, and all three have concrete answers.

Mike’s favorite use is as “a sparring partner” – taking a half-baked interview guide, set of questions, or hypothesis and poking holes in it “to strengthen that idea,” which is “not the same as actually going out and doing the work” but useful for early scrutiny. He frames synthetic users as a possible “early canary” or “hardening step” in a staged model of product risk: show an idea to a coworker, then a dogfood group, then real users. Synthetic feedback can catch holes “before you actually go talk to real people.” But he draws a bright line: “I would be really concerned if you went to market with a product that had never seen real users.”

“To experiment, you need subjects,” Zack says – and rather than a middle ground inside classical research, he sees the arena itself moving. Research “will become much more involved in the earlier parts of the product lifecycle,” bringing users – real and synthetic – into experimentation and prototyping, with synthetic data providing subjects for “rapid, large-scale experimentation” that can then shape the product downstream. When people call synthetic data antithetical to research, he hears a misunderstanding of where it fits. The point is to inject a deep understanding of the user and their organization context into a product development process that requires speed and scale due to the non-deterministic nature of LLMs.

Connor is willing to stake his credibility on exactly one synthetic-user use case today: winnowing. “Maybe I have ten different design options; I use synthetic users to knock out five, so I only have to A/B test five rather than ten.” You don’t need to capture individual nuance to spot the worst options – “the collective is really the one that’s doing a good job saying those suck.” He’ll also pre-test instruments with it, the same way he already hardens a research report by having “five agents” – a head-PM persona, a head designer, a head critic – critique it, then a further agent find the patterns across their notes. The trap to avoid is “just listening to what the synthetic user said” and shipping the change unexamined.

What only real people give you

On the limit, they finally agree – each naming it differently. For Mike, it’s entropy. “Talking to real people introduces entropy into your design process,” surfacing the unexpected; LLMs, “designed to minimize surprise by producing the most likely response,” are good at validating a plausible hypothesis but poor at “illuminating weaknesses in your model.” He puts a number on it: a vendor claiming synthetic users replicate human input with “85% accuracy” is missing the point, because “it’s that last 15% that really makes a huge difference” – the moment an individual says something you weren’t expecting and it shifts how you see the problem. Conor frames it as averages versus extremes, and adds a sharper claim: critical and “net-new” thinking are “one of the last remaining skills” AI can’t yet do, so a synthetic user can surface the recurring patterns in your past work but not “the biggest challenge those users have faced.” Even Zack agrees on the floor: a synthetic user “is never going to replicate exactly the nuance of a human,” because people evolve over time in ways “imperceptible even to humans themselves.” His team’s goal is to provide the LLM with a deep understanding of the user and their work, in a way that can be leveraged in the AI product building process.

Coming up next

Three researchers, three conclusions – enrich but don’t replace, build it deep enough to trust, separate the tools and demand real evidence – and one shared opinion: the unpredictable, entropy-generating human being that no simulation has yet learned to make surprising.

In the next episode, we continue with another corner of the practice being remade around AI.