Generative AI as a tool for market research

What could possibly go wrong with a minor product tweak? Anthropic recently learned the hard way after a pricing experiment with its Claude Code tool went viral on social media. The resulting developer backlash forced the company to backtrack on implementing changes to its subscription model.

Like many companies do, Anthropic was simply testing the waters on product plans without shouldering the time and cost of a formal market research program. What it might have considered instead is using artificial intelligence to model the response and engage in a bit of preliminary intelligence gathering. 

MIT Sloan School of Management associate professor John Horton makes the case that large language models can effectively deliver informal market insights that can later be refined through further investigation and more formal research practices.

“Trying to simulate the effects of decisions before they are made is what market research has always done,” said Horton, presenting in MIT Sloan’s speaker series about how AI is changing management practices. “The idea of doing something that normally would have been done with people and trying it with AI agents is doable now.”

Simulating human response

Horton said that traditional market research is in flux: Response rates have plummeted, it takes too long to get answers, and quality is on the decline, among other challenges. Improving those conditions requires several actions, including acquiring richer and higher-quality data, making better use of that data through more sophisticated analysis, and generating models that can simulate preferences and help organizations arrive at better answers. Horton said that generative AI and AI agents can positively impact all three.

For example, AI can automate the process of gathering information, saving time and money while also serving as a more neutral actor for eliciting negative feedback than a human-to-human interchange could. AI is also versed at taking unstructured data, such as video content, and making it useful for an increasingly wide array of data science tasks. “If this lets us do some qualitative research at scale or gain access to sources of data that have previously been locked, that’s pretty exciting,” Horton said.

Harnessing AI modeling capabilities to simulate human behaviors and responses represents a key breakthrough. Whether it’s building a model of a particular persona or enabling role-playing, Horton said, AI has high utility for market research, despite inherent flaws. But users must have confidence that models are close approximations to real-world scenarios, he noted. Horton’s own research has shown that “if you have a good proxy that you could explore akin to bench science, this would be a very powerful tool. You can see anomalies or findings that might guide you towards something worth digging into.”

AI Executive Academy

In person at MIT Sloan

Register Now

Horton made several recommendations to help organizations maximize the value of AI-enabled market intelligence without getting led astray. Among them:

Don’t take AI as the last word. AI simulations are not meant to compete with gold-standard market research practices; rather, they’re a means of providing directionally correct feedback or to highlight the second-order implications of decisions. Most organizations are not performing this kind of informal and early intelligence gathering, and AI can help fill that gap.

Iterate. When experimenting with AI research practices, organizations must use different types of models and experiment by changing rank order and prompt wording. These nuances change results and can enrich insights and exploration.

Ensure that results easily replicable and testable. If people expect to make important decisions based on the insights an LLM generates, they’ll want to be able to see how the calculations were made. To find these simulations credible, they’ll need the ability to audit answers and access information such as which models and prompts were used. “Even if you’re not going to dig in, you’d have the capability of doing it,” Horton said.

He anticipates that as the category and practices mature, more companies will use AI as a quick way to garner insights for minimal cost. He described it as a dress rehearsal for expensive research, enabling users to preemptively find problems and narrow the field of options. “Companies might build and test three or four variants of something, but they’re not going to test 700,000,” he said. “Being able to pre-process and surface problems is really powerful.”

Read Horton’s research: “Automated Social Science: Language Models as Scientist and Subjects”

John Horton is the Chrysler Associate Professor of Management and an associate professor of information technologies at the MIT Sloan School of Management. He is also an adviser to Anthropic and the cofounder of Expected Parrot, an open-source platform for designing, running, and analyzing research with digital personas and people. His research focuses on the intersection of labor economics, market design, and information systems. He is particularly interested in improving the efficiency and equity of matching markets.