How generative AI ‘persuasion bombs’ users — and how to fight back

The standard advice for managing artificial intelligence risks such as hallucinations and unreliable outputs is to keep a human in the loop.

But a new study that tracked AI use among Boston Consulting Group employees suggests that rather than solving problems related to AI, putting a human in the loop introduces a new issue. 

When the consultants at BCG tried to validate a large language model’s suggestions for a particular business case, the LLM reiterated its position — and the harder people challenged it, the harder the LLM defended its original answer. Instead of considering pushback and appearing to work toward the best solution, the LLM dug in its heels and acted like a salesperson, pushing its suggestions even when they were wrong.

“We saw the human’s act of real-time validation with generative AI triggering this persuasive counter-response by the LLM,” said MIT Sloan School of Management professor Kate Kellogg, one of the researchers. “The very tool that was supposed to be the solution to one set of problems actually activated a different problem.”

The study, which Kellogg conducted with colleagues at Harvard University and the University of Warwick, highlights a new barrier to human-AI collaboration and an uncomfortable question for any organization betting on human oversight to keep AI honest: What if the AI is better at persuasion than humans are at resistance?

AI fights back with “persuasion bombing”

The study tracked 72 BCG employees as they used GPT-4 to complete a problem-solving task: analyzing a fictitious company’s clothing brands and recommending one for investment. Working at the level of each conversation, the researchers logged every exchange (4,339 prompts in all). Many exchanges involved humans probing the LLM’s suggestions by asking for further clarification or justification. The researchers also categorized the persuasive tactics the LLM used in its responses.

“We were able to observe these very micro tactics used by dozens of professionals as they tried to validate the LLM’s outputs,” Kellogg said. “And what we found is that the LLM responded to questions or pushback in three escalating ways, which is what we call persuasion bombing.”

• First, the LLM ratcheted up the intensity of its recommendation, flooding the conversation with statistics and information that functioned to support its initial conclusion. 
• If the consultants pushed further, the LLM switched to a more overtly emotional register featuring apologies, flattering language, and renewed assurances of effort and transparency. But it still did not waver from its initial conclusion.
• Finally, if the consultants continued to question the LLM’s results, its responses drew on a widening range of rhetorical approaches: advancing claims about credibility, reinforcing logical arguments, and deepening rapport with the user. The interaction gradually shifted from a joint decision-making process into something closer to a sales pitch.

3 persuasive tactics used by AI 

The researchers found that the LLM moved well beyond sycophancy, shifting its emphasis and persuasive strategies in response to the particulars of the conversation. It essentially argued its case along three dimensions that align with Aristotle’s classical framework for rhetoric:

• Ethos, or appeals to credibility. When challenged, the LLM apologized for errors while reframing them as minor, showed its work through conspicuous calculations and structured reasoning, or deflected responsibility (“I apologize for the confusion — it seems you did not provide any financial data”). These responses presented the LLM as a reliable analyst.
• Logos, or appeals to logic. The LLM presented data-driven comparisons, cited figures that projected precision, and repeatedly reframed its analysis in ways that supported its underlying reasoning. These moves risked raising contested or weakly supported data points consistent with its original conclusion.
• Pathos, or appeals to emotion. The LLM mirrored professionals’ language, affirmed their inputs (“You are correct in your assessment”), and used inclusive “we” language that framed the exchange as a partnership while projecting confidence about outcomes that encouraged emotional buy-in.

Standard methods of validation, like asking questions or calling out inconsistencies, assume that both sides are prepared to answer with honesty. The LLM, which was trained to be engaging, responded instead with persuasion, the researchers found. 

Generative AI Business Sprint

Attend Online

REGISTER NOW

Building systemic defenses

Given generative AI’s capabilities, LLMs are now being used in situations where the outcome is critical or financially significant. For example, health care organizations use LLMs to summarize radiology reports for patients, and consulting firms use them to advise clients. Steven Randazzo, a visiting researcher at Harvard University and a coauthor of the paper, was recently in conversation with pharmaceutical executives about their use of LLMs.

“When they were talking about adoption within their organizations, they were saying, ‘Well, we have humans in the loop, so what’s the big deal?’ The assumption is that if people are in the workflow, then the risk is neutralized,” Randazzo said. “But now we see that this safety check is prone to persuasion bombing, where an LLM campaigns for its position with escalating rhetoric, and the human is persuaded — or just simply beaten down enough — to accept the output.”

Companies should work at two levels to prevent this influence:

• At the individual level, train employees to recognize an LLM’s persuasive tactics. Encourage fact-checking outside the chat interface, and use prompt engineering to request neutral, academic responses rather than confident, narrative ones.
• At the organizational level, deploy “judge agents,” LLM-based systems tasked specifically with critiquing other AI outputs and raising counterpoints. Running these evaluators parallel to production systems, rather than relying solely on episodic human interrogation, enables scalable oversight.

The researchers’ findings suggest that oversight of generative AI doesn’t just need to be reinforced but redesigned. 

“We know now that the logic of systems like GPT-4 is designed for adoption and stickiness, anchoring on the user’s first interaction, affirming, and escalating persuasion when challenged, which is fundamentally at odds with what we’d want in a system where the human is supposed to exercise independent critical judgment,” said University of Warwick professor Hila Lifshitz, a co-author of the paper. “The entire human-in-the-loop architecture is compromised.”

Read the study: “GenAI as a Power Persuader” 

Kate Kellogg is the David J. McGrath Jr. Professor of Management and Innovation at the MIT Sloan School of Management. Her research focuses on helping organizations and knowledge workers develop and implement AI systems to improve decision-making, collaboration, and learning. 

Steven Randazzo is a visiting research fellow at the Harvard University Laboratory for Innovation Science and a PhD candidate at the University of Warwick. He studies generative AI and its impact on knowledge work. 

Hila Lifshitz is a professor of management at Warwick Business School and a faculty affiliate at the Harvard University Laboratory for Innovation Science. Her research focuses on understanding scientific and technological innovation and knowledge creation processes in the digital age. 

The paper’s other authors are Akshita Joshi, a PhD candidate at Harvard Business School; Fabrizio Dell’Acqua, a postdoctoral researcher at Harvard Business School; and Karim Lakhani, a professor at Harvard Business School.