Where to look for generative AI risks

Generative artificial intelligence is opening doors to unprecedented productivity and growth opportunities. Yet it’s also amplifying business risk, especially as organizations advance from experimentation — including exploring AI agents — to full-scale deployment. 

New research from the MIT Center for Information Systems Research can help business leaders take stock of the generative AI risk space — the set of components that introduce risk, including training data and user prompts. 

Drawing on 62 interviews with data and technology executives, MIT CISR researchers Nick van der Meulen, Hippolyte Lefebvre, and Barbara Wixom identified which generative AI components lead to risk and what leaders should know to respond effectively. 

Two types of generative AI risk 

The researchers identified two different types of generative AI risk, each of which requires different management approaches. 

Embedded risks are inherent to the technology because they are built into the foundation models an organization adopts. These risks are shaped by training-data quality, model behavior, and performance drift introduced by vendor updates. They are not fully within an organization’s direct control.

Enacted risks come from the choices organizations make about how they deploy, configure, and use generative AI. They encompass everything from designing system prompts to implementing safeguards against attacks and granting permissions to agents. 

Where to look for generative AI risks 

To illustrate potential risks, the researchers used the example of a hiring manager using generative AI tools to draft a job description for a new role. This task involves multiple AI components, each associated with a distinct set of issues that could cause problems or derail intended outcomes, such as the following.

Training data: Foundation models are trained on massive datasets culled from the internet. While training data includes millions of job descriptions and resumes, it could also incorporate outdated HR practices, biased language, or inaccurate information. That means there’s a chance that the output won’t map to the hiring manager’s industry or region or that it reflects outdated norms rather than current industry best practices.

Foundation models: The large language models that form the backbone of generative AI don’t generate the same responses, even when given the exact same inputs. These models are also capable of generating hallucinations, or plausible-sounding content that is factually incorrect. Lack of transparency into model behavior can cause additional problems for a hiring manager if they can’t discern why a particular output was generated, let alone easily diagnose and correct errors.

User prompts: An LLM’s response is only as good as the prompt. Without clear directions — in this case, examples of what a good job description looks like — the output likely won’t meet expectations. The hiring manager could also introduce risks if they unknowingly include confidential data, proprietary strategies, or personally identifiable information in their prompts. 

System prompts: Enterprise-grade generative AI tools are architected with a hidden system prompt that enforces organizational context and safety guardrails and sets tone. A poorly engineered system prompt can create a single point of failure that can lead to errors and security vulnerabilities. Conversely, a system prompt that is too rigid can result in a boilerplate job description that is a turn-off to top candidates.

AI Executive Academy

In person at MIT Sloan

Register Now

Advanced uses introduce additional threats

Many organizations are moving beyond off-the-shelf tools and basic generative AI solutions to more sophisticated deployments that integrate with proprietary data and systems, expanding the risk space. 

Retrieval-augmented generation employs internal data assets to augment and provide context to core foundation models. In the hiring manager example, a RAG approach would enhance the foundation model with an organization’s existing job descriptions, compensation data, and HR policies to further fine-tune results.

Because RAG relies on internal data, it inherits any existing data quality issues in the underlying data foundation. For many companies, that’s a big problem to address because their data is fragmented and distributed across systems.

In addition, RAG can expose previously undetected control gaps in vector databases, where internal data is often indexed. For example, RAG might surface sensitive data that users could technically access before but previously had no way of finding. In the hiring manager example, a query for salary benchmarks might inadvertently retrieve executive compensation data.

AI agents, which can act autonomously within defined boundaries, also elevate risk. In the HR recruiter scenario, an AI agent might automatically pull requirements from similar roles, check salary data, and post the resulting job listing to multiple sites without any human intervention. 

When agents are able to access multiple tools and data sources, organizations can quickly lose visibility into what data is flowing where — and what decisions agents are making on their own. As multiple agents coordinate on complex tasks, firms may face autonomy creep — a situation in which agents are authorized to perform more tasks but lack proper oversight or accountability. 

An action plan for success

To shrink the risk space, organizations should do the following::

1. Map exposure across the entire risk space with an inventory of every generative AI tool and solution in use. For each one, document the foundation model in play, the process for how system prompts are designed and maintained, what data assets are connected, and where human review is required. Establish accountability and permission structures. 
2. Adapt different approaches to managing embedded risks and enacted risks. Embedded risks require proactive engagement with vendors and partners, including independent evaluation and contractual requirements for transparency and change notification. Enacted risks require a framework of internal governance capabilities and technical controls. 
3. Assign clear ownership of ongoing risk assessment and establish audit trails that log the prompts, outputs, and human interventions at each step where generative AI influences decisions. 

Even if foundation model operations are a black box, the generative AI risk space cannot remain opaque. Given the high stakes, organizations must be intentional about understanding and mitigating risk to succeed in this new era.

Read the research briefing: “Mapping the generative AI risk space”

Nick van der Meulen is a research scientist at the MIT Sloan Center for Information Systems Research. He conducts academic research that targets the challenges of senior-level executives, with a specific interest in how companies need to organize themselves differently in the face of continuous technological change. He is one of the faculty members who teaches the MIT Sloan Executive Education course AI Executive Academy. 

Hippolyte Lefebvre is a research collaborator with MIT CISR, an assistant professor in management information systems at University College Dublin, and an affiliated researcher with the Competence Center Corporate Data Quality. His research focuses on improving data and AI management in multinational firms.

Barbara Wixom is a principal research scientist at MIT CISR. Since 1994, her research has explored how organizations generate business value from data assets. Her methods include large-scale surveys, meta-analyses, lab experiments, and in-depth case studies. She teaches the MIT Sloan Executive Education course Data Monetization Strategy: Creating Value Through Data.