Building cyber-resilient AI in the enterprise

Key Takeaways:

• AI’s rapid expansion is creating new security vulnerabilities.
• Large language models introduce unpredictable outputs that attackers can manipulate.
• Prompt injection, social engineering, and data exfiltration are emerging threats.
• Zero-trust and least-privilege controls help secure AI environments.
• Incident response and continuous monitoring are essential for mitigating AI breaches.

AI Breaches Are Different

Enterprise AI deployments are scaling faster than any other software category in history, now commanding 6% of the $300 SaaS market. According to McKinsey & Company, 88% of businesses have applied AI to at least one task. In the rush to remain competitive, many organizations inadvertently neglect security considerations, paving the way for new types of breaches.

Adversaries have quickly learned to exploit AI’s unique attack surface. Large language model (LLM) applications differ from traditional software in how they accept and process user input. Rather than producing consistent results from identical inputs, LLM outputs can vary due to factors such as temperature settings and updates to the underlying model. This variability makes it difficult to verify that vulnerabilities have been patched and remain closed.

Moreover, attackers do not always need direct infrastructure access to exfiltrate data. By skillfully manipulating AI models, threat actors can trigger malicious actions, including data leakage. Methods such as prompt injection, instruction hacking, and data poisoning disrupt model performance and confidentiality.

Building a Cyber-Resilient AI Environment

Given the significant consequences of AI breaches, security must be embedded into AI design. Threat modeling for large language models should begin early, focusing on vulnerabilities like prompt injection, indirect attacks, and data leakage via retrieval-augmented generation (RAG). Authorization rules should function at each retrieval point, ensuring identity permissions cover not only user interface elements but also underlying databases and search layers.

Least-privilege access is another cornerstone. Connectors that link various systems to the AI environment must be strictly limited, using allowlists and robust constraints on agent execution. Human intervention is necessary before irreversible actions, such as financial transactions or outgoing correspondence, can occur.

Enterprises should also implement zero-trust controls, maintaining the assumption that external content may be hostile until proven otherwise. Data loss prevention systems can help prevent users from pasting sensitive content into AI tools where it could be inadvertently leaked. Alongside vetting software supply chains for AI workflows, organizations must maintain logs and monitor for suspicious query patterns or surges in retrieving high-value data.

Logging, monitoring, and incident response protocols are crucial when anomalies arise. AI breaches often unfold subtly, with small amounts of data leaked over time. This reality makes an established incident response guide indispensable. Actions may involve taking tools offline, rotating tokens, purging search indexes, and verifying whether any confidential information was compromised.

As AI continues to transform operations across industries, organizations must recognize that innovation without robust security can be a costly gamble. Enterprises that invest in cyber-resilient AI today will be better positioned to avoid breaches while fully harnessing AI’s transformative potential.