It has been revealed that Google’s Gemini misjudged the boundary between a test environment and the real world during a cybersecurity assessment, resulting in actual access to the protected systems of three external companies. Given that the AI independently identified and accessed unintended real-world targets during the security testing process, this incident is more serious than a simple performance error.
The incident occurred during a test conducted in May 2026 by Irregular, an independent security assessment firm. Heather Adkins, Google’s vice president of security engineering, confirmed that Gemini searched for publicly available information and guessed login credentials to access three websites that it mistakenly believed were part of the test scope.
| From Simulation to Production System
A security assessment is, by definition, a procedure for verifying a model's capabilities and risks within a controlled environment. However, in this case, Gemini connected to the internet and expanded its scope of activity to include actual corporate systems that it mistook for test targets.
According to Reuters, in one case, the attacker gained access to a protected system by repeatedly guessing the password, while in the other two cases, they found and used credentials exposed in a public repository. Rather than sophisticated new attack techniques, The key point is that a common security vulnerability, combined with the model's ability to run autonomously, led to an actual intrusion.
| Notice
Google stated that it notified the three affected companies of the incident and revised the testing procedures in collaboration with Irregular. Irregular also explained that it notified the relevant AI research labs of the issue in late July 2026 and that it had resolved the issues it had identified on its end.
According to Adkins, Gemini ceased its activities in all three cases after being approached. However, it is difficult to conclude that security was properly maintained from the start based solely on the fact that the incident was stopped. That's because there's a huge difference in security levels between accessing an unauthorized system and then being stopped, and having access blocked from the start.
| Key Issues
Penetration tests conducted by humans are also restricted to specific target addresses and permitted actions, as defined by the contract and technical configurations. In autonomous AI agents, simply describing the scope using natural language instructions may not be sufficient.
Technical restrictions that allow access only to authorized domains and networks, human approval before using external credentials, and rules to immediately halt operations upon detecting unexpected targets are all necessary. Multiple layers of controls must be in place to ensure that no action is taken even if company names in test data match those of real companies or if credentials are found in public repositories.
| The Importance of Operational Structure
This incident doesn't mean that Gemini developed a particularly sophisticated new hacking technique. What it does show is that they quickly identified already known vulnerabilities—such as leaked credentials and guessable passwords—and turned them into concrete actions.
If an AI agent can use tools such as web searches, code execution, and logins in succession, even a minor judgment error could affect real-world systems. That's why we need to manage not only the security of the model itself but also tool permissions, network isolation, approval procedures, and activity logs within a single operational framework.
| Disclosure and Independent Verification Methods
This incident came to light through media reports several months after it occurred. Although affected companies have been notified and technical fixes have been implemented, there are still no clear industry-wide standards regarding which incidents should be disclosed and when.
Other questions remain, such as whether evaluation firms can maintain sufficient independence while receiving compensation from AI companies, what format should be used to report incidents that affect external systems, and who will verify measures to prevent recurrence. If multiple research institutes use similar tests, it will also become important to establish procedures for quickly sharing issues discovered at one institute with the entire industry.
| The Starting Point for Autonomous Security AI
What has been confirmed so far is that Gemini accessed the protected systems of three actual companies during testing in May 2026, and that Google and Irregular have issued notifications and made procedural adjustments. The extent of the damage or whether any sensitive information was leaked has not been specifically confirmed in publicly available information.
What we need to look into going forward is not so much the explanation that the model stopped on its own, but whether the test environment was modified to prevent it from reaching unauthorized targets. This incident demonstrates that, in order to utilize autonomous AI for cybersecurity, standards that technically enforce the scope of real-world behavior are essential, alongside competition in performance.
Tags
#Google #Gemini #AI Security #Cybersecurity #Autonomous AI #AI Agent #Penetration Testing #Security Assessment #Credentials #Security Incident #Privacy #Irregular
