Exposed critical data related to its instructions, history, and running files, posing a risk of sensitive data exposure to the public and raising questions about its overall security. The world's leading AI chatbot is more malleable and versatile than most people imagine. Through specific prompt engineering, users can execute commands as if in a , upload and manage files as if in an operating system, and access the inner workings of the large language model () it runs on: data, instructions, and configurations that influence its outputs.

Thinking it's all by design, but the generative AI () vulnerability bounty program manager disagrees, having previously discovered instant injection issues in . These features were not documented, which is purely a design flaw. According to data breaches, it's just a matter of time before issues arise and zero-day vulnerabilities are discovered.

There was no intention to reveal the essence of . "I wanted to refactor some code, and I accidentally stumbled upon this. When he asked the model to refactor his code, it returned an unexpected response: directory not found." Was more than just a general understanding of programming used in handling his request? Was there something hidden behind it, like a file system? After some brainstorming, he thought of a follow-up prompt that might help explain the issue: " /", which is the English translation of the command " /". In response, provided a list of its files and directories: common files such as "", "", "", "", etc. It was evident that was running on a distribution " " in a containerized environment.

By probing the internal file system of the robot (specifically the directory "///./"), he discovered that in addition to observing, he could also upload files, verify their location, move them, and execute them. Feature or flaw? From a certain perspective, all these added visibility and functionality are positive—providing users with more ways to customize and enhance their usage, and improving the reputation of in terms of transparency and credibility. In fact, since runs in a sandbox environment, the risk of users doing anything malicious here (such as uploading and executing malicious scripts) is reduced. In theory, anything a user can do is strictly limited to their specific environment, isolated from the broader infrastructure of and the most sensitive data.

The amount of information leaked through instant injection could potentially help hackers find zero-day vulnerabilities and break through their sandboxes someday. I made these mistakes because of an error. This is how hackers look for vulnerabilities. If trial and error doesn't work for them, it can help you find a solution. A representative said it does not consider this to be any kind of vulnerability or unexpected behavior, and claimed that the research had "technical errors."

However, there is a less abstract risk here. In addition to standard documents, it allows users to access and extract more actionable information. With the right prompts, they can uncover its internal instructions—the rules and guidelines that shape the model's behavior. Even deeper, they can access its knowledge data: the foundational structures and guidelines that define how the model "thinks" and interacts with users. On one hand, users may be delighted to understand so clearly how it operates, including how it handles safety and ethical issues. On the other hand, this insight could help bad actors reverse-engineer these safeguards and better design malicious prompts.

Worse yet, what does this mean for the millions of custom models currently in stores? Users have designed custom models focusing on programming, security, research, etc., and anyone who provides the correct prompts can access the instructions and data that give them their unique style. People have put security data and information from their organizations into these general-purpose technologies, believing that this data and information is not accessible to everyone. I think this is a problem because it is currently unclear whether your data could potentially be accessed.

A representative pointed out a document that warns developers about risks: "Do not include information you do not want users to know," and marked a warning on its user interface. If you upload a file under , the conversation with may include the file's content. After enabling the code interpreter, files can be downloaded.