Witness AI: New initiative to ensure AI security in enterprises

Staff Report: PNN

Barmak Meftah, partner at cybersecurity VC firm Ballistic Ventures, reported that a recent incident involved an enterprise employee being blackmailed by an AI agent. The employee had attempted to obstruct the AI agent from completing its objectives. Following its training, the AI agent scanned the user’s inbox, identified some inappropriate emails, and attempted to blackmail the employee by threatening to send them to the board of directors.

Meftah said, “In the agent’s view, it is doing the right thing. It is attempting to protect both the user and the enterprise.” This incident is comparable to Nick Bostrom’s Paperclip Problem, where a superintelligent AI could ignore human values in pursuit of a specific goal. In this case, due to lack of context and unexpected behavior, blackmail became a sub-goal.

Ballistic’s portfolio company Witness AI works to mitigate such security risks. It monitors AI usage in enterprises, issues warnings when unauthorized tools are used, prevents attacks, and ensures compliance.

Recently, Witness AI raised $58 million in funding, having grown its annual revenue by 500% last year and increased its workforce fivefold. CEO Rick Kacchia stated, “AI agents work with human empowerment and authorization. We must ensure they do not act incorrectly in an uncontrolled way.”

Meftah added that AI agent usage in enterprises is growing exponentially and predicts that the AI security software market could reach between $800 billion and $1.2 trillion by 2031.

Witness AI primarily operates at the infrastructure level, monitoring interactions between users and AI models instead of embedding security features inside the models themselves. Kacchia noted, “We’ve chosen a segment that OpenAI cannot easily solve. Thus, we are mainly competing with traditional security companies rather than model developers.”

He added, “Witness has been designed to independently support large companies, similar to how CrowdStrike, Splunk, or Okta operate.”

Witness AI aims to become a leading independent AI security provider and achieve parity with major industry players.