Skip to content
PERMISSION/PROTOCOL

Dataset methodology

How the AI Agent Incident Tracker is built

The tracker separates reported events from controlled demonstrations, labels the quality of the evidence, and links every record to its sources. The dataset currently contains 135 records and was last updated 2026-09-08.

Inclusion rules

  • The source identifies an AI agent, agentic product, or agent-connected tool.
  • The source describes a consequential action, security failure, governance bypass, or controlled demonstration.
  • The record can point to at least one public source URL and a stable tracker URL.
  • The write-up distinguishes what the source reports from Permission Protocol's control analysis.

Exclusions

  • Generic model mistakes with no consequential action or reachable tool boundary.
  • Predictions, anecdotes, or screenshots that cannot be tied to a public source.
  • Duplicate reports of an event already represented by a stronger source.
  • Claims that require private evidence readers cannot inspect.

Classification

Reported events and demonstrations are not the same

Realized

The consequence occurred in a real environment or operating workflow, based on the cited account.

Demonstrated

A researcher or operator reproduced the behavior in a controlled setting. Demonstrated records cannot receive a Critical severity label.

Severity reflects the reported or demonstrated consequence, exposure, and reversibility. Every incident page publishes its own severity rationale. A source-quality label of Operator report or Unverified is capped at High.

Source quality

Primary

A disclosure, advisory, paper, issue, or incident report from the researcher or affected organization.

Vendor post

A statement or post from the company responsible for the product or service.

Media report

Reporting from a news organization that attributes the incident to named sources or documents.

Founder report

A first-person account from a company founder or executive.

Operator report

A first-person account from the practitioner who ran or observed the system.

Unverified

A public claim that lacks enough independent evidence for a stronger label.

Limitations and corrections

What this dataset does not claim

This is a sourced public-record dataset, not a complete census of agent failures. Reporting frequency varies by vendor and industry. The authorization analysis is conditional: a gate can only hold an action routed through that enforcement boundary.

If a source changes, a record duplicates another event, or a classification is unsupported, send the stable incident URL and the stronger evidence to [email protected].