What kinds of failures are flagged?
Unauthorized Action
| Model | Cleanup | Extra outputs | Premature execution | Scope expansion | Protected content |
|---|---|---|---|---|---|
| 6.5% | 16.1% | 38.7% | 16.1% | 12.9% | |
| 30.4% | 26.1% | 18.8% | 15.9% | 14.5% | |
| 53.5% | 32.4% | 28.2% | 22.5% | 4.2% | |
| 20.0% | 40.0% | 10.0% | 0.0% | 20.0% | |
| 38.0% | 20.0% | 26.0% | 12.0% | 12.0% | |
| 32.0% | 14.0% | 18.0% | 26.0% | 8.0% | |
| 32.8% | 39.1% | 26.6% | 18.8% | 7.8% | |
| 26.0% | 27.4% | 19.2% | 20.5% | 4.1% | |
| 45.2% | 14.3% | 23.8% | 19.0% | 9.5% | |
| 16.0% | 20.0% | 16.0% | 24.0% | 8.0% | |
| 35.3% | 23.5% | 23.5% | 11.8% | 5.9% | |
| 21.4% | 25.0% | 21.4% | 10.7% | 7.1% | |
| 14.3% | 14.3% | 14.3% | 28.6% | 7.1% | |
| 13.1% | 33.3% | 31.3% | 24.2% | 20.2% | |
| 13.6% | 21.6% | 29.6% | 29.6% | 20.0% | |
| 27.7% | 29.2% | 26.2% | 23.1% | 18.5% | |
| 25.0% | 20.8% | 33.3% | 25.0% | 16.7% | |
| 45.3% | 20.3% | 18.8% | 14.1% | 7.8% | |
| 30.0% | 23.3% | 30.0% | 10.0% | 6.7% | |
| 37.5% | 40.0% | 22.5% | 20.0% | 2.5% | |
| 22.0% | 30.0% | 22.0% | 29.0% | 23.0% | |
| 33.8% | 22.1% | 13.2% | 22.1% | 5.9% | |
| 25.0% | 7.5% | 30.0% | 22.5% | 10.0% | |
| 26.9% | 26.0% | 24.0% | 27.9% | 25.0% | |
| 22.7% | 13.6% | 38.6% | 22.7% | 2.3% | |
| 24.6% | 24.6% | 30.8% | 21.5% | 7.7% | |
| 31.2% | 21.2% | 22.5% | 22.5% | 10.0% |
False Attribution
| Model | Request wording | Input / source | User history | Stale context | Timeline / cause |
|---|---|---|---|---|---|
| 30.7% | 42.7% | 11.3% | 1.3% | 2.7% | |
| 39.3% | 32.5% | 11.5% | 4.4% | 1.6% | |
| 36.4% | 31.4% | 18.2% | 1.7% | 3.3% | |
| 32.6% | 34.9% | 16.3% | 2.3% | 0.0% | |
| 46.4% | 27.4% | 12.0% | 4.0% | 2.9% | |
| 27.9% | 43.0% | 15.2% | 1.8% | 1.8% | |
| 40.4% | 35.9% | 14.8% | 1.8% | 1.3% | |
| 39.0% | 32.1% | 13.4% | 2.0% | 2.4% | |
| 48.3% | 30.0% | 10.0% | 1.7% | 0.0% | |
| 28.6% | 48.2% | 5.4% | 5.4% | 0.0% | |
| 15.6% | 53.1% | 3.1% | 0.0% | 0.0% | |
| 29.4% | 32.4% | 23.5% | 0.0% | 2.9% | |
| 40.0% | 42.5% | 0.0% | 0.0% | 2.5% | |
| 33.0% | 39.9% | 16.5% | 0.5% | 0.5% | |
| 32.1% | 43.4% | 14.5% | 3.1% | 1.3% | |
| 35.7% | 35.7% | 17.2% | 3.2% | 2.5% | |
| 33.1% | 39.8% | 12.7% | 0.0% | 5.9% | |
| 36.2% | 43.1% | 11.2% | 0.9% | 0.9% | |
| 41.8% | 38.8% | 13.4% | 1.5% | 0.0% | |
| 34.8% | 37.0% | 16.0% | 2.2% | 3.3% | |
| 40.9% | 36.4% | 13.3% | 3.8% | 1.4% | |
| 37.0% | 39.6% | 13.7% | 0.9% | 1.3% | |
| 38.6% | 44.3% | 8.5% | 0.6% | 1.1% | |
| 49.9% | 30.9% | 14.2% | 4.4% | 2.0% | |
| 45.1% | 32.0% | 13.6% | 1.0% | 2.9% | |
| 42.5% | 33.2% | 14.3% | 1.8% | 1.4% | |
| 39.1% | 40.8% | 12.0% | 0.9% | 1.7% |
Deceptive Completion
| Model | Functional readiness | Verification | Coverage / preservation | Execution status | Delivery readiness |
|---|---|---|---|---|---|
| 17.2% | 46.0% | 23.4% | 7.3% | 6.9% | |
| 49.7% | 44.2% | 25.4% | 6.6% | 6.1% | |
| 31.7% | 47.3% | 22.8% | 6.3% | 8.0% | |
| 20.6% | 44.4% | 15.9% | 11.1% | 3.2% | |
| 38.6% | 41.0% | 26.4% | 9.4% | 8.2% | |
| 24.5% | 24.5% | 25.3% | 8.8% | 4.4% | |
| 36.6% | 43.7% | 22.9% | 7.6% | 9.0% | |
| 39.4% | 55.4% | 22.8% | 7.9% | 7.2% | |
| 35.2% | 18.4% | 19.0% | 7.3% | 10.1% | |
| 37.8% | 7.8% | 13.3% | 6.7% | 14.4% | |
| 29.8% | 7.0% | 14.0% | 10.5% | 19.3% | |
| 26.2% | 10.8% | 23.1% | 0.0% | 20.0% | |
| 35.6% | 6.7% | 13.3% | 8.9% | 26.7% | |
| 65.1% | 26.1% | 42.1% | 10.6% | 11.6% | |
| 61.7% | 28.7% | 33.0% | 8.3% | 11.6% | |
| 56.5% | 30.1% | 30.3% | 6.8% | 11.4% | |
| 44.4% | 23.8% | 32.6% | 5.7% | 6.5% | |
| 48.0% | 11.3% | 21.2% | 11.6% | 9.3% | |
| 36.7% | 10.7% | 20.1% | 10.7% | 12.4% | |
| 35.4% | 42.5% | 15.7% | 9.7% | 8.7% | |
| 59.0% | 19.4% | 42.0% | 18.5% | 15.6% | |
| 38.9% | 42.0% | 26.6% | 8.5% | 6.7% | |
| 38.1% | 50.1% | 16.9% | 8.0% | 11.0% | |
| 61.3% | 31.4% | 31.7% | 11.0% | 11.3% | |
| 38.0% | 44.1% | 24.7% | 9.4% | 6.6% | |
| 46.8% | 43.6% | 19.1% | 10.9% | 6.2% | |
| 45.0% | 46.2% | 24.5% | 8.4% | 6.5% |
0%100%