Judgment in the Age of AI · Article 41
In manual use, one wrong answer usually affects one decision. Under automation, the same mistake can reach thousands of objects within minutes. Risk is not merely the error rate; it also includes speed, scale, authority and detection delay.
Estimate blast radius with four multipliers
Impact ≈ consequence per action × volume × executable authority × time before detection
A small loss per event can become substantial when volume is large and nobody notices for a long time. Conversely, a small number of high-consequence actions still calls for strong confirmation.
Put fuses into the system
| Fuse | Purpose |
|---|---|
| Rate and value limits | Cap the reach of one abnormal run |
| Sample review | Find a pattern before every case is affected |
| Anomaly threshold | Pause when distributions, reversals or complaints change sharply |
| Gradual release | Validate in a small population first |
| Kill switch and rollback | Restore a known state quickly |
Before automating, write down how much the worst single run could do. A system whose reach cannot be measured, stopped or reversed is not ready to operate unattended.
References
- Cyber.gov.au: Careful Adoption of Agentic AI Services
- NIST: Generative Artificial Intelligence Profile
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.