Judgment in the Age of AI · Article 39
A polished conclusion may be wrong before analysis begins. Duplicate rows, missing values, mixed units, filters and sampling bias can all change the result. When auditing AI analysis, trace where the data came from before inspecting what was calculated.
Walk through five data stations
- Origin: Who collected it, for what purpose, over which period and population?
- Cleaning: What was deleted, imputed, merged or recoded?
- Definitions: What are the unit, meaning and denominator of each field?
- Calculation: Can formulas, filters and groups be reproduced independently?
- Interpretation: Is the conclusion descriptive, correlational or causal, and does it exceed the sample?
Catch errors with three control totals
| Control | Example |
|---|---|
| Rows | Do counts before and after import or deduplication match expectations? |
| Totals | Do groups add to the whole, and do amounts reconcile? |
| Known sample | Does a hand calculation of several rows match the AI result? |
Ask AI to deliver more than a conclusion: require a data dictionary, cleaning record, executable formulas or code, and a list of unresolved questions. Analysis without a reproducible path is only a story that sounds evidenced.
References
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.