After AI Enters the Workflow · Season Three: “When AI Starts Acting for the Organisation” · Article Five
Two customers visit the same company’s website on the same day and ask about the same service.
The first sees a concise explanation, a lower-cost option and a button to book an adviser. The second sees a complex table, a more expensive bundle and a qualification questionnaire that must be completed first. Neither knows that the page differs, and neither can compare it with a version they were never shown.
The company may call this design personalisation. AI uses browsing history, location, device, earlier purchases and predicted need to arrange the most relevant experience. It may genuinely help by removing irrelevant information and providing suitable language or channels. The same mechanism can also decide who receives help first, who sees a discount, who must provide additional proof and who is quietly directed away.
The issue is not simply that personalisation uses customer information. It is what personalisation changes. If AI adjusts expression, risk is relatively limited. If it silently changes price, eligibility, waiting or access to people, personalisation has become differential treatment.
Ask what is personalised before asking how much data is used
Organisations often discuss personalisation as one feature. It includes at least four levels:
| Level | Example | Main benefit | Main risk |
|---|---|---|---|
| Expression | Change language, type size or content order | Easier understanding | Stereotyping and omitted information |
| Assistance | Remember preferences or offer accessible channels | Less repeated work | Excess collection and frozen errors |
| Recommendation | Rank products, services or next steps | Lower search cost | Hidden alternatives and commercial bias disguised as relevance |
| Treatment | Change price, eligibility, priority or scrutiny | More granular allocation | Unfair difference, proxy discrimination and inability to compare |
The first two can often be governed through clear choice, easy correction and data minimisation. The latter two require more scrutiny because the system does not merely help someone understand an existing service. It shapes the opportunity the person can receive.
“We do not use sensitive attributes” does not finish the analysis. Postcode, purchase pattern, language, device, working hours and social behaviour may correlate with age, disability, ethnicity, income or family circumstance. A model can reproduce a similar division through proxy variables without ever receiving a protected attribute.
Identical final outcomes can conceal unequal processes
Difference is not limited to approval and refusal. Two customers may eventually purchase the same service, while one submits more evidence, waits longer, repeatedly verifies identity or never sees the lower-cost option. Friction changes who can persist through the process.
A personalisation audit should therefore examine more than final conversion. It should compare:
- options displayed and their order;
- prices, discounts and fees;
- fields and documents required;
- steps and waiting time to reach a person;
- repeated classification as high risk or low value; and
- ability to correct a profile and reset personalisation.
If the organisation optimises only clicks, sales or service cost, AI will learn distinctions helpful to those targets rather than distinctions compatible with fair service. Values omitted from the objective do not appear merely because the model is sophisticated.
Australia’s AI Ethics Principles treat human-centred values, fairness, privacy, transparency and contestability as connected. Department of Industry, Science and Resources, “Australia’s AI Ethics Principles” Personalisation needs those principles together: service may become more relevant without removing a customer’s ability to understand, compare and challenge treatment.
Knowing more may mean becoming more confident in a wrong profile
Customer data are often treated as objective fact, but profiles mix observation, inference and institutional response.
“No purchase during the past three months” is a record. “Price sensitive” is an inference. “Not worth routing to an adviser” is a treatment produced from that inference. They should not be merged. The customer may not have purchased because the site was inaccessible, care responsibilities intervened or an earlier request was wrongly refused. If the model interprets the result as preference and then reduces offers or help, a loop forms: fewer opportunities produce lower engagement, and lower engagement confirms the original low-value classification.
Profiles also expire. When someone moves, changes work, recovers from illness or alters a language preference, an old inference may continue to govern new treatment. If the organisation cannot state when a profile was updated, permit correction and remove unnecessary inferences, personalisation fixes the past as identity.
The Australian Privacy Principles require covered entities to manage personal information transparently and impose obligations concerning collection, use, quality, access and correction. Office of the Australian Information Commissioner, “Australian Privacy Principles guidelines” Scope and legal duties depend on the entity and information, but the design question remains: why is the business entitled to use this information to change treatment, and how can the customer know and correct it?
A recommendation does not neutrally discover “the best” option
AI recommendation requires a definition of best: cheapest for the customer, most suitable over time, easiest to understand, or most profitable for the business? When the system optimises relevance and revenue together, it should not present paid ordering or margin preference as pure assistance.
The problem resembles identifying advertising in search. Customers should know whether an option is recommended because of their need or the company’s sales objective. A system may combine both, but some boundaries should remain: do not hide a basic option, do not let commission become the sole ordering rule, and do not exploit reduced comparison capacity in urgent situations.
The Australian Competition and Consumer Commission’s guidance on false or misleading claims focuses on the overall impression and notes that omission of important information may mislead. ACCC, “False or misleading claims” AI personalisation creates a special difficulty because each customer receives a different overall impression. The business needs records of versions and ranking reasons rather than reviewing one “standard page”.
Fairness evaluation must manufacture comparability
Personalisation prevents real customers from comparing experiences, so the organisation must create comparison deliberately.
One method is paired testing: construct profiles that are as similar as possible except for one potentially relevant characteristic and compare price, options, waiting and verification. Another is group outcome analysis: examine persistent differences in access to people, discounts, evidence requests and completion. A third is path replay, reconstructing the page, recommendation and rule version a particular customer saw.
Each method has limits. Paired cases cannot cover every interaction. Group averages hide intersectional circumstances. Gathering real group data can create privacy risk. Fairness evaluation is therefore not a single certification. It combines design review, recurring tests, complaints, staff observations and customer research.
The NIST AI Risk Management Framework asks organisations to identify affected groups, intended and unintended uses in context, and to measure and manage risk continuously. NIST AI Resource Center, “AI RMF Core” A personalisation context cannot be described only as “improving customer experience”. It should list the treatments that may change and who bears the error.
Offer choice without transferring all governance to the customer
An option to turn off personalisation is useful, but it does not excuse an unfair model. Customers may not know what the setting changes and should not need to investigate preferences merely to obtain a standard price or speak with a person.
Three forms of control are needed:
- Non-negotiable institutional limits: do not alter eligibility using inappropriate variables, hide necessary information or use an unexplainable high-consequence profile.
- Customer-selected preferences: language, content density, contact method and remembered service interests.
- Reviewable inferences: when risk, value or recommendation materially changes treatment, permit explanation and human review.
Defaults also matter. Requiring customers to find an opt-out is not free choice where the system collects far more than service delivery requires. Data minimisation reduces privacy risk and the opportunity for models to create distinctions from irrelevant features.
Beneficial personalisation expands customer capability
The most valuable personalisation often reduces barriers rather than predicting who deserves service. It can provide suitable language, restore an unfinished form, highlight material relevant to a stated need, remember accessibility preferences and ask rather than infer when uncertain.
A directional test is useful: does the feature give the customer more capacity to understand, choose and complete, or give the institution more capacity to filter, price and persuade? The two can coexist, but risk increases where the main benefit is institutional influence that the customer cannot see or correct.
“Relevant” must also remain separate from “permitted”. A variable may predict purchase extremely well without being appropriate for service priority. Information may improve profit without being worth collecting for that purpose. The ability of data to create a distinction does not explain why the distinction is justified.
Conclusion: personalise assistance, not rights in secret
Knowing more about customers can make service easier, faster and more intelligible. It also lets institutions differentiate price, choice, friction and attention in ways customers cannot observe.
The minimum principle is:
Personalisation may adapt expression and assistance. When it changes price, eligibility, priority, verification burden or access to people, the organisation must explain its reasons, compare treatment across groups, let customers correct material inferences and provide review independent of the same profile.
The best personalisation does not quietly decide what kind of person a customer is. It makes it easier for the customer to say what they need now. The first turns prediction into identity; the second turns technology into service.
Primary sources and further reading
- Department of Industry, Science and Resources: Australia’s AI Ethics Principles
- Office of the Australian Information Commissioner: Australian Privacy Principles guidelines
- Australian Competition and Consumer Commission: False or misleading claims
- NIST AI Resource Center: AI RMF Core
Continue reading: Explore the After AI Enters the Workflow series.
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.