Do Customers Have a Right to Know They Are Talking to AI?

Geoffrey Chen

After AI Enters the Workflow · Season Three: “When AI Starts Acting for the Organisation” · Article Two

A face appears in the lower corner of a website. “Hello, I’m Mia. How can I help?” A customer asks about a delayed refund and receives an immediate answer. He explains that he has just lost his job and needs more time. Mia replies, “I understand your situation.” When he asks whether the extension has been approved, Mia says, “Yes, I have taken care of that for you.”

Only after the company refuses the extension does the customer discover that Mia was not a service employee but an AI system. The company points to a clause in the corner of its terms saying that some responses may be generated by automated technology. The customer’s objection is not limited to an incorrect answer. Throughout the conversation, he believed that the other party could understand an exception, possessed internal authority and was making a decision for the company.

Do customers have a right to know they are speaking with AI? The short answer should be: when AI identity would reasonably change how a person understands the exchange, discloses information, relies on a promise or seeks help, the person should know before making those choices.

That does not mean every email touched by a spelling tool needs a technical report. It means transparency should be proportionate to the consequences of the interaction.

“AI is used” is not a complete explanation

A statement that a service “may use AI” leaves at least four questions unanswered:

  • Is the current interaction with a person, an AI system or a combined process?
  • Can the AI search and explain, or can it also approve, refuse and alter an account?
  • Does the answer come from material approved by the organisation or from a general model’s inference?
  • How can the matter reach a person with authority when the answer does not fit the customer’s situation?

Those facts change behaviour. Someone may be willing to describe a health, financial or family problem to an employee governed by professional duties, internal discipline and a defined role, while refusing to place the same information into a system whose use and retention are unclear. A customer may treat “approved” from a service officer as a decision, but treat the same word from an automated assistant as a status that needs confirmation.

Transparency must therefore cover more than the existence of technology. It concerns identity, capability, authority, source boundaries and exit.

Australia’s AI Ethics Principles say that, where appropriate, people should be informed when AI materially affects them and connect transparency with contestability and accountability. Department of Industry, Science and Resources, “Australia’s AI Ethics Principles” These are principles rather than a single permission regime for every commercial interaction, but the direction is sound. Disclosure should enable understanding and challenge, not merely satisfy a labelling exercise.

Customers need five layers of transparency

The information a customer needs can be divided into five layers. Not every setting needs the same length of notice, but consequential interaction cannot stop at the first layer.

Layer one: identity

State clearly at the beginning that this is an AI assistant. Do not bury the fact in terms or a privacy page. Names, faces, simulated typing and phrases such as “I understand how you feel” should not be designed to make a customer believe that a human is present.

Layer two: task scope

Tell the person what the system can do: “I can explain published policy, check an order status and help prepare a request.” A positive scope statement is more useful than a generic warning that AI can make mistakes.

Layer three: authority boundary

Explain what it cannot do, particularly whether it can approve a refund, vary a contract, give final professional advice or respond to an emergency. The boundary must appear before the customer decides to rely on the answer, not only after failure.

Layer four: information and records

When personal information is requested, the customer should know why it is needed, how it will be used and whether it will enter an account record or improve a system. Australian Privacy Principle 1 requires covered entities to manage personal information openly and transparently and maintain a clearly expressed privacy policy. Office of the Australian Information Commissioner, “APP 1—Open and transparent management of personal information” An AI interface should not make conventional privacy information harder to reach.

Layer five: human escalation and challenge

Provide a functioning route to a person. Explain which responses are informational, which actions have been recorded on the account and how review can be requested. A “speak to a person” button that returns the customer to the same bot is not escalation.

Disclosure must occur at the right moment

Transparency that arrives too late does not preserve choice. If identity is disclosed after sensitive information has been provided, the initial disclosure decision cannot be undone. If lack of authority appears only after a customer relies on a price or deadline, the boundary has failed in the same way.

Three moments normally matter:

  1. Before the conversation: disclose identity.
  2. Before collecting information: explain purpose and recording.
  3. Before a decision or commitment: restate authority and offer human confirmation.

Article 50 of the European Union’s AI Act includes transparency obligations for certain systems intended to interact directly with natural persons, subject to its scope, conditions and exceptions. It generally requires people to be informed that they are interacting with AI unless that fact is obvious from the circumstances. European Union, Regulation (EU) 2024/1689 This is not a universal Australian rule, but it expresses an important design judgement: notice should be timely and intelligible rather than confined to a compliance document behind the interaction.

Not every AI assistance needs the same label

If an employee uses AI to check spelling, search internal material or arrange call notes, the customer may not need a new pop-up each time. The employee still understands the customer, selects the response and owns the communication; AI has not occupied the relationship’s identity.

Disclosure should become stronger as three things change:

  • AI directly produces what the customer sees;
  • AI selects an action rather than merely preparing material; and
  • the outcome begins to affect money, rights, safety, professional advice or complaint handling.

A practical test is: if the customer knew this was AI, would that knowledge reasonably change what they say, what they trust or what they do next? If it probably would, identity should not be hidden.

“Obvious” should not be declared by designers alone. A robot icon may be clear to a technically familiar user and obscure to someone using assistive technology, a second language or a service under stress. Transparency needs testing with real users, not only a legal review of the interface.

Pretending to be human creates an additional risk

Some systems use a human name, photograph, typing delays and emotional language to make conversation feel natural. Natural language is not the problem; deception is. Design intended to cause a customer to misjudge identity, qualification or authority exploits information the institution has and the customer lacks.

The Australian Competition and Consumer Commission’s guidance on false or misleading claims emphasises the overall impression and notes that omission of important information can also mislead. ACCC, “False or misleading claims” Whether a particular AI interface breaches law depends on the facts, but a disclaimer in one location does not automatically neutralise a contrary impression created by the whole interaction.

Better design can remain friendly without borrowing a false human identity: “I’m Mia, the company’s AI service assistant. I can check status and explain standard policy, but I cannot approve exceptions. When needed, I can transfer this conversation to a service officer.” That statement is more useful than a cold machine number and more honest than pretending to be a person.

A human channel must possess capabilities the AI lacks

If a service officer can only repeat the answer already produced by AI, the customer has received a change of voice rather than a review. Human escalation matters because a person can examine fuller facts, interpret the purpose of policy, approve an exception, correct a record and accept responsibility for an explanation.

The organisation should monitor how many conversations request a person, how long transfer takes, which issues repeatedly escalate, whether customers must repeat information and whether human staff can actually change the result. High escalation may reveal a narrow system, but it may also show that the AI occupies a service position unsuitable for automation.

Nor should “speak to a person” become a penalty. Customers who decline AI should not automatically wait longer or pay more. Automated service can be a convenient entrance, but a human path is part of the service for consequential, complex and accessibility-sensitive matters—not a privilege reserved for a premium tier.

Conclusion: customers need the true structure of the relationship, not the model name

Most customers do not need parameter counts, supplier versions or architecture diagrams. They need to know who is responding, what supports the answer, whether the system can act for the company and how to reach someone able to take responsibility.

The minimum principle is:

When AI directly occupies a customer-facing role, or when its identity can affect disclosure, reliance, consent or challenge, the organisation should disclose that identity before interaction and explain purpose, authority and the human route before collecting information or approaching an important outcome.

Disclosure is not a disclaimer. It is relationship design. It lets a person continue with knowledge of what the other party is, what it can do and where it must stop. An organisation that wants AI to be trusted cannot first obtain trust through ambiguous identity and then require customers to bear the cost of misunderstanding alone.

Primary sources and further reading

Continue reading: Explore the After AI Enters the Workflow series.

Discover more from Geoffrey Chen

Subscribe to get the latest posts sent to your email.