When Does a Personal AI Have the Right to Speak for You?

Geoffrey Chen

After AI Enters the Workflow · Season One: “From Answering Questions to Participating in Work” · Article 11

A personal AI has read years of someone’s writing, messages and notes. It can reproduce her tone, remember her preferences and draft replies that friends recognise as “exactly like her”. While she is busy, the system answers routine invitations. Later it accepts a commercial proposal, explains her position in a dispute and sends condolences to a bereaved friend.

At what point did assistance become representation?

The system’s resemblance to its user is not enough. A skilled impersonator may sound like someone without having authority to speak for them. Conversely, an authorised lawyer or assistant may speak validly without imitating the principal’s style at all.

The right to represent a person comes from identity, authorisation, scope and accountability—not from similarity. As personal AI becomes more capable, these distinctions must be visible to recipients as well as users.

Similar style does not create the same identity

A model can learn recurring vocabulary, sentence length, preferred greetings and typical positions. It can predict what a person might say in familiar circumstances. That is a behavioural approximation, not the continuing person.

The individual may have changed her mind. She may make an exception because of a relationship or a new fact. She may wish to remain silent even when the model can generate a plausible reply. She may regard a private pattern in old messages as an error rather than a preference to preserve.

Personal identity includes the capacity to revise commitments and to decide that the past should not control the present. A system trained or configured on previous behaviour naturally emphasises continuity. Representation must preserve the person’s right to introduce discontinuity.

This is why “the AI knows me” should never be treated as “the AI may bind me”. Prediction of preference and authority to act are separate properties.

Five levels of representation

Personal AI activity can be divided into at least five levels.

At the first level, private assistance, the system organises notes, suggests wording or prepares options that only the user sees. It does not communicate externally.

At the second, drafting, it prepares a message for the user to review and send. The user remains the immediate speaker.

At the third, routine communication, it sends within a narrow pre-authorised domain: acknowledging receipt, arranging an available meeting time or supplying standard information.

At the fourth, negotiation or commitment, it proposes terms, accepts obligations, spends money, changes appointments with consequences or states the user’s position in a dispute.

At the fifth, personal or public representation, it speaks in emotionally, professionally or politically significant contexts where the communication itself expresses relationship, judgement or identity.

The move between levels should not happen silently as the system becomes more accurate. Each level requires different authorisation, disclosure and review. A user who permits calendar scheduling has not thereby authorised contractual acceptance or intimate correspondence.

First establish who is acting, then what they may do

Digital identity systems distinguish authentication from authorisation. Authentication establishes that an actor or credential corresponds to an identity; authorisation determines which actions that actor may perform.

NIST’s Digital Identity Guidelines address identity proofing, authentication and federation for digital services. NIST, “Digital Identity Guidelines” Personal AI introduces an additional layer: the user may authenticate successfully and then delegate limited operations to software.

The recipient needs to know whether a message came directly from the person, from an authorised AI acting within scope or from an unauthorised imitation. The service needs to know which account and permissions are involved. The user needs a reliable record of what was done under the delegation.

Strong authentication alone is not enough. A system may securely prove that it operates from the user’s account while exceeding the authority the user intended to grant.

What acceptable authorisation should contain

Authorisation for a personal AI should be granular and understandable. It should define:

  • purpose: the kinds of tasks for which representation is allowed;
  • audience: the people, organisations or channels involved;
  • actions: whether the system may draft, send, schedule, negotiate or commit;
  • limits: money, time, subject matter, sensitivity and frequency;
  • duration: when the authority begins, expires or must be renewed;
  • disclosure: what recipients will be told about AI involvement;
  • records: what will be logged and visible to the user;
  • revocation: how the user can stop future action immediately;
  • escalation: which uncertainty or consequence requires direct approval.

Defaults should favour the least authority necessary. Expanding scope should require a deliberate choice, not emerge from accumulated convenience.

Privacy must also be considered. A highly personalised assistant may need access to messages, contacts, health details, location or financial information. NIST’s Privacy Framework provides a structure for governing and communicating privacy risk, including data processing and individual control. NIST, “Privacy Framework”

The data needed to imitate a person may be much broader than the data needed to perform a particular task. Personalisation should not become a justification for unlimited collection.

Transparency cannot be one universal disclaimer

Should every AI-assisted message say “written by AI”? The answer depends on what the disclosure is meant to achieve.

If AI only corrects grammar in text the user reviewed, a prominent warning may add little. If an autonomous system conducts a negotiation, offers advice or produces synthetic audio that appears to be the person, disclosure becomes materially important.

Article 50 of the EU AI Act establishes transparency obligations for specified AI interactions and synthetic content, with details depending on the type and context of system. EU AI Act, Article 50

A useful disclosure should help the recipient answer relevant questions: Was this content generated or materially altered by AI? Did the person approve this specific communication? Is the system authorised to make a commitment? How can a recipient reach a human if the matter is sensitive or disputed?

A generic footer cannot answer all of these. Transparency should be matched to consequence and should not falsely imply that disclosure cures excessive delegation.

Provenance can prove a history, not a mind

Content provenance standards can help show where digital material came from and what transformations were recorded. C2PA Content Credentials provide a technical approach for attaching cryptographically verifiable provenance information to media. C2PA, “Content Credentials Explainer” The technical specification defines assertions, manifests and validation mechanisms. C2PA, “Content Credentials Technical Specification”

Such systems can help a recipient establish that content was created or processed by particular tools and that recorded provenance has not been altered. They cannot prove that the represented person agrees with the message, that the delegation was normatively appropriate or that no relevant event occurred outside the recorded chain.

Provenance is evidence about production history. Authorisation is a relationship between a person, an agent, a purpose and a permitted act. Both matter, but they solve different problems.

The more convincing the imitation, the more important the difference

An imperfect assistant reminds recipients that they are interacting with a tool. A highly realistic voice, face or writing style can erase that distance. The recipient may disclose sensitive information, accept a commitment or experience an emotional communication as if the person were directly present.

This increases the duty to protect difference. Systems should not use resemblance to bypass consent or exploit trust. Sensitive messages should require direct review. Recipients should have a way to request the person. The system should not invent feelings, memories or personal experiences merely because they match a style profile.

The user also needs protection from the model’s version of the user. Stored preferences should be inspectable and correctable. The system should distinguish direct instructions from inferred tendencies. A past pattern should never become an invisible rule that narrows future choice.

Conclusion: representation comes from authorisation, not resemblance

A personal AI may become remarkably good at predicting how someone writes and what they usually choose. That capability can make assistance more useful. It does not create a right to speak, promise or decide on the person’s behalf.

The central rule is:

An AI may represent a person only within an explicit, limited and revocable delegation whose scope is visible to the user and appropriately clear to the recipient.

Routine, reversible communication can often be delegated. Commitments, disputes, intimate relationships and public positions require stronger approval and disclosure. Identity verification, permission design, provenance, privacy controls and activity records must work together.

The goal is not to prevent personal AI from speaking. It is to ensure that when it speaks, recipients can distinguish assistance from agency, prediction from consent and stylistic resemblance from the person’s present judgement.

Primary sources and further reading

Continue reading: Explore the After AI Enters the Workflow series.

Discover more from Geoffrey Chen

Subscribe to get the latest posts sent to your email.