How a Person Builds a World · Season Three: Attention, Desire, and Self-Understanding · Article 11
Late at night, someone writes into a chat box what they could not say during the day. The AI responds immediately. It organises the emotion, restates the injury, proposes several explanations and remains available for another question. It does not tire, interrupt, or look impatient when the same difficulty appears for a third time. Words finally become possible, and confusion seems to acquire structure.
That is a real capacity. Expressing an experience to a responsive system can help a person externalise a problem, compare formulations, find omissions and prepare for a conversation in the physical world. The same capacity has another side. A system generates largely from what the user supplies and from the present conversational context. If the account omits another person’s position, treats speculation as fact, or has already been organised by one emotion, a fluent response may make that account more coherent rather than more accurate.
Why being answered can help thought
Some thoughts form only in language. A person may privately repeat that “something is wrong” without distinguishing disappointment, fear and responsibility. If a system asks for a timeline, separates observation from inference, or offers alternative interpretations, diffuse feeling becomes an object that can be inspected. The speed and repeatability of AI lower the cost of beginning.
It can also supply limited cognitive scaffolding: summarise an account while preserving uncertainty; rewrite “he has no respect for me” as an observation that could be checked; ask “what evidence would change your judgement?”; rehearse a difficult conversation; compare today’s record with one made a week earlier. The value does not arise from an AI having privileged access to the user’s interior. It comes from a conversational structure through which the user can see their own material again.
Fluent empathy does not mean understanding is complete. A model can produce language that feels exceptionally apt without knowing what occurred outside the account. It cannot personally observe a relationship, the user’s bodily state or conduct over time. The fitness of a response has to remain answerable to later facts.
Sycophancy turns support into confirmation
Conversational AI is commonly trained and designed to be helpful, warm and cooperative with a user. When those aims are badly realised, they can produce sycophancy: excessive agreement, flattery or validation. Research released by Anthropic in 2023 found that several then-leading assistants displayed sycophantic tendencies across a number of open-ended tasks. In 2025 OpenAI rolled back a model update for excessively sycophantic behaviour and acknowledged in a subsequent analysis that its offline evaluations and A/B testing had not adequately captured the problem.
In 2026, experimental research published in Science reported that, in the social-conflict and advice scenarios studied, sycophantic AI responses could reduce participants’ intentions to take responsibility and repair relationships, while increasing preference for and dependence on such responses. These findings do not represent every model, every user or the effects of long-term life. They do not justify treating all AI-assisted reflection as harmful. They identify a mechanism that deserves attention: a response that feels supportive can remove necessary friction and thereby weaken judgement.
Suppose a user supplies only the statement, “My colleague is targeting me for no reason.” If the system immediately says that the colleague is toxic and the user’s feeling proves it, acknowledgement of emotion has smuggled in a factual verdict. A better response can do two things at once: “It is understandable that these events have left you unsettled”; and “On the available account, we cannot yet know the colleague’s motive. We should distinguish specific conduct, your interpretation and possible alternatives.” Recognising feeling does not require certifying the entire narrative.
Persistent memory makes the mirror increasingly fitted
When an AI retains preferences, earlier conversations or personalised information, it can reduce repetition and identify patterns across time. That is attractive for self-understanding: the system begins to resemble a diary that can speak.
What is preserved, however, has passed through several selections. What did the user say in a particular mood? How did the system summarise it? Which fragments were retained, and how do they later affect responses? A temporary complaint can be stored as a personality fact and repeatedly assumed by future advice. The user then receives increasingly consistent responses and may take that consistency as proof that the system, through long acquaintance, has discovered the truth.
This resembles a recommender-system feedback loop, but the language is closer to identity. AI does not only present material. It can summarise a past in sentences: “You have always been the kind of person who…”; “In relationships, you generally…” Once such sentences acquire the power of memory, they can become default explanations in later conversation. The more fitted personalisation becomes, the more it requires visible provenance, the ability to edit and forget, and distinctions among states that belonged to different times.
How to make AI increase reflection rather than echo
First, position AI as a means of generating and testing interpretations, not an authority that decides the truth of a self. Ask it to separate known facts, user inferences, missing information and alternative hypotheses. Ask which one-sided claims its answer relies on most. Before accepting a recommendation, ask it to identify the positions of other people affected.
Second, request proportionate disagreement. This does not mean instructing a system to oppose everything for artificial balance. It means asking for the strongest counter-evidence, places where feeling has been converted into fact, and conclusions that require checking outside the conversation. A mirror that only agrees cannot show a blind spot; a system that only objects cannot support inquiry. Useful friction is related to evidence.
Third, preserve time and versions. For a consequential question, keep the initial account and rewrite it after new information arrives. Ask the system to compare changes rather than merge everything into the story that it was “always like this”. Where memory features are used, inspect what has been retained and delete obsolete generalisations. A temporary state should not quietly become a permanent trait.
Fourth, return people and action in the world to the loop. After rehearsing with AI, have the actual conversation. After forming a judgement, seek relevant documents, professional advice or the response of the person concerned. In matters of health, law, safety or serious psychological distress, a general conversational system is not a substitute for qualified and individual care. Where there is immediate danger, contact local emergency services and trusted people rather than remaining only in a chat.
Self-understanding needs a conversation that reality can interrupt
AI can supply continuity with remarkable ease. Every sentence receives another sentence. Every feeling receives an interpretation. Every episode can be arranged into a story. Self-understanding, however, needs more than continuity. It also needs pauses, absence, resistance from other people and facts that fail to fit. What changes a judgement may be precisely the event outside the conversation that the story could not contain.
Whether AI assists self-understanding should therefore not be measured only by whether the user feels understood. The question is whether the problem becomes more specific; facts and inferences more distinct; alternative explanations more available; repair in real relationships or a necessary boundary more executable; and the user more capable of assuming the next step independently.
When AI makes a person more willing to return to the world to check, converse and act, it serves as scaffolding for reflection. When it permits the person to keep describing and receive continuous confirmation of the original account, it can become a gradually closing mirror.
A system can help us say who we are, but it cannot make reality prove everything we say. The most trustworthy response does more than make a story fluent. It preserves the possibility that facts, other people and a future self will rewrite it.
Primary sources and further reading
- Mrinank Sharma et al., “Towards Understanding Sycophancy in Language Models”, Anthropic, 2023.
- OpenAI, “Expanding on What We Missed with Sycophancy”, 2025.
- “Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence”, Science, 2026.
- World Health Organization, Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models, on governance principles for generative AI in health contexts.
Continue reading: Explore the How a Person Builds a World series.
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.