When an External AI Service Changes, Has the Organisation’s Promise to Customers Changed Too?

After AI Enters the Workflow · Season Three: “When AI Starts Acting for the Organisation” · Article Eleven

A business promises on its website that its AI assistant answers only from an approved knowledge base, shows sources for important questions and transfers matters beyond scope to staff. The system is tested at launch and meets the stated standard.

Three months later, the model provider updates the default model. A retrieval service changes its ranking, the safety filter moves to a new version and the interface reduces maximum context. The business changes no code. Product name and website description remain the same, but customers begin receiving more confident answers, fewer citations and omissions from long documents.

The supplier may regard this as continuous improvement. For the business, behaviour has departed from the promise made to customers.

“We did not change the system” is not an adequate response. If an external component can change what customers receive, a supplier change is a potential change to the institution’s product. It must be detected, evaluated and, where necessary, reauthorised.

Customers rely on a stable service boundary, not a model name

Customers usually do not know or need the underlying model version. They rely on the business’s promises concerning sources, purpose, response, privacy, human escalation and remedy.

A provider may warrant API availability without guaranteeing style, citation frequency, refusal boundary or stable behaviour for the same input. If the business treats a technical service-level agreement as a customer-service guarantee, infrastructure can remain “healthy” while customer experience changes materially.

The organisation should express its promises as observable outcomes: answers come from named sources; the system stops without support; material facts display citations; specified topics reach a person; unapproved tools are never invoked. Those properties can be retested after an external change.

An AI workflow can change in seven places

Behaviour can move without an internal code release through:

1. Foundation model: capability, refusal, language and context treatment.
2. System instruction: provider defaults or safety policy.
3. Knowledge retrieval: chunking, indexing, ranking and result count.
4. Safety filter: input and output categories and thresholds.
5. Tool interface: parameters, permission, timeout and error behaviour.
6. Data and infrastructure: region, retention, logging and subcontractors.
7. User interface: citations, warnings, escalation and history display.

A register containing only the model name cannot reconstruct the service a customer used. The complete version needs identifiers, configuration and effective time for all seven, plus known unknowns the supplier does not expose.

The UK National Cyber Security Centre’s secure operation and maintenance guidance calls for monitoring behaviour, managing updates, collecting logs and preparing incident response. NCSC, “Secure operation and maintenance” For a business dependent on external AI, monitoring must extend beyond its own release records to the supply chain.

“May update at any time” does not replace institutional judgement

Cloud services need to evolve; a business cannot freeze every model forever. Procurement terms and technical arrangements should nevertheless cover:

• advance notice for defined changes;
• version pinning or delayed migration;
• old-version support and emergency rollback;
• changes to data use, retention, region and subcontractors;
• availability of logs, evaluations and incident information;
• supplier evidence after material change; and
• export of data and configuration on termination.

A provider’s right to modify service may be standard, but the business cannot passively accept every change in a consequential use case. Where visibility, version control or rollback are inadequate, the institution must narrow purpose, add controls or choose another service.

ISO/IEC 42001 includes suppliers, risk, change, performance evaluation and continual improvement within AI management systems. ISO, “ISO/IEC 42001—Artificial intelligence management systems” Buying AI is not a completed component purchase. It is continuing management of a changing dependency.

Detect behavioural drift rather than waiting for notice

Provider notice can be incomplete, and a “non-material update” may be material in one domain. Organisations need independent regression evaluation.

A test set can preserve common tasks, boundary cases, questions without answers, sensitive topics, languages, long documents, prompt injection and human escalation. It should compare:

• factual and source-grounded accuracy;
• appearance and correctness of citations;
• refusal and escalation against policy;
• whether tone turns advice into commitment;
• differential change among customer groups; and
• latency, cost and failure pattern.

The objective is not identical wording. Generative behaviour varies. Institutional boundaries should remain stable. Answers may become more natural without gaining refund authority; retrieval may become faster without introducing unapproved material.

The NIST AI Risk Management Framework emphasises continuing measurement, monitoring and response rather than evaluation only before release. NIST AI Resource Center, “AI RMF Core” Supplier drift demonstrates why one successful test cannot authorise an external component forever.

Classify change by customer promise, not supplier version number

Three levels are useful.

An operational change does not alter commitments—for example, performance improvement with stable behavioural tests. It can be recorded and allowed.

An important change affects accuracy, citation, refusal, cost or customer path while staying inside approved purpose. It requires regression evaluation and business-owner confirmation, and may require updated explanation.

A material change alters data use, authority, customer treatment, human escalation, safety or legal risk. It should be treated as a new deployment with renewed impact assessment and approval, and restricted until complete.

The provider cannot classify business effect alone. A small release may degrade understanding of a critical industry term. A new feature may enable tools by default. Significance belongs to the institution’s context.

Does change require review of earlier results?

A new version may reveal that an earlier model created a systematic error. The institution then asks which past customer outcomes are affected. Conversely, a faulty new version requires identification of every relevant output between activation and containment.

Customer results need linkage to the complete workflow version. Without lineage, the business knows that a provider was used on a date but cannot identify which configuration produced an answer.

Retrospective review should follow consequence. Internal drafts may need no reopening. Outputs that affected price, eligibility, complaint or safety require more scrutiny. Identified people should receive retraction and remedy rather than merely a rollback to the old model.

Tell customers about changed boundaries, not every technical release

Forwarding all supplier release notes creates noise. Customers need changes that affect choice and reliance: new data use, removal of citations, reduced human support, new decision authority or earlier results requiring review.

Notice should state effective time, effect, required customer action and contact route. A line silently changed in terms or privacy policy does not ensure that customers understand a different service.

Minor performance updates do not need notification fatigue. The organisation needs a threshold judged jointly by business, risk and customer responsibility, not only by whether the API still responds.

Australia’s AI Ethics Principles concerning transparency, reliability and safety, contestability and accountability also apply to change. Department of Industry, Science and Resources, “Australia’s AI Ethics Principles” Transparency is not every parameter. It is capability and limitation meaningful to affected people.

Supplier governance requires an exit capability

A business may reject a change yet be unable to migrate because prompts, evaluations, knowledge indexes, logs and tools are tied to one platform. Contractual termination exists, but operational choice does not.

An exit plan belongs at adoption, not after incident. It includes exportable knowledge and records, supplier-independent test sets, an interface layer, a human fallback, migration testing and evidence of deletion from the former service.

The business also needs degraded mode. If the advanced model is unavailable, it may offer static search, restrict service to low-risk questions or route to people rather than preserve full function on an untested fallback.

Exit need not be immediate or costless. It must allow the institution to stop where supplier behaviour conflicts with customer promise. Without it, the supplier roadmap ultimately determines what the customer was promised.

Conclusion: suppliers change components; institutions maintain promises

Continuous change is normal for external AI. The issue is whether the institution can translate that change into its own judgement about risk, service and responsibility.

The final principle is:

Any supplier update capable of changing customer-facing facts, citations, authority, treatment or human access is a potential change to the institution’s service. The organisation must maintain its promise through complete version records, independent regression evaluation, graduated approval, customer-notice thresholds and a viable exit plan.

A business may outsource model, hosting and tools. It cannot outsource “what we provide to customers” into a variable it cannot see. The supplier changes the product; the institution decides whether the changed product may continue to operate under its name.

Primary sources and further reading

• UK National Cyber Security Centre: Secure operation and maintenance
• ISO: ISO/IEC 42001—Artificial intelligence management systems
• NIST AI Resource Center: AI RMF Core
• Department of Industry, Science and Resources: Australia’s AI Ethics Principles

Continue reading: Explore the After AI Enters the Workflow series.