When Models, Instructions and Knowledge Bases Keep Changing, How Should AI Workflows Be Versioned?

After AI Enters the Workflow · Season Two: “From Personal Tool to Organisational Capability” · Article 11

Last week an AI assistant explained the refund policy correctly. This week it repeatedly omits an exception. The team checks the instructions and finds no change. The knowledge base appears normal. Eventually they discover that the provider updated the model and a new guide altered retrieval ranking. The system’s name remained the same; the operating combination did not.

Conventional software can normally identify a release and code commit. AI workflow behaviour emerges from models, instructions, data, retrieval, tools, parameters and external services. Recording only “we use this AI product” is insufficient to reproduce an outcome or restore a safe configuration.

Version the complete configuration, not the model alone

An AI run may depend on the model and provider version, system instructions and templates, a knowledge snapshot and indexing method, tool definitions and permissions, parameters, post-processing rules and human stages.

Changing any element can alter the result. Editing one document appears to be content maintenance but can reorder retrieval. Adding a tool gives an agent a new path. A provider-side update may change behaviour without an internal code release.

The organisation needs an identifiable workflow-configuration version that binds component references together. If a model cannot be pinned, retain the provider identifier, call date and all version information available under the service.

Distinguish content, technical and purpose changes

Not every change requires the same approval.

Correcting a spelling error in policy is a low-risk content update. Replacing an embedding model or retrieval algorithm is a technical change. Allowing a drafting system to send messages automatically changes purpose and permission.

The classification should determine evaluation and approval. Content updates trigger tests of affected answers. Technical updates rerun baselines and failure cases. Changes to purpose, data or action authority require renewed impact and risk assessment.

If every edit enters one slow process, teams will bypass it. If every edit is treated as routine maintenance, a material expansion of authority can happen invisibly.

Give every version a readable configuration card

A configuration card should record intended and prohibited uses, component versions, authoritative data and currency, main evaluation results, known limitations, approver, release date, monitoring measures and rollback version.

Model-card research proposed documenting intended use, evaluation conditions and limitations of models. Google Research, “Model Cards for Model Reporting” Datasheets document the sources, composition, collection and recommended uses of datasets. Microsoft Research, “Datasheets for Datasets”

An organisational configuration card extends these ideas to the whole workflow. It is not merely a long technical record for developers. It lets business owners and auditors understand what the current system actually is.

Bind evaluation results to a version

“The system achieved 95 per cent accuracy” has meaning only with a configuration, test set and date. A new release cannot inherit the previous release’s approval automatically.

Each release should rerun critical baselines: ordinary work, prior severe failures, edge groups, permission tests, tool failure and stopping conditions. Impact analysis may justify fewer tests for a narrow change, but the team should record why the untested components are unaffected.

The test set also needs versions. Adding difficult cases may lower the score without indicating that the system became worse; the standard became stronger. Both the old baseline and the new criterion should remain intelligible.

Production monitoring must detect gradual and sudden change

Not every change is a release. User questions, customer populations, external sources and attacks drift. The same configuration can meet a different world.

The NCSC’s secure-operation guidance calls for monitoring model and system behaviour, inputs, anomalies and data drift, and for supporting users in responding to model change through mechanisms such as preview access and versioned APIs. NCSC, “Secure operation and maintenance”

Monitoring should extend beyond service uptime. Refusal rates, human overrides, error types, group differences, retrieved sources and complaints can reveal changing behaviour. Thresholds should trigger suspension or return to a known good state.

Rollback includes data and permissions

Switching to an earlier model while retaining a new knowledge index and broader tools is not a full rollback.

A recoverable release includes the component inventory, configuration, data snapshot or reproducible index and matching permissions. Constantly changing customer records cannot all be frozen, but their structure, sources and changes can be retained so past outcomes remain explainable.

The NCSC secure-development guidance recommends tracking, authenticating and version-controlling AI assets and retaining the ability to restore a known good state after compromise. NCSC, “Secure development”

Rollback also needs an authorised owner. A backup is not recovery capacity if business, IT and the provider wait for one another during an incident.

Change logs must address affected people

Internal logs preserve technical detail. Users need to know how behaviour changed: which tasks are affected, whether output formats changed, whether older results require review and whether temporary human checking has increased.

Consequential external systems may need revised transparency statements, partner notification or employee retraining after material change. Version control should make change appropriately visible rather than hide it.

The NIST AI RMF treats monitoring, feedback and improvement as lifecycle work. NIST AI RMF Core Continual improvement becomes continual unpredictability unless version and responsibility remain connected.

Emergency change and retirement are also version-control problems

A security incident or severe failure may require a team to disable a tool, withdraw permissions or replace a source before full evaluation is possible. The organisation should predefine an emergency-change route: who may act immediately, how scope is limited, when deferred testing occurs and who informs business owners and affected people.

Temporary fixes need expiry. Otherwise, a bypass introduced during an incident remains permanently and becomes technical debt no one understands. Returning to normal operation should include removal of temporary data, permissions and caches.

Retirement also needs a versioned plan. Record when use stops, which evidence must be retained, how downstream processes migrate, when old interfaces close and who handles unresolved cases. Cancelling a subscription does not remove copied instructions, exported data or dependent work.

The lifecycle runs from the first experiment to the safe removal of the final dependency. A system that can be released but not retired is not a capability the organisation fully controls.

These records also protect institutional memory. When staff and suppliers change, a future team should not have to reconstruct from fragments why a model was selected, why an exception was accepted or which downstream process still depends on an old output. Version history is therefore part of operational continuity, not merely a convenience for developers.

The version boundary must also include the operating environment. Permission settings, retrieval parameters, safety filters, interface timeouts and external-tool versions can change behaviour even when the model does not. A release record containing only a model name cannot reconstruct the system that actually ran. The complete configuration should bind these dependencies to the business rules and trigger proportionate regression testing whenever a material dependency changes.

Conclusion: a changeable system must explain how it changed

Dynamic models do not justify abandoning version control. The more components and updates a workflow contains, the more important it is to identify the complete configuration at a moment in time.

The minimum principle is:

Every production AI use case must be able to state what is running now, how it differs from the previous release, which tests it passed, who approved it, how it will be rolled back, and which prior outcomes may require review.

The objective is not to keep AI unchanged. It is to ensure that change does not destroy explanation, audit and recovery. An organisation owns a system only when it can change that system safely.

Primary sources and further reading

• NCSC: Secure development
• NCSC: Secure operation and maintenance
• Google Research: Model Cards for Model Reporting
• Microsoft Research: Datasheets for Datasets
• NIST AI Resource Center: AI RMF Core

Continue reading: Explore the After AI Enters the Workflow series.