Your company ran its AI model through a full governance review in Q1 — privacy assessment, legal sign-off, risk classification, the works. Since then, the model has been fine-tuned four times on production logs and customer interaction data. It is now Q4, and a regulatory inquiry has arrived asking about training data provenance.
The model your compliance team certified hasn't existed for eight months. Nobody lied. Nobody cut corners. This is simply what a well-intentioned workflow produces when governance was designed for a world where models stay still.
Enterprise AI governance rests on a silent architectural assumption: that the model you reviewed at time T is the model running at time T+6 months. That assumption is now routinely false, and five pillars of assurance are collapsing quietly beneath organisations that haven't noticed yet.
The Assumption That Broke Everything Else
Traditional enterprise governance was built to audit artifacts — versioned software releases, bounded training datasets, point-in-time model cards. These are discrete, inspectable objects. You approve them, document them, and move on. Change management workflows, compliance sign-off processes, data lineage records — all of it was designed with this artifact model in mind.
Continual learning breaks this entirely. And continual learning, for the purposes of this article, does not mean exotic research techniques. It means scheduled fine-tuning on production logs, which thousands of companies are already running. It means RAG systems with live document ingestion. It means agent frameworks that update their own memory stores based on outcomes. The moment your model's behaviour or weights change based on operational data — on any cadence, through any mechanism — you have exited the world your governance framework was built for.
The practical consequence is that your governance objects go stale while your model keeps moving. The model card describes what the model was. The legal sign-off references what the model was. The data lineage documentation traces what the model was trained on. The model itself no longer reflects any of it.
The Five Assumptions
Assumption 1: "We Know What Data Trained This Model"
Data lineage documentation is almost always written once, for the initial training run, and then treated as a living archive — something you add to during a major release, not something that updates automatically with each fine-tuning cycle.
Each fine-tuning cycle adds data: production logs, customer support transcripts, domain documents, user feedback. This data typically receives significantly less scrutiny than the original training set — lighter cleaning, lighter review, and sometimes no formal review at all, because the reasoning is "we're just updating the model, not training a new one."
By the time a data subject submits a Subject Access Request — a right guaranteed under GDPR Article 15 — asking how their data was used to train your system, the honest answer is often: we don't know exactly. You know what went into the original model. You have incomplete records of what went into cycles two through six. This is not a small documentation gap. It is a live DSAR vulnerability, and regulators are beginning to pursue it in earnest.
What organisations need — and almost none have — is a training provenance graph: a living artifact that captures every data source, every transformation, and every cleaning step contributing to every model version, updated automatically when a fine-tuning run completes. This is not exotic engineering. It is the model-world equivalent of a dependency lock file. Most organisations building software would not ship without one. Most organisations running fine-tuning loops don't have anything close.
Assumption 2: "Our Privacy Review Covers This Deployment"
The privacy review happened once, before deployment. It assessed a specific dataset at a specific point in time and issued a clearance. Then the model went into production and began absorbing customer interaction data through quarterly fine-tuning cycles.
Here is the technical reality that most enterprise data teams treat as a research edge case: de-identified training data does not produce a de-identified model. Fine-tuning on domain-specific data — even when individual records have been anonymised — creates new identifiable patterns in the model's weights. The model becomes a compressed representation of behavioural patterns that can, in the right query context, reconstruct or contextually re-identify individuals. The clearance was issued for a dataset. The risk now lives in weights that nobody has re-reviewed.
This failure mode is compounded by a common practice: the original training dataset receives rigorous anonymisation, but subsequent fine-tuning data — production logs, support tickets, interaction histories — gets lighter treatment, because the team reasons that "we already cleared the base model." That reasoning is wrong in two directions simultaneously. The base model clearance does not carry forward to new data inputs. And lighter anonymisation of fine-tuning data introduces re-identification risk that did not exist at original deployment.
Upstream anonymisation is a risk reduction measure, not a compliance solution. And that risk reduction degrades with each fine-tuning cycle that uses less rigorously cleaned data than the one before it.
Assumption 3: "The Model We Approved Is the Model We're Running"
Legal sign-off, procurement approval, and compliance certification are all tied to a specific model artifact reviewed at a point in time. Fine-tuning creates a new artifact. In practice, almost no organisation routes the updated model back through approval, because it is mentally categorised as "the same model, just updated" — the same reasoning that allows minor software patches to bypass security review.
Consider a common production architecture: logs flow to an S3 bucket, a weekly cron job pulls those logs, a fine-tuning script runs against the base model, the updated model gets pushed to the serving endpoint, and traffic switches over. This pipeline typically has excellent technical monitoring — latency, error rates, quality scores — and zero governance gates. The engineering team built it. Governance was never consulted on the cadence.
The result is a chain-of-custody problem that compounds with every cycle. The model the legal team approved no longer exists. Something else is serving traffic under its name. If an incident occurs — a harmful output, a privacy breach, a discriminatory recommendation — the investigation traces back to a model artifact that is not the one that caused the problem.
There is a specific technical failure mode that sharpens this risk: catastrophic forgetting, where fine-tuning on new data degrades capabilities the model previously demonstrated. A model that passed your safety evaluations before its most recent fine-tuning cycle may not pass them now. When the re-evaluation cadence is slower than the fine-tuning cadence — which it almost always is — you are running a model in production under a safety certification that applies to a model that no longer exists.
Assumption 4: "Our Risk Classification Is Still Accurate"
Risk classification typically happens once, at the point of procurement or initial deployment. It is tied to the stated use case, not the model's evolving capabilities. A customer-facing summarisation tool gets classified as medium risk. That classification is written into a register somewhere and reviewed annually, if at all.
As the model fine-tunes on customer interaction data over the following eighteen months, its capabilities shift in ways the original classification did not anticipate. The same tool — same API endpoint, same interface, same stated purpose — may now be generating responses that function as implicit personalised recommendations, based on behavioural patterns absorbed through production use. The risk profile has changed. The classification has not. Nobody triggered a reclassification because nobody built a mechanism to trigger one.
This is particularly acute in regulated industries. A financial services model deployed as a "document summarisation assistant" that subsequently absorbs enough interaction data to begin producing outputs that function as financial guidance has crossed a regulatory boundary that the original risk classification did not contemplate. The capability drift happened gradually, through normal operation, and was invisible to the governance process — because the governance process only looks when asked.
Assumption 5: "We're Not a Model Provider — We're Just Using One"
This is the most consequential assumption and the least discussed in practitioner circles. When a company fine-tunes a general-purpose AI model on proprietary data, it may — depending on how the arrangement is structured and what the fine-tuning produces — legally become a provider of that model under applicable regulation, not merely a deployer of someone else's.
Under the EU AI Act, the "substantial modification" threshold for GPAI models combines compute-based measures with consequence scanning. A mid-market company doing sector-specific fine-tuning in healthcare, legal, or financial services — domains where proprietary data can produce meaningful capability changes — may cross this threshold without realising it. The compliance obligations that attach to model providers are dramatically more extensive than those that apply to deployers.
Large AI vendors have regulatory affairs teams tracking these boundaries. A 60-person fintech fine-tuning a hosted model through a cloud provider's API does not. The fine-tuning job runs. The new endpoint goes live. Nobody checked whether the company's legal classification relative to that model just changed.
The risk compounds further when the cloud provider manages the fine-tuning process and returns an API endpoint. The fine-tuned weights may not be inspectable or portable. If the vendor changes their data processing terms or is acquired, the company may hold compliance obligations for a model whose weights they cannot access or audit — a situation with no clean resolution.
One Thing Worth Acknowledging
The mainstream framing — that static deployments are well-governed and continual learning introduces new problems — flatters the baseline too generously. In reality, static model governance is largely performative across the mid-market. Most companies have no meaningful model card, no genuine data lineage, and no adversarial evaluation beyond vendor-provided safety filters. Continual learning does not create a governance gap. It reveals one that was always there, by making the consequences more immediate and harder to defer.
This matters because the answer is not to avoid continual learning. A properly instrumented fine-tuning loop — with immutable model versioning, automated re-evaluation on each training run, governed data ingestion pipelines, and explicit governance gates tied to compute and data thresholds — is more auditable than a static model that has been quietly drifting through prompt engineering and retrieval index updates for a year. The problem is not dynamic models. The problem is static-world assumptions applied to dynamic-world architectures.
Where to Start This Week
Pick one production AI system that has been fine-tuned since its initial deployment. Ask your team two questions.
First: can you produce a complete list of every data source that has contributed to the current model, including all fine-tuning cycles — not from memory, from documentation?
Second: is the model currently serving traffic the same artifact that received legal or compliance sign-off?
If you cannot answer both with documented evidence, you have found your governance gap. Not a theoretical one. A live one, attached to a system in production right now.
Start there. Build the provenance graph before the next fine-tuning cycle runs. Establish an immutable versioning convention so every model update produces a distinct, auditable identity. Define a threshold — compute-based, data-volume-based, or calendar-based — that triggers a governance review before an updated model goes live. None of this requires new tooling. It requires someone being responsible for it, and a process that treats each fine-tuning run as what it actually is: a new model, not an old one with a fresher coat of paint.