The Compliance Gap Nobody Is Auditing Yet

Your engineers are running large language models on their laptops right now. Not through a sanctioned cloud API with access controls and audit logs — locally, using tools like Ollama or LM Studio, pulling models like Llama 3, Mistral, or Microsoft’s Phi directly onto company-issued hardware. No data leaves the network, so your cloud access security broker stays silent. No API key gets flagged, so your DLP tool reports nothing. From a monitoring perspective, nothing happened. From an ISO AI compliance perspective, you have a serious problem.

This is not a theoretical risk. Ollama alone has been downloaded over five million times. LM Studio reports millions of active users. The capability to run a fully functional 7-billion-parameter model locally — one capable of summarizing quality reports, drafting nonconformance responses, or analyzing process data — is now a single terminal command away. The people doing this are not malicious actors. They are your best engineers solving real problems faster than your approved tooling allows.

This article makes a direct argument: on-device AI inference is the new shadow IT, and it is quietly creating compliance exposure inside ISO 27001 and ISO 9001 frameworks that most quality managers and CISOs have not started auditing. By the time it shows up in an external audit finding, the remediation cost will be significantly higher than building governance now.

What On-Device Inference Actually Means for ISO-Certified Operations

On-device inference means a complete AI model — weights, runtime, and all — is installed and executed on a local machine. There is no external server involved. When an engineer runs ollama run llama3, that model processes inputs and generates outputs entirely within the endpoint. It is functionally equivalent to installing licensed software, except the “software” can read, summarize, classify, and generate text about anything it is given — including controlled quality documents, supplier data, or internal process specifications.

ISO 27001: Which Annex A Controls Are Silently Violated

ISO 27001 Annex A contains several control categories that on-device AI directly intersects without triggering any existing enforcement mechanism. Annex A.8 (Asset Management) requires that information assets be inventoried and owned — locally installed model weights processing company data are assets that virtually no organization has catalogued. Annex A.8.24 (Use of Cryptography) and A.8.10 (Information Deletion) become ambiguous when model context windows temporarily hold sensitive data with no documented retention policy.

Annex A.5.10 (Acceptable Use of Information) and A.6.3 (Information Security Awareness) require that employees understand what constitutes acceptable use of company data. If your acceptable use policy does not mention local AI inference — and most written before 2023 do not — you have a policy gap that is also an audit gap. An ISO 27001 auditor asking whether employees have clear guidance on AI tool use will not accept a policy that predates the tools by two years.

Annex A.8.32 (Change Management) is the control most frequently violated without awareness. When an engineer uses a local model to draft a quality procedure or analyze a process dataset, that AI-assisted output can enter controlled documentation without any change record reflecting AI involvement. The change management trail is incomplete, and the organization cannot demonstrate the integrity of its documented information.

ISO 9001: How Local AI Outputs Enter Documented Processes Without Validation

ISO 9001 Clause 7.5 governs documented information — its creation, control, and maintenance. The standard requires that organizations ensure documented information is suitable, adequate, and protected. When a quality engineer uses a local LLM to draft a corrective action report or summarize audit findings, and that output is incorporated into the QMS without validation, Clause 7.5 is technically breached. There is no evidence of review, no record of the tool used, and no verification that the AI-generated content is accurate.

Clause 8.5.1 (Control of Production and Service Provision) requires that organizations use suitable monitoring and measuring resources and implement controlled conditions. In manufacturing environments where local AI is being used to analyze production data or suggest process adjustments, this clause demands that the tool producing recommendations be validated — a requirement almost no team using Ollama on a workstation has met.

The Data Residency Illusion — Local Does Not Automatically Mean Compliant

The most common mistake quality managers make when first encountering on-device AI is concluding that because data stays on the machine, it stays compliant. This conflates data residency with data governance, and they are not the same thing. Data can be physically local while still violating classification, handling, retention, and access control requirements.

Consider a scenario common in manufacturing: a process engineer feeds a supplier nonconformance report into a local LLM to generate a response draft. The data never leaves the building. But who authorized that use? Was the supplier data classified as confidential? Is there a retention policy for the model’s temporary context? Can the organization prove the final response was human-reviewed and not used verbatim? ISO AI compliance requires answering all of these questions — local inference makes each one harder, not easier, to answer.

Why Your Current Governance Stack Cannot See This Risk

Traditional security monitoring was architected around a core assumption: sensitive data moves across network boundaries when it is at risk. DLP tools watch for data exfiltration. CASBs monitor cloud application access. SIEMs correlate network events. On-device inference breaks every one of these assumptions cleanly. The data does not move. The application does not connect to an external service. There is no anomalous network event to correlate. Your entire monitoring stack was designed for a threat model that on-device AI does not fit.

What Your SIEM and DLP Tools Are Missing

A SIEM detects on-device AI risk only if it has been specifically configured to flag process execution events for tools like Ollama, LM Studio, or llama.cpp — and almost none are. Standard log sources (firewall, proxy, identity provider) generate zero signal when inference runs locally. DLP tools that scan outbound traffic will not see a local model reading a quality document, because no file is transmitted. The activity is invisible to the tool by design.

Endpoint Detection and Response (EDR) platforms can detect process execution, but they are typically tuned for malicious behavior — not for policy-violating legitimate use. An engineer running Llama 3 to summarize an internal audit report does nothing that looks suspicious to CrowdStrike or SentinelOne. The process runs, completes, and terminates without any behavioral indicator that a security tool would flag. The local AI governance risk is a policy compliance problem, not a malware problem, and most security stacks are not configured to enforce policy at that level.

How Model Weights and Outputs Move Without Triggering Standard Alerts

Model weights are large binary files — typically 4GB to 40GB — downloaded from Hugging Face or the Ollama registry over standard HTTPS. To a proxy or firewall, this looks identical to any large software download. There is no DLP signature for “7-billion-parameter language model.” Once downloaded, weights reside on the endpoint and can be copied to USB drives or shared drives without triggering content-inspection rules, because they contain no readable sensitive data — they are numerical parameters, not documents.

Outputs are the more immediate compliance concern. A local model generating a quality report summary, a supplier communication draft, or a process recommendation produces a text output that the user then pastes into a Word document, email, or QMS record. That handoff — from AI output to official document — is completely unmonitored and unrecorded. Your ISO AI compliance evidence trail has a gap precisely at the point where the risk is highest.

Where ISO Frameworks Win — and Where They Were Never Designed to Reach

ISO 27001 and ISO 9001 are not useless for AI governance — far from it. Their underlying logic (document your controls, evidence your outputs, manage your risks, audit your compliance) applies directly to AI risk. The problem is not the philosophy of the standards. The problem is that the specific control language was written before generative AI existed as an operational tool, and several structural gaps are now visible that require supplemental policy rather than creative reinterpretation.

ISO Controls That Map Well to AI Risk With Minor Extension

Risk assessment processes under ISO 27001 Clause 6.1 and ISO 9001 Clause 6.1 apply cleanly to AI tools — you assess the risk, document it, and implement controls. Extending your existing risk register to include a row for “local AI inference on endpoints” is straightforward and auditable. Similarly, supplier management controls (ISO 27001 A.5.19–A.5.22, ISO 9001 Clause 8.4) can be extended to cover open-source model providers and inference runtime vendors with minimal structural change.

Documented information controls (ISO 9001 Clause 7.5) provide a workable foundation for AI output governance — you simply need to add explicit requirements that AI-assisted documents are labeled, reviewed, and approved before entering the QMS. This requires policy additions, not framework overhaul. Training and competence requirements (ISO 9001 Clause 7.2, ISO 27001 A.6.3) map directly to AI literacy requirements for employees authorized to use local models.

The Three Areas Where ISO Guidance Runs Out and You Need a Supplemental Policy

• Model validation and fitness-for-purpose: ISO 9001 requires that tools and equipment be validated for their intended use. Neither standard provides specific guidance on what “validation” means for a generative AI model — you need an internal policy defining acceptable accuracy thresholds, use-case restrictions, and human review requirements before AI outputs enter controlled processes.
• Inference activity logging: ISO 27001’s logging controls (A.8.15, A.8.16) assume that systems generate logs by default. Local inference tools do not. You need a supplemental requirement that any approved local AI tool must be configured to produce auditable logs of sessions, inputs (or input categories), and output dispositions — even if this requires custom scripting or wrapper tooling.
• Model lifecycle management: Neither standard addresses the governance of model versioning, deprecation, or replacement. A model used in quality analysis last quarter may have been superseded by a newer version with different behavior. Without a model lifecycle policy, you cannot demonstrate that your AI-assisted processes are reproducible or stable — a direct ISO 9001 concern.

A Practical Governance Playbook for On-Device AI in Manufacturing and Quality Ops

The goal is not to eliminate local AI use — that is neither achievable nor desirable. The goal is to bring it inside your ISO compliance boundary with controls that are proportionate, auditable, and sustainable. The following three-step sequence can be completed in 30 to 90 days and does not require new software purchases to start.

Step 1: Discovery — Inventory What Models Are Already Running on Endpoints

Start with a point-in-time endpoint sweep. Query your endpoint management platform (Intune, JAMF, or equivalent) for the presence of directories associated with local AI tools: ~/.ollama, ~/LMStudio, ~/.cache/huggingface. Also search for installed applications matching known inference runtimes. This will surface a baseline inventory of who is running what, without requiring any employee to self-report — which they will not do if they suspect the response will be punitive.

Complement the directory sweep with a network traffic review. Pull 90 days of proxy logs and filter for outbound requests to ollama.com, huggingface.co, and lmstudio.ai. Match download events to endpoint identities. This gives you a picture of model acquisition even if the models have since been deleted. Document the inventory as a formal risk register entry — that record becomes your baseline evidence for ISO AI compliance purposes.

Step 2: Policy — Define Approved Local Models, Use Cases, and Output Handling Rules

Write a Local AI Use Policy that covers four elements: approved models (by name and version), approved use cases (e.g., internal document drafting — not customer-facing communications), data classification restrictions (e.g., no Confidential or above data as model input without CISO approval), and output handling requirements (all AI-assisted documents must be reviewed and approved by a named human before entering any controlled process or QMS record).

Keep the approved model list short and deliberate. Starting with one or two validated models — for example, Mistral 7B for internal drafting, Phi-3 Mini for structured data summarization — is better than a broad approval that you cannot support with validation evidence. Each approved model should have a one-page use case specification and a designated owner responsible for monitoring version changes.

Step 3: Audit Trail — Create Lightweight Logging That Satisfies ISO Evidence Requirements

ISO auditors require evidence of control operation, not perfection. For local AI governance, a lightweight logging approach works: require that any use of an approved local model for a quality-related task be recorded in a simple log (date, user, model used, use case category, output disposition — reviewed/approved/discarded). This can be a shared spreadsheet initially, migrated to a ticketing system or QMS field as volume justifies.

For Ollama specifically, session logging can be enabled via environment variable configuration and logs routed to your SIEM using a lightweight agent. This creates an automatic audit trail without relying on user self-reporting. Pair this with a quarterly review process — a named quality or IT owner reviews the log, identifies any out-of-policy use, and documents the review. That review record is your ISO evidence of ongoing control operation.

What Most Organizations Get Wrong When They Try to Lock This Down

The instinct to respond to shadow AI with a blanket ban is understandable and wrong. It is wrong practically — motivated engineers will use personal devices, VPNs, or cloud-hosted alternatives that create far worse compliance exposure. It is wrong strategically — the productivity gains from local AI in manufacturing and quality functions are real, and blocking them hands a competitive advantage to organizations willing to govern the risk rather than avoid it.

Misconception: Blocking Model Downloads Solves the Problem

Blocking huggingface.co and ollama.com at the proxy level stops naive downloads from managed devices. It does not stop engineers from downloading models on personal hotspots, transferring them via USB, using pre-downloaded weights already on their machines, or switching to cloud-based alternatives like ChatGPT or Claude that create a different and often larger data governance problem. You have not eliminated on-device inference risk — you have driven it underground and made it less visible.

The organizations that handle this well treat the download as a leading indicator, not the risk itself. The risk is uncontrolled use of AI outputs in quality-critical processes. Focusing governance energy on the output handling and documentation requirements — rather than the download — addresses the actual ISO AI compliance exposure and does not create adversarial dynamics with your best technical talent.

Misconception: ISO Certification Already Covers AI Tools by Default

This misconception is widespread and dangerous. Being ISO 27001 or ISO 9001 certified means your defined scope, at the time of your last audit, was compliant with the standard. It does not mean new tools introduced after that audit are covered. Certification is a point-in-time assessment of a defined scope — and “employees using local AI models on endpoints” almost certainly falls outside the scope of any certification granted before 2024.

Auditors are increasingly aware of AI risk. ISO has published AI-specific standards (ISO/IEC 42001) and is actively integrating AI risk language into updates to existing frameworks. Organizations that assume their existing certification provides cover for AI tool use are accumulating audit findings that will surface at the worst possible time — during a customer audit, a regulatory review, or a contract renewal that requires demonstrated ISO compliance.

On-Device AI Governance Is the Next ISO Audit Frontier — Get Ahead of It Now

The trajectory here is clear. ISO/IEC 42001 (AI Management Systems) was published in December 2023. Accreditation bodies are training auditors on AI risk. Large manufacturers and their Tier 1 suppliers are beginning to include AI governance requirements in supplier qualification questionnaires. The organizations that treat on-device AI governance as an optional future concern are building a compliance debt that will become due faster than they expect.

What a Mature AI Governance Posture Looks Like Inside an ISO-Certified Operation

A mature posture does not mean a restrictive one. It means local AI use is inventoried, policy-governed, logged, and periodically audited — the same standard applied to any other tool used in quality-critical processes. Approved models are documented. Use cases are defined. Outputs that enter controlled processes have a human review record attached. The evidence exists. The auditor can see it. The organization can defend it.

Operationally, this looks like a Local AI Use Policy reviewed annually, a model registry maintained by IT or quality with CISO sign-off on additions, session logging routed to existing SIEM infrastructure, and a standing agenda item in quarterly management review meetings covering AI tool use and any identified policy deviations. This is not a large governance apparatus — it is proportionate controls applied consistently, which is exactly what ISO frameworks reward.

The organizations that build this now — before external auditors start asking for it — will have two advantages: they will not face surprise findings, and they will have demonstrated to customers and regulators that they govern emerging technology risks proactively. In ISO AI compliance as in quality management, the cost of building the process is always lower than the cost of remediating the failure.