Why Banks Self-Host LLMs: Data Risk & Governance

About three months after I joined a bank to help build an enterprise LLM platform, one question kept coming up: cloud models are already mature, so why would a bank still run its own?

There is no single answer. Self-hosting gives an institution more control over data flows and system configuration, but it also brings responsibility for GPUs, model updates, security patches, capacity planning, and operations. Cloud services reduce the infrastructure burden, but require a closer review of data processing, outsourcing, service location, retention, and contractual responsibility.

The deployment model should not come first. The more useful sequence is to define the use case and classify the data, then decide which workloads may use an external service, which must remain in a controlled environment, and which compensating controls each option requires.

The FSC guidance calls for risk governance, not one deployment model

Taiwan's Financial Supervisory Commission published its Guidelines for the Use of Artificial Intelligence in the Financial Industry in 2024. The document applies a risk-based approach and states that it is administrative guidance rather than a binding rule. It covers six areas: governance and accountability, fairness and human-centered values, privacy and customer rights, robustness and security, transparency and explainability, and sustainable development.

The guidance does not say that every financial institution must self-host AI. When a third-party service is involved, the focus is due diligence, contracts, data protection, responsibility, and ongoing oversight. For generative AI, institutions are expected to choose controls that match the data and the risk of the use case. A closed deployment is one possible control, not the only acceptable answer.

That distinction matters. Treating self-hosting as proof of compliance can hide weaknesses in output review, permissions, and monitoring. Choosing a cloud service only because it is convenient can just as easily understate data-transfer and third-party risk.

Classify the data before choosing the architecture

One LLM platform may handle public documents, internal operating procedures, and customer information. These data classes should not share the same access and transmission rules.

A practical starting point is to separate the use cases:

Summarization, translation, and general Q&A over public information.
Internal document search and workflow assistance.
Analysis involving customer, transaction, or employee data.
High-impact decisions that affect credit, investments, customer rights, or regulatory obligations.

For each case, check whether the data may leave the institution's controlled environment, whether a third party may retain it, whether human review is required, and what harm an incorrect output could cause. A low-risk task over public data may be suitable for an external service. Sensitive information and high-impact decisions may justify internal deployment, de-identification, a dedicated endpoint, or another stricter control.

The main value of self-hosting is control over the data path

In our environment, vLLM, the application platform, and the vector database run inside a controlled network. The goal is not technical independence for its own sake. It is to bring prompts, retrieved documents, model outputs, and logs under the bank's existing identity, audit, and incident-handling controls.

Self-hosting does not solve privacy by itself. If every employee can search every document, the vector database has no permission boundaries, or logs retain complete customer input, the system is still risky even without an internet connection.

I put the first round of privacy hardening in this order:

Control access to models, knowledge bases, and tools by identity and group.
Make retrieval inherit the access rules of the source document.
Mask, replace, or block sensitive fields with guardrails.
Limit which prompt, response, and retrieval fields enter logs, and define their retention period.
Keep human review and traceable records for high-risk outputs.

These controls are usually more direct than starting with a complex privacy technology. Fully homomorphic encryption, differential privacy, and trusted execution environments all have valid use cases, but their priority depends on the threat model, performance requirements, and the gaps in existing controls.

Treat the vector database as a sensitive data layer

An embedding may look like an anonymous list of numbers, but that is not a safe assumption. Research has shown that under specific models and attack conditions, an attacker may reconstruct parts of the source text or recover sensitive information from text embeddings.

The vector database should therefore receive protections comparable to those applied to the source documents: network isolation, encryption, authentication, fine-grained authorization, and access auditing. If a user cannot read a document, semantic similarity should not give that user one of its chunks.

One common mistake is to search the entire collection first and ask the model to decide whether the result is allowed. Permission checks should take effect before or during retrieval, not after the model has already received the content.

PII masking needs both rules and models

Names, phone numbers, national identifiers, account numbers, and addresses have different structures. Regular expressions alone miss personal data expressed in natural language. A named-entity recognition model alone can miss fixed-format fields or classify them incorrectly.

A more reliable design uses rules for data with clear formats, then uses entity recognition to cover names, organizations, addresses, and other semantic entities. Masked output should retain auditable type markers such as [ACCOUNT_ID], so downstream tasks do not lose all structure.

PII detection should not exist only at the user-input boundary. External documents, RAG chunks, model outputs, and tool parameters may all carry sensitive information. The controls need to follow the data through its lifecycle.

An air-gapped network reduces some risks and raises operating costs

When an environment cannot reach PyPI, model repositories, or public CDNs, every update needs another process. Packages, model weights, and container images are usually downloaded outside, scanned, recorded in a software bill of materials, and transferred through an approved path.

This reduces uncontrolled outbound connections, but it does not eliminate supply-chain risk. The risk moves to package provenance, image builds, offline transfer procedures, and update delays. Without version pinning, vulnerability scanning, signature verification, and an emergency patch process, a closed network can remain on a known-vulnerable release for too long.

The difficult part of a self-hosted platform is rarely the first successful inference. It is the continuing work: GPU capacity, model versions, permission changes, logging, failover, patching, and incident recovery all need an owner.

Federated learning is not required for every self-hosted LLM project

Federated learning is useful when data is distributed across organizations or environments, cannot be centralized, and still needs to support joint model training. If the immediate goal is inference and RAG inside one bank, and the data can already be processed under existing permissions, federated learning is usually not a first-phase requirement.

That does not mean a single bank will never need it. A large institution may have internal data silos, while cross-institution fraud or risk models may involve data that cannot be directly shared. The decision should start from a real collaboration objective, a privacy threat, and expected model value, not from the fact that the technique is advanced.

How to decide whether to self-host

Before choosing a deployment model, answer these questions:

What data will the model access, and where may that data travel?
Does the third party retain inputs, outputs, or telemetry?
Who may use the model, knowledge bases, and tools?
If the model is wrong, what happens to customer rights or regulatory obligations?
Can the institution maintain GPUs, model versions, and security updates?
During an incident, can the team obtain enough logs and stop the service?

If contracts, dedicated endpoints, masking, and access controls can manage the risk, a cloud service may be reasonable. If the data cannot leave a controlled environment, the system must integrate internal authorization, or the third party's terms do not meet the requirements, self-hosting has a stronger case.

This is not a permanent cloud-versus-on-premises decision. Financial institutions often need a tiered architecture in which different data and tasks follow different paths, with clear ownership, controls, and audit evidence for each path.

Before the next deployment discussion, make a table with five columns: data type, allowed destinations, users, impact, and required controls. It produces a more actionable decision than choosing a model or infrastructure first.

Frequently asked questions

Must a bank self-host every LLM?

No. The FSC guidance is risk-based and includes requirements for due diligence and oversight of third-party AI services. The deployment model should reflect data sensitivity, the use case, the provider's terms, and the institution's ability to operate the system.

Can an embedding be treated as anonymous data?

It should not be. Research shows that text embeddings may leak source text or sensitive attributes. The practical risk depends on the model, the attacker's capabilities, and the data, but the vector database should still be protected as a sensitive data layer.

Does a single bank need federated learning?

For internal inference and RAG, permissions, data governance, and evaluation usually come first. Federated learning becomes relevant when data cannot be centralized but several environments or institutions have a real need to train a model together.

Sources and version notes

About the author

KJ Huang (KJH) is a Taiwanese software engineer and technical leader with more than eight years of experience across AI, blockchain, gamification, cybersecurity, and finance. His current work focuses on enterprise self-hosted LLM platforms, RAG, and AI governance. More at kjhuang.com.

This article is part of the AI Engineering in Banking series.

Also in this series: RAG Optimization in Practice and Why RAG Can't Answer "How Many?".

Chinese version: 銀行為什麼自架 LLM？.