Building Secure AI Infrastructure for Healthcare and Governed Organizations
Where models run, how PHI stays inside your boundary, and why security decisions shape AI architecture long before the first prompt.
Secure AI infrastructure for a healthcare or governed organization starts with one decision: where the model runs and what data is allowed to cross that line. Get the boundary right and access control, network isolation, and audit logging fall into place around it. Get it wrong and no amount of policy language will protect the protected health information already sitting in a vendor's logs.
We design AI systems for organizations that answer to HIPAA, HITRUST, SOC 2, and similar regimes. The pattern below reflects how we approach those builds.
Decide where the model runs first
Three deployment surfaces cover almost every healthcare AI build we take on, and the right one depends on the sensitivity of the data and the latency you can tolerate.
- On-premise. The model and the data never leave hardware you control. This fits organizations with strict data residency requirements, existing GPU capacity, or a board that will not accept PHI traversing a public network. The tradeoff is that you own the scaling, patching, and capacity planning.
- In-cloud. The model runs inside your own cloud tenant on Microsoft Azure, Amazon Web Services, Google Cloud, DigitalOcean, or Vultr, isolated from the provider's general inference services. You get elasticity and managed hardware while keeping data inside an account you govern.
- Hybrid. Sensitive inference stays On-premise while burst workloads, model training, or non-PHI tasks run In-cloud. Most mid-adoption organizations land here because it lets them move fast on low-risk work without exposing the regulated data.
The mistake we see most often is treating model hosting as an afterthought, calling a public inference endpoint during a pilot, then discovering at audit time that PHI left the building months ago. Choose the surface before you choose the model.
Keep PHI inside the boundary
Once the surface is set, the work becomes drawing a hard line around protected health information and proving nothing crosses it without authorization.
A few practices carry most of the weight:
- Minimize what the model ever sees. Strip or tokenize identifiers before data reaches the model when the task does not need them. A summarization model rarely needs a patient's full name and date of birth in the same prompt.
- Disable vendor training and retention. When you do use a hosted model, contract and configure it so your prompts and completions are not retained or used for training. Confirm this in writing, then verify it in the network behavior.
- Encrypt everywhere. TLS for data in transit, provider-managed or customer-managed keys for data at rest. Healthcare auditors expect both, and the key management story matters as much as the encryption itself.
- Treat embeddings and vector stores as PHI. An embedding derived from a clinical note still carries the information in that note. Your retrieval database deserves the same controls as your primary record system.
Control who and what gets access
Access control in an AI system covers more than human logins. The model, its tools, and the services it calls all need identities and scoped permissions.
We build around least privilege on three layers.
- People. Role based access through your existing identity provider, with multi-factor authentication and short-lived sessions. Clinicians, analysts, and administrators see only what their role requires.
- Services. The application that calls the model authenticates with its own credential, not a shared key pasted into a config file. Rotate it automatically.
- Tools the agent can invoke. If the model can query a database or call an internal API, that capability is a permission. Scope each tool to the minimum it needs and log every call.
A scoped tool permission for an agent that schedules appointments captures this directly. The agent is allowed to do two things, read availability and create a booking, and it is explicitly denied the ability to read patient history or delete a record. Its data scope is fenced to a single clinic, so it can only act on records tied to the caller's own clinic. The only fields it can see are the ones the task requires, the appointment time and the provider, and nothing else from the patient record is in reach. Every call the agent makes is logged. Read what it may do, write only the one thing it should, and a record of each invocation. That is the whole permission.
Isolate the network around the model
Network design is where a secure architecture either holds or leaks. The goal is simple to state: the model and its data plane should not be reachable from the open internet, and outbound traffic should go only where you intend.
Concrete steps we apply:
- Place inference, the vector store, and the record systems in private subnets with no public IP addresses.
- Reach hosted cloud AI services over private endpoints so traffic stays on the provider backbone rather than the public internet.
- Default outbound traffic to deny, then allow only the specific destinations the system needs. This stops a compromised component from quietly shipping data out.
- Put a gateway in front of the model that enforces request size limits, rate limits, and input filtering before anything reaches inference.
Log everything an auditor will ask about
When a HIPAA or SOC 2 auditor reviews an AI system, they want to answer one question: who did what, to which data, and when. Your logging has to make that answer trivial to produce.
We capture, at minimum, the identity behind every request, the action taken, the data scope touched, and the timestamp. Logs go to append-only, tamper-evident storage separate from the systems that generate them, with retention that matches your compliance obligations. We log the fact of a PHI access, never the PHI itself, so the audit trail does not become a second copy of the sensitive data.
| Event | Captured | Retention |
|---|---|---|
| Model inference request | identity, tool, data scope, timestamp | per policy, typically 6 years for HIPAA |
| PHI access | who, which record class, timestamp | per policy |
| Permission change | actor, before and after, timestamp | per policy |
| Failed authorization | identity, attempted action | per policy |
Why security shapes the architecture, not the other way around
The reason we settle the boundary, access model, and network design before writing application code is that retrofitting security into a running AI system is expensive and rarely complete. A boundary drawn on day one is a wall. A boundary added in month six is a patch over decisions already made.
This is the difference between an AI pilot that passes audit and one that has to be rebuilt. Healthcare and governed organizations cannot afford the rebuild, so we treat the security model as the foundation the rest of the system stands on.
Working with us
We build secure AI infrastructure for healthcare providers, SaaS companies, and other governed organizations, whether you are already running models, partway through adoption, or just starting to scope the work. If you want a second set of eyes on where your data goes and how your AI system holds up under audit, reach out and we will walk through it with you.
Published Jun 9, 2026 · 6 min read