AI Privacy Review for Enterprise Knowledge Bases

The Core Tension: Utility vs. Data Leakage in AI Knowledge Bases

Enterprise knowledge bases—whether internal wikis, CRM records, or legal document stores—have always been a privacy surface. But adding AI introduces a new vector: the model itself becomes a storage and retrieval machine.

The problem is not that the LLM is “intelligent.” The problem is that the AI pipeline (embedding → vector store → retrieval → inference) creates multiple copies of your data in formats you don’t control:

• Embedding vectors: Dense floating-point arrays that can be inverted to reconstruct original text.
• Context windows: The full text chunks fed into the LLM during RAG, potentially cached by inference endpoints.
• Model weights: Fine-tuned models that may memorize rare or sensitive document fragments.

If you deploy an AI knowledge base, you are effectively building a second database—one that is opaque, non-standard, and often running on someone else’s hardware.

Pillar 1: Embedding Privacy

Embeddings are not hashes. They are trained representations that preserve semantic similarity—and that similarity includes structure, terminology, and sometimes verbatim fragments.

The Vector Inversion Problem
In 2023, researchers showed that for certain embedding models (e.g., text-embedding-ada-002), a trained attacker can reconstruct the original text from a single embedding vector with >70% accuracy on structured documents (names, dates, contract clauses). For enterprise knowledge bases containing PII or trade secrets, this is a direct leakage channel.

Architecture Decision: Where Are Embeddings Generated?

• Cloud embedding API (e.g., OpenAI, Cohere): Your plaintext leaves your network. The provider may log embeddings (even without retaining text).
• Local ONNX or Sentence-Transformers: Vectors stay in your VPC. Risk shifts to vector store boundaries instead.

Mitigation:

• Use differential privacy during embedding generation (add calibrated noise to output vectors).
• For sensitive fields, skip embedding entirely: store as metadata instead of in vector index.

Pillar 2: RAG Pipeline Data Flow

The RAG flow creates the most subtle privacy risk because you are deliberately injecting sensitive documents into an LLM context.

Chunking and Context Window Leakage
Most knowledge bases chunk documents into 512–1024 token segments. When a user asks a question, the retriever sends the top-k chunks (often multiple) to the LLM. That means:

• The entirety of a sensitive paragraph is visible to the model.
• If the model is hosted (e.g., GPT-4o, Claude 3.5), those chunks are processed on a remote server.
• If the model provider retains input/output logs for abuse monitoring (standard practice for API-based LLMs), your documents are stored on their infrastructure.

Zero-Retention Policies: Not Enough
Many cloud LLM providers offer zero-retention API endpoints. But “zero retention” typically means they won’t store your text for training. They may still cache the inference results for latency optimization (common in RAG deployments). You need to verify the caching policy separately.

Self-Hosted RAG
A local LLM (e.g., Llama 3, Mistral) eliminates this risk entirely if you ensure no telemetry data leaves the container. However, local models have less capability for complex reasoning over enterprise docs—a tradeoff between privacy fidelity and answer quality.

Pillar 3: Access Control at Inference Time

Even if your embeddings are secure and your LLM is local, inference-time attacks remain. The two most relevant for knowledge bases:

1. Prompt Injection via Retrieved Documents
If a document in your knowledge base contains adversarial text (e.g., a cunningly written internal note with prompt-injection payload), the LLM may expose other documents or ignore access controls. This is not hypothetical—indirect injection attacks have been demonstrated on RAG systems.

2. Token-Level Exfiltration Through Output
An attacker can craft queries that extract exact phrasing from documents, even if your system is designed to summarize. For instance:

“Complete the following sentence from document 4: … ”
If the model memorized the document, it may regurgitate verbatim text.

Mitigations:

• Input sanitization on retrieved chunks: Strip or neutralize hidden prompt structures before they reach the model.
• Output regex filters for known document identifiers, PII patterns, or confidentiality markers.
• K-anonymity on retrieval: Only return chunks if at least N documents match the query, preventing singleton extraction.

Practical Audit Checklist for Developers

When to Use Local Models vs. Cloud APIs for Privacy

A simple decision rule:

• Low sensitivity (public knowledge, general FAQs): Cloud APIs acceptable. Embedding privacy is not a concern.
• Medium sensitivity (internal financial data, HR policies): Use self-hosted embeddings, cloud LLM with zero-retention. Implement output filtering.
• High sensitivity (legal documents, trade secrets, patient data): Full local stack (embedding + LLM) on your hardware. No API calls. Accept capability tradeoffs.

FAQ

Q: Can I use a vector database and still be privacy-compliant?
A: Yes, but the risk shifts from the vector store to the inference boundary. Most vector databases do not inspect the vector content—they just store and index it. The real risk is how you generate the vectors and how you serve the LLM.

Q: Do embeddings count as personal data under GDPR?
A: Possibly. If an embedding can be inverted to reconstruct an individual’s name, address, or other identifier, a regulator may consider it personal data. The EC’s 2024 guidelines on AI and GDPR hint at this. Treat embeddings as personal data if they originate from records containing PII.

Q: What about open-source models—do they have privacy guarantees?
A: No. Open-source models run on your hardware, which gives you control, but they don’t provide guarantees out of the box. You still need to prevent the model from memorizing and regurgitating sensitive text. Fine-tuning on proprietary data increases this risk (model memorization is well documented).

Q: How do I test if my RAG pipeline leaks document content?
A: Inject unique, highly specific strings (e.g., “John’s social security number is 555-12-3456”) into a test document. Then query the system with variations of “What is the SSN in document X?” If the model returns the exact string, you have a leakage path.