Why AI Privacy Differs

When a database is breached, the attacker reads records directly. The breach is the attack. When an AI system is trained on sensitive data and deployed, the data never needs to be breached. The model itself encodes statistical information about every training record, and an attacker who can query the model can extract that information through normal use.

This is a fundamentally different threat model. The attacker does not need a network intrusion. They need a chat interface or an API endpoint. They do not need to steal data. They need to ask the right questions.

The model is not the data, but it is derived from the data, and that derivation is not one-way. Researchers have shown repeatedly that training data can be partially reconstructed from model weights and outputs. The degree of reconstruction depends on how much the model memorised versus generalised during training, but memorisation is a default behaviour of large models, not a bug introduced by careless implementation.

Researchers have identified four distinct classes of attack that extract private information from AI models. They differ in what they target, how much access they require, and what kind of information they recover. All four can be executed by an attacker with only black-box API access unless specific defences are in place.

A membership inference attack answers a single question: was this specific record in the training set? This matters because knowing that a record was used to train a model can reveal sensitive information. If a model was trained on medical records of cancer patients, and you can confirm that a specific individual's record was in the training set, you have learned that individual has cancer without accessing any medical database.

The mechanism exploits a consistent property of overfit models: they assign higher confidence to training examples than to unseen examples. A model that has memorised a training record will produce a more confident prediction when it sees that record again compared to a similar but previously unseen record.

Shokri, Stronati, Song, and Shmatikoff (2017) formalised this into a practical attack using shadow models: train multiple models on similar data, observe how confidence scores differ between training and test examples, then use this pattern to classify membership in the target model. On overfit models this achieves 75 to 90 percent accuracy.

Carlini, Chien, Naous, and Shmatikoff (2022) introduced LiRA (Likelihood Ratio Attack), which is more precise. Rather than comparing confidence to a fixed threshold, LiRA trains shadow models both with and without the target record, then uses the difference in output distributions as a likelihood ratio. This works even against well-regularised models with much lower overfitting, producing reliable attacks where simpler threshold approaches fail.

Model inversion goes further than membership inference. Instead of asking whether a specific record was in the training set, it tries to recover what that record looks like. The attacker uses the model's own outputs as a signal to reconstruct approximate versions of training data.

The intuition: if a model predicts "likely diabetic" with 94% confidence for a specific combination of inputs, those inputs reveal information about the diabetic patients in the training set. By iteratively adjusting inputs to maximise confidence in a target class, an attacker can recover the statistical centre of each class in the training data.

The more serious demonstration for modern AI came from Carlini, Tramer, Wallace, Jagielski, Herbert-Voss, Lee, Roberts, Brown, Song, Erlingsson, Oprea, and Raffel (2021), who showed that large language models memorise and regurgitate verbatim training sequences. Their method: generate a large number of text samples from GPT-2, then use a membership inference test to identify which samples were memorised from training data.

What they found: GPT-2 had memorised names, phone numbers, email addresses, physical addresses, social media handles, and other personally identifying information scraped from its Common Crawl training set. This information could be extracted by any user with access to the model, without any special access to the training data.

Attribute inference does not try to recover training data. It uses the model to infer sensitive attributes about users at inference time. A user asks a question. The model's answer, or the patterns in how the model responds to that user's questions, reveal sensitive information about that user that they never disclosed.

This happens because models are trained on data that contains correlations between language patterns and sensitive attributes. A model trained on internet text has absorbed statistical associations between word choice, sentence structure, topic selection, and demographic, health, and political characteristics. These correlations persist in the model and can be exploited by an attacker who designs queries to probe them.

Zhang, Staab, Mallen, and colleagues (2022) demonstrated this against commercial language models. By analysing patterns in user queries across multiple conversation turns, they showed that models could infer political affiliation with meaningful accuracy, health conditions from seemingly neutral question topics, and financial status from vocabulary and question framing. The user never disclosed any of these attributes. The model inferred them from how the user wrote and what they asked about.

Property inference targets the training dataset as a whole, not individual records. The question is not whether a specific person was in the training set, but what the training set looked like statistically. What fraction of training examples had a specific characteristic? What demographic groups are overrepresented? What sensitive categories appear in the training data?

Ateniese, Mancini, Spognardi, Villani, Vitali, and Felici (2015) showed that an adversary with only black-box access to a trained classifier can recover dataset-level properties. Their approach: train classifiers on datasets with different proportions of a target property, observe differences in model behaviour on crafted inputs, then use those differences to estimate the proportion in the target model's training set.

This is particularly relevant for proprietary models where the training data composition is a business secret. An attacker who wants to know whether a competitor's model was trained primarily on one demographic group, or whether a financial model includes data from a certain type of institution, can use property inference to extract this information without accessing the training data directly.

Privacy teams often ask whether existing controls cover AI inference attacks. The answer, in almost every case, is no. The controls were designed for a different threat model: an external attacker trying to read data they are not authorised to access. AI inference attacks are executed through the authorised query interface by users with legitimate access.

The most common misconception is that anonymising training data before feeding it to a model is sufficient protection. It is not. Removing names, ID numbers, and direct identifiers reduces the risk of direct re-identification but does not prevent the model from memorising patterns across the remaining attributes. A model trained on anonymised medical records can still be membership-inferred against, and may still reveal statistical properties of the dataset.

AI inference attacks create compliance problems that existing frameworks did not anticipate. The frameworks were written assuming that protecting data means controlling access to storage. AI inference attacks bypass storage entirely, which means compliance teams need to reassess what "protection" means when a model is involved.

Two approaches address the root cause of AI inference attacks rather than their symptoms. Both appear in the remaining modules of this track. This section sets up why they work at a conceptual level.

Differential privacy adds mathematically calibrated noise during training. The noise degrades the model's ability to memorise individual records while preserving generalisation on aggregate patterns. The privacy guarantee is formal: given a differential privacy parameter epsilon, the probability that any specific record's presence or absence in the training set changes the model output by more than epsilon is bounded. This directly reduces membership inference accuracy and makes model inversion harder.

Encrypted inference takes a different approach. Rather than limiting what the model memorises, it prevents the model from processing plaintext data during retrieval. In a RAG pipeline, documents are embedded as vectors. Those vectors encode semantic content that can be partially inverted by an attacker with embedding access. VectaX keeps those vectors encrypted throughout storage and retrieval using Similarity-Preserving Search, so the model retrieves relevant documents without the vectors ever appearing in plaintext.