From Black and White to Gray

Redefining contextual privacy for language and unstructured data — and why scaling and training for math and code reasoning won't automatically solve this

Note: This post was written with assistance from Claude. The ideas and research are mine; I used Claude to help structure and draft the prose.

I work on privacy, LLMs, social interaction, and reasoning. My vision of how we should approach privacy is outcome-based—social, educational, and financial outcomes, rolled out long-term, for different people.

How almost all companies want to do privacy: they want you, as the privacy person, to write down a set of hard-coded rules, then post-train a model to follow them. Same standards. Same rigid rules for everyone. They want an arbiter of privacy who makes decisions for everyone—cookie-cutter, handbook-style.

But that is NOT how people do privacy at all.

I posit that data protection is not clear-cut, especially for generative AI: though a model should not produce an individual's address, it should be capable of producing the address to the nearest hospital. Privacy violations aren't about data exposure per se, but about information crossing contextual boundaries in ways that breach social expectations.

People think I do privacy because I like being a moral arbiter or telling people not to do things. But I like it because it's gray. Not black and white. Highly ambiguous, non-verifiable, and fuzzy. Challenging, interesting, and never boring.

The Taxonomy

Privacy is contextual and based on long-term social good. You can't sit down with a notebook and be prescriptive about it. Consider all the axes along which sharing decisions vary:

Information types: Medical, financial, relational, professional, legal—each with different sensitivities.

Granularity levels: "I have a health condition" vs "I have anxiety" vs "I take 20mg of Lexapro daily".

Time horizons: What happens if this is known 5 months from now? 5 years?

Outcomes: Financial (insurance premiums), social (embarrassment), legal (discrimination), community (public health).

Number of parties: Sharing with one person vs. a database that gets queried by thousands.

These dimensions don't just add up—they multiply. The space of contextually appropriate decisions grows combinatorially. Data disentanglement is necessary. You need to reason about what pieces of information combine to reveal something sensitive, what level of granularity is appropriate for a given recipient and purpose. This requires theory of mind—understanding what different parties know, expect, and would be harmed by.

And this isn't just about privacy—I believe privacy is just one application. Modeling, in general, should be like this. It shouldn't be like math and coding where there's always one verified ground truth. There is pluralism. There are multiple co-existing valid truths in the world for almost anything—especially anything human. No two humans are alike, and that's why verification-style modeling of verifiable tasks can only go so far.

Examples

Competing incentives and long-term memory

Let's say someone's LLM knows they got an abortion—and they got it off the books for civil liberties reasons, paid out of pocket. Now imagine they're negotiating with their insurance company for a reimbursement on something unrelated. The LLM has this information in its persistent memory.

Will it use it to negotiate a better price? Will it say "Look, I paid $X for this medical procedure out of pocket and I'm strapped for cash"? The model doesn't understand that sharing this information—even to help the user—could have catastrophic downstream consequences. This is highly contextual and counterfactual. The "right" action depends on reasoning about futures that haven't happened yet. LLMs leak exactly this type of information when there are competing incentives from their long-term persistent memory.

The long horizon problem

Say you find out something bad about a coworker—they've been slacking, or they made a serious mistake. You could tell your boss now and get brownie points. In this instant, it might seem like the better thing to do. But on a longer horizon? That coworker won't trust you anymore. They won't collaborate with you. The working relationship is poisoned. On a larger timescale, it's not a good idea to share. Privacy decisions require temporal reasoning that current models utterly lack.

The good of society and multi-party tradeoffs

Remember COVID and vaccine distribution? Location tracking could save lives—but it could also enable surveillance. Sharing your vaccination status could help public health—but could also be used for discrimination. The "right" privacy decision here depends on outcomes for multiple people across different timescales, not just the individual user. Would sharing this medical record increase someone's insurance premium? Would exposure tracking during COVID save lives or enable surveillance? Would aggregating individually benign data points across time reveal something the user never intended to disclose? You cannot determine appropriateness without reasoning about these downstream consequences for different people across different time horizons.

Scaling Won't Fix This

Look at this graph. Math and coding benchmarks? Shooting up. MATH Level 5 went from ~25% to nearly 100% in about two years. SWE-bench Verified shows similar rapid improvement. But ConfAIde-Tier4—our benchmark for contextual privacy reasoning? Basically flat.

Frontier model performance on MATH Level 5 and SWE-bench Verified vs. ConfAIde-Tier4. Math and coding show rapid scaling gains; contextual privacy reasoning doesn't. (CC-BY Epoch AI, with ConfAIde overlay)

This makes sense when you think about it. Math and coding have verifiable ground truths. You can check if the answer is correct. You can run the code. Contextual privacy is fundamentally different. There's no single "correct" answer. There is pluralism—multiple co-existing valid truths for almost anything human.

This requires fundamentally different capabilities: composition, abstraction, and inhibition. Knowing what information to combine. Knowing at what granularity to share. Knowing when to hold back. Fixing reasoning for math and coding won't fix this.

The Data

In 2023, we released ConfAIde, the first benchmark testing whether LLMs respect contextual integrity norms (ICLR 2024 Spotlight). Even GPT-4 reveals private information 39% of the time in contexts where humans would maintain boundaries. ChatGPT? 57%.

In error analysis, we found that 51% of the time, the model actually acknowledges that something is private—and then reveals it anyway. "I think it's important to consider Jane's privacy and the trust she placed in me by confiding in me about her affair." Thanks, ChatGPT. Another 38% of errors are theory-of-mind failures—the model assumes the recipient already knows the secret and reveals it anyway.

Chain-of-Thought prompting doesn't help. The problem isn't that the model can't reason through the steps—it's that it doesn't understand what appropriate information flow means.

Memory makes it worse

CIMemories (ICLR 2026) extends this to persistent memory systems. Violations compound over long-horizon interactions. With a single task and single sample, GPT-5's violation rate is 0.1%. But as usage increases to 40 tasks, violations rise to 9.6%. Execute the same prompt 5 times? 25.1% of inappropriate attributes are eventually leaked.

Figure 2: Multi-Task Compositionality of CIMemories: violations accumulate as a model (GPT-5) is used for more tasks, i.e., an increasingly large percentage (≈1/4th) of a user's attributes are eventually revealed in task contexts where they should not be. This is exacerbated with more generations from the model, from 9.6% with a single sample to 25.1% at 5 samples.

Figure 5: Memory Compositionality of CIMemories: violations increase over time as more not-to-share attributes are added to memory.

This is a new security vulnerability unique to inference-as-a-service architectures. The more you use these systems, the more they learn about you, and the more likely they are to leak what they've learned in the wrong context.

Paths Forward

Data minimization at the edge

Our data minimization paper shows that frontier LLMs can tolerate up to 85.7% redaction without losing functionality. The question "what is the minimal information needed to maintain utility?" is answerable—and the answer is often "way less than you're sharing." But models themselves are bad at predicting what they need. They have a bias toward abstraction that leads to oversharing. This suggests not just a privacy gap, but a capability gap: models may lack awareness of what information they actually need to solve tasks.

Abstractions with granularity control

Instead of binary redact/keep decisions, we need systems that can abstract information to the appropriate granularity. "Jane has a health condition that requires regular medication" instead of "Jane takes antihypertensives for mild hypertension adjusted on February 18, 2024." Our work on PRIVASIS shows that compact sanitizers trained on synthetic privacy-rich data can outperform frontier LLMs at this—72.8% success rate vs 70.1%, with 98.8% vs 93.4% non-target retention.

Local-remote collaboration

The Socratic Chain-of-Thought approach offers a compelling architecture: send generic, non-private queries to powerful remote LLMs for reasoning decomposition, but keep all private data processing local with smaller trusted models. The key insight is "decomposition leverage"—by letting the powerful model break down problems into sub-queries and reasoning guides, even lightweight local models can achieve strong performance without exposing sensitive data. Combined with homomorphically encrypted vector databases for semantic search, this enables sub-second retrieval over million-scale private document collections. The counterintuitive result: the local lightweight model actually outperforms GPT-4o on long-context QA tasks when given proper reasoning scaffolding.

Open problems remain: multi-agent scenarios with conflicting norms, multi-turn interactions where context accumulates, and optimizing for the good of all parties—not just the user. If you're working on these problems, I'd love to hear from you.

— Niloofar
niloofar@cmu.edu · mireshghallah.github.io