Privacy-Preserving ML: Differential Privacy & Federated Learning

#1. Why Privacy Matters in ML

Machine learning models aren't just statistical summaries — they can memorize and reproduce training data in ways that directly violate user privacy.

#Model Memorization Attacks

#Regulatory Landscape

The EU AI Act specifically classifies models trained on biometric, health, and financial data as high-risk, requiring documented privacy controls — which in practice means DP or FL certification for training pipelines.

#2. Differential Privacy

Differential privacy provides a mathematical guarantee: the output of a computation is approximately the same whether or not any single individual's data is included. This prevents inference attacks because the model genuinely doesn't "know" whether any specific record existed.

#Formal Definition: (ε, δ)-DP

A randomized mechanism M satisfies (ε, δ)-differential privacy if, for all pairs of adjacent datasets D and D' (differing by exactly one record), and all possible outputs S:

Pr[M(D) ∈ S] ≤ exp(ε) · Pr[M(D') ∈ S] + δ

• ε = 0: Perfect privacy — outputs are identical regardless of data (useless model)
• ε = 0.1: Very strong privacy — attacker gains almost no information about any individual
• ε = 1: Strong privacy — typical for sensitive healthcare applications
• ε = 8–10: Moderate privacy — common for ML models on less sensitive data
• ε = ∞: No privacy — standard training

#Gaussian Mechanism

The Gaussian mechanism adds calibrated noise to a function's output to achieve DP. For a function f: D → ℝᵈ with L2 sensitivity Δf (maximum change in output when one record changes):

M(D) = f(D) + N(0, σ² · Iᵈ)

This is the basis of all practical DP mechanisms. The key tradeoff: smaller ε → larger σ → more noise → lower model quality.

#DP-SGD: Differentially Private Stochastic Gradient Descent

Standard SGD leaks privacy through gradients — even a single gradient update can reveal whether a specific record was in the batch. DP-SGD (Abadi et al., 2016) makes each SGD step differentially private through two modifications:

Step 1 — Per-sample gradient clipping

Compute per-sample gradients gᵢ (not the batch average). Clip each to L2 norm C:

g̃ᵢ = gᵢ / max(1, |gᵢ|₂/C)

This bounds the sensitivity: no single sample can move the gradient by more than C.

Step 2 — Gaussian noise injection

Add noise calibrated to the clipping bound:

g_noisy = (1/B) · (Σᵢ g̃ᵢ + N(0, σ²C²I))

Where B is the batch size and σ is the noise multiplier.

Privacy accounting — the moments accountant

Composing T DP-SGD steps doesn't multiply the cost linearly. The moments accountant (Rényi DP framework) tracks the exact privacy cost of composition, yielding tighter bounds than naive composition. For a dataset of size n, sampling probability q = B/n, noise multiplier σ, and T steps:

ε_total ≈ q · √(2T · ln(1/δ)) / σ  (simplified Gaussian DP bound)

#Utility-Privacy Tradeoff

DP adds real accuracy cost. Typical impact on image classification:

The 2–5% accuracy cost at ε=3–8 is generally acceptable for high-value regulated applications. Below ε=1, the utility penalty becomes prohibitive for most production use cases.

#Privacy Budget Tracking

DP budget depletes every time you query the data — training, validation, hyperparameter tuning, and evaluation all consume budget. Best practices:

1. Fix ε, δ, and δ at the start — treat them as immutable design decisions with legal sign-off
2. Budget tuning separately — use public data or synthetic data for hyperparameter search
3. Track with a privacy ledger — log every operation that touches private data with its ε contribution
4. Never reuse a depleted budget — retrain from scratch rather than continue training past ε limits

#3. Federated Learning

Federated learning (FL) trains a shared model across many clients (phones, hospitals, bank branches) without centralizing raw data. Each client trains locally and shares only model updates (gradients or weight deltas).

#Horizontal vs Vertical FL

Most production FL deployments are horizontal.

#FedAvg: The Core Algorithm

FedAvg (McMahan et al., 2017) is the standard FL algorithm. Each round:

1. Server broadcasts current global weights W_t to a random subset of K clients
2. Each client k runs E epochs of local SGD on its private dataset Dₖ, computing Wₖ_t+1
3. Clients send weight updates ΔWₖ = Wₖ_t+1 - W_t to the server
4. Server aggregates with weighted averaging by dataset size nₖ:

W_t+1 = W_t + Σₖ (nₖ/n) · ΔWₖ

Where n = Σₖ nₖ. This is identical to full-batch gradient descent when E=1 and all clients participate — the power of FedAvg is that E > 1 (multiple local steps) dramatically reduces communication rounds.

#Non-IID Data Challenges

In practice, client data is non-IID — each hospital's patient population, each phone's user behavior, each branch's customer demographics are all different. This breaks FedAvg's convergence assumptions:

• FedProx: Adds a proximal regularization term ||W - W_global||² to each client's loss, keeping local models close to the global model
• SCAFFOLD: Uses control variates to correct for client drift — each client maintains a local correction term
• FedNova: Normalizes local updates by the number of local steps before aggregation
• Personalized FL: Allows each client to maintain a small personal adapter on top of the global model (essentially FL + LoRA)

#Communication Compression

FL's main bottleneck is communication: a 7B model sends 28 GB of weight updates each round. With 1,000 hospital clients each round, that's 28 TB of data transfer per global step. Compression is mandatory:

In practice, FL pipelines combine top-k sparsification (send only the largest k% of gradient elements) with error feedback (accumulate the dropped gradients locally for the next round).

#4. When to Use What

Choosing the right privacy tool depends on the threat model, data location, and regulatory requirement:

#Decision Framework

Use Differential Privacy when

• Training data is centralized or centralizable
• The threat is membership inference or model inversion
• GDPR/HIPAA requires provable privacy guarantees for model outputs
• You can tolerate 2–10% accuracy reduction
• Examples: user behavior models, click-through prediction, medical image classifiers

Use Federated Learning when

• Data cannot leave its origin (contractual, regulatory, or practical reasons)
• Multiple organizations want to collaborate without sharing raw data
• The dataset is too large to centralize (edge devices at scale)
• Examples: keyboard prediction across 1B phones, cross-hospital disease models, cross-bank fraud detection

Use Secure Multi-Party Computation (SMPC) when

• You need provable cryptographic privacy, not just statistical privacy
• Vertical FL is required (different features, same users)
• Compliance requires zero-knowledge proofs of computation
• Trade-off: 10–1,000× compute overhead vs FL

#Healthcare Case Study: Cross-Hospital Rare Disease Model

A model for rare pediatric cancers needs data from 50 hospitals, each with 200–1,000 patient records. Centralizing data violates HIPAA BAAs and individual hospital governance rules. Solution stack:

1. Horizontal FedAvg with 50 client hospitals
2. Local DP with ε=1, δ=1e-5 on each client's gradient
3. Secure aggregation (SMPC) on the server — hospital updates are aggregated without the server seeing individual updates
4. Privacy accounting logged and auditable for IRB compliance

#5. GDPR Right to Erasure and Machine Unlearning

GDPR Article 17 gives EU citizens the "right to erasure" — the right to have their data deleted. For traditional databases, this is simple: delete the row. For ML models, it's not. A model trained on a deleted user's data still "contains" information about them.

#The Machine Unlearning Problem

ΔW ≈ H⁻¹ · ∇L(xᵢ, W)

Where H is the Hessian of the loss. Computing H⁻¹ exactly is intractable for large models, but Kronecker-factored approximations (K-FAC) make it feasible for models up to ~1B parameters.

#Practical Compliance Strategy

Regulators have not yet defined what "sufficient" unlearning means for ML models. Current best practice for GDPR compliance:

1. Document that the data was not used (before training, apply a deletion queue to training data)
2. Use SISA training if erasure requests are expected to be frequent (>1/month)
3. For LLMs, approximate unlearning is often accepted when paired with a strong privacy filter (RAG-based retrieval block that prevents the deleted user's info from surfacing)
4. Log all erasure requests and model responses for regulatory audit

#6. Code: DP-SGD with Opacus on a Simple Classifier

"""
DP-SGD training with Facebook Opacus.
Opacus wraps a standard PyTorch training loop and handles:
  - Per-sample gradient computation
  - Gradient clipping
  - Noise injection
  - Privacy accounting (Rényi DP / moments accountant)
"""

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from opacus import PrivacyEngine
from opacus.validators import ModuleValidator


# ── 1. Define a simple classifier ──────────────────────────────────────────

class TabularClassifier(nn.Module):
    def __init__(self, input_dim: int, num_classes: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, num_classes),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)


# ── 2. Prepare synthetic data ───────────────────────────────────────────────

n_samples, n_features, n_classes = 10_000, 32, 2
X = torch.randn(n_samples, n_features)
y = torch.randint(0, n_classes, (n_samples,))

dataset = TensorDataset(X, y)
# Opacus requires that batch_size is fixed (no drop_last=False with non-uniform batches)
loader = DataLoader(dataset, batch_size=256, shuffle=True, drop_last=True)


# ── 3. Initialize model and optimizer ──────────────────────────────────────

model = TabularClassifier(input_dim=n_features, num_classes=n_classes)

# Opacus validates that the model has no unsupported layers (e.g., BatchNorm).
# Replace BatchNorm with GroupNorm if needed.
errors = ModuleValidator.validate(model, strict=False)
if errors:
    model = ModuleValidator.fix(model)  # auto-replace incompatible layers

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()


# ── 4. Attach Opacus PrivacyEngine ────────────────────────────────────────

# Key hyperparameters:
#   max_grad_norm (C): clipping threshold per sample
#   noise_multiplier (σ): noise added per step, relative to C
#   target_epsilon: desired final privacy budget (computed post-hoc or pre-committed)
#   target_delta: failure probability (set << 1/n_samples)

privacy_engine = PrivacyEngine()

model, optimizer, loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=loader,
    epochs=10,
    target_epsilon=3.0,    # ε = 3: strong privacy guarantee
    target_delta=1e-5,     # δ < 1/n_samples = 1/10,000
    max_grad_norm=1.0,     # C: clip per-sample gradient L2 norm to 1.0
)

print(f"Noise multiplier σ = {optimizer.noise_multiplier:.3f}")


# ── 5. Training loop ───────────────────────────────────────────────────────

def train_epoch(model, loader, optimizer, criterion, device="cpu"):
    model.train()
    total_loss = 0.0
    total_correct = 0

    for X_batch, y_batch in loader:
        X_batch, y_batch = X_batch.to(device), y_batch.to(device)

        optimizer.zero_grad()
        logits = model(X_batch)
        loss = criterion(logits, y_batch)
        loss.backward()
        optimizer.step()  # Opacus handles clipping + noise injection here

        total_loss += loss.item() * len(y_batch)
        total_correct += (logits.argmax(dim=1) == y_batch).sum().item()

    return total_loss / len(loader.dataset), total_correct / len(loader.dataset)


for epoch in range(1, 11):
    loss, acc = train_epoch(model, loader, optimizer, criterion)

    # Query the privacy accountant — ε grows each epoch
    epsilon = privacy_engine.get_epsilon(delta=1e-5)
    print(f"Epoch {epoch:2d} | Loss: {loss:.4f} | Acc: {acc:.3f} | ε = {epsilon:.3f}")


# ── 6. Privacy budget summary ─────────────────────────────────────────────

final_epsilon = privacy_engine.get_epsilon(delta=1e-5)
print(f"\nFinal privacy guarantee: (ε={final_epsilon:.3f}, δ=1e-5)-DP")
print("Interpretation: an adversary cannot determine if any individual")
print(f"was in the training set with better than e^{final_epsilon:.1f}≈{torch.exp(torch.tensor(final_epsilon)):.1f}x prior odds.")


# ── 7. Non-private baseline for comparison ────────────────────────────────

# To quantify the privacy-utility tradeoff, train the same model without DP:
# model_base = TabularClassifier(n_features, n_classes)
# optimizer_base = torch.optim.Adam(model_base.parameters(), lr=1e-3)
# loader_base = DataLoader(dataset, batch_size=256, shuffle=True)
# ... standard training loop ...
# Compare accuracy_base vs accuracy_dp — typically 2-5% gap at ε=3