Model Merging: LoRA, SLERP & Combining Fine-Tuned Models

#1. LoRA from First Principles

#The Core Decomposition

For a pre-trained weight matrix W₀ ∈ ℝ^(d×k), LoRA learns a residual:

W = W₀ + ΔW = W₀ + A · B

Where:

• A ∈ ℝ^(d×r) — randomly initialized, trained
• B ∈ ℝ^(r×k) — initialized to zero (so ΔW = 0 at step 0)
• r ≪ min(d, k) — the rank, typically 4–64

#Rank Selection: r = 8 vs r = 64

Rank is the core tradeoff knob:

Higher rank captures more expressive updates but approaches full fine-tuning cost. Empirically, r = 8 matches full fine-tuning on most instruction-following tasks while using 0.1% of the parameters.

#Alpha Scaling

• α = r: no scaling — unit effective LR
• α = 2r: amplifies adapter contribution (common for domain adaptation)
• α = 1: dampens adapter (common when base model is already close)

#Which Layers to Adapt

Not all layers benefit equally from LoRA:

Most implementations default to adapting only Q and V projections, which covers 80% of LoRA's effectiveness.

#2. QLoRA: 4-Bit Fine-Tuning for Massive Models

LoRA still requires the frozen base model to fit in GPU memory — a 65B model needs ~130 GB in fp16, far beyond a single A100 (80 GB). QLoRA (Dettmers et al., 2023) solves this with aggressive quantization layered under LoRA adapters.

#NF4: NormalFloat 4-bit

Standard int4 quantization distributes bit patterns uniformly across the value range, wasting precision in the distribution tails. NF4 (NormalFloat 4) is information-theoretically optimal for normally distributed weights — it places quantization levels to minimize expected squared quantization error for Gaussian-distributed values.

The 16 NF4 levels are positioned at the 1/17, 2/17, ..., 16/17 quantiles of a unit normal distribution. For transformer weights, which are approximately N(0, σ²), this recovers ~0.1% more accuracy than symmetric int4 at the same bit width.

#Double Quantization

The quantization constants themselves (one per block of 64 weights) are stored in fp32. For a 65B model, these constants total ~2 GB — not free. QLoRA quantizes the constants too (a second round of quantization), saving ~0.5 GB with negligible accuracy loss. This is "double quantization."

#Paged Optimizers

#The Result

QLoRA makes Llama-3 65B fine-tuning fit on a single 80 GB A100. On consumer hardware (24 GB RTX 4090), it enables 13B fine-tuning that would otherwise need 4× A100s.

#3. PEFT Methods Compared

Parameter-Efficient Fine-Tuning (PEFT) is the umbrella for all methods that fine-tune a small number of parameters while keeping the base model frozen. LoRA is the dominant approach, but three others appear frequently in papers and production systems.

#IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations)

Instead of learning additive weight residuals, IA³ learns element-wise scaling vectors that multiply activations at specific points (key, value, and FFN intermediate). With only 3 vectors per transformer layer, IA³ uses 10–100× fewer parameters than LoRA but achieves competitive performance on classification tasks. It does not merge cleanly into weights (must be applied at inference), adding a small latency overhead.

#Prefix Tuning

Prefix tuning prepends k trainable "virtual tokens" to the key and value sequences of every attention layer. These prefix vectors are optimized while the model is frozen. At r = 10 (10 prefix tokens), it adds ~0.1% parameters but:

• Cannot be merged into base weights (inference overhead)
• Struggles on generation tasks with long outputs
• Outperforms LoRA on some structured prediction tasks

#Prompt Tuning

Prompt tuning adds trainable embeddings only to the input layer (not every attention layer). With fewer parameters than prefix tuning, it's the most parameter-efficient PEFT method — and the most sensitive to initialization. Performance degrades sharply below ~7B parameters; for smaller models, LoRA is consistently better.

#Comparison Table

#4. Model Merging

Once you have multiple LoRA adapters (or multiple full fine-tunes), merging combines their capabilities without serving multiple models. The key insight: neural network weights live in a high-dimensional space; models fine-tuned from the same base often lie in a "linear mode connectivity" basin where interpolation produces valid models.

#Weight Averaging (Model Soup)

The simplest merge: average corresponding weights across models.

W_merged = (1/n) Σᵢ Wᵢ

Wortsman et al. (2022) showed that averaging fine-tuned variants of the same architecture improves accuracy by 2–3% over any single model on ImageNet. This "model soup" works because averaging in weight space approximates ensemble averaging in function space — but only when models share a base and the loss landscape between them is convex (they're in the same "basin").

#SLERP (Spherical Linear Interpolation)

Linear interpolation through weight space can exit the high-probability region of the loss landscape. SLERP interpolates along the surface of a hypersphere — respecting the "norm" of each weight vector rather than pulling toward the origin.

For two weight vectors w₀ and w₁ and interpolation factor t ∈ [0, 1]:

SLERP(w₀, w₁, t) = sin((1-t)θ)/sin(θ) · w₀ + sin(tθ)/sin(θ) · w₁

where θ = arccos(w₀·w₁ / (|w₀||w₁|)).

SLERP produces smoother interpolation paths than linear averaging for high-dimensional weight vectors, resulting in 5–15% better merged model quality on benchmarks. It's the default method in mergekit for two-model merges.

#TIES-Merging (Trim, Elect, Merge)

When merging more than two models, individual weight updates may conflict — one fine-tune pushes a weight positive, another pushes it negative, and averaging cancels both improvements. TIES-Merging handles this in three steps:

1. Trim: For each adapter ΔWᵢ = Wᵢ - W₀, zero out the bottom k% of magnitudes (keep only the top 20% by default). Small updates are noise.
2. Elect: For each weight position, take a majority vote on the sign of the update across all adapters. The elected sign is the consensus direction.
3. Merge: Average only the adapters that agree with the elected sign. Discard conflicting updates.

This typically recovers 80–90% of each model's individual capability in the merged result, versus 50–70% for naive averaging when tasks are diverse.

#DARE (Drop and Rescale)

DARE takes a simpler probabilistic approach: randomly drop a fraction p of the delta parameters (ΔW = W - W₀), then rescale the survivors by 1/(1-p) to preserve expected magnitude.

mask = torch.bernoulli(torch.ones_like(delta_W) * (1 - p))
delta_W_dare = mask * delta_W / (1 - p)

DARE works because fine-tuned deltas are sparse in practice — most of the "meaningful" update is concentrated in a small fraction of parameters. Dropping 90–99% of delta parameters and rescaling often preserves 95%+ of task performance, making it a powerful tool for reducing interference before TIES-Merging.

In practice, DARE is typically applied before TIES — "DARE-TIES" is the dominant state-of-the-art merging strategy for multi-model combinations as of 2025.

#5. When Merging Works and When It Fails

#Conditions Favorable to Merging

• All models fine-tuned from the same base checkpoint (same architecture, same base weights)
• Tasks are complementary rather than contradictory (e.g., coding + math + reasoning)
• Fine-tunes used similar data volumes (prevents one adapter dominating)
• Models trained with similar learning rates and regularization

#Common Failure Modes

#The "Merge Coefficient" Heuristic

When using SLERP, the t parameter controls how much each model contributes. A practical heuristic:

• Start with t = 0.5 (equal blend)
• Evaluate on your primary task benchmark
• Shift t toward 0 or 1 by 0.1 increments to rebalance
• Final t is rarely outside [0.3, 0.7] for well-matched models

#6. Code: Merging Two LoRA Adapters with SLERP

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel

def slerp(w0: torch.Tensor, w1: torch.Tensor, t: float) -> torch.Tensor:
    """Spherical linear interpolation between two weight tensors."""
    # Flatten to 1D for dot product, then reshape back
    orig_shape = w0.shape
    w0_flat = w0.flatten().float()
    w1_flat = w1.flatten().float()

    # Compute angle between the vectors
    cos_theta = torch.dot(w0_flat, w1_flat) / (
        torch.norm(w0_flat) * torch.norm(w1_flat) + 1e-8
    )
    cos_theta = cos_theta.clamp(-1.0, 1.0)
    theta = torch.acos(cos_theta)

    # If vectors are nearly parallel, fall back to linear interpolation
    if theta.abs() < 1e-4:
        return ((1 - t) * w0 + t * w1).reshape(orig_shape)

    sin_theta = torch.sin(theta)
    coeff0 = torch.sin((1 - t) * theta) / sin_theta
    coeff1 = torch.sin(t * theta) / sin_theta

    return (coeff0 * w0_flat + coeff1 * w1_flat).reshape(orig_shape)


def merge_lora_adapters_slerp(
    base_model_name: str,
    adapter_path_a: str,
    adapter_path_b: str,
    output_path: str,
    t: float = 0.5,
):
    """
    Merge two LoRA adapters trained on the same base model using SLERP.
    
    Args:
        base_model_name: HuggingFace model ID for the base model
        adapter_path_a: Path to first LoRA adapter (peft format)
        adapter_path_b: Path to second LoRA adapter (peft format)
        output_path: Where to save the merged model
        t: Interpolation factor (0.0 = all model A, 1.0 = all model B)
    """
    print("Loading base model...")
    base_model = AutoModelForCausalLM.from_pretrained(
        base_model_name,
        torch_dtype=torch.float32,  # Use fp32 for clean arithmetic
        device_map="cpu",
    )

    # Merge adapter A into a copy, extract its weights
    print("Materializing adapter A weights...")
    model_a = PeftModel.from_pretrained(base_model, adapter_path_a)
    model_a = model_a.merge_and_unload()  # Fuses A·B into W₀

    # Merge adapter B into another copy
    print("Materializing adapter B weights...")
    base_model_b = AutoModelForCausalLM.from_pretrained(
        base_model_name,
        torch_dtype=torch.float32,
        device_map="cpu",
    )
    model_b = PeftModel.from_pretrained(base_model_b, adapter_path_b)
    model_b = model_b.merge_and_unload()

    # SLERP between corresponding weight matrices
    print(f"SLERP interpolation at t={t}...")
    state_dict_a = model_a.state_dict()
    state_dict_b = model_b.state_dict()
    merged_state_dict = {}

    for key in state_dict_a:
        w_a = state_dict_a[key]
        w_b = state_dict_b[key]

        if w_a.dtype in (torch.float32, torch.float16, torch.bfloat16):
            # SLERP for floating-point weight matrices
            merged_state_dict[key] = slerp(w_a, w_b, t).to(w_a.dtype)
        else:
            # For embeddings or integer tensors, use model A's values
            merged_state_dict[key] = w_a

    # Load merged weights and save
    print("Saving merged model...")
    model_a.load_state_dict(merged_state_dict)
    model_a.save_pretrained(output_path)
    print(f"Merged model saved to {output_path}")


# Example usage:
# merge_lora_adapters_slerp(
#     base_model_name="meta-llama/Meta-Llama-3-8B",
#     adapter_path_a="./lora-adapter-coding",
#     adapter_path_b="./lora-adapter-math",
#     output_path="./merged-coding-math",
#     t=0.5,
# )


# ── DARE preprocessing before TIES ─────────────────────────────────────────

def dare_delta(base_weights: dict, finetuned_weights: dict, p: float = 0.9) -> dict:
    """
    Apply DARE (Drop and Rescale) to a delta weight dict.
    
    Args:
        base_weights: state_dict of the base (frozen) model
        finetuned_weights: state_dict of the fine-tuned model
        p: fraction of delta parameters to randomly drop (0.9 = drop 90%)
    
    Returns:
        Pruned and rescaled delta dict
    """
    dare_deltas = {}
    for key in finetuned_weights:
        if key not in base_weights:
            dare_deltas[key] = finetuned_weights[key]
            continue

        delta = finetuned_weights[key].float() - base_weights[key].float()

        if delta.dtype in (torch.float32,):
            # Random binary mask: keep each element with probability (1-p)
            mask = torch.bernoulli(torch.ones_like(delta) * (1 - p))
            # Rescale survivors to preserve expected magnitude
            dare_deltas[key] = (mask * delta / (1 - p)).to(finetuned_weights[key].dtype)
        else:
            dare_deltas[key] = finetuned_weights[key]

    return dare_deltas