FusionLM: Model Merging
for Enterprise Finance AI

§1 Motivation

Enterprise AI teams at financial institutions have accumulated a quiet structural problem. Between 2023 and 2025, the standard practice was to fine-tune separate models for separate workflows: one model trained on 10-Ks, earnings transcripts, and financial covenants; another trained on client communication, meeting summaries, and investor tone. Both models perform well within their respective domains. Neither handles the other's domain reliably.

Any real analyst workflow — drafting a client memo that explains a DCF analysis, or writing a risk summary that is both numerically precise and accessible to a non-technical LP — requires both capabilities simultaneously. Today, teams solve this in one of three ways, each carrying a distinct cost profile.

§2 Problem Statement

The core problem is a mismatch between how models are trained and how analysts actually work. Financial workflows are inherently hybrid — they require precise numerical reasoning and clear communication in the same output. Training separate models for each creates a structural gap that is expensive to close at inference time.

The table below shows the financial services teams where this specialist-silo pattern is most prevalent and the specific failure mode each team reports.

The two-model pipeline (Figure 2, Approach A) is the most common workaround. It fails in three measurable ways.

Before committing to weight merging, four alternative architectures were evaluated. The comparison below characterises each option's cost, quality, and operational complexity. Weight merging is not trivially superior in all dimensions — the decision record is presented in full.

§3 Merge Techniques

FusionLM implements three merge algorithms, each operating on the delta between fine-tuned weights and the shared base model. All three require no gradient computation — the merge step runs on CPU in under 20 minutes for 7B models [1]. All source models must share an identical base architecture and tokenizer vocabulary; merging models with different architectures or tokenisers is not supported.

§4 Architecture

FusionLM orchestrates the full merge workflow through six stages. The system wraps mergekit (arcee-ai) [1] and adds a benchmark harness, per-layer interference profiler, and configuration recommendation engine. All merge computation runs on CPU; no GPU is required for the merge step.

Merge behaviour is fully controlled by a YAML configuration file passed to mergekit. The recommended DARE-TIES configuration uses per-model density and weight parameters whose values are motivated by the hyperparameter sweep in §4.3 and the per-layer interference profile in §7.

The density and weight parameters for the DARE-TIES merge were selected by evaluating a grid of 9 configurations against the FinQA development split. The selected configuration (density=0.6, weight_quant=0.4, weight_comms=0.6) is highlighted. The asymmetric weighting (0.4 vs 0.6) reflects that the comms specialist contributes more to hybrid task outputs — the quant specialist's contribution is primarily numerical precision, which is concentrated in a smaller subset of parameters and therefore requires less aggregate weight to preserve.

§5 Configuration Walkthrough

The walkthrough below demonstrates the FusionLM merge pipeline configuration for four scenarios. Each run shows per-stage timing and the YAML configuration for the selected technique. Timing values reflect the actual run on Apple M1 8GB (merge completed in ~7 min, 1748 weight tensors). Real MMLU benchmark results from Kaggle T4 eval are reported in §6.2. Select a scenario and run.

// select a scenario and run walkthrough

§6 Evaluation

All benchmark results presented in §6.2 are reproducible from the following exact configuration. The evaluation uses lm-evaluation-harness v0.4.2 with fixed random seed 42. Results are single-run; confidence intervals will be added in v0.3 with 5-seed variance reporting.

§7 Per-Layer Interference Analysis

Task vector interference is the primary failure mode of naive weight averaging. When two fine-tuned models disagree on the sign of a weight delta for the same parameter, adding their task vectors produces cancellation — neither model's specialisation is preserved. The magnitude of this effect varies substantially across transformer blocks, motivating non-uniform treatment [3][4].

The chart below shows the fraction of parameters in each transformer block where the quant and comms task vectors disagree in sign (sign conflict rate). Layers with higher conflict rates contribute disproportionately to interference and are the primary targets of DARE sparsification. Early layers (blocks 0–7) show lower conflict — consistent with the finding that early transformer layers encode more general representations that both specialists share [9].

This per-layer profile motivates the density=0.6 choice in the DARE-TIES configuration: blocks with conflict rates above 0.45 benefit most from 40% delta pruning before TIES sign election. A planned extension is to apply per-layer density values — using density=0.4 for the highest-conflict blocks and density=0.8 for the lowest-conflict early layers.

§8 Deployment Architecture

Production deployment of FusionLM targets Google Cloud Platform, using Vertex AI Model Serving as the inference control plane and GKE with GPU node pools as the data plane. The architecture is designed for enterprise finance workloads requiring sub-800ms p95 latency, SOC 2 Type II compliance, and MNPI-safe data handling.

Three observability planes are required for production finance inference: the infrastructure plane (latency, throughput, GPU utilisation), the model quality plane (domain drift detection, output length distribution), and the compliance plane (request logging for audit, PII detection, MNPI flag rate).

§9 Failure Mode Taxonomy

Weight merging introduces failure modes distinct from those of the pipeline architecture. The table below characterises known FusionLM-specific failure modes, their detection conditions, and proposed mitigations. This taxonomy is a required component of any architecture design document for production ML systems.

§10 Limitations and Open Problems

References

1. [1]Goddard, C. et al. (2024). "Arcee's MergeKit: A Toolkit for Merging Large Language Models." EMNLP 2024 Industry Track. arXiv:2403.13257.
2. [2]Touvron, H. et al. (2023). "LLaMA 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288. — Joint fine-tuning GPU cost estimates.
3. [3]Yu, L. et al. (2023). "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch." arXiv:2311.03099. — Introduces DARE; reports stability up to p=0.9 drop rate and per-layer interference patterns.
4. [4]Yadav, P. et al. (2023). "TIES-Merging: Resolving Interference When Merging Models." NeurIPS 2023. arXiv:2306.01708. — Introduces task vectors, sign conflict resolution, and TIES algorithm.
5. [5]Chen, Z. et al. (2021). "FinQA: A Dataset of Numerical Reasoning over Financial Data." EMNLP 2021. arXiv:2109.00122. — Primary benchmark for quantitative financial reasoning.
6. [6]Zhu, F. et al. (2021). "TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance." ACL 2021. arXiv:2105.07624. — Primary benchmark for table+text numerical QA.
7. [7]Hendrycks, D. et al. (2021). "Measuring Massive Multitask Language Understanding." ICLR 2021. arXiv:2009.03300. — MMLU Finance subset (5-shot protocol).
8. [8]Lin, S. et al. (2022). "TruthfulQA: Measuring How Models Mimic Human Falsehoods." ACL 2022. arXiv:2109.07958. — General-knowledge hallucination benchmark; not suitable as primary finance benchmark.
9. [9]Raghu, M. et al. (2017). "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability." NeurIPS 2017. arXiv:1706.05806. — Representational similarity methods; early layer sharing finding.
10. [10]Kornblith, S. et al. (2019). "Similarity of Neural Network Representations Revisited." ICML 2019. arXiv:1905.00414. — CKA methodology for cross-model representational analysis.
11. [11]Ilharco, G. et al. (2023). "Editing Models with Task Arithmetic." ICLR 2023. arXiv:2212.04089. — Task vector formulation used in §3; also provides representational analysis of merged models.
12. [12]Hu, E. et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022. arXiv:2106.09685. — LoRA adapter loading latency characterisation referenced in §2.3.
13. [13]Kwon, W. et al. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. arXiv:2309.06180. — vLLM continuous batching and KV cache management referenced in §8.