Forward-Only01
Observe component activations
Decompose a weight matrix with SVD. The sum of the activations induced by its rank-1 components reconstructs the standard output.
Y = XWT = ∑i Y(i)
Forward activations provide the search signal.
NeurIPS 2026/Efficient LLM Adaptation
Efficient LLM Adaptation
with Forward-Only Passes
Use the geometry of activations to find the weights worth reducing.
Efficient adaptation, powered by forward passes.
1 Lund University, Sweden · 2 Google DeepMind, USA
* Correspondence: baichuan.huang@eit.lth.se · † Advisory contribution
A forward pass reveals what to reduce.
01 / OBSERVE
Decompose. Then inspect.
W = UΣVT
Each singular component induces an activation. Together, they recover the original forward pass.
02 / SELECT
The singular spectrum
Larger singular valuesSmaller singular values →
03 / ADAPT
Y = XWT
Standard forward passInspect angle concentration to identify weight matrices with a high dispersion score.
Illustration, not experimental data. The retained rank is selected on validation data.No backpropagated gradients in the adaptation search.
36.7×
Average search speedup
GPT-J · relative to LASER (std)
1.9×
Lower peak memory
GPT-J · relative to LASER-Grad
+11.3 pp
Average accuracy gain
GPT-J · over direct inference
57B
Parameters in largest model
Qwen2-57B-A14B · MoE
Table 2: GPT-J, eight datasets, hierarchical variant. Speedup is derived from computational complexity; peak memory is analytically estimated. Model scale: Table 4.
01 / Overview
Adaptation can start
with the forward pass.
Large-scale LLM inference motivates a simple question: can the inference process itself enable efficient task-specific adaptation? This work proposes a forward-only approach that uses activations to identify which weight matrix merits rank reduction.
The method measures the angle concentration of activations induced by each singular value decomposition (SVD) component and compares it with that of the full hidden states. This exposes components with a dispersing effect and guides a search for promising weight matrices. Higher-order components with smaller singular values are then selectively removed, while lower-order components are retained.
Across diverse tasks, the approach offers competitive accuracy with lower estimated peak memory than gradient-based search and lower computational cost than exhaustive search. Extended experiments cover LLMs with up to 57B parameters, alongside applications to fine-tuning and steering prompts.
The key is knowing which weights to reduce—and using forward passes to find them.
Read the original abstract
Large language model (LLM) inference serving is undergoing rapid growth and large-scale deployment, motivating us to rethink how the inference process itself can be leveraged to enable efficient task-specific LLM adaptation. In this paper, we propose a forward-only approach for efficient LLM adaptation with forward-only passes in LLM inference serving. We exploit the angle concentration of activations induced by each singular value decomposition (SVD) component to measure its contribution—dispersed or concentrated—to the angle concentration of the hidden states. Based on this, our forward-only search then efficiently identifies the weight matrix with the highest dispersion that merits rank reduction. We then selectively remove the higher-order components and retain the lower-order components in SVD. Empirical results across diverse datasets demonstrate the competitive accuracy of our forward-only approach, while theoretical analysis shows lower peak memory and greater speedup than the gradient-based approach and greater speedup than the exhaustive search approach. The experiments further present the robustness and generalization of our forward-only approach to various LLMs with up to 57B parameters. The code is included in the supplementary material.
02 / Method
From activation geometry
to selective rank reduction.
Three steps connect a standard forward pass to a task-adapted model. The search uses angle concentration as a heuristic for selecting candidate weights.
02
Rank weights by dispersion
Compare each component’s angle concentration with the full hidden states. Aggregate negative differences among the last m components.
Δi = AC(Y(i)) − AC(Y)
score(W) = −∑i=r−m+1…r min(Δi, 0)
score(W) = −∑i=r−m+1…r min(Δi, 0)
A larger score identifies a more dispersed candidate.
03
Validate the retained rank
Evaluate rank-reduced candidates. Hierarchical search uses a small validation subset before evaluating the top candidates on the full validation split.
Ŵ = UΣ̂VT
Retain the top k% singular components
Retain the top k% singular components
Validation accuracy determines the final candidate.
AC = mean pairwise cosine similarity between token activations. See §2 and Algorithm 1.Explore the empirical results ↓
03 / Results
Competitive accuracy.
A smaller adaptation budget.
Compare GPT-J across eight downstream tasks. Select a dataset and metric to inspect the accuracy–efficiency tradeoff.
Accuracy (%) · higher is better
0255075100%
Accuracy reported in Table 2. LASER and LASER-Grad use the standard validation setting; ours uses hierarchical search. Results vary by task.
All eight datasets · accuracy and search speedup
| Dataset | Baseline | LASER | LASER-Grad | Ours | Our speedup |
|---|
Accuracy in %. Speedup relative to LASER (std), from the computational-complexity analysis. Table 2.
Measured runtime and memory · RoBERTa core computation
FEVER · average sequence length 27
Memory change: 130 MB vs 343 MB.
Average execution time: 1.00 s vs 3.34 s.
Bios Profession · average sequence length 100
Memory change: 235 MB vs 1,062 MB.
Average execution time: 1.31 s vs 5.55 s.
Ours vs LASER-Grad, Appendix K.1 / Table 22. These measurements cover the core angle-concentration or gradient calculation, rather than end-to-end adaptation time or total peak memory.
04 / Generalization
Across architectures.
Up to 57B.
The forward-only approach extends to dense, mixture-of-experts, and multimodal model architectures. Table 4 reports text-task results on Epistemic Reasoning using the standard variant.
| Model | Baseline | LASER | Ours | Speedup |
|---|---|---|---|---|
| LLaMA2-7B | 44.8 | 63.4 | 63.5 | 6.4× |
| DeepSeekV1-7B* | 55.1 | 61.4 | 61.8 | 29.6× |
| Mistral-7B | 54.8 | 61.4 | 61.4 | 9.1× |
| Qwen2-7B* | 51.4 | 58.3 | 58.3 | 27.6× |
| Gemma-7B | 38.3 | 60.2 | 61.4 | 11.1× |
| Gemma4-E2B-it | 48.5 | 60.5 | 60.2 | 11.6× |
| Qwen2-57B-A14B* | 39.9 | 62.1 | 62.1 | 13.9× |
Accuracy in %. Speedup is complexity-derived, relative to LASER.
* Chat / instruction versions: DeepSeekV1-7B-Chat, Qwen2-7B-Instruct, Qwen2-57B-A14B-Instruct. Source: Table 4.
* Chat / instruction versions: DeepSeekV1-7B-Chat, Qwen2-7B-Instruct, Qwen2-57B-A14B-Instruct. Source: Table 4.
Figure 4 / Interactive
A warm start for fine-tuning
On RoBERTa / Bios Gender, fine-tuning the selected matrix trains 1.89% of parameters and reaches 95.1 ± 0.01% accuracy after 20 epochs. Fine-tuning all input FFN matrices trains 22.7% of parameters and reaches 95.6 ± 0.02% after 10 epochs (Table 15). Figure 4 below compares warm start, cold start, and the lower bound over the first 10 epochs.
/ 10
Replay training, drag the epoch slider, or click the chart to inspect a training window.
Warm − cold: rank-reduction gainCold − lower: weight-selection gainWarm − lower: combined gain
Figure 4 · RoBERTa / Bios Gender. Curves and μ ± σ bands reconstructed from the PDF figure; displayed values are approximate. Gain areas are computed here by integrating differences between the plotted mean curves over the displayed epochs (pp·epoch). The larger gain fills are distinct from the narrow uncertainty bands.From weight updates to steering prompts
On RoBERTa / Bios Profession, the paper derives task-level steering prompts from the selected weight update. Accuracy reaches 72.0%, compared with 71.9% for the weight update. See Appendix J / Table 21.
05 / Resources
Explore the research.
Implementation
The manuscript states that code is included in the supplementary material. Contact the corresponding author for access.
Contact Baichuan Huang ↗06 / Citation
Build on this work.
NeurIPS 2026 · Conference paper
@inproceedings{huang2026foadaptllm,
title={Efficient LLM Adaptation with Forward-Only Passes},
author={Huang, Baichuan and Balashankar, Ananth and Aminifar, Amir},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
year={2026},
url={https://openreview.net/forum?id=ux9eaZ3bxm},
}