Thinking Machines - Connectionism Notify me• Sep 29, 2025essay

LoRA Without Regret

John Schulman(guest)

Get the full transcript next time Thinking Machines - Connectionism releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on John Schulman.

LoRA vs. Full Fine-Tuning Performance Parity Architectural Best Practices for LoRA

12 quotes

Concerns Raised

LoRA underperforms full fine-tuning on very large datasets that exceed the adapter's capacity.
In some settings, LoRA is less tolerant of large batch sizes than full fine-tuning.
The theoretical understanding of why the optimal LoRA learning rate is 10x that of full fine-tuning is still incomplete.

Opportunities Identified

Replacing FullFT with LoRA in most post-training scenarios to reduce cost and complexity without sacrificing performance.
Leveraging LoRA for highly efficient reinforcement learning on large language models.
Using LoRA to enable multi-tenant serving, allowing for cost-effective deployment of many customized models.
Making fine-tuning accessible on smaller hardware configurations due to LoRA's reduced memory footprint.

Key Themes

Research Findings12

Low-rank adaptation (LoRA) underperforms full fine-tuning on very large datasets that exceed the storage capacity of LoRA's parameters.

For supervised fine-tuning on small-to-medium-sized instruction-tuning and reasoning datasets, Low-rank adaptation (LoRA) performs the same as full fine-tuning.

Low-rank adaptation (LoRA) performs better when applied to all weight matrices, especially the MLP and Mixture-of-Experts (MoE) layers.

For reinforcement learning using policy gradient algorithms, Low-rank adaptation (LoRA) performs equivalently to full fine-tuning, even with ranks as low as 1.

The optimal learning rate for Low-rank adaptation (LoRA) is consistently 10 times higher than for full fine-tuning (FullFT) across both supervised learning and reinforcement learning.

Low-rank adaptation (LoRA) enables multi-tenant serving, where a single inference server can keep many adapters in memory and sample from them simultaneously in a batched way.

Inference engines vLLM and SGLang implement multi-tenant LoRA serving.

Low-rank adaptation (LoRA) can be trained on a hardware layout only slightly larger than what is used for sampling due to its lower memory usage compared to full fine-tuning.

Low-rank adaptation (LoRA) is less tolerant of large batch sizes than full fine-tuning, paying a larger penalty in loss as batch size increases.

The performance penalty Low-rank adaptation (LoRA) experiences with large batch sizes is not mitigated by increasing the LoRA rank.

Applying Low-rank adaptation (LoRA) only to attention layers underperforms applying it to all layers, even when the number of trainable parameters is matched by using a higher rank.

The 1/r scaling factor in the Low-rank adaptation (LoRA) parametrization makes the optimal learning rate approximately independent of the rank.

Topics

Low-Rank Adaptation (LoRA)Parameter Efficient Fine-Tuning (PEFT)Full Fine-Tuning (FullFT)Reinforcement Learning (RL)Supervised Fine-Tuning (SFT)Learning Rates Hyperparameter Tuning Model Architecture Mixture of Experts (MoE)MLP Layers Attention Layers Compute Efficiency FLOPs Multi-Tenant Serving Training Dynamics Llama 3 Qwen3

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

LoRA Without Regret

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

LoRA vs. Full Fine-Tuning Performance Parity

Architectural Best Practices for LoRA

Hyperparameter Tuning and Dynamics

LoRA's Efficacy in Reinforcement Learning

Operational and Compute Efficiency

Research Findings12

Topics