Soft Prompts / Adapters

Soft prompts and adapters are PEFT routes that freeze the backbone and train only a few extra parameters: 1) Prefix-Tuning prepends learnable key/value vectors before attention in every layer (~0.1%–2% of params); 2) P-Tuning (v2) prepends continuous prompt vectors to input embeddings (or each layer's input), reparameterized by MLP/LSTM for stability; 3) Adapter inserts a bottleneck after Transformer sub-layers: down-project $d \to r$, up-project $r \to d$, with nonlinearity and a residual connection. Together with LoRA (frozen $W$, low-rank delta $\Delta W = BA$; rank r=8 cuts parameters by ~100×) they form the four mainstream PEFT methods.

multi-task / multi-tenant serving (hot-swapping prompt/adapter sets on one base), memory-constrained fine-tuning, rapid experimentation.

full fine-tuning has prohibitive parameter and storage cost. The three routes trade efficiency against quality: Prefix/P-Tuning modify attention inputs, closer to prompt semantics but slightly less stable; Adapters use residuals to stay near-identity at init and approach full-FT quality; soft prompts use even fewer parameters than LoRA with a somewhat lower ceiling. Multiple extra-parameter sets coexist on one base — the server stores only ~0.1% more parameters per task/tenant, giving each its own adaptation.

Prefix-Tuning vs P-Tuning: what differs?

Why is the Adapter bottleneck effective (residual + nonlinearity)?

Soft prompts vs LoRA: comparison and how to choose?

Why freeze the backbone and train only extra params? Benefits & risks?

Multi-task serving: how do multiple prompts/adapters coexist?