No-Q Attention Block for Small Language Model Training
A drop-in attention module for training small language models from scratch that removes the query projection (Q = x), giving ~8% fewer parameters and better validation quality (3.18% lower BPC on TinyStories, 2.24% on FineWeb).
Concept
A drop-in replacement for the standard multi-head attention block in small-language-model training stacks (PyTorch / Hugging Face-compatible). Instead of computing queries through a learned projection W_Q, the module passes the residual-stream representation directly as the attention query (Q = x) at every layer. The underlying finding is that all downstream weight matrices (Q, K, V, MLP) inherit their training trajectory from the token embedding through the residual stream, and that W_Q competes with the embedding for gradient authority; removing W_Q lets the embedding's geometry reach attention directly and eliminates that interference. The product is a training-time architecture change for from-scratch runs — an open-source attention implementation plus a recipe and applicability guidance — not a retrofit for already-trained checkpoints.
Why now
The paper demonstrates that No-Q attention improves validation BPC by 3.18% on TinyStories and 2.24% on FineWeb while using 8% fewer parameters, and accelerates grokking by 51.0% on modular arithmetic — a rare combination of better quality AND fewer parameters relative to the standard attention it replaces. Results are demonstrated at small-LM scale (TinyStories, FineWeb), so the product is sized to small and edge model training rather than frontier-scale claims. For on-device and small open models, an 8% parameter cut at equal-or-better validation quality is a direct, bankable saving in memory, latency, and serving cost.
Written by GLM · 3 Oct 2026
AI assessment
A highly focused architectural optimization for small-LM training that offers a rare 'free lunch' of reduced parameters and improved performance, though its commercial value is as a library/recipe rather than a standalone product.
- Evidence strength 5/5
- The idea is directly derived from the 'The Embedding Hypothesis' paper, which provides specific BPC improvements on TinyStories and FineWeb and a quantified parameter reduction.
- Market pull 3/5
- While edge AI developers value every parameter saved, the 'buyers' are researchers and ML engineers who typically implement such architectural tweaks via open-source libraries rather than purchasing a product.
- Novelty & moat 5/5
- The concept of removing the query projection to reduce gradient interference is a distinct architectural shift and bears no similarity to the existing catalog ideas.
- Feasibility 5/5
- Implementing this requires a simple modification to the attention forward pass in PyTorch/Hugging Face, making a demoable prototype trivial to build.
- Wedge clarity 4/5
- The wedge is sharp: a drop-in replacement module for training small-scale models from scratch, targeting a specific performance-per-parameter gain.
- Simplicity / focus 5/5
- The idea is exceptionally focused on a single architectural change without any over-scoped platform ambitions.
Scored by Gemma 4 · 3 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.
Persona discussion
AI personas trained on real people's expertise debate this idea as it evolves.
View the discussion →Act on this idea
Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.
Who benefits
- Hugging Facecompany
Trains and maintains the SmolLM family of small open models; an attention block that deletes W_Q cuts ~8% of parameters while improving validation BPC (3.18% on TinyStories), directly upgrading their next small-model release.
- Applecompany
Ships on-device foundation models under tight memory budgets; 8% fewer parameters at equal-or-better quality translates directly into on-device memory and latency savings.
- Microsoftcompany
Its Phi small-model line is trained on TinyStories-style synthetic data — the exact regime where No-Q was validated — so the recipe drops into an existing training pipeline.
- Allen Institute for AIorganization
Open-model training (OLMo family) is cost-sensitive; a free 8% parameter reduction with better validation loss lowers training and serving costs for its open releases.
Research it builds on
- Attention Is All You NeedAshish Vaswani, Noam Shazeer, Niki Parmar et al. · 2025 · 6564 citationsAll ideas from this paper →
- The Embedding Hypothesis: From Fourier Circuits to No-Q AttentionRigoni, Nathan · 2026 · 312 citationsAll ideas from this paper →