AWS priming cuts hybrid model pretraining cost by over 99%
Can we change an AI model’s architecture without retraining it from scratch?
Amazon Web Services researchers introduced priming, a method that reuses pretrained Transformer weights to build hybrid state space models, cutting pretraining cost by more than 99% and using under 0.5% of the source model's token budget. A 32-billion-parameter Gated KalmaNet hybrid matched its source Transformer on reasoning while decoding up to twice as fast at 128,000-token context. Code and models were released under Apache 2.0.
Cheap, controlled architecture experiments could accelerate the shift away from pure Transformers for long-context agents, where KV-cache memory costs dominate serving economics.
This site provides summaries and links to the originals only.
Source: Amazon Science · amazon.science