Skip to content
Amazon Science·Selected AI score75

AWS priming cuts hybrid model pretraining cost by over 99%

Can we change an AI model’s architecture without retraining it from scratch?

Summary

Amazon Web Services researchers introduced priming, a method that reuses pretrained Transformer weights to build hybrid state space models, cutting pretraining cost by more than 99% and using under 0.5% of the source model's token budget. A 32-billion-parameter Gated KalmaNet hybrid matched its source Transformer on reasoning while decoding up to twice as fast at 128,000-token context. Code and models were released under Apache 2.0.

Why it matters

Cheap, controlled architecture experiments could accelerate the shift away from pure Transformers for long-context agents, where KV-cache memory costs dominate serving economics.

This site provides summaries and links to the originals only.

Source: Amazon Science · amazon.science