When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching Haoran Zheng

 Transformers allocate uniform computation to every position, regardless of difficulty. State Space Models (SSMs) offer efficient alternatives but struggle with precise information retrieval over a long horizon. Inspired by dual-process theories of cognition (Kahneman, 2011), we propose AMOR (Adaptive Metacognitive Output Router), a hybrid architecture that dynamically engages sparse attention only when an SSM backbone is "uncertain"--as measured by prediction entropy. Compared to standard transformers, AMOR gains efficiency by projecting keys and values from SSM hidden states (Ghost KV), reusing the SSM's O(n) computation rather than requiring O(n^2) attention at every layer. On small-scale synthetic retrieval tasks, AMOR outperforms both SSM-only and transformer-only baselines, achieving perfect retrieval accuracy while engaging attention on only 22% of positions. We validate that prediction entropy reliably signals retrieval need, with a gap of 1.09 nats (nearly half the entropy range) between retrieval and local positions. Additionally, our approach provides interpretable adaptive computation, where routing decisions can be understood in information-theoretic terms.


AMOR




Comments

Post a Comment

Popular posts from this blog

  A New Kind of Hidden Networking Science Building off the success of the program  Resilient Anonymous Communication for Everyone (RACE) , which recently released its code on  GitHub , PWND 2  will develop formal models of emergent communication pathways (AKA weird networks) to fundamentally improve the deployment and detection of robust and resilient hidden networks. What is RACE? RACE is an open source project aimed at developing technologies to provide metadata-anonymous, secure, and resilient messaging for users around the world. RACE provides anonymity by routing messages through an overlay network of volunteer servers using cryptographic algorithms that prevent a malicious subset of these servers from determining who is messaging whom. RACE uses specialized networking protocols to prevent connections between individual members of the network from being detected or blocked. RACE is built to run in a dockerized linux environment and on Android devices. For more d...
Read more

AlphaProteo generates novel proteins for biology and health research

AlphaProteo: A New AI System for Protein Design  A powerful tool for biological research In a recent paper published in Nature, researchers from DeepMind introduced AlphaProteo, a new AI system for protein design. AlphaProteo can design proteins that bind to specific target molecules, which could be helpful for drug discovery, disease understanding, and more. AlphaProteo is better than other methods at designing these proteins. In one experiment, AlphaProteo designed binders that were 10 times stronger than those designed by other methods. AlphaProteo is still under development, but it has the potential to revolutionize many areas of science. What is AlphaProteo? AlphaProteo is a deep learning system that was trained on a large dataset of protein structures. The system can learn the rules that govern how proteins fold and how they interact with other molecules. This allows AlphaProteo to design new proteins that have specific properties, such as the ability to bind to a particular ...
Read more