Energy-Based Transformers are Scalable Learners and Thinkers

Hello, welcome. It is July 30th, 2025, and we're in Active Inference Guest Stream 115.1 with Alexei Gladstone, who will be presenting and discussing energy-based transformers in the future of scaling. There will be a presentation followed by a discussion, so if you're watching live, please write any questions in the live chat. Otherwise, thank you, Alexei et al., for the paper and the work and for joining, and looking forward to the presentation. Thank you. Thanks for having me, Daniel. Excited to start this presentation. So let's go ahead and get started. And I'm going to ask a philosophical question at the start of this, just to get everyone thinking. And we'll circle back to this at the end of the presentation. But that question is, would you trade off 100 times the amount of computation, so burning your GPUs for 100 times longer, for 1% better generalization, right? So generalizing 1% better out of distribution, tiny amount, right? And you might say, this question sounds absolutely ludicrous. Who in their right mind would spend 100 times the amount on computation just to do 1% better? And maybe we're already actually seeing it, right? So this is an example with opening ISO 1 and it thinking for a very long time to generalize better, albeit maybe not 1% better. But this is kind of the insight that I think will lead you into the ideas behind energy-based transformers that we are going to dive into now. So to give some contextualization, there's been lots of interest recently in what are called thinking and reasoning models broadly in AI, especially in language modeling. So in language models, particularly with large language models, we've seen tons of models. In fact, this slide is now outdated because Grok 4 and Claude 4 Sonnet have come out. So many models, including open source Chinese models, have been used to basically perform this reasoning or thinking for longer during inference. And the same thing has actually happened in Visions. There's been lots of interest in scaling diffusion models at test time as well for image generation or for video generation. So broad interest from the field in basically making models perform better at inference by kind of doing search or thinking or reasoning or test time compute, whatever you'd like to call it. But today, I'm going to take a perspective on these approaches that these approaches are ultimately unsatisfying in the quest for human level intelligence. And you might say, why? These approaches have had widespread success. I mean, I use O1 every day and these models have been largely successful in doing great in math and coding benchmarks. So why are these approaches unsatisfying? And what is my argument today? Well, from the intellectual perspective of achieving human level intelligence, we need approaches that are modality agnostic because humans are able to reason and think over any modality. And the current approaches do not have that characteristic. Another challenge is problem agnosticness. So current approaches very much rely on, especially in language, verifiable rewards, i.e. domains or problems such that they can be easily verified with a binary reward. An example of this is coding or math, where we can verify very easily whether or not a coding or a math solution is correct. And ultimately, this kind of hints at another problem, which is the reliance of these approaches on external supervision. So us models, or sorry, us humans are able to get to human levels intelligence without any external rewards or verifiers or supervision. We're able to learn how to think on any modality or any problem without this type of supervision. So I'm going to make an argument today that if we want human level intelligence, right, so if we want artificial general intelligence or artificial super intelligence, whatever you'd like to call it, we really do need systems that learn to think on their own, on any modality problem, and without any additional supervision, just like us humans,…