Thermodynamic Linear Algebra

Hello and welcome. It's November 11th, 2024. We're in Active Inference Guest Stream number 92.1 with Max Eifer. We're going to be discussing a variety of thermodynamic and Bayesian topics. It should be very exciting. If you're watching live, feel free to write any questions in the live chat. Max will go through some sections of several papers and I will read questions and ask questions. So thank you, Max, for joining. Looking forward to hearing about this. Take it away. Great. Yeah, thank you for the introduction and thanks for having me on. Really excited to talk about this stuff. So there are a few papers I want to talk about. One called Thermodynamic Linear Algebra about kind of a novel hardware approach to solving linear algebra problems. Another one called Thermodynamic Bayesian Inference, which is similarly about using these new thermodynamic hardware devices for Bayesian inference problems. But before I get into that, I just wanted to make a note on kind of like the story of this work and how it evolved because I think it's pretty interesting. So I'm working at a company, a startup called Normal Computing. And where we started off was just thinking about what's kind of the best hardware paradigm for AI and especially probabilistic machine learning, because these are some of the most challenging tasks computationally and seen as some of the most necessary to be able to do kind of robust AI without hallucinations. And a big approach to that is probabilistic machine learning. And a lot of our approach was inspired by this report that came out of a workshop in 2019 called Thermodynamic Computing. And I'm sharing it. Can you see my screen here, what I'm sharing? Okay. Yeah. So there's this one quote here in the introduction that I think really summarized the point. It says, somewhat paradoxically, while avoiding stochasticity in hardware, we are generating it in software for various machine learning techniques at substantial computational and energetic cost. So one of the big kind of takeaways from the work they did in this workshop was there was kind of a consensus that emerged that we're working so hard to suppress the randomness that comes up naturally in our hardware. And then we need randomness actually to run various randomized algorithms, especially in probabilistic machine learning. And so if there were some way to just use the randomness that's naturally there that we know about due to thermodynamics to achieve those algorithmic goals, maybe we would save a lot of energy basically. So that was kind of one of the ideas we were inspired by. But the first step along this journey was, so there was an earlier paper where we basically proposed circuits that are capable of sampling from a normal distribution. And I'll get more into the math of why that happens. You know, why do circuits that look like this allow you to sample from a multivariate Gaussian? But so this was kind of the first work that was published at Normal actually before I started working there called Thermodynamic AI and the fluctuation frontier. But how you can think of this is this is the electronic analog of a system of coupled harmonic oscillators. So you can just think about a bunch of masses that are coupled together by springs and that are also damped. So they have friction acting on them and there's also noise acting on them. So kind of damped system of harmonic oscillators driven by noise. That's the kind of system that we're talking about in these works. And we're just kind of looking at this electronic analog, which is completely analogous, completely isomorphic to that picture in terms of masses on springs. So one of the first problems we were interested in solving was solving a linear system of equations. So you can think of this as if I have a few planes in an n dimensional space. Here I have three planes in a three dimensional space. And in general, I'll have n planes in an n dimensional space. And I want to find the one point where they…