Active Inference for Self-Organizing Multi-LLM Systems: A Bayesian Thermodynamic Approach to Adaptation

Hello and welcome everyone. It's January 16th, 2025. We're in Active Inference Guest Stream number 95.1 with Ritvik Praki discussing Active Inference for Self-Organizing Multi-LLM Systems, a Bayesian Thermodynamic Approach to Adaptation. So Ritvik, thank you for joining. Looking forward to your presentation and sharing more and discussing. So thank you to you. Great. Thank you. Thank you, Daniel, for having me. Yeah. So I'm going to talk about kind of Active Inference for self-organizing multi-LLM systems. So basically making LLM agents better with Active Inference. So to motivate the problem a little bit, current multi-LLM agents kind of have a lot of complex parameters. For example, prompts, agent architecture, tool use. Like these are things that you can't just kind of fix with hyperparameter optimization. You have to find better, more intelligent ways to improve these in time. And that's not really a thing right now. Also, another thing is that LLMs themselves are used as a vector adaptation for the agent in many ways. So for example, in the Voyager paper, LLMs are used to acquire new skills and then use those skills to explore the environment of Minecraft further, which is useful, but also quite limited. And then the third thing is that there's no formalized mechanism of balancing the exploration and exploitation trade-off. Right now, it's kind of just using LLMs internal abilities to go about that, but there's no formalized mechanism. And that kind of limits a lot of the potential. Now, our solution is using Active Inference to kind of overcome a lot of these problems. So the first thing is that the best possible prompt for different LLMs within the agent can be learned over time. These things are something that we can figure out from the environment. We can do search to figure out what are the best new prompting techniques. So we can test out different prompts to see which ones give better scores on certain metrics. So these things can be learned over time. And the Active Inference agent can help with that because now the Active Inference agent becomes the vector adaptation, allowing for real learning through belief updates and structural updates versus just solely in-context adaptation, which is what LLMs can do. So we're actually learning and we're actually adding information and updating beliefs within our model. And Active Inference acts as the formalized mechanism to balance the explore-explore boundary, allowing for a more principled understanding of uncertainty, and also giving us the option to verify optimality. See if the path of selecting actions is actually an optimal one or not, which we can't do with LLMs directly. So Active Inference works through a generative model. And the generative model we typically use for active inference in discrete time is called the Partially Observable Markov Decision Process, POMDP. Now it's Markov because current states are only influenced directly by past states and it's partially observable as opposed to a regular MDP because the agent doesn't have the ability to directly observe the states. It must go through an observation, which allows it to update beliefs about the states without any direct observation of the states. So within Active Inference, we have several matrices that we put into these POMDPs to encode the actual information that we're learning from the environment or that is contained within the environment. So the first thing is the A matrix or the likelihood matrix. This is basically just showing the relationships between the likelihood, showing the relationships between states and observations. So what are the chances that this observation is related to this state or this state produces this type of observation? The B matrix is showing the relationship between states at T plus one and states at T, time T. So basically, what is the probability that this state will occur at this time, given that this state occurred at a past time? Now the C matrix encodes the…