Value of Information and Reward Specification in Active Inference and POMDPs

Hello and welcome. It's September 13th, 2024 and we're in Active Inference Model Stream 14.1 with Ranwei talking about value of information and reward specification in Active Inference. There will be a presentation then we'll have a discussion and looking forward to people's comments and questions. Thank you again for joining and looking forward to the presentation. Yeah sure so I'll just dive into it. Yeah so thanks again for the invitation and so this paper is about the value of information and then some aspects about reward specification in Active Inference but it also touches on PAMDPs. And so the motivation is very simple so I was simply trying to understand the expected free energy objective for action planning in Active Inference. And the nice thing about the EFE objective which makes it an appealing objective for a variety of people in a variety of domains is that it has a very intuitive decomposition as the sum of the expected value term and epistemic value term. And then so the expected value term is this term that encourages the agent to obtain reward or achieve their preferred state of the world because it's simply the cross entropy between the predicted future observations given some actions and the preferred observations P-tilde. The second term epistemic value is what makes active inference or make it what makes EFE very interesting. So the epistemic value term is defined as the expected KL divergence between future future posterior so belief about future states given future observations and the predicted distribution of future states. So what this term does is it quantifies how far is the future posterior from the future prior which essentially quantifies the amount of belief update. So if you optimize this term, then it essentially encourages the agent to take actions that would lead to a higher amount of belief update. So a higher amount of information obtained about the environment. So experimentally, people have found that optimizing this objective led to some very interesting behavior. So on the left hand side here, we have an example from one of Alex's papers where he compared the what we call the state coverage of reward maximizing agents versus active inference agents in this simple environment called the mountain car, where the state of the agent is represented by its position and velocity. So we see that active inference agents has a much larger coverage of the state space. So it spreads more territory of the state space versus the reward maximizing agents. So what this potentially allows the agent to do is to learn a better and more comprehensive model of the environment, which it also turns out that having such a model enables better action selection and actually achieving higher reward. So on the right hand side, we have an example from one of my papers. So I did a lot of study on modeling human driving behavior. And this is one of the examples where we have a car called the Eagle vehicle driving down the road. And then on the side of the road, we have a large truck, which potentially obscures the view of a pedestrian. But we don't actually know whether the pedestrian exists or not exists. But if the pedestrian does exist, we assume or rather the Eagle vehicle believes that it's going to cross the road. So we don't want to cause any conflict with the pedestrian who's trying to cross the road. So what this Eagle vehicle does as a consequence of optimizing EFV is that it will nudge slightly to the left of the road to gain view of the pedestrian. And if it sees that the pedestrian doesn't exist, then it will return to the center of the road and carry on with the optimal speed without ever having to stop for the pedestrian. So these behaviors are very interesting. And it highlights the value of encouraging agent to explore and gain information about the environment. And this is usually understood as the main differentiator between active inference and some of the other frameworks for decision making,…