Natural Neural Structure in Artificial Neural Networks

Hello and welcome. It's September 18th, 2023, and it's ACTIMF Guest Stream 57.1 with Andy Keller. We're going to be talking about natural neural structure for artificial intelligence. There will be a presentation followed by a discussion. So if you're watching live, please feel free to write questions in the live chat. Otherwise, thank you, Andy, for this. Really looking forward to it and to you for the presentation. Yeah, thanks so much. Thanks for having me. I'm super excited to be able to present this stuff with the Active Inference Group. I'm a fan and very interested. So hopefully, yeah, get to have a good discussion and see what you guys think about it. So my name is Andy. I'm finishing up my PhD, supervised by Max Welling at the University of Amsterdam. I'm starting a postdoc at Pervert after this. So I'll start out just talking about the goal of my work in general is to try to bring modern artificial intelligence closer to more human-like generalization. And so what we mean by this is maybe some sort of structured generalization or maybe more familiar to the Active Inference Committee, like a structured world model, which we believe that humans have. And the way that we propose to do this is by integrating natural neural structure into artificial intelligence. So first, let's define what we mean by structured generalization. So I think it's fairly uncontroversial to say that modern machine learning generalizes beyond its training set in the traditional sense. So, for example, even the earliest artificial neural networks, multilayer perceptrons could be trained on data sets of images like this and achieve high accuracy. Then when they're presented with a held out test set of images that they've never seen before, they can still classify them relatively easily with the same level of accuracy. And this is what we typically call generalization. However, even fairly early on, it was noticed that these systems really struggle with small shifts or deformations applied to the images. For example, So we think, why is this surprising? And I argue it's really due to our innate ability to perform this type of structure generalization that this example is a failure. So, for example, this shift is nearly imperceptible to us and we handle it automatically. Whereas in these systems, it's very clearly a major problem. So in words, we can say that structure generalization is a generalization to some symmetry transformations of the input, or in this case, the symmetry transformation is a small shift that leaves the digit class unchanged. So the obvious question then is, what precisely do we mean by this natural structure? And why do we think that this would help us with these settings? So first, let's talk about what we mean by natural neural structure. One way to talk about structure or any type of bias in a system is an inductive bias. And so an inductive bias can loosely be defined as an a priori restriction of a set of realizable hypotheses when you're doing model selection. More colloquially, we can call this something like before seeing any data, it's a restriction of what and how you can learn. So very broadly, this can include anything from model class to optimization procedures or even hyperparameters. And in some sense, they really define what is possible to learn. And it defines generalization in that you actually can't generalize beyond a training set without having some inductive biases. And this is explained more thoroughly in this paper by David Wolpert. So what we mean by natural inductive biases then is biases that stem from the restrictions and limitations that are faced by natural systems, by the nature of having to live in the real world. For example, the brain has many efficiency constraints and physical constraints by nature of its construction. And following this logic, then these constraints are really playing some role in our generalization abilities, which currently exceed modern artificial intelligence, as…