Backpropagation-Free 4D Continuous Ant-Based Neural Topology Search

Hello and welcome. This is ACT-INF Guest Room 52.1 on August 10th, 2023. We have Ahmed Elsaid, Alexandra Arorbia, and Travis DeSalle. Ahmed's going to give a presentation and following we'll have some reflections and discussions. So thank you all for joining and Ahmed to you for the presentation. Okay. Thanks so much for having me. Today I'm going to discuss the methods that we came up to solve new architecture search and new evolution problems. The methods that are end-colony optimization-based solutions. So as the title here is mentioning, end-colony optimization for new architecture search and new evolution. This work is a collaboration between me, my previous advisor, Dr. Travis DeSalle, and my co-advisor, Dr. O'Rourke-Axander Arorbia. I'm currently an assistant professor in University of North North Carolina, Wellington. And moving on to the next slide. So as an overview of the things that I'm going to discuss today, I'll try to take, give a bird eyes view for what is new evolution, why we need it, and from there I'll try to discuss other methods that is, and that is based on end-colony optimization, which is called end-based neural topology search, or ENDS for short. And after that, I will discuss how we advanced with this idea by introducing a continuous ENDS, or CANDS for short, which is, which could thread off this free search space and replace it with a continuous search space. And after that, I'm going to also say what we did with the three dimensions and the continuous ENDS to have a back-propagation-free CANDS. And later, I'll discuss three of the points that we are considering for future work. So, in the machine learning learn, as the structure, the neural network structures get, got deeper and deeper, people were trying to optimize the structures to have better performance. And people in different realms, or different problem domains, used to borrow the best performing neural architectures or structures, and try to modify a little bit to work for their problem. And they tried to tweak some of the features of the structure and compare these different tweaks. And then they say that we found the best performing structure. But to actually find an absolute optimum structure, concerning that solution, it could be an NP-hard problem, because to reach that solution, they had to try all the combinations of the different structural elements. And because we have massive structures in these deep neural networks, it could be an NP-hard problem, because we don't have computational power or enough time to actually train and test all these structures, all these structures constructed from these different combinations of structural elements. So, the alternative way to do this is to actually try to do, to apply a meta-rearistic method, to converge to a neural-to-op-com solution. That is much better than not optimizing the structure or relying on a kind of like a random search, right? A random search can get us better performing neural network, but it's not going to give us a near-to-optimal solution, or to converge to a near-to-optimal solution. So, a meta-rearistic method would give us an automated method and also can converge to a near-to-optimal solution to this structural problem. So, the way that people approached NAS was by trying to mimic how optimization is done in nature, right? So, the first method, the first way they thought of it was trying to mimic how living organisms evolve in nature, using genetic-based algorithms like the Darwinian genetic evolution. And it started with NEAT, NEAT, it's a short for Neuroolution of Augmending Topologies, and it relied on genetic algorithms, where also that concept was applied for in most of the NAS on neural evolution methods, and exam is one of them. Travis came up with this method, and it became one of the state-of-the-art methods in NAS. So, in such methods, we tried to again mimic genetic evolution by introducing new structural elements or removing structural elements or…