The most basic task is locomotion, in which the agent is rewarded for the distance it travels during an episode. The direct descendant of each agent inherits the parentвЂ™s genotype and goes through mutations that can create new limbs, remove existing limbs, or make small modifications to characteristics, such as the degrees of freedom or the size of limbs.Д®ach agent is trained with reinforcement learning to maximize rewards in various environments. Their design space is called Universal Animal (Unimal), in which the goal is to create morphologies that learn locomotion and object-manipulation tasks in a variety of terrains.Д®ach agent in the environment is composed of a genotype that defines its limbs and joints. Simulating evolutionДЇor their framework, the researchers used MuJoCo, a virtual environment that provides highly accurate rigid-body physics simulation. вЂњDERL opens the door to performing large-scale in silico experiments to yield scientific insights into how learning and evolution cooperatively create sophisticated relationships between environmental complexity, morphological intelligence, and the learnability of control tasks,вЂќ the researchers wrote. None of the learned parameters are passed on across generations. DERL uses Darwinian evolution to search the morphological space for optimal solutions, which means that when a new generation of AI agents are spawned, they only inherit the physical and architectural traits of their parents (along with slight mutations). Within the DERL framework, each agent uses deep reinforcement learning to acquire the skills required to maximize its goals during its lifetime. вЂњOur goal is to elucidate some principles governing relations between environmental complexity, evolved morphology, and the learnability of intelligent control,вЂќ they wrote in their paper. In their new work, the researchers at Stanford aim to bring AI research a step closer to the real evolutionary process while keeping the costs as low as possible.

While these approaches speed up the process and reduce the costs of training and evolving AI agents, they also limit the flexibility and variety of results that can be achieved. Yet another approach is to train different AI subsystems separately (vision, locomotion, language, etc.) and then tack them on together in a final AI or robotic system. Another shortcut is the use of Lamarckian rather than Darwinian evolution, in which AI agents pass on their learned parameters to their descendants.

For example, they fix the architecture or physical design of an AI or robotic system and focus on optimizing the learnable parameters. It would need a lot of parallel and sequential trial-and-error cycles.ДЄI researchers use several shortcuts and predesigned features to overcome some of these challenges. An AI system that would want to recreate intelligent life in the same way that evolution did would have to search a very large space of possible morphologies, which is extremely expensive computationally. Studying the evolution of life and intelligence is interesting, but replicating it is extremely difficult. Based on the selection pressures caused by the environment, the descendants of those first living beings evolved in many directions. Interestingly, all these species descended from the first life-form that appeared on Earth several billion years ago. Humans have an upright posture that frees their arms and lets them see the far horizon, hands and nimble fingers that can manipulate objects, and a brain that makes them the best social creatures and problem solvers on the planet.

Sea turtles have flippers to swim with and a magnetic field detector system to travel very long distances. Bats have wings to fly and an echolocation apparatus to navigate dark spaces. Mosquitos are equipped with thermal vision to spot body heat. Across many generations, every animal species has gone through countless cycles of mutation to grow limbs, organs, and a nervous system to support the functions it needs in its environment. In nature, the body and brain evolve together. The findings can have important implications for the future of AI and robotics research. Called вЂњdeep evolutionary reinforcement learning,вЂќ or DERL, the new technique uses a complex virtual environment and reinforcement learning to create virtual agents that can evolve both in their physical structure and learning capacities. In a new paper published in the scientific journal Nature, AI researchers at Stanford University present a new technique that can help take steps toward overcoming some of these limits.

How to evolve