Hunting for Insights: Investigating Predator-Prey Dynamics through Simulated Vision and Reinforcement Learning

The natural world is full of fascinating and complex interactions between predators and prey, with each constantly adapting and evolving to survive. As researchers seek to better understand these dynamics, visual intelligence has emerged as a critical field of study, allowing us to gain new insights into how animals perceive and react to their environments.

In this work, we investigate the role of vision in prey-predator settings by leveraging reinforcement learning to train agents in simulated environments. Specifically, we are interested in how the field of view and the region of binocular vision affect the strategies and effectiveness used by predators and prey to hunt or evade respectively. We train predator and prey agents with varying vision fields in non-trivial environments with obstacles. The vision fields we experiment with are inspired by typical real-world predators and prey.

To evaluate the effectiveness of our approach, we both quantitatively evaluate agent performance in new environments and qualitatively observe the emergent strategies and behaviours of the agents under different configurations. Our work builds upon existing literature in this area to gain a deeper understanding of how differences in visual perception can influence predator-prey interactions.

The exploration of predator-prey dynamics in the context of vision has received relatively little attention in the existing literature. Research on simulated predator-prey systems commonly focuses on population dynamics and deals with very simple environments. At the same time, there have been major advancements in multi-agent reinforcement learning (MARL), making it possible to discover interesting group dynamics in complex environments. Given this, we present below some prior work related to this project.

Given the existing gaps in the literature for the study of predator-prey dynamics in the context of vision, our work aims to address them by incorporating various vision fields to represent different predators and prey. Through this approach, we investigate the impact of diverse visual perspectives on the emergent behaviours of the agents, providing valuable insights into the adaptive strategies employed in predator-prey interactions.

In this section, we present the methodology utilized for this paper. We first discuss the environment setup, after which we detail how the predator and prey agents are created. Finally, we talk about how the agents are optimized.

We create a 3D environment using the Unity game development engine [3]. The selection of Unity as the engine for creating the environment instead of a different simulator is motivated by several key factors. Mainly, Unity's ML-Agents package [4] offers support for the OpenAI Gym framework [5], which is widely used in reinforcement learning research. This way, Unity enables interaction between the environment and the learning algorithms, allowing for efficient experimentation and evaluation of predator-prey agents.

Moreover, the engine offers a wide range of built-in tools, such as physics simulations, ray tracing, and collision. Finally, Unity provides a user-friendly and intuitive development environment, making it accessible to researchers with varying levels of expertise.

We present a diverse set of simulated environments aimed at replicating various scenarios involving predator-prey interactions. Our objective is to uncover a wide range of behaviors for the predators and prey. To achieve this diversity, we incorporate obstacles such as trees and rocks within the environments. Additionally, we introduce walls along the boundaries to confine the agents to the designated training area. This confinement allows us to concentrate the actions of an agent and observations within a controlled environment.

Furthermore, we integrate physics-based simulations and collision detection mechanisms into the environment. By assigning colliders to the predator and prey agent models, we ensure accurate detection of interactions with the environment and other agents. Each object in the environment is assigned a corresponding tag, such as "obstacle", "predator", or "prey", enabling efficient identification.

To speed up the learning process, we design the initial training environment ('Control' shown below) to be free of obstacles. This approach eliminates potential obstructions that could slow down training progress in the early stages. Subsequently, we create three additional training environments ('Forest', 'Escape Room' and 'Rocky') featuring distinct obstacle types: small trees, large rocks, and a split doorway. These environments allow the agents to acquire the skills necessary for navigating through increasingly complex situations involving obstacles, and should allow the prey agent to learn to exploit occlusions.

Whilst the prey and predator have opposing objectives, they share similar characteristics. Both agents operate within a continuous action space with two degrees of freedom: rotation about the \(y\)-axis and forward/backward movement. Each degree of freedom allows the agent to select a value between \([-1, 1]\), determining the direction and intensity of movement. The agents' behavioural parameters consist of maximum movement and rotation speeds, which scale this intensity value. Notably, the predator possesses an additional box collider located at its face, resulting in the prey's death upon collision. Conversely, the prey lacks this collider as its primary goal involves evading the predator.

In order to simulate various vision fields, we employ a modified version of the ML-Agents Ray Perception sensor. This sensor uses a collection of ray casts originating from a central point. Several parameters are available to manipulate the vision, including ray angles (to establish the field of view), ray length (to determine the depth of field), number of rays (to regulate ray density/resolution), and a list of tags for filtering detected objects. Each ray conveys the following information: a flag indicating whether it collided with a tagged object, a one-hot encoded vector identifying the object type, and the normalized distance to the hit object relative to the ray length. Consequently, the sensor's output comprises a flattened vector encapsulating the information encoded within each ray.

To control the region of binocular vision, we have extended this sensor to incorporate a parameter representing the number of depth rays. We make the simple assumption that agents can see the presence and type of an object across their entire field of view, but only have depth perception in their binocular region. Consequently, we exclude depth information for any rays falling outside this region by assigning a distance of \(-1\). Using this modified Ray Perception sensor, we define two distinct sensor types: predator-style and prey-style, which draw inspiration from the characteristic vision traits observed in real-world predators and prey [16], respectively.

Utilizing the modified Ray Perception sensor offers distinct advantages for this paper by providing precise control over the simulated vision fields. Customizable parameters such as field of view and depth of field enable alignment with the specific requirements of the prey and predator agents in the simulated environment. This level of control ensures that the sensory input accurately reflects the desired characteristics of the vision of the agent. Furthermore, using the modified Ray Perception sensor instead of raw camera inputs enhances training efficiency by reducing the computational burden of processing high-dimensional image data.

The setting of predator-prey dynamics can be framed as a two-player dynamic game [6], [7]. This formulation captures the interactions and influence of the agents on the state of the game, providing a framework for studying predator-prey dynamics based on a common utility function. The game is only played for a set number of steps, until it is terminated.

In our formulation, the state and action spaces align with the descriptions provided in the previous sections, allowing for a consistent representation of the game dynamics. The transitions between states are determined by the underlying physics engine, ensuring a realistic simulation of the predator-prey environment.

$$ G_\text{pred} = \sum_{i=1}^{T}\left[\frac{𝟙(\text{prey caught at } t)}{N_\text{prey}} - \frac{1}{T}\right] $$ $$ G_\text{prey} = - G_\text{pred} $$

The reward structure employed in our proposed framework, as illustrated in the figure above, incorporates a constant time penalty applied to the predator at each time step, and a positive reward if a prey is caught. This deliberate design choice serves to incentivize the predator to capture prey at the earliest opportunity. Conversely, the reward for the prey is formulated in the opposite manner, establishing a zero-sum game. The range of rewards is bounded within the interval \([-1, 1]\). This reward specification enables a natural distinction between favorable and unfavorable outcomes of both agents.

Furthermore, since the predator-prey game exhibits an inherent asymmetry, it can be expressed as an asymmetric zero-sum game. Unlike games such as soccer, where teams have shared objectives, the predator and prey agents pursue conflicting goals. The asymmetry arises from the distinct policies used by the different agents, resulting in strategic dynamics that differ from those observed in symmetric games.

Overall, the nature of the game poses additional challenges, as it suffers from many of the problems in competitive multi-agent learning [8]. Multi-agent reinforcement learning in competitive scenarios poses several challenges [9]. In this paper, we employ a prominent technique called "self-play" [10], [11] to address these difficulties. The diagram below provides a high-level overview of this approach.

Initially, either the predator or prey agent is trained, while the model of the other agent remains frozen. After a predetermined number of iterations, the frozen model is trained, and the other agent's model is frozen. However, a potential issue arises from the bias introduced by repeatedly playing against the most recent model. This bias can lead to overfitting and poor generalization. To mitigate this problem, we incorporate an ELO ranking system as done in MuZero [12], enabling models to compete against earlier versions of their opponents. By doing so, we reduce the impact of bias, leading to improved generalization and robustness in the learned policies.

Our work investigates the role of vision in simulated predator-prey interactions. By varying the vision fields of predator and prey agents and training them using reinforcement learning, we make the following observations:

• Depth information is crucial for predators, while prey also benefit from depth perception to a lesser extent.
• Prey require a wide field of view to navigate while detecting predators in their periphery, whereas predators primarily look in the direction of their movement and thus don't require as wide a FOV.
• Memory is crucial to visual perception in predator-prey interactions.

These observations support the understanding of why real-world predators prioritize depth perception and prey prioritize a wide field of view in their vision systems.

Although the initial results are promising, our approach has several limitations. Real-life predator prey interactions are incredibly complex, and it’s almost impossible to capture all these complexities in a simple simulated environment such as this. The most pertinent limitations and potential future work include:

• The predator and prey agents sometimes displayed untrained or random behaviour, suggesting a need for further training. For comparison, OpenAI's hide and seek environment required ~\(2.7\) million episodes for seekers to consistently chase hiders, even with the aid of a curriculum learning approach. Our training lasted only around \(10\) million steps (order of magnitude of ~\(10^4\) episodes). Although training on OpenAI's scale is not feasible, additional training or alternative strategies such as curriculum learning could improve the agents.

• Memory is crucial to complement vision in predator-prey scenarios, however our agents had virtually no memory. This, coupled with the lack of positional information to understand when they were stuck, greatly affected the agents performance. To address these issues, we could increase the observation stack to give the agents more memory, whilst including past positional information as an additional observation. Using a recurrent neural network instead of a vanilla multi-layer perceptron could also better model the temporal information.

• Depth perception seems to play an important role for both predators and prey, but our simplified sensor assumes no depth perception outside of the binocular region which isn't strictly true. There are various monocular depth cues as well (as shown in the figure below) that both predators and prey in reality could make use of in their periphery. Thus, using a more complex sensor as such adding noisy depth estimates rather than no depth information in the monocular regions could help improve learning.

• We simplified the action space for our agents by restricting them to backward/forward movement and rotation. In reality, predators and prey have more degrees of freedom, such as the ability to turn their heads and look in different directions while moving. This additional freedom, combined with memory, could for example provide advantages for prey with predator vision. They could look backward to assess the predator's location and then look forward again to escape, similar to how humans (who have predator-style vision) might behave when being chased.

Furthermore, in this initial work, we only investigate a discrete set of vision fields. However, a similar setup could also be used to instead optimise the vision fields for each agent's goal of evading/hunting, given some physical constraints.

1. Park J, Lee J, Kim T, Ahn I, Park J. Co-Evolution of Predator-Prey Ecosystems by Reinforcement Learning Agents. Entropy. 2021; 23(4):461. https://doi.org/10.3390/e23040461.
2. Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.
3. Haas, J. K. (2014). A history of the unity game engine.
4. Juliani, A., Berges, V. P., Teng, E., Cohen, A., Harper, J., Elion, C., ... & Lange, D. (2018). Unity: A general platform for intelligent agents. arXiv preprint arXiv:1809.02627.
5. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). Openai gym. ArXiv Preprint ArXiv:1606.01540.
6. Tsai, Dorian, Timothy L. Molloy, and Tristan Perez. "Inverse two-player zero-sum dynamic games." 2016 Australian Control Conference (AuCC). IEEE, 2016.
7. Başar, T., & Olsder, G. J. (1998). Dynamic noncooperative game theory. Society for Industrial and Applied Mathematics.
8. Buşoniu, L., Babuška, R., & De Schutter, B. (2010). Multi-agent reinforcement learning: An overview. Innovations in multi-agent systems and applications-1, 183-221.
9. Shoham, Y., Powers, R., & Grenager, T. (2003). Multi-agent reinforcement learning: a critical survey (Vol. 288). Technical report, Stanford University.
10. Bai, Y., Jin, C., & Yu, T. (2020). Near-optimal reinforcement learning with self-play. Advances in neural information processing systems, 33, 2159-2170.
11. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., ... & Hassabis, D. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140-1144.
12. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., ... & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604-609.
13. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48).
14. Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020). Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research, 21(1), 7382-7431.
15. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. ArXiv Preprint arXiv:1412.6980

16. Land, M. F., & Nilsson, D. E. (2012). Animal eyes. OUP Oxford.