This post will provide a general introduction to the biology of predator-prey systems: I will start with the evolution of predator and prey and their consequential co-evolution in predator-prey systems. The focus will then move to mechanisms to avoid attack and how it results in a complex signalling system called aposematism. I will present the scientific background to the emergence of aposematism and lay out the open questions which motivate the following chapters.

Predator-prey systems

There are three fundamental principals which define any biological system. These principals are: reproduction, selection, and mutation.

It has been long recognised that life comes in a profuse variety of shapes, forms, and traits. But as vast as the differences are there are underlying similarities which apparently connect all living things. These similarities are evidence for their descent from common ancestors.

Modern evolutionary theory is the aggregate of many subsidiary ideas from anticipation of nature to genuine interpretation of nature (Osborn, 1896). The notion of joining diversity with similarity in an attempt to explain biological systems goes all the way back to ancient Greek philosophy. In a time when the world was considered to be static Anaximander suggested a dynamic and changing world and is considered as evolution’s most ancient proponent. The next great progress towards an inductive theory of evolution based on laws of nature was by Lamarck’s transmutation theory of 1809. Lamarck proposed that organisms adapt to their local environment through inherited changes over generational time. But Lamarck did not provide a workable mechanism with his theory and is better known today for his flawed principle of inheritance of traits acquired by the parental generation from use or disuse. The heritability of acquired characteristics was later called Lamarckism. Furthermore, Lamarck’s transmutation theory was not able to change the prevailing concept of fixed species as it was widely opposed for its lack of empirical evidence.

Darwinism and evolution by natural selection

The situation changed dramatically with Charles Darwin. Charles Darwin was a naturalist and sailed on board the ship Beagle around the world where he collected and documented flora and fauna. It was his vast collection which made him recognise the differences and similarities of species on an unprecedented scale. He concluded his realisations with the writing of his book ‘On the origin of species’. Darwin’s main achievement was to present a workable mechanism based on empirical evidence: his theory has the same notion of natural evolution. But in a crucially important difference, he proposed that the observed variation in traits is innate and not acquired. Thus, the variation in traits results in an unequal adaptation to the environment and, consequently, some organisms will survive and reproduce more successfully. It has to be noted that Alfred Russell Wallace formulated independently an almost identical theory.

In summary, the central points of modern evolutionary theory are:

  1. organisms having innate variations in their traits and characteristics,
  2. well adapted organisms are more likely to survive and reproduce, which
  3. leads to better adapted organisms.

The focus of current research lays on the search for different factors of the natural law called evolution. Interesting aspects of evolution are for example the effects of sexual selection or the evolution in more complicated environments where we observe the effects of interactions between species and organisms.


As a matter of fact natural environments are intrinsically complex. This complexity derives on the one hand from the high entanglement of organisms interacting in competitive relationships with each other. On the other hand, natural environments are also defined by their dynamics of constant change. Thus, evolution in natural environments is defined by the dynamic competitive relationships of organisms. Typically, evolution results in multiple species which successively adapt in response to their adaptations. As an example, predators and prey evolve together as the prey is part of the predator’s environment and vice versa: the predators rely on their prey as a food source and evolve necessary traits in order to feed on their prey efficiently. Common traits found in predators are therefore speed, a good sense of sight, hearing, and smell, or specifically adapted mouthparts. Likewise, the prey evolves means to avoid predation such as speed, crypsis, deterrents, and good senses to detect predators. This phenomena is called co-evolution (Janzen, 1980):

Co-evolution may be usefully defined as an evolutionary change in a trait of the individuals in one population in response to a trait of the individuals of a second population, followed by an evolutionary response by the second population to the change in the first.

Evolutionary game theory

A modern framework to describe and analyse evolutionary models is in the form of a game. In a game the environment is modelled based on individuals, strategies, and payoffs. The payoff an individual receives from taking part in the game depends on the individual’s strategy but equally on the strategy of all other individuals taking part in the game. The aim in such a game is to reason about the optimal strategy under the assumption that all individuals taking part in the game are rational and try to maximise their individual payoffs. As the optimal strategy of a single individual in such a game depends reciprocally on all the other individual strategies the task of defining optimality seems almost impossible at first. The modern field of game theory holds a wide body of methodologies and frameworks which addresses optimality in games with the Nash equilibrium probably being the most fundamental concept (Nash, 1951). The Nash equilibrium defines a set of optimal strategies in a game of multiple players which do not cooperate and individually maximise their payoffs. Within such a set of strategies no individual can gain any improvement of their own payoff by independently changing their strategy. In the special case of a stable Nash equilibrium changes to an individual’s strategy do not impact the optimality of the other individuals strategies either.

The notion of evolutionary game theory applies the game theoretical framework to biology where individuals might not reason about their optimal behaviour but show different forms of strategies or adaptations due to their genetic variations. The payoff within evolutionary games is represented by fitness gains or losses with natural selection being the driving force of optimising strategies or adaptations in the evolutionary games. Additionally, the underlying population dynamics of interacting individuals can be ignored as being a separated layer to the evolutionary model. This allows a focus on static games which does not alter the outcome of the analysis in most cases within the biological context: finding the best strategy or adaptation for a specific environment represented by the evolutionary game.

An optimal strategy within a biological context is termed an evolutionarily stable strategy or ESS and is closely related to the stable Nash equilibrium. An ESS is defined as the strategy \( (S) \) that no different/ mutant strategy \( (T) \) can invade (assuming that a population is playing the ESS uniformly and within the influence of natural selection) (Maynard Smith, 1974) which is the case if and only if one of the following conditions hold:

\[ \begin{aligned} E(S,S) &> E(T,S)\text{, or} \\ E(S,S) = E(T,S) &\land E(S,T) > E(T,T) ~~ \forall~ T \neq S\text{.} \end{aligned} \]

This has been a simplistic and brief introduction to evolutionary game theory only. Other posts will discuss models of defended prey and signalling that defence to their predators. Such models use continuous strategies and fall into the category of non-linear games. I refer to the other posts for a detailed discussion of the ESS in non-linear games with continuous strategies and effects of finite populations using a method called adaptive dynamics.

How to avoid attack

Actually, the vast majority of species are under predatory risk in their natural habitat and targeted by predators as part of the food web. Through the process of evolution by natural selection manifold mechanisms have emerged to avoid predation. As Fisher (1930) argued, it is the ubiquitous presence of anti-predator adaptations which shows that predation plays a significant role in the ecology and evolution of ecosystems.

Primary and secondary defences

It is common to classify anti-predator adaptations into ‘primary defences’ and ‘secondary defences’. By definition, primary defences operate before an attack by reducing the frequency of detection with predators, as in disruptive colouration, countershading, and crypsis, or by reducing the risk of attack given detection, for example by warning colouration, morphological adaptations, chemical defences, mimicry, and aggregation (Robinson, 1969; Edmunds, 1974; Ruxton et al., 2004). Complementarily, secondary defences such as toxins or unpalatable substances reduce the risk of falling prey in an encounter with predators. This general classification, however, is not without limitations as there are interesting grey areas when it comes to warning signals or mimicry, for example. Importantly, anti-predator adaptations are not discrete and independent traits but continuous and interacting.

A common alternative is to categorise mechanisms of anti-predator adapta- tions into functional groups of

  • avoiding detection (which includes avoiding encounters),
  • avoiding attack (or falling prey in an attack), and
  • deceiving predators.

Given that predation plays a significant role in natural selection, avoiding detection by predators seems like a strategy most obviously favoured by evolution through natural selection. In fact, crypsis is a widely found adaptation to prevent detection by predators. But the broader question of ‘Why are species selected for a specific form of defence over another?’ is an important starting point for the discussion of conspicuous warning signals in the next chapter. It will be crucial for our understanding of evolutionary dynamics to define defence as involving some kind of ‘cost-benefit’ trade-off.

Tollrian and Harvell (1999) proposed a framework of five general categories for analysing fitness cost in secondary defences which has been generally adopted for anti-predator adaptations:

Allocation cost
or internal cost which arises from allocating limited resources to the erection, maintenance, and operation of a defence.
Environmental cost
or external cost which arise from interactions with the environment in relation to the defence.
Opportunity cost
or indirect cost which arises from missed alternatives caused by the defence. Adaptations such as crypsis or seasonal behaviour can limit an individuals options in respect to foraging or mating.
Design cost
or self-damage cost arises typically in the context of chemical defences to prevent auto-toxicity. But the design is not limited to chemical defences alone.
Plasticity cost
relates to inducible defences which allow an individual to re- spond to changing predatory risk in its environment. The cost arises from the deployment of sensory systems for example.

Concluding, most defences, if not all, incur some kind of fitness cost or other trade-offs. Thus defences require clear benefits which outweigh their costs where the reduced predation is not necessarily the only advantage. I will discuss further benefits in regard to overcoming crypsis in the initial evolution of aposematism.


On the one hand, the benefits of primary defences to prevent detection are evident, especially, when we assume the risk of predation to be high. On the other hand, prey may have secondary defences which commonly involve the expression of toxins or deterrent substances. These secondary defences are not directly observable by the predator and the benefits of such defences are not self-explanatory. A predator might attack the defended prey nevertheless for a lack of aversive information.

Hence, many defended prey species use conspicuous signals – either visual, audible or behavioural – in combination with their otherwise non-observable secondary defences to warn predators. This pairing of a toxic secondary defence and a conspicuous warning signal is known as aposematism (Poulton, 1890). The most commonly associated warning signal is ‘warning colouration’ but other signals are known such as conspicuous sounds, behaviours, and odours. An example is the family of poison dart frogs, Dendrobatidae, which are native to Central and South America. The species have brightly coloured skin and are at least to some degree toxic.

Yellow-banded poison dart frog, Dendrobates leucomelas, an ex- ample of aposematic display. (picture in public domain taken from Wikimedia commons)

Aposematism is a primary defence and the benefit of avoiding well-defended prey seems to be mutual and obvious. Additionally, signalling is omnipresent and fundamental within biological systems which might make it appear as trivial. But this is far from the truth and aposematism has been the focus of much research by the scientific community in the light of evolutionary theory. In particular, the initial evolution of these warning signals in ancestrally cryptic populations has been much debated because a novel conspicuous mutant has to overcome the loss of protection of crypsis, which is maintained by its conspecifics. Furthermore, anti-apostatic selection by inexperienced predators results in rare mutants being predated relatively more often (Lindström et al., 2001).

Previous models mainly focused on the questions of the initial evolution of aposematism and there are two established arguments which aid the appearance of conspicuous signals:

  1. The Predator’s perception and cognitive processes possess specific prop- erties which promote aposematism. For example the usage of aposematic signals as warning flags improves discrimination in educated predators and enhances the learning of unprofitability (Keehn, 1959). Other possi- ble factors could be dietary conservatism (Lee et al., 2010; Thomas et al., 2003) or a shifted peak of the aversive information so that more conspicu- ous prey individuals are favoured (Leimar et al., 1986; Yachi and Higashi, 1998; Gamberale-Stille and Tullberg, 1996).
  2. The opportunity cost of crypsis in combination with the reliability of hon- est warning signals of well-defended prey drove the evolution of apose- matism and consequently the evolution of predator psychology (Sherratt, 2002).

Other more general factors which can further aid the initial evolution of aposematism are spatial aggregation and kin selection. Both factors are widely applied to introduce a more significant number of mutants in theoretical models to overcome the problems of the initial evolution of new traits. I will discuss these factors in more detail in the next chapter where they find application in a model of co-evolution.

A second major issue in the theoretical treatment of aposematism is the problem of dishonesty: it may be beneficial for an undefended individual to use a warning signal too to avoid being attacked (mimicry), which in turn can undermine the effectiveness of the signal. When this kind of dishonesty occurs across species, it is known as Batesian mimicry; when it happens within species, it is known as automimicry (Ruxton et al., 2004). A second form of dishonesty arises when there is continuous variation in toxicity within or between species. In some theoretical treatments, prey with weak secondary defences may choose to signal brightly, in order to compensate for their lack of repellence to predators. Hence key questions of current importance in aposematism theory focus on the questions: how conspicuous should a signal be, and how much should be invested into secondary defences (Speed and Ruxton, 2007; Speed et al., 2010; Longson and Joss, 2006; Ruxton et al., 2009)? Moreover, which factors influence evolutionary stability of honest signalling and what is the role of mimics in maintaining or destabilising aposematic display (Gamberale-Stille and Guilford, 2004)?

Leimar et al. (1986): Mechanisms of aversive learning in aposematism

Although many aspects of signalling systems are understood, a key element missing from the current theory is the incorporation of learning such as the role of aversive learning in particular. The importance of this was described in an earlier work by Leimar et al. (1986) which is still one of the most relevant frameworks for the evolution of aposematism.

The key contribution of Leimar et al. are elements of predator psychology contributing to a process of aversive learning in aposematism which could ex- plain the evolution of aposematism assuming that secondary defences involve some cost.

The model defines an inhibitory gradient h which generalises the aversive experience from encounters with n prey individuals of a specific morph \( (x_i, y_i ) \) to generalised attack probabilities \( g(x) \): \[ g(x) = e(x) \prod_i [1 - h(x,x_i,y_i)]^{n_i}\text{,} \] with \( e(x) \) being the excitatory gradient of a naive predator, \( x \) being the colouration, and \( y \) being the degree of unprofitability of prey. The specific nature of the generalisation gradient promotes conspicuous prey through the application of a peak-shift (Figure as defined in Leimar et al., 1986).

(a) This chart shows the generalisation of aversive information from an encounter with a specific morph \( (x_1, y_1 ) \) through the inhibitory gradient \( h(x, x_1, y_1 ) \).

(b) This chart shows the attack probability \( g(x) \) after n encounters with prey of morph \( x_1=0.5 \) and \( y_1=0.2 \). The graph for \( n=0 \) shows the excitatory gradient \( e(x) \) of a naive predator. The graph for \( n > 0 \) show the peak-shift resulting in a bias towards the avoidance of more conspicuous prey.

Elements of predator psychology describing a process of aversive learning as defined in the previous Equation following the definition in Leimar et al. (1986) with \( x \) being the colouration and \( y \) being the degree of unprofitability of prey.

The peak-shift is a psychological phenomenon which results in a bias towards avoidance of more conspicuous prey following an aversive encounter. There is a growing body of empirical evidence which supports the assumption of biased generalisation, and it might play an important role in the stability and initial evolution of aposematism (Yachi and Higashi, 1998).

Other main findings by Leimar et al. (1986) were that aposematism can initially evolve in an otherwise cryptic prey population if there are some elements of kin-selection or a change in environment making crypsis less effective.

Sherratt (2002): The co-evolution of aposematism.

Even though the peak-shift phenomenon promotes the initial evolution of aposematism it requires a pre-existing and universal psychological response to conspicuousness. The question of ‘Which came first: conspicuous signals or specific properties of predator psychology?’ has not been resolved. Furthermore, the generalisation bias itself does not explain why predators show this particular psychological property. Most importantly, aposematism occurs within many different species, over a wide range of taxa, and conspicuous signals manifest themselves over a diverse set of sensory systems. It is unlikely that all this can be explained by a common pre-existing generalisation bias. It is perhaps more feasible to assume that the generalisation bias has arisen from a common selective pressure.

Sherratt (2002) addresses these questions by introducing a model of co- evolution of multiple predators and prey. The difference to previous models is that the foraging behaviour of predators is subject to selection itself. This allows aposematism to evolve through a process of co-evolution of predator and prey.

Sherratt concludes that a novel conspicuous prey item has most likely been encountered by previous naive predators as it is easily detected. For this reason, novel conspicuous prey is likely to be defended to have survived previous encounters with predators.

Likewise, a naive predator which moves into an environment with experienced predators should avoid conspicuous prey as it is easily detected by the other predators and is likely to be defended to have risen to greater numbers.

As soon as the predators have adapted to the correlation of defences and conspicuous signals aposematism arises rapidly, rather than gradually, through runaway co-evolution.

Sherratt identifies further factors in his model which aid the initial association of defences with conspicuous signals such as the opportunity cost of crypsis and the aggregation of prey.

Speed and Ruxton (2007): The vast variety of aposematic solutions

We saw that the initial evolution of aposematism itself already presents the scientific community with a wide range of challenges. That is why it might be not surprising that most of the theoretical work is concerned about the factors which drove the initial evolution of aposematism. Nevertheless, recent efforts have tried to explain the evolutionary stability of aposematism in the light of growing empirical studies. The interest in evolutionary stability itself is not new, with Leimar et al. (1986) already discussing evolutionarily stable strategies (ESS) within their framework. The reason for the later neglect of questions regarding ESS in many models was mainly due to the complexity of the task at hand: empirical studies of aposematic species show a vast variety of aposematic solutions. A broad study by Summers and Clough (2001) showed a positive correlation between conspicuousness and toxicity in the poison dart frog family, Dendrobatidae, which supports the ideas of aposematism being a costly handicap signal indicating fitness advantages. But when Darst et al. (2006) revisited three specific dart frog species they actually found the reverse case of negatively correlated toxicity and a established theoretical framework to treat these findings consistently did not exist. The next chapter will present recent work which addresses evolutionary stability within a theoretical framework of co-evolution to fill this gap (Broom et al., 2006; Broom et al., 2008; Teichmann et al., 2014b).

In order to explain the variety of aposematic solutions Speed and Ruxton (2007) introduce a model incorporating marginal costs of both display \( \psi \) and secondary defences \( \zeta \). The model predicts optimal values of conspicuousness \( C \) and defence \( D \) of a focal prey population in an environment of multiple predators and other cryptic and undefended non-focal prey populations.

The probability of attack given detection \( P_\text{(Att)} \) is a combination of inherent wariness regarding conspicuousness \( W_{(C)} \) with repellence \( R \) from previous encounters with other defended prey individuals represented by the average toxicity of the focal group \( D^* \): \[ P_i(Att) = Att_\text{min} (1-Att_\text{min}) W(C_i) R(D^*)\text{.} \] The probability of being killed in an attack \( P_\text{(Kill)} \) is derived from the individual’s level of defence \( D \) and the conspicuousness of its individual display \( A \): \[ P_i(Kill) = Kill_\text{min} (1-Kill_\text{min}) K(A_iD_i)\text{.} \] The specific functions used by Speed and Ruxton (2007) are Gompertz functions. After the end of a season the fitness of surviving prey is calculated considering the fecundity costs of secondary defences and display: \[ F_i = exp(-(\psi A_i + \zeta D_i))\text{.} \]

The Figure presents the main results of the model: the variety of aposematic solutions can be explained with the variation of the marginal cost of displays for different prey species (a). Additionally, the model predicts positive as well as negative correlation of aposematic display \( A \) and defence \( D \) (b).

(a) This chart shows the variety of aposematic solutions as a result of the marginal cost of display \( \psi \). 1) \( \psi=1 \), 2) \( \psi=0.1 \), and 3) \( \psi=0 \). All other parameters as in Speed and Ruxton (2007).

(b) This chart shows examples of correlation between the optimal investment into defence \( D \) and display \( A \). 1) shows negative correlation in dependency of the effectiveness of defence in surviving an attack \( Kill_min \), 2) shows negative correlation in dependency of marginal cost of display \( \psi \) with constant marginal cost of defence \( \zeta \), and 3) shows positive correlation if \( \psi \) and \( \zeta \) vary linearly at the same time. All other parameters as in Speed and Ruxton (2007).

The variety of aposematic solutions as described by Speed and Ruxton (2007).

Problem formulation

As it turns out, the aspects of aposematism are manifold and far from straightforward. The main interest in the past laid in the initial evolution of aposematism and there is a wide body of theories available today. Nevertheless, the nature of aposematism beyond the initial evolution has still many open questions and remains a challenging research area. I have presented seminal models which have made significant progress on these exciting and open questions around aposematism and have influenced the field greatly by laying the foundations of much research which followed. In particular, the importance of the role of aversive learning was described in the earlier work by Leimar et al. (1986), but was omitted from later modelling developments in order to simplify analytical tractability. However, interesting questions arise as to how a predator incorporates the information gained from an encounter with prey into a generalised approach to predation and defence. In summary, the main aspects of aposematism the field has a great interest in are:

  • the properties of aposematic solutions and their stability,
  • the role of co-evolution,
  • the influence of predator psychology in particular aversive learning, and
  • the consequences of aposematism for predator and prey populations in regards to their fitness and foraging behaviour.