Direct and indirect fitness effects of plant metabolites, and genetic constraints, limit evolution of allelopathy in an invading plant

Abstract Invading species encounter novel communities of consumers, pathogens and competitors. Both phenotypic plasticity and rapid evolution can facilitate invasion across these heterogenous communities. However, adaptive evolution can be constrained by adaptive phenotypic plasticity and the genetic architecture of traits under selection. We measured phenotypic plasticity and quantified genetic variation for growth, leaf chlorophyll a (Chl a ), leaf glucosinolate concentration and lifetime fitness among 22 naturally inbred seed families of Alliaria petiolata (garlic mustard) collected across its invasive range in eastern North America. After growing a self‐pollinated generation in a uniform common garden to reduce maternal effects, we reared second‐generation plants in a 2‐year greenhouse and field experiment with soil from an uninvaded habitat. We estimated selection gradients and causal factors affecting lifetime fitness of A. petiolata when reared alone, with an intraspecific competitor, and under interspecific competition with naïve Acer saccharum (sugar maple) saplings. We defined Total Metabolite Production (TMP) as the first principal component of Chl a and glucosinolate concentration and Relative Glucosinolate Investment (RGI) as the second principal component. TMP accounted for 84% of variation in these two traits, with significant plasticity across growing environments ( p < 0.001), but non‐significant broad‐sense heritability ( H 2 = 2.91; p = 0.08). Path analysis revealed that plastic phenotypes with higher TMP had an indirect positive effect on A. petiolata fitness via a direct, negative effect on the performance of A. saccharum competitors. In contrast, RGI was significantly heritable ( H 2 = 16.91, p < 0.001) with no detectable plasticity across treatments. Elevated RGI among A. petiolata genotypes had a direct, positive effect on A. saccharum performance and an indirect negative effect on A. petiolata fitness. Synthesis. A proposed model of allelopathy‐mediated evolution during invasion is unlikely in A . petiolata due to (i) low heritability and high plasticity for glucosinolate production and (ii) a fitness cost with selection to reduce glucosinolate production under interspecific competition. Overall, we find no evidence for allelopathy or glucosinolate evolution in response to competition, but heritability despite fitness costs suggests that glucosinolates play other important ecological roles.