Disproportionate influence of site disorder on the evolution of magnetic phases in the anti-Heusler alloy <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msub> <mml:mi>Al</mml:mi> <mml:mn>2</mml:mn> </mml:msub> <mml:mi>MnFe<

Anti-Heusler alloys, being a new addition to the Heusler alloys family, exhibit atomic disorders, and almost all of them are reported as a re-entrant spin-glass system. Although such spin-glass feature is generally attributed to the inherent atomic disorder, a comprehensive and extensive investigation on the individual roles of different types of disorders in magnetic interactions remains lacking for any of the reported anti-Heusler systems. As an illustrative case, we have carried out an in-depth experimental as well as theoretical investigation of structural, magnetic, and transport properties of a polycrystalline anti-Heusler alloy, Al$_2$MnFe. While the major atomic disorder is found to be among Fe and Mn atoms, which are randomly distributed among the two octahedral sites, 4$a$ and 4$b$ (B2-type disorder), a relatively small fraction ($\sim$12\%) of Mn atoms also replace Al atoms at the tetrahedral 8$c$ site. Magnetically, the system undergoes two transitions: a paramagnetic to a ferromagnetic transition at $T_{\rm C}\sim$113~K, followed by a spin-glass phase transition below $T_{\rm f}\sim$20~K. Here, the magnetic moment is primarily confined to Mn atoms. Very interestingly, our theoretical analysis reveals that the ferromagnetic spin arrangement remains rather robust in spite of the 50\% disorder of moment-carrying Mn atoms between the two octahedral sites, but a much smaller ($\sim$12\%) cross-distribution of Mn atoms between octahedral and tetrahedral sites are sufficient to impose a reentrant spin-glass state at low temperature. Our analysis brings forth the importance of understanding the role of individual types of swap-disorder on magnetic properties in the anti-Heusler family of materials.