Exploring anomalous flavour-changing neutral tqh transitions at future muon colliders: insights from $$\mu ^{+}\mu ^{-}\rightarrow t\bar{q}h$$ interactions

Abstract Flavour-changing neutral current (FCNC) interactions between the Higgs boson and the top quark are subject to significant suppression within the Standard Model. Consequently, these interactions can only attain an observable level through the influence of Beyond the Standard Model (BSM) physics effects. It is therefore of great importance to investigate these rare interactions, both to test the limits of the Standard Model and to reveal new physical signatures. At this point, the high sensitivity, clean experimental environment and low systematic uncertainties provided by muon colliders allow for a detailed study of such rare processes. In this context, the present study analyses the signal of the process $$\mu ^+\mu ^- \rightarrow t\bar{q}h$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msup> <mml:mi>μ</mml:mi> <mml:mo>+</mml:mo> </mml:msup> <mml:msup> <mml:mi>μ</mml:mi> <mml:mo>-</mml:mo> </mml:msup> <mml:mo>→</mml:mo> <mml:mi>t</mml:mi> <mml:mover> <mml:mrow> <mml:mi>q</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>¯</mml:mo> </mml:mrow> </mml:mover> <mml:mi>h</mml:mi> </mml:mrow> </mml:math> and the background processes $$t\bar{t}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>t</mml:mi> <mml:mover> <mml:mrow> <mml:mi>t</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>¯</mml:mo> </mml:mrow> </mml:mover> </mml:mrow> </mml:math> , $$tW\bar{b}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>t</mml:mi> <mml:mi>W</mml:mi> <mml:mover> <mml:mrow> <mml:mi>b</mml:mi> </mml:mrow> <mml:mrow> <mml:mo>¯</mml:mo> </mml:mrow> </mml:mover> </mml:mrow> </mml:math> , WWh and WW at both in $$\sqrt{s} = 1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msqrt> <mml:mi>s</mml:mi> </mml:msqrt> <mml:mo>=</mml:mo> <mml:mn>1</mml:mn> </mml:mrow> </mml:math> TeV and $$\sqrt{s} = 3$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msqrt> <mml:mi>s</mml:mi> </mml:msqrt> <mml:mo>=</mml:mo> <mml:mn>3</mml:mn> </mml:mrow> </mml:math> TeV energy levels. Analyses at the 1 TeV energy level reveal the observability of FCNC interactions even at low energy conditions and achieve the limit $$\text {Br}(t \rightarrow qH) = 1.96 \times 10^{-4}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mtext>Br</mml:mtext> <mml:mrow> <mml:mo>(</mml:mo> <mml:mi>t</mml:mi> <mml:mo>→</mml:mo> <mml:mi>q</mml:mi> <mml:mi>H</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> <mml:mo>=</mml:mo> <mml:mn>1.96</mml:mn> <mml:mo>×</mml:mo> <mml:msup> <mml:mn>10</mml:mn> <mml:mrow> <mml:mo>-</mml:mo> <mml:mn>4</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> . This result corresponds to an improvement of approximately 1.5 times compared to the current limits of the ATLAS experiment, which are $$3.0 \times 10^{-4}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.0</mml:mn> <mml:mo>×</mml:mo> <mml:msup> <mml:mn>10</mml:mn> <mml:mrow> <mml:mo>-</mml:mo> <mml:mn>4</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> . On the other hand, at the 3 TeV energy level, the most stringent limit was calculated under the luminosity $$\mathcal {L} = 20 ab^{-1}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>L</mml:mi> <mml:mo>=</mml:mo> <mml:mn>20</mml:mn> <mml:mi>a</mml:mi> <mml:msup> <mml:mi>b</mml:mi> <mml:mrow> <mml:mo>-</mml:mo> <mml:mn>1</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> , yielding a value of $$\text {Br}(t \rightarrow qH) = 1.1 \times 10^{-4}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mtext>Br</mml:mtext> <mml:mrow> <mml:mo>(</mml:mo> <mml:mi>t</mml:mi> <mml:mo>→</mml:mo> <mml:mi>q</mml:mi> <mml:mi>H</mml:mi> <mml:mo>)</mml:mo> </mml:mrow> <mml:mo>=</mml:mo> <mml:mn>1.1</mml:mn> <mml:mo>×</mml:mo> <mml:msup> <mml:mn>10</mml:mn> <mml:mrow> <mml:mo>-</mml:mo> <mml:mn>4</mml:mn> </mml:mrow> </mml:msup> </mml:mrow> </mml:math> . This result coincides an improvement of approximately 3 times with respect to the current ATLAS experimental limits. The findings of the study underscore the potential of muon colliders in the search for new physics and the pivotal role they can play in the study of rare processes such as Higgs-top quark interactions. Consequently, the results are expected to provide a crucial foundation for both experimental and theoretical studies and to make a significant contribution to the search for BSM physics.