Nonholomorphic <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msub><mml:mi>A</mml:mi><mml:mn>4</mml:mn></mml:msub></mml:math> modular invariance for fermion masses and mixing in <mml:math xmlns:mml="http://www.w3.org/1

Addressing the fermion flavor structures using modular invariance is a challenging task in the framework of quark-lepton unification. Building on recent applications of modular symmetry in nonsupersymmetric models, we propose the first renormalizable <a:math xmlns:a="http://www.w3.org/1998/Math/MathML" display="inline"><a:mi>S</a:mi><a:mi>U</a:mi><a:mo stretchy="false">(</a:mo><a:mn>5</a:mn><a:mo stretchy="false">)</a:mo></a:math> grand unified theory incorporating level 3 nonholomorphic modular symmetry, <e:math xmlns:e="http://www.w3.org/1998/Math/MathML" display="inline"><e:msub><e:mi mathvariant="normal">Γ</e:mi><e:mn>3</e:mn></e:msub><e:mo>≃</e:mo><e:msub><e:mi>A</e:mi><e:mn>4</e:mn></e:msub></e:math>. This framework constrains Yukawa couplings to polyharmonic Maaß forms, significantly reducing the number of free parameters while enhancing the predictive power of the models. We present a comprehensive analysis of fermion masses and mixing while tackling key grand unified theory (GUT) queries such as gauge coupling unification and proton decay. Beyond the minimal <h:math xmlns:h="http://www.w3.org/1998/Math/MathML" display="inline"><h:mi>S</h:mi><h:mi>U</h:mi><h:mo stretchy="false">(</h:mo><h:mn>5</h:mn><h:mo stretchy="false">)</h:mo></h:math> framework, the Higgs sector incorporates the <l:math xmlns:l="http://www.w3.org/1998/Math/MathML" display="inline"><l:mn>4</l:mn><l:msub><l:mn>5</l:mn><l:mi>H</l:mi></l:msub></l:math> dimensional Higgs field crucial in differentiating the masses of down quarks and charged leptons, and the fermion sector is extended with three right-handed neutrinos enabling neutrino masses via the type-I seesaw mechanism. We analyze two benchmark models with distinct modular weight and <n:math xmlns:n="http://www.w3.org/1998/Math/MathML" display="inline"><n:msub><n:mi>A</n:mi><n:mn>4</n:mn></n:msub></n:math> charge assignments. The predicted effective Majorana mass <p:math xmlns:p="http://www.w3.org/1998/Math/MathML" display="inline"><p:msub><p:mi>m</p:mi><p:mrow><p:mi>β</p:mi><p:mi>β</p:mi></p:mrow></p:msub></p:math> values align with current neutrinoless double-beta decay experiments, and the effective neutrino mass <r:math xmlns:r="http://www.w3.org/1998/Math/MathML" display="inline"><r:msub><r:mi>m</r:mi><r:mi>β</r:mi></r:msub></r:math> is within the reach of future beta decay searches. The predicted sum of neutrino masses, <t:math xmlns:t="http://www.w3.org/1998/Math/MathML" display="inline"><t:mo>∑</t:mo><t:msub><t:mi>m</t:mi><t:mi>i</t:mi></t:msub></t:math>, satisfies the upper bound set by recent cosmological observations. The gauge coupling unification is achieved through a light scalar triplet <v:math xmlns:v="http://www.w3.org/1998/Math/MathML" display="inline"><v:msub><v:mi>ϕ</v:mi><v:mn>3</v:mn></v:msub><v:mo>∼</v:mo><v:mo stretchy="false">(</v:mo><v:mn>3</v:mn><v:mo>,</v:mo><v:mn>3</v:mn><v:mo>,</v:mo><v:mo>−</v:mo><v:mn>1</v:mn><v:mo>/</v:mo><v:mn>3</v:mn><v:mo stretchy="false">)</v:mo></v:math> and a scalar octet <z:math xmlns:z="http://www.w3.org/1998/Math/MathML" display="inline"><z:msub><z:mi>ϕ</z:mi><z:mn>5</z:mn></z:msub><z:mo>∼</z:mo><z:mo stretchy="false">(</z:mo><z:mn>8</z:mn><z:mo>,</z:mo><z:mn>2</z:mn><z:mo>,</z:mo><z:mn>1</z:mn><z:mo>/</z:mo><z:mn>2</z:mn><z:mo stretchy="false">)</z:mo></z:math> belonging to the <db:math xmlns:db="http://www.w3.org/1998/Math/MathML" display="inline"><db:mn>4</db:mn><db:msub><db:mn>5</db:mn><db:mi>H</db:mi></db:msub></db:math> Higgs, while proton decay constraints require that the contribution of the <fb:math xmlns:fb="http://www.w3.org/1998/Math/MathML" display="inline"><fb:mn>4</fb:mn><fb:msub><fb:mn>5</fb:mn><fb:mi>H</fb:mi></fb:msub></fb:math> Higgs to the up-quark mass matrix remains highly suppressed.