Kitaev interaction and possible spin liquid state in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msub> <mml:mi>CoI</mml:mi> <mml:mn>2</mml:mn> </mml:msub> </mml:math> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <m

Kitaev materials are of great interest due to their potential in realizing quantum spin liquid (QSL) states and applications in topological quantum computing. In the pursuit of realizing Kitaev QSL, a Mott insulator with strong bond-dependent frustration and weak geometric frustration is highly desirable. Here we explore Kitaev physics in the van der Waals triangular antiferromagnet (AF) CoI$_2$, through the spin-orbital states and Wannier function analyses, exact diagonalization and density matrix renormalization group study of the electronic structure and magnetic properties. We find that the high-spin Co$^{2+}$ ion is in the $J_\mathrm{eff}=1/2$ state because of strong spin-orbit coupling, and the weak trigonal elongation and crystal field contribute to the observed weak in-plane magnetic anisotropy. The strong $t_{2g}$-$e_g$ hopping via the strong Co 3$d$-I 5$p$ hybridization gives rise to a strong Kitaev interaction ($K_1$) at the first nearest neighbors (1NN), and the long Co-Co distance and the weak $t_{2g}$-$t_{2g}$ hoppings determine a weak Heisenberg interaction $J_1$. The resultant $|K_1/J_1|$ = 6.63 confirms a strong bond-dependent frustration, while the geometric frustration due to the 3NN Heisenberg interaction $J_3$ gets involved, and they all together result in the experimental helical AF order in CoI$_2$. We then propose to suppress the $J_3$ using a partial Mg substitution for Co, and indeed we find that Co$_{2/3}$Mg$_{1/3}$I$_2$ has the much reduced geometric frustration but hosts the robust bond-dependent frustration, and thus it would be a promising Kitaev material being so far closest to the QSL state.