On decidability of the product of subgroups membership problem for nilpotent groups

Abstract The product of subgroups membership problem for a finitely generated group 𝐺 is the decision problem, where, given two or more finitely generated subgroups <m:math xmlns:m="http://www.w3.org/1998/Math/MathML"> <m:msub> <m:mi>K</m:mi> <m:mi>i</m:mi> </m:msub> </m:math> K_{i} , <m:math xmlns:m="http://www.w3.org/1998/Math/MathML"> <m:mrow> <m:mi>i</m:mi> <m:mo>=</m:mo> <m:mrow> <m:mn>1</m:mn> <m:mo>,</m:mo> <m:mi mathvariant="normal">…</m:mi> <m:mo>,</m:mo> <m:mi>m</m:mi> </m:mrow> </m:mrow> </m:math> i=1,\ldots,m , of 𝐺 and a group element 𝑔, it is asked whether <m:math xmlns:m="http://www.w3.org/1998/Math/MathML"> <m:mrow> <m:mi>g</m:mi> <m:mo rspace="0.111em">∈</m:mo> <m:mrow> <m:msubsup> <m:mo>∏</m:mo> <m:mrow> <m:mi>i</m:mi> <m:mo>=</m:mo> <m:mn>1</m:mn> </m:mrow> <m:mi>m</m:mi> </m:msubsup> <m:msub> <m:mi>K</m:mi> <m:mi>i</m:mi> </m:msub> </m:mrow> </m:mrow> </m:math> g\in\prod_{i=1}^{m}K_{i} . In this paper, we prove that, for any finitely generated nilpotent group 𝑁 of class two, the product of two subgroups membership problem is decidable. Note that this problem is equivalent to the problem of determining the non-emptiness of the intersection of two cosets of finitely generated subgroups. We also show that, for a sufficiently large direct power <m:math xmlns:m="http://www.w3.org/1998/Math/MathML"> <m:msup> <m:mi mathvariant="double-struck">H</m:mi> <m:mi>n</m:mi> </m:msup> </m:math> \mathbb{H}^{n} of the Heisenberg group ℍ, there exists a product <m:math xmlns:m="http://www.w3.org/1998/Math/MathML"> <m:mrow> <m:mi>K</m:mi> <m:mo rspace="0.111em">=</m:mo> <m:mrow> <m:msubsup> <m:mo>∏</m:mo> <m:mrow> <m:mi>i</m:mi> <m:mo>=</m:mo> <m:mn>1</m:mn> </m:mrow> <m:mn>4</m:mn> </m:msubsup> <m:msub> <m:mi>K</m:mi> <m:mi>i</m:mi> </m:msub> </m:mrow> </m:mrow> </m:math> K=\prod_{i=1}^{4}K_{i} of four finitely generated subgroups for which the membership problem is algorithmically undecidable. The proof of the last assertion is based on the undecidability of Hilbert’s 10th problem and interpretation of Diophantine equations in nilpotent groups. The question of decidability of the membership problem for a product of three finitely generated subgroups in a nilpotent group of class two remains open.