Minimal signatures with undecidability of representability by binary relations

Abstract A semigroup of binary relations (under composition) on a set X is complemented if it is closed under the taking of complements within $$X\times X$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>X</mml:mi> <mml:mo>×</mml:mo> <mml:mi>X</mml:mi> </mml:mrow> </mml:math> . We resolve a 1991 problem of Boris Schein by showing that the class of finite unary semigroups that are representable as complemented semigroups of binary relations is undecidable, so composition with complementation forms a minimal subsignature of Tarski’s relation algebra signature that has undecidability of representability. In addition we prove similar results for semigroups of binary relations endowed with unary operations returning the kernel and cokernel of a relation. We generalise to signatures which may include arbitrary, definable operations and provide a chain of weaker and weaker signatures, each definable in the previous signature, each having undecidability of representability, but whose limit signature includes composition only, which corresponds to the well known, decidable and finitely axiomatised variety of semigroups. All these results are also proved for representability as binary relations over a finite set.