Multiplicity and small representations for some Gelfand pairs

Let [Formula: see text] be the semidirect product associated to the Gelfand pair [Formula: see text], where [Formula: see text] is a connected and simply connected nilpotent Lie group, and [Formula: see text] is a compact subgroup of the automorphism group, [Formula: see text], of [Formula: see text]. Under some assumptions on the pair [Formula: see text], we give a precise connection between the spherical representations and the small representations of [Formula: see text], which are the irreducible representations of [Formula: see text] that cannot be Hausdorff separated from the trivial one-dimensional representation [Formula: see text] of [Formula: see text] (called the cortex of [Formula: see text] and denoted by [Formula: see text]). More precisely, we show that the set of the small representations of [Formula: see text] is strictly contained in the closure of the set of the spherical representations of [Formula: see text] (with respect to the Fell topology). Furthermore, we prove that any spherical representation of [Formula: see text] cannot be a small representation. In the special case, ([Formula: see text] is abelian) we show that the converse of this assertion is not true.