Ulnar-mammary syndrome (UMS) is characterized by ulnar defects, and nipple or apocrine gland hypoplasia, caused by TBX3 haploinsufficiency. Signs of hypogonadism were repeatedly reported, but the mechanisms remain elusive. We aim to assess the origin of hypogonadism in two families with UMS. UMS was suspected in two unrelated probands referred to an academic center with delayed puberty because of the evident ulnar ray and breast defects in their parents. Clinical, biochemical and genetic investigations proved the existence of congenital normosmic IHH (nIHH) associated with pituitary hypoplasia in the two probands who were heterozygous for novel TBX3 pathogenic variants. The mutations co-segregated with delayed puberty, midline defects (nose, teeth and tongue anomalies) and other variable features of UMS in the two families (absent axillary hairs and nipple hypoplasia, asymmetrical features including unilateral ulnar or renal abnormalities). The combined analysis of these findings and of the previous UMS reports showed delayed puberty and other signs of hypogonadism in 79 and 37% of UMS males, respectively. Proband 1 was followed up to adulthood with persistence of nIHH. In conclusion, UMS should be suspected in patients with delayed puberty and midline defects, including pituitary hypoplasia, in the presence of mild cues for TBX3 mutation, even in the absence of limb malformations. In addition, TBX3 should be included among candidate genes for congenital nIHH.