The episode reveals why Simon is so determined to never marry or have children through a series of flashbacks:

By pretending to be "glued to each other's side," Simon avoids mothers looking to marry off their daughters, while Daphne becomes a "prize" coveted by other men who suddenly see her as more desirable because a Duke wants her.

in any room I entered. Come. You can speak. I understood you well enough. And I will help you to overcome this stammer of yours. . Scraps from the loft Bridgerton S01E02 Recap - TV Tropes

When young Simon developed a stutter, his father rejected him as a "failure" and sent him away to be hidden from society.

A close friend of his mother, Lady Danbury, intervened, helping Simon find his voice and gain the confidence to eventually confront his father.

Simon's mother died during childbirth after finally delivering a male heir. His father, Lord Hastings, was only concerned with the boy's ability to carry on the family name and showed no grief for his wife.