This would be completely understandable had he not messed everything else up. Glad he didn’t want to do that with a literal child and he even made a garden thinking he was marrying a woman… But he then treated the child like dirty laundry. He completely forgot her and her place in the castle. Sure bring in a mistress but keep it on the low so you aren’t insulting your child wife and giving the servants fuel for fodder. The one who forgot the purpose of the marriage is him.