We investigated the effect of estradiol add-back therapy (EAT) on brain activation related to cognitive function and affect in addition to putative changes in gray and white matter volume in testosterone depleted participants with prostate cancer. We conducted a randomized controlled, double-blinded trial in which 40 patients received 0.9 mg of transdermal estradiol per day for 6 months or matched placebo. Anatomical MRI and three functional MRI (fMRI) scans were obtained for the emotion recognition task, verbal memory task, and visuospatial memory task. Activation in corresponding cognitive and affective brain networks was demonstrated for all tasks. Longitudinally, there was no difference in brain activation, reaction time, or accuracy in response to the fMRI tasks between the EAT group and placebo group at 6 months. In addition, there was no detectable change in whole-brain gray or white matter volume or in hippocampal volume between the two groups after 6 months. This study supports earlier findings that EAT does not improve verbal memory or affect and has no immediate effect on hippocampal volume in testosterone depleted patients with prostate cancer.