The objective of this study was to investigate whether there is a bidirectional association between testosterone concentrations and insulin resistance, in a prospective population study. A random population sample of 1400 men, aged 30–74, was examined in 2002–2005 in southwestern Sweden and followed up in 2012–2014 (N = 657). After excluding subjects without information on sex hormones and insulin resistance, 1282 men were included in the baseline study. Fasting measurements of plasma glucose, insulin and hormones were performed. Insulin resistance was defined using HOMA-Ir. Mean age at baseline was 47.3 ± 11.4 years. From the follow-up survey 546 men were included, mean age 57.7 ± 11.6 years. Low concentrations of total testosterone at baseline were significantly associated with high logHOMA-Ir at follow-up in a multivariable model including age, waist–hip ratio, physical activity, alcohol intake, smoking, LDL, CRP, hypertension, diabetes and logHOMA-Ir at baseline as covariates (β = −0.096, P = 0.006). Similar results were observed for bioavailable testosterone. Men within the lowest quartile of total testosterone at baseline had significantly higher logHOMA-Ir at follow-up than other quartiles (Q1 vs Q2 P = 0.008, Q1 vs Q3 P = 0.001, Q1 vs Q4 P = 0.052). Multivariable analysis of the impact of insulin resistance at baseline on testosterone levels at follow-up revealed no significant associations regarding testosterone concentrations (β = −0.003, P = 0.928) or bioavailable testosterone (β = −0.006, P = 0.873), when adjusting for baseline concentrations of total testosterone, age, waist–hip-ratio, LDL, CRP, physical activity, alcohol intake, smoking, hypertension and diabetes. Low testosterone concentrations at baseline predicted higher insulin resistance at follow-up, but high insulin resistance at baseline could not predict low testosterone at follow-up.