The literature on the health-related quality of life (HRQOL) after rectal cancer is growing, however, a comparison between patients with nonadvanced disease (NAD), locally advanced rectal cancer (LARC), locally recurrent rectal cancer (LRRC) and a normative population has not been made. Data on the sexual functioning of patient groups is also scarce. We compared (i) the HRQOL of patients with NAD, LARC, or LRRC, with a special focus on sexual functioning and (ii) the HRQOL of the three treatment groups with a normative population. The EORTC QLQ-C30 and QLQ-CR38 were completed by 80 patients with NAD, 292 LARC patients and 67 LRRC patients. The normative population (n = 350) completed the EORTC QLQ-C30 and the Sexual Functioning and Sexual Enjoyment scales of the CR38. LRRC patients reported a lower Physical Function, Social Function, Future Perspective, Sexual Functioning and more Pain compared with LARC and NAD patients. Also, LRRC patients had a worse Body image than NAD patients and a lower Male Sexual Functioning than LARC patients. More than 75% of men and 50% of women were sexually active preoperative, compared with less than 50% and less than 35% postoperative. Male LRRC patients had more problems with erectile or ejaculatory functioning and felt less masculine than NAD or LARC patients. Women did not differ on Lubrication, Dyspareunia and Body Image. About 10% of patients used aids in order to improve erectile functioning (men) or lubrication (women). The treatment groups reported a lower HRQOL and sexual functioning compared with the normative population.