Stigma can hamper full inclusion of people with intellectual disabilities in society. For other minority groups, higher levels of familiarity with these groups have been shown to relate to lower levels of stigma, whereby emotions can play a mediating role. However, concerning people with intellectual disabilities, there is limited knowledge regarding the general public’s levels of familiarity, its relationship with stigma, and the role of emotions in this relationship. A cross-sectional survey was conducted among a nationally representative sample of the Dutch population (n = 892). The relationship between levels of familiarity and different measures of stigma representing cognitive, emotional, and behavioral aspects of stigma (i.e., attributions, emotions, discrimination) was examined. Emotions (fear, anger, and sympathy) were studied as a mediator in the relationship between familiarity and discrimination (i.e., social distance, and intention to help). Participants who reported no familiarity in real life with people with intellectual disabilities (30.6% of the population sample) demonstrated higher levels of stigma (attributions, emotions, discrimination) than participants who reported any form of real-life familiarity (69.4% of the sample). Fear was found to be the most important mediator of the relationship between familiarity and discrimination. The findings stress the continuing importance to advocate for people with intellectual disabilities to be recognized and become known within society. This increased familiarity might reduce stigma and increase positive experiences of inclusion. Support workers may have a vital role in this process. The importance of fear in the relationship between familiarity and discrimination may inform future research and antistigma interventions.