In this paper, we survey different granular computing (GrC) applications to the field of cognitive mapping by highlighting how fuzzy cognitive maps (FCMs) have been augmented with different types of information granules such as intervals, fuzzy sets, fuzzy clustering, rough sets, and grey sets. These information granules have been integrated into core FCM components such as their set of concept nodes, the causal links among these concepts or their underlying inference mechanisms. We discuss the advantages and limitations brought forth by these granular cognitive maps (GCMs) as well as their reported applications, with especial emphasis on time series analysis and pattern classification scenarios. To the best of our knowledge, this is the first time that GCMs stemming from a variety of granular constructs are systematically reviewed. We hope this survey inspires further research endeavors in the exciting interplay between GrC and intelligent systems.