In this paper, we develop a paleoeconomic model of the co-evolution of economic specialization and encephalization—the common physiological measure of intelligence as reflected by brain mass relative to total body mass. Our economic analysis links ecological and social intelligence theories of increased encephalization in early hominins through a model in which both economic and ecological feedbacks jointly determined the evolutionary incentives. We focus on degrees of specialization affected by coordination costs with and without market exchange. Our results suggest encephalization would be a process characterized by diminishing returns to behavioral advances. In terms of the long-running debate in economics over whether specialization increases or decreases intelligence, our results suggest from an evolutionary perspective the answer depends on economic/social institutions and how these influence ecological interactions.