Can human judgement be machine-sourced? An approach to measure the perceptual dimensions embedded in software

Considering the challenges of measuring the perceptual dimensions of the IT artifact, we propose a computational method for developing latent concepts through the lines of code that make up a software artifact. The proposed “machine-sourcing human judgement” approach, combines a novel technique of extracting the semantic properties (meanings) of the code from the software engineering literature with the machine-learning techniques used in the IS literature. Using the illustration of open source software (OSS), we demonstrate that the ‘contextual’ and ‘economic’ limitations of evaluating the creativity of OSS code contributions can be overcome through our approach. The performance of the proposed approach is tested by using a labelled dataset of code contributions created by two experienced OSS developers. We find that our approach of using semantic properties from the “software code” matches in performance to evaluating “textual descriptions” of the code. Potential methodological improvements and future research opportunities are also discussed.