Description
In this talk, we present our commentary “[Towards replication in computational cognitive modeling: A machine learning perspective](https://link.springer.com/article/10.1007/s42113-019-00055-w)”, which explores the challenges of reproducibility in machine learning, and draws analogies to (future) endeavors in cognitive modeling (and beyond). Advancements in machine learning have raised concerns about the lack of rigor and robustness in achieving reproducible results. Surprisingly, despite the inherent characteristics of the field being conducive to open science, reproducibility remains a significant challenge. Our paper presents three key challenges hindering reproducibility in machine learning. Firstly, the inertia of cultural change presents a barrier to implementing best practices. We emphasize the need for a cultural shift and proper credit attribution to incentivize reproducibility. Secondly, the scalability of producing reproducible code poses a persistent issue. The extensive requirements for documentation and modular code creation demand significant effort and infrastructure familiarity. This challenge disproportionately affects smaller research groups and impedes the adoption of open science.Thirdly, interpretability of reproducible research is crucial. The need for rigorous structure and standardized documentation becomes paramount to facilitate efficient analysis and interpretation of research findings. Moreover, we discuss the downstream effects of reproducibility efforts on the research community. We highlight instances where well-established models outperformed complex counterparts, emphasizing the importance of critically assessing benchmarks and task formulations.
Our talk ends with a discussion of the current state of affairs in machine learning; specifically pertaining the recent focus on generative AI. The computational scale and resources required to drive these research lines have promoted an industry-centered culture that is often antithetical to open source and transparency efforts. This leave us with the question: What will the role of academia become?
| Period | 5 Sept 2023 |
|---|---|
| Event title | Open to Complexity: Symposium on Open Science in the Social Sciences and Humanities |
| Event type | Seminar |
| Location | Tilburg, NetherlandsShow on map |