From human-in-the-loop to institutional oversight of judicial decision support systems

Human oversight is recognised as a key governance mechanism for artificial intelligence and is mandated by the EU AI Act for systems that present a high risk to individuals’ health, safety, or fundamental rights. Nevertheless, the concept has attracted criticism, particularly regarding its practical effectiveness and the extent to which it may serve to legitimise the indiscriminate deployment of AI in the public sector, including within the judiciary. This paper explores what effective human oversight requires in judicial contexts, assessing whether judges possess the capacities envisaged by the AI Act to meaningfully supervise AI systems. It argues for a shift from individual to institutional oversight of judicial AI, contending that responsibility should not rest solely with judges, who may lack full control over such systems. Instead, oversight ought to be institutionalised through dedicated oversight bodies that integrate both human design and human review practices.