Deep Neural Networks (DNNs) are transforming fields like Particle Physics, but their black box nature challenges scientific understanding. This talk explores whether DNNs can serve as vehicles for scientific understanding and what conditions must be met for this to occur.

I propose a minimal account of scientific understanding distinguishing between understanding as a state and a process. Scientific understanding requires both (moderate) factivity and usability. Additionally, representational devices must be intelligible and allow reliable inferences to provide understanding. The problem is that DNNs themselves are often not understood, making it difficult to use them as vehicles for scientific insight.

A promising direction lies in eXplainable AI (XA)), but many methods focus on how DNNs work rather than what they reveal about physics. To regain understanding, researchers must extract meaningful structure from DNNs, not just interpret their internal mechanics. However, a fundamental dilemma remains: intelligiblee methods often solely confirm known physics, while novel insights frequently lack clear scientific interpretation. I argue that bridging this gap is crucial for making DNNs genuine tools for scientific understanding and will discuss ongoing efforts in Particle Physics to tackle this challenge.